Page 1 of 1

Mini PC as search bot / server for Awasu

Posted: Fri Aug 25, 2017 2:46 pm
by awasu.user
At the last time I think about making small server for Awasu. Free webservers with Python support are too limited with external websites access. I want flexibility with installing another modules too. The best option is buy cluster time and CPU power, but is out of my range. I think about something different.

Expectantions:
- it's not a huge price - it's under 300 US dolars, but the best option is around price Rasberry PI
- it has enough CPU power to make calculation in reasonable time
- it has GBit Ethernet onbuild
- it has not HDMI, camera and multimedia extensions and etc.
- it has minimum 2GB RAM, but more is the best option
- it has posibility fast read and write data on disk / flash card
- it is small that I can get it with my laptop in laptop bag
- it is a power efficient and it is under 50W, the best option is few WATs only.

How it should works?

It is a simply text system (for install python). On it is runned asbolute minimum like chipset drivers, controllers for I/O etc to best performance. On this small server is installed Python. Machine has two modules:

Search Bot - it's getting information from Internet and make text processing on them fx. make summary of pages, count frequencies of words etc.
Web Server - it is out API to connection with Awasu. Awasu get data as RSS, or when I run my scripts on laptop I can mix data from Awasu and from Search Bot by web API call.

All this stuff have to be coded in Python (because I start like it a more and after communication with Awasu Support this languange is more inpiring me than others :D ).

Summary

At the summary I'm looking minimal OS with platform to run python 3 without other applications and GUI to best performance. Hardware can save few gigabytes dictionarys and use them in reasonable time. Hard drive space 16-32 GB is max here. Idea is that this machine is running 24/7 and getting data and process them, but when API call it is a stop and return processed data. When I go out I can plug in a switch and use in another place.

I hoped I wrote clear here what I am looking for.

Could you some one suggested me any soultion here?

Re: Mini PC as search bot / server for Awasu

Posted: Sat Aug 26, 2017 12:11 am
by support
This is an intriguing idea. It's common for larger Awasu installations to offload some of the work to other computers - e.g. channel hooks transfer new content to an external database (e.g. MySQL or Postgres) running on another server, and then have a search server (e.g. something Lucene-based) indexing that, perhaps running on another server itself - but I get your desire to keep costs down :-)

I've always preferred Banana Pi's over Raspberry Pi's, since they are more powerful, although that's less of an issue with the new rPi 3. I've written a few articles on how to set them up here and here., but it never occurred to me to integrate one of these things with Awasu :idea: I use Banana Pi Pro's, but there are more powerful models e.g. the Banana Pi M3 with 8 (!) cores and 2GB of memory. Java is a big fat pig, so I'm not sure if this will be enough to run Lucene, but it'll sure be a fun project to find out :-), and even if not, there are lighter alternatives.

These things run off an SD card, but they degrade over time, so you don't want to be writing to them too much, especially things like a database or search index, but the bPi's have 3 USB ports, so it'd be no problem just plugging in an external hard drive.

If you prefer Windows, there are things like the Intel Compute Stick, although I don't have any experience with these.

Re: Mini PC as search bot / server for Awasu

Posted: Sat Aug 26, 2017 7:41 am
by awasu.user
support wrote:This is an intriguing idea. It's common for larger Awasu installations to offload some of the work to other computers - e.g. channel hooks transfer new content to an external database (e.g. MySQL or Postgres) running on another server, and then have a search server (e.g. something Lucene-based) indexing that, perhaps running on another server itself - but I get your desire to keep costs down


I think about reverse aproach :D Idea is not tranfer data from Awasu but extend data in Awasu fx. it's making hot list of articles with summary and concept (fx. Donald Trump = president USA) based on dbpedia to find relation. It is more oriented in metasearch around news.

I spend time to find target solution. The best option is Intel NUC - it's a configuration with i7 6th gen CPU up to 32GB RAM enough to process searching and processing data in quite compicated stuff, but price is out start budget. On the first step I think about making some skeleteon with inexpensive prototype electronic with enough power to run OS and python scripts. On the beginning I'll start from scratch and it will be network "Hello world!", so invest in a huge priced hardware not make sense now.

support wrote:I've always preferred Banana Pi's over Raspberry Pi's, since they are more powerful, although that's less of an issue with the new rPi 3. I've written a few articles on how to set them up here and here., but it never occurred to me to integrate one of these things with Awasu I use Banana Pi Pro's, but there are more powerful models e.g. the Banana Pi M3 with 8 (!) cores and 2GB of memory. Java is a big fat pig, so I'm not sure if this will be enough to run Lucene, but it'll sure be a fun project to find out , and even if not, there are lighter alternatives.


I think on start about Raspberry, but I think 1GB RAM is not enought to run more complicate scripts. I woudn't add electronic extension to this stuff, but only concentrate on coding stuff. Banana PI M3 need some hardware updates like heatsink and fun, because is throttling and switch cores. Python is not multithascing interpretated that I don't now that extra 4 core make a difference here.

support wrote:These things run off an SD card, but they degrade over time, so you don't want to be writing to them too much, especially things like a database or search index, but the bPi's have 3 USB ports, so it'd be no problem just plugging in an external hard drive.


Interesing that Banana Pi M3 is about twice faster on writing on SD than using SATA.

support wrote:If you prefer Windows, there are things like the Intel Compute Stick, although I don't have any experience with these.


It has not have RJ45. For me it must have, because my solution will be connected to switch.

Re: Mini PC as search bot / server for Awasu

Posted: Sat Aug 26, 2017 8:46 am
by support
awasu.user wrote:On the first step I think about making some skeleteon with inexpensive prototype electronic with enough power to run OS and python scripts.

We'll definitely need a write-up if you get this going! :-)

awasu.user wrote:I think on start about Raspberry, but I think 1GB RAM is not enought to run more complicate scripts.

Meh, just build a cluster :hysterical:

awasu.user wrote:Python is not multithascing interpretated that I don't now that extra 4 core make a difference here.

I found having multiple cores made a huge difference - the UI (even if it's just a console) is still responsive, even if the machine's working hard.

Re: Mini PC as search bot / server for Awasu

Posted: Sat Aug 26, 2017 9:22 am
by awasu.user
support wrote:
awasu.user wrote:On the first step I think about making some skeleteon with inexpensive prototype electronic with enough power to run OS and python scripts.

We'll definitely need a write-up if you get this going! :-)


I'm stucking on defining minimal resurce needed to start. Too much is wasting money, too small is wasting time. Eh, I'll probably choose Banan PI M3.

support wrote:Meh, just build a cluster


Sure, I need one room in home or I can leave in a garrage :D

support wrote:I found having multiple cores made a huge difference - the UI (even if it's just a console) is still responsive, even if the machine's working hard.


I only think about console. GUI is not necessery here.