awasu.user
Posts: 81
Joined: Fri Jan 06, 2017 12:50 pm

Postby awasu.user » Sat Mar 18, 2017 9:42 am

When I started test with Awasu application bit a FeedDeamon in updates times. Now time to finish updating channels is longer than before. I add few channels, but is not question few minutes longer but fx. 30-40 minutes to updates ~300 channels. Is a posoibility that it can be too large database to working with and Awasu is slowing down? I use in my laptop SSD disk (~240/240 MB read/write) so it could not be a bottle neck for performance. I've 16GB RAM and i7-2620M CPU so I think is good configuration to comfortable update app.

Yes, I have some apps like AV running background but Awasu process use 132MB RAM and less CPU that my e-mail client. I can't figure out what is going on because Awasu slowing down few times. I have time to drink a tea, eat breakfeast and I still waiting for update.

User avatar
support
Site Admin
Posts: 3021
Joined: Fri Feb 07, 2003 12:48 pm
Location: Melbourne, Australia
Contact:

Postby support » Sat Mar 18, 2017 10:01 am

It's almost certainly the search index.

The way it works, it's much slower adding 100 documents one-at-a-time, than adding 100 documents as a single batch. The default configuration for Awasu is one-at-a-time, so that documents will be returned as search results as soon as they arrive, but this gets inefficient once the search index grows in size, so you need to switch to adding them in batches.

To confirm that this is what's happening, open the Program Options, go to the Archive tab and tick "Restrict search index updates", and select "During channel blackouts". If you don't have any channel blackouts configured, this means that Awasu will never update the search engine.

Then, in the morning, update all your channels - this should now be faster. Once it's done, go back to the Program Options, and you'll see a button to manually update the search index. Run this, and the new content will appear in the search results.

User avatar
support
Site Admin
Posts: 3021
Joined: Fri Feb 07, 2003 12:48 pm
Location: Melbourne, Australia
Contact:

Postby support » Sat Mar 18, 2017 10:19 am

Another thing to try is reduce how long content is archived for (Program Options, Archive tab, Keep content for). Awasu deletes content as it gets old, so the less there is in the database, the faster things will run.

awasu.user
Posts: 81
Joined: Fri Jan 06, 2017 12:50 pm

Postby awasu.user » Sat Mar 18, 2017 9:31 pm

support wrote:go to the Archive tab and tick "Restrict search index updates", and select "During channel blackouts".

I change option as you sugested two post above get me a lof of crushes. After rebuild database Awasu start working normaly and update content. Is that a normal behaviour?

Questions
1. How calculate optimal period to archive feed contens? Is any option that error and trial way?
2. After changing settings Feed from Search query will be work normally or have I manually update them to get new content?

User avatar
support
Site Admin
Posts: 3021
Joined: Fri Feb 07, 2003 12:48 pm
Location: Melbourne, Australia
Contact:

Postby support » Sat Mar 18, 2017 11:52 pm

awasu.user wrote:I change option as you sugested two post above get me a lof of crushes. After rebuild database Awasu start working normaly and update content. Is that a normal behaviour?

No, crashes are never normal :) It's just a coincidence you got a corruption in the search index at the same time you changed the setting.

awasu.user wrote:1. How calculate optimal period to archive feed contens? Is any option that error and trial way?

It's really a question of what you're trying to do. RSS feeds are usually used for short-lived content, so if you don't care about anything older than a month, that's what you set it to. There's a global setting in Program Options, but it can also be set for individual channels.

awasu.user wrote:2. After changing settings Feed from Search query will be work normally or have I manually update them to get new content?

Short answer: yes, you have to update the search index before new content will be returned as results.

Awasu stores content in its archive database so that you can view it, even after it's been removed from the publisher's feed (the setting we're talking about above controls how long content is kept in the archive database before it's deleted). The search index is something separate, it contains a summary of each archived feed item, so that searches can be made quickly.

So, the normal behaviour is:
  • New feed item is received.
  • Add it to the archive database.
  • Add it to the search index.
  • Another new feed item is received.
  • Add it to the archive database.
  • Add it to the search index.
  • etc...
This allows new items to appear in search results immediately, but the process is a bit slow.

So, the way to speed things up is:
  • New feed item is received.
  • Add it to the archive database.
  • Another new feed item is received.
  • Add it to the archive database.
  • etc...
  • Update the search index with all the new items that have been received since the last update.

This is quicker, but until you do the last step, new items won't appear in search results, because they're not in the search index yet.

User avatar
kevotheclone
Posts: 239
Joined: Mon Sep 08, 2008 7:16 pm
Location: Elk Grove, California

Postby kevotheclone » Sun Mar 19, 2017 7:35 am

I also subscribe to 350+ feeds on much older desktop hardware, and Awasu updates all channels in a few minutes.

I do not have "Restrict search index updates" checked.

I have "Keep content for" set to 4 months as the default.
I have "Keep content for" set to 1 day for "deal of the day" type of Channels, since their special offer is only good for 1 day, but this is only 5 - 10 Channels.
I have "Keep content for" set for a longer period of time, for those bloggers who don't post much anymore, but whose posts I greatly value; this way after 4 months their old posts don't re-appear as new again.

Please don't :lol: to hard, the SSD is from 2016, the NVIDA video card is from 2010 or 2011, and the rest of the PC is from 2008.
  • CPU: Intel Core2 Quad Q6600 @ 2.40GHz (Passmark 2977 score) http://www.cpubenchmark.net/cpu.php?cpu ... 40+2.40GHz
  • 4GB Dual-Channel DDR2 RAM @ 399MHz
  • 1023MB NVIDIA GeForce GTX 550 Ti
  • 476GB Samsung SSD 850 PRO SCSI Disk Device
    The SSD is a SATA III, but my Motherboard is only a SATA 1.

User avatar
support
Site Admin
Posts: 3021
Joined: Fri Feb 07, 2003 12:48 pm
Location: Melbourne, Australia
Contact:

Postby support » Sun Mar 19, 2017 8:18 am

kevotheclone wrote:Please don't :lol: to hard, the SSD is from 2016, the NVIDA video card is from 2010 or 2011, and the rest of the PC is from 2008.

No, it's good to hear that Awasu still runs well on older hardware :)

But this reminded me of another possible explanation: virus checkers are notorious for slowing things down. Awasu uses the disk a lot, and every time it writes something out, the virus checker steps in to take a look see. So, try disabling virus checking on the local data directory (this is where most of the disk activity happens).

Another possible explanation is if you turn your computer off overnight. I leave my computer, and Awasu, on more-or-less 24x7, so there's a steady stream of content arriving over time. If you don't have Awasu running overnight, when you start it up in the morning, there will be a flood of new content, which all needs to be archived and indexed i.e. things will run a bit more slowly because Awasu is working so hard.

awasu.user
Posts: 81
Joined: Fri Jan 06, 2017 12:50 pm

Postby awasu.user » Mon Mar 20, 2017 6:18 am

support wrote:It's really a question of what you're trying to do. RSS feeds are usually used for short-lived content, so if you don't care about anything older than a month, that's what you set it to. There's a global setting in Program Options, but it can also be set for individual channels.
(...)
This is quicker, but until you do the last step, new items won't appear in search results, because they're not in the search index yet.


I want based on leads and headlines find trending topic based on the most used phrases in internal Awasu DB. I've some feeds that repeats the same articles - I want filter them out from new and try figure out what are leading topics based on category. For this purpuse I start learning Python. I try in the future coding plugin returning result to Awasu and generated searched based o keywords. I start a subject so it can take a while. For me best solution is get articles minimum 3 months old, optimal 6 months, max 1 year old to get info how it that change in time. I hope this is posible because I'm not mathematican :-)

*

So feeds generated from search will not updates on new settings, but only when I click on them?

kevotheclone wrote:I also subscribe to 350+ feeds on much older desktop hardware, and Awasu updates all channels in a few minutes.

I do not have "Restrict search index updates" checked.

I have "Keep content for" set to 4 months as the default.
I have "Keep content for" set to 1 day for "deal of the day" type of Channels, since their special offer is only good for 1 day, but this is only 5 - 10 Channels.
I have "Keep content for" set for a longer period of time, for those bloggers who don't post much anymore, but whose posts I greatly value; this way after 4 months their old posts don't re-appear as new again.

Please don't :lol: to hard, the SSD is from 2016, the NVIDA video card is from 2010 or 2011, and the rest of the PC is from 2008.


I use old laptop too so we are in the same chair. My only difference is integrated VGA and more RAM (16GB). I must think about your settings, because we have to similar result but I see difference on my machine. After changing settings I get few times faster. I need think about storage DB options how set them optimal for me.

User avatar
support
Site Admin
Posts: 3021
Joined: Fri Feb 07, 2003 12:48 pm
Location: Melbourne, Australia
Contact:

Postby support » Mon Mar 20, 2017 7:01 am

awasu.user wrote:
support wrote:So feeds generated from search will not updates on new settings, but only when I click on them?

Search channels run a search query, the same as if you did one yourself from the UI, and returns the results as a feed.

So, if you're restricting search index updates, you need to first update the search index (as described above), then update the search channel. Just opening the channel won't be enough, since it will show you whatever was in the feed the last time it was updated, it needs to update (i.e. actually run the search query). If you set the channel to automatically update every 5 minutes, then it will receive any new search results, by itself, within a few minutes of you updating the search index.


Return to “Awasu - General Discussion”

Who is online

Users browsing this forum: No registered users and 2 guests