awasu.user
Posts: 81
Joined: Fri Jan 06, 2017 12:50 pm

Postby awasu.user » Sat Jul 22, 2017 12:46 pm

Intro

To get all articles in channels I use API call in format:

http://localhost:2604/channels/get?form ... xxx&id=100

Result: HTML file and making processing data.

Workaround

Set report template with structure and check processing by grouping in Awasu:

Code: Select all

title;url;published;channelname{%REPEAT% Channels-IfGroupingItems}{%?GROUP-ITEMS-BY-CHANNEL%}{%REPEAT% FeedItems}
{%ITEM-METADATA% name!};{%ITEM-METADATA% url encode=attr};{%ITEM-METADATA% timestamp};{%CHANNEL-METADATA% name}{%ENDIF%}{%/REPEAT%}{%/REPEAT%}


I get CSV, read them and I can process it.

Request

Add posibility return channel data (articles, published, title, url etc.) as JSON and XML file by calling
$/channels/get

User avatar
support
Site Admin
Posts: 3021
Joined: Fri Feb 07, 2003 12:48 pm
Location: Melbourne, Australia
Contact:

Postby support » Sat Jul 22, 2017 1:11 pm

There are several ways you can get this information, depending on exactly what you want, and how you want to do it.

Reports can return content in any format, including JSON and XML (e.g. see MetaChannel.template), so you could create a report that includes the channels you want. This would be really tedious if you only wanted to do 1 channel at a time, but since you're doing all this from a script, you can use $/reports/create to create one, $/reports/get to generate it, then $/reports/delete to clean it up.

Alternatively, if you have the item ID's you want, you can use $/feedItems/get to retrieve them. This API call has a feature where you can pass in a report config as the POST data, and Awasu will use that to generate the response.

Or, if you know you're always going to be doing 1 channel at a time, you could create a new channel template that generates the channel's page as XML or JSON instead of HTML, temporarily re-configure the channel to use that template, call $/channels/get, then configure the channel back.

awasu.user
Posts: 81
Joined: Fri Jan 06, 2017 12:50 pm

Postby awasu.user » Sat Jul 22, 2017 4:31 pm

From beginning I start with definition:
Channel pack - few channels on the same name fx. "culture", "politics".

I want by one API call get unread articles from channel pack fx. "culture". I need URL, title, publishing date and Channel Name. General idea is use more RAM than Hard disk to this operation to protect HD from devastation. The worst scenario as I calculated is write 1TB data / day what is too much for my SSD.

Why one API call? From my observation is faster.

User avatar
support
Site Admin
Posts: 3021
Joined: Fri Feb 07, 2003 12:48 pm
Location: Melbourne, Australia
Contact:

Postby support » Sun Jul 23, 2017 1:13 am

awasu.user wrote:Why [not]one API call?

Because Awasu has been written over a period of many years, and things that might make sense now maybe didn't back then :-) The idea was always to let information be extracted out of Awasu based on content, not on structure e.g. "what stories are there about subject X within the past 3 days?", not "give me information based on how I've organized my feeds".

I'll see what I can do to add something to Awasu to give you what you want; the suggestions I gave earlier are ways to do what you want with Awasu as it is now.

awasu.user wrote:General idea is use more RAM than Hard disk to this operation to protect HD from devastation.

I used to worry about SSD writes as well, but these days, SSD's are good enough that they are likely to outlast the rest of your computer and you don't really need to worry about it :-) The new build I'm going to send you soon will go a long way to reduce writes (it's search index updates that causing all this disk activity, and the new build reduces this).

One option you could really consider is to set up a RAM disk and run Awasu from that (I use DataRAM). Configure your data files to live in a specified directory, and move your existing files (Local and Roaming data) into that directory.

Then, when you want to run Awasu, copy those files to your RAM disk, point the DEBUG.INI file to the new files, and Awasu will run using those files. When you're done, copy the files back from the RAM disk to your SSD. Awasu will really fly if you do it like this ::-): The only down-side is that if your PC crashes, you will lose those files on the RAM disk (although you will still have the ones on the disk, so you will only lost the work you've done since the last copy).

Edit: BTW, the DataRAM program has an option to save/load the contents of the RAM disk to a disk file, so you don't even have to worry about copying the Awasu data files around. If you're going to do things like this, you might also want to consider running Awasu in portable mode, so that everything is on the RAM disk, so that it's all self-contained (this also means you don't have to worry about things like absolute vs. relative file paths, what drive letter you mount the RAM disk on, etc.)

awasu.user
Posts: 81
Joined: Fri Jan 06, 2017 12:50 pm

Postby awasu.user » Tue Jul 25, 2017 11:05 am

support wrote:Because Awasu has been written over a period of many years, and things that might make sense now maybe didn't back then The idea was always to let information be extracted out of Awasu based on content, not on structure e.g. "what stories are there about subject X within the past 3 days?", not "give me information based on how I've organized my feeds".

I'll see what I can do to add something to Awasu to give you what you want; the suggestions I gave earlier are ways to do what you want with Awasu as it is now.


Yes, as I read on your blog Awasu has a long history. For me crucial to make me happy with API is make it more flexible with output format and using directory structure. It's logical steps that user organize channels in folders tree and get data with this structure as base to aditional work. It's sense organising data in folders.

Your sugestion about DataRAM is versy intersting. I'll think about it. For start a day crucial for me is make as fast as posible channel data for some processing stuff. Now I have to run Awasu afternoon and early night to speed up this and make more actual data from search channels.

User avatar
support
Site Admin
Posts: 3021
Joined: Fri Feb 07, 2003 12:48 pm
Location: Melbourne, Australia
Contact:

Postby support » Tue Jul 25, 2017 11:45 am

awasu.user wrote:Your sugestion about DataRAM is versy intersting. I'll think about it.

If you use the DataRAM feature to save the contents of the RAM disk to a file, make sure Awasu is not running when you do it. Otherwise, you might save files while Awasu is updating them i.e. they will be saved to disk in an inconsistent state.

BTW, I'm doing final testing on a new build for you now, and it should make a difference. Don't forget me to send the performance logs I asked for earlier.


Return to “Awasu - Feature Requests”

Who is online

Users browsing this forum: No registered users and 4 guests