herkimer
Posts: 3
Joined: Wed Oct 08, 2008 11:20 am

Postby herkimer » Wed Aug 05, 2009 7:57 pm

OK, I am able to upload html files with my FTP client, and the pages look fine. Can anyone help me with a method to generate a new page for content when one becomes full (or at a certain size). The pages ra egetting too large. I need some way to spread the content over several pages.

Thanks in advance!

User avatar
support
Site Admin
Posts: 3022
Joined: Fri Feb 07, 2003 12:48 pm
Location: Melbourne, Australia
Contact:

Postby support » Thu Aug 06, 2009 12:39 pm

herkimer wrote:I need some way to spread the content over several pages.

This is a tricky one. I can't think of any way to get Awasu to do it and this would be a very tricky feature to add.

You could generate the file onto your PC, chop out all the content except for what you want on the first page, then FTP that up as page1.html. Then do the same thing for page2.html, page3.html, etc. It's a bit of a manual process but you could write a program that did it for you (either on the PC or on your web server).

User avatar
kevotheclone
Posts: 239
Joined: Mon Sep 08, 2008 7:16 pm
Location: Elk Grove, California

Postby kevotheclone » Thu Aug 06, 2009 11:03 pm

herkimer, What do you mean by "too large"? Is it too much vertical scrolling to where the page is getting ugly to work with or is the page really too large to load in the browser?

What are you exporting? All feed item data via a Channel Report?

If it is just too much vertical scrolling, I might have an alternative to multiple pages for you.

Also, what kind of coding skills do you have?
Client side JavaScript? Ever try jQuery?

Your Channel Report Template could create a large page, with client-side JavaScript, that could show/hide groups of feed items based upon some criteria: Channel Name, date, etc. This could reduce/eliminate excessive vertical scrolling.

Server side coding using ASP/JSP/PHP?

Your Channel Report Template could create an text-based data source using the MetaChannel.template or an template of your own creating JSON, tab-separated values, etc.
An ASP/JSP/PHP page could read the data source and show various groups of feed items based upon some criteria: Channel Name, date, etc.
Microsoft's ADO can serialize Recordsets to an XML format; a Channel Report Template could generate this same XML format and ADO code could de-serialize it back into a Recordset object. .NET has similar capabilites with the DataSet object. I'm not a PHP expert, but I know it has a text-based data format that is very similar to JSON, so you could output this format and I believe PHP could read it and work with it using "higher level code".


Give me some details and I'll try to help you.
In the meantime I'll try to sneak in a feature request... :wink:

User avatar
kevotheclone
Posts: 239
Joined: Mon Sep 08, 2008 7:16 pm
Location: Elk Grove, California

Postby kevotheclone » Fri Aug 07, 2009 5:31 am

Warning, potential feature request disguised as a reply...

One thing that I've thought of, but personally don't need at this time, is the ability to attach one or more :idea: Channel Report Hooks :idea: to a Channel Report. Just like Channel Hooks these would be pluggable programs that Awasu users would create that would most likely process the output of their Channel Reports. The only events I can really think of are BeginChannelReportUpdate and EndChannelReportUpdate and really I think 99% of the usage would be the with the EndChannelReportUpdate event.

Inside my twisted brain
I don't just think of Channel Reports as writing output files that are immediately viewable by humans (e.g. HTML), I also think of Channel Reports as being a way to export feed item data, en masse, into machine readable formats (XML, tab-separated values, etc.). I've hinted at it here, and Taka has mentioned it several times over the eons of Awasu existence. When he talks about "machine-to-machine communication" and "end-to-end information management" Awasu doesn't have to be at either one of those "ends", it could be somewhere in the middle;
    1) intelligently retrieving focused feed items via Search Channels
    2) automatically saving found items to one or more Workpad
    3) an Awasu user can (optionally) add notes to feed items and manually add/remove Workpad items
    4) the composite Workpad can be re-published to a new data source via the MetaChannel.template or a similar template of your own creation which will be further analyzed/processed by some other program in the chain.

Upcoming in Awasu version 2.4.1 you can create JSON data that can easily be processed by any language capable of working with JSON (scroll to the bottom of the page and pick you're favorite language). JSON is built into JavaScript and I'm working on a interactive (not static) Channel Report that has been called "insanely cool" by a guy that has written some "insanely cool" software himself, so I trust he know's what he's talking about.

As I mentioned in my last post there are also XML formats that Awasu Channel Reports can produce that can be de-serialized into database recordset objects (if anybody needs an example of these let me know).

So we can output a wide variety of text formats using Awasu's Channel Reports, but once generated we can't automatically trigger some external program to process the data, unless you use the Channel Report's email notification to trigger the external process, but most of us don't trigger our processes via email.

I know you can do some of the same things with Channel Hooks except that: 1) Channel Hooks would get called many times compared to a Channel Report Hook, 2) manually attaching a lot of Channel Hooks is tedious, and 3) we've already discussed Global Channel Hooks and decided that it's a bad idea.

So a Channel Report Hook would be a much cleaner way to process a lot of feed item data. I know this feature would probably not be an easy one to implement, but I also think that this feature wouldn't be impossible to implement since Awasu already has that tried-and-true Channel Hook code that could be adapted for this purpose. But I'm sure Taka is going to need to hear from more voices than just mine before he'd consider adding Channel Report Hooks to an already feature-packed Awasu.

Oh yeah... I almost forgot... herkimer a Channel Report Hook could be able to take your one big file and chop it up into smaller files and link them accordingly. See that didn't sound like a feature request did it? :wink:

User avatar
kevotheclone
Posts: 239
Joined: Mon Sep 08, 2008 7:16 pm
Location: Elk Grove, California

Postby kevotheclone » Fri Aug 07, 2009 7:37 am

Warning, yet another potential feature request disguised as a reply...

Push vs. Pull
In my last post I pondered Channel Report Hooks as symbiotic a way of processing newly exported Channel Report data. In that scenario Awasu "pushes" the Channel Report data to a file on your LAN or via FTP and it "pushes" a process to run.

There is an alternative... The Pull :idea:
In this scenario you have a program that could be internal to Awasu (e.g. an Application Plugin) or external to Awasu. Instead of the data getting "pushed" to where the program can access it and the program getting "pushed" into action; in this scenario the program is run first and it requests ("pulls") the Channel Report's data from Awasu. Maybe something along the lines of an HTTP request might work: http://awasuserver:2604/runreports?reportname=

This technique would work well for Application Plugins since they run from within Awasu and would probably work well for intranet based scenarios, but due to firewall restrictions it might not work well over the internet.

Of course, there are lots of details to work out since a Channel Report can write to a file, email a file, FTP a file, and display the results within Awasu. Are there any scenarios in which some of these actions should not happen; like if the request is made from a different IP address than the one Awasu is running on should the Channel Report's results be displayed within Awasu... probably not? Also what should the call to "runreports" return via HTTP, just an HTTP return code or should the actual report data be returned as well. And should Awasu allow multiple reports to be run in one call, maybe with a delimited list of Channel Report names or a wildcard (*) to run all reports?

I did previously "suggest" this HTTP API to Taka via email a few months ago and he didn't say "No!", so he might be willing to implement it in a future release. I know he's busy right now with the 2.4.1 release, and I'm busy with a bunch of things, so I don't really want to try to hammer out the details right now. I wouldn't have brought this idea to the forum right now, except that the ability to remotely run Channel Reports might be able solve herkimer's problem. So if any of you Awasu Power Users are listening, keep this in the back of your minds and maybe we can discuss it some day in the future.

Oh yeah... I almost forgot... herkimer in this scenario you would have a program that you run that would call Awasu and run a Channel Report to extract the necessary data as one big file and chop it up into smaller files and link them accordingly. See that didn't sound like a feature request did it? :wink:

I promise this is it... for today, gotta get some Zzzzzzzzzzzzzzz :sleepy:

User avatar
support
Site Admin
Posts: 3022
Joined: Fri Feb 07, 2003 12:48 pm
Location: Melbourne, Australia
Contact:

Postby support » Sat Aug 08, 2009 9:36 am

kevotheclone wrote:[One thing that I've thought of, but personally don't need at this time, is the ability to attach one or more :idea: Channel Report Hooks :idea: to a Channel Report. Just like Channel Hooks these would be pluggable programs that Awasu users would create that would most likely process the output of their Channel Reports. The only events I can really think of are BeginChannelReportUpdate and EndChannelReportUpdate and really I think 99% of the usage would be the with the EndChannelReportUpdate event.

This is a really amazing idea :bow: It wouldn't be so much hooks as it would be "run this program after generating the report" (what would you need to do <i>before</i> running a report?). This idea is already present in Awasu - you can configure a program to be run after saving Awasu sync data - and the intent is the same, to provide a way to do some post-processing. In the case of sync, it was to allow different ways (other than the built-in ones) of sending the sync data file somewhere and for reports it could be the same thing, or to do some post-processing on the report file (or both!).

kevotheclone wrote:When he talks about "machine-to-machine communication" and "end-to-end information management" Awasu doesn't have to be at either one of those "ends", it could be somewhere in the middle

When I say "end-to-end", the idea is that Awasu retrieves information from one endpoint and transfers it to another, perhaps doing some analysis and processing on it along the way. Awasu has always been in the middle, as an "information hub."

kevotheclone wrote:I know this feature would probably not be an easy one to implement

Actually, simply running a program after the report had been generated would be very easy to implement. Definitely in the next release :clap:

User avatar
support
Site Admin
Posts: 3022
Joined: Fri Feb 07, 2003 12:48 pm
Location: Melbourne, Australia
Contact:

Postby support » Sat Aug 08, 2009 9:42 am

kevotheclone wrote:In this scenario you have a program that could be internal to Awasu (e.g. an Application Plugin) or external to Awasu. Instead of the data getting "pushed" to where the program can access it and the program getting "pushed" into action; in this scenario the program is run first and it requests ("pulls") the Channel Report's data from Awasu. Maybe something along the lines of an HTTP request might work: http://awasuserver:2604/runreports?reportname=

Holy moly, this is even cooler! We add something to Awasu's HTTP interface so that a report can be run by "opening" a URL. So, for example, people could add a link to an intranet page that, when clicked on, sends a request to Awasu to generate a report, which it does, then it runs a program to do some post-processing on it and then uploads it or emails it somewhere. That's just insane! :blink:

User avatar
kevotheclone
Posts: 239
Joined: Mon Sep 08, 2008 7:16 pm
Location: Elk Grove, California

Postby kevotheclone » Sun Aug 09, 2009 2:22 am

support wrote:Awasu has always been in the middle, as an "information hub."

Yes I probably didn't make my point clear enough; and in quoting you I should have added "information hub". Believe it or not I was trying to reiterate this fact. Awasu has always been in the middle, although I think if you are producing HTML with a Channel Report for human consumption, then Awasu is a little closer to one end; where if you are producing a data format (XML, JSON. Tab-separated-text, etc) that will be processed by someone/something else then Awasu is even more in the middle. You've always had that vision, and have provided the MetaChannel.template as an example of a "data" output format, but sometimes with our repetitive usage patterns can get ourselves trapped in a mental "box" that we have trouble thinking outside of.

My point was that in those rare instances when Awasu's Template Processing Engine/Channel Reports can't produce exactly want you need, you should try to publish the output to a data file and process it externally.

You basically told herkimer the same thing except that you didn't stress that a nice well formed ATOM feed, via the MetaChannel.template, would generally be easier to chop up than an HTML file. I guess I'm really into exportable data side of Awasu.


Push
Yes a full blown Channel Report Hook system is probably more than anyone needs, so a single program after the Channel Report completes should be good enough. Unless somebody can think of something that they need in a config file I dont see a reason to make this any more complicate than a single program run after the report completes.

Thanks for considering it in the next release (2.4.2).


Pull
My primary reason for wanting this "Run Channel Report" HTTP API is to be able to call it from an Application Plugin to get all of the "new" or "unread" feed item data for processing. There's an App. Plugin which I haven't told you about yet (I'll do so in the Plugins forum soon), basically it will analyze and help to summarize the feed data fed to it. There may actually be 3 different variations of this App. Plugin, but that'll spawn a whole conversation in itself. I've previously hinted at it here.

To quote you once again:
support wrote:One of the key principles behind Awasu's design is to make it easy for the user to manage large amounts of information coming in.


I think this App. Plugin supports this principle. ::-):

I can still create this App. Plugin without the "Run Channel Report" HTTP API, but the user would have to run the appropriate Channel Report first and then run this App. Plugin; so the "Run Channel Report" HTTP API just makes this App. Plugin easier to run.

Yet another crazy idea (don't worry I'll have to implement this one)
I might have a way to make vertically scrolling a large page more interesting.

Remember that "Timeline" Channel Report Template? Timelines are usually displayed horizontally, but there is a switch in the Simile timeline code to make the timeline display vertically. So maybe I could hack together a fairly narrow vertical timeline running in the right 1/4 of the page that is sync'd with the feed items displayed in the left 3/4 of the page; new feed items at the top, oldest feed items at the bottom. I'm not sure I can get the synchronized scrolling working, but it'd be interesting to try, I'll put it on my list of things to do. :whip:

jerrymartin
Posts: 27
Joined: Mon Mar 09, 2009 6:29 pm
Contact:

Postby jerrymartin » Wed Sep 02, 2009 7:17 pm

Wow! Listening to you two brainstorming is always an amazing, entertaining, and often mind boggling event. Some of the features you are adding, plan to add, and ponder about adding to Awasu are very interesting and cool. My needs and what we are doing at Leading Edge Software are very relevant to what you are doing here.

When I first started using Awasu, it was my intention to have Awasu create piles of content for some of my and some of my clients web sites. Awasu was simply going to send posts to my sql database via the sqlhook, and/or email them via the sendmailhook, and/or ftp them to my server. In the case of the sql and send mail a cron would take over and put all this data where it belongs. Using the FTP idea I had the channel reports formatted as rss, xml, atom, etc and the websites would display the information using appropriate scripts, parsers, etc. This was all a very interesting and a promising path, and if I was a script programmer I could have taken it to the next level, but I'm not so I didn't. The bottom line is that I got it all to work, but there were a few things I couldn't do and some problems.

Problems:
1. My local version of mysql got corrupted and my Awasu SqlHook stopped working. No matter how many times I tried to reinstall both of them it still wouldn't work. My recollection is that the SqlHook needs a local version of mysql installed or at least one of its components to run, am I wrong on that? However that has been my experience. I don't want to have to install mysql on my local system to do what I need to do, nor do I want my clients to have to install it either. So the sqlhook didn't work for me, and for what you guys have been talking about I'd say that sending data to a sql data base is a pretty important feature. Awasu's sends data to a sql database and any sql capable application can access it and do what it wants with it and Awasu is done with and has done its job well.

2. Then SendEmail worked too, but I found that it really slowed down Awasu to the point that my updates were taking too long. We're talking about sending 300+ emails at a time. I even set up separate channels that didn't get updated with all the others and then I would update those channels manually but that took too much time as well. Those channels where generated from ohter channel reports. At some point I could have set my SendEmail Channels to run after the main update but you never really know how long those updates are going to take so effecient timing could be an issue. Either way sending out all those emails seemed to be taking a toll on the system.

3. Ftping worked quite well, and with scripts in place on the server I was able to parse, convert, and display whatever was in those files on my website. It was all completely automated and working well, and for those that know a little about php, html, etc. it could be a good solution.

The problem with all of the options above is that there are many many variables involved. One of them is, do you really want to do this to begin with because you are throwing tons of data onto you're website without knowing what is actually in it or what it is going to look like. And what about categorizing it, or attaching an Author to it, and even editing it, changing it's time stamp and whatever else you would want to do with your content. btw. I had separate channel reports for separate categories on my blog site and that way the data was tagged and catagorized appropriately.

Plus, you also need to think about the legalities of all of this. It is legal to post some one else's articles for the sake of review or commentary as long as you add some value to it but just to post their post on your site is not legal without their permission or the appropriate licensing. Also, if you are adding content to your site for the sake of having content for the search engines to find your not going to benefit from it much because search engines know and consider where the original content came from and will rank the originators way higher then yours as it should be. That is if you don't spin it or add some value to it. Content Spinning in Code Orange? (in the works)

So to benefit from automated content flowing into your website you have to add to it and/or modify it, and add some value to it.

Solutions:

So all that being said I needed to come up with my own solution and Code Orange was born.

I won't get into all of the details about Code Orange but you can take a look at it at http://www.leadingedgesoftware.net/category/products/code-orange/ but what is relevant to this conversation is the fact that...

Code Orange will import Awasu Channel Reports, xml, atom, rss, plain text, html, etc from a local file or from a remote file available via ftp, http, html and either immediately send it to your blog or allow you to quickly look at each record review it, edit it, spin it, decide what blog it is supposed to go to, what categories and tags it belongs too, etc, etc.

***in our testing just yesterday Code Orange imported a channel report with 276 posts in it in less then a minute. Parsing maybe took 15 seconds, the rest was adding the records to the database.***

btw. Code Orange has a timer mechanism so that it can be set to do a lot of these things automatically.

Code Orange can also send out HTTP requests and process that data as well. So were watching the whole HTTP API thread. :)

So when it comes to the post processing side of this equation I think Code Orange will be the place to be. I am a huge fan of Awasu and all of the things you guys are trying to put into it, as well as the great support you guys offer, and that is why we are building many features into Code Orange specifically for Awasu.

If you have any questions post here if you wish or PM or email me.

Now back to work.

Jerry


Return to “Awasu - General Discussion”

Who is online

Users browsing this forum: No registered users and 4 guests