Wow! Listening to you two brainstorming is always an amazing, entertaining, and often mind boggling event. Some of the features you are adding, plan to add, and ponder about adding to Awasu are very interesting and cool. My needs and what we are doing at Leading Edge Software are very relevant to what you are doing here.
When I first started using Awasu, it was my intention to have Awasu create piles of content for some of my and some of my clients web sites. Awasu was simply going to send posts to my sql database via the sqlhook, and/or email them via the sendmailhook, and/or ftp them to my server. In the case of the sql and send mail a cron would take over and put all this data where it belongs. Using the FTP idea I had the channel reports formatted as rss, xml, atom, etc and the websites would display the information using appropriate scripts, parsers, etc. This was all a very interesting and a promising path, and if I was a script programmer I could have taken it to the next level, but I'm not so I didn't. The bottom line is that I got it all to work, but there were a few things I couldn't do and some problems.
1. My local version of mysql got corrupted and my Awasu SqlHook stopped working. No matter how many times I tried to reinstall both of them it still wouldn't work. My recollection is that the SqlHook needs a local version of mysql installed or at least one of its components to run, am I wrong on that? However that has been my experience. I don't want to have to install mysql on my local system to do what I need to do, nor do I want my clients to have to install it either. So the sqlhook didn't work for me, and for what you guys have been talking about I'd say that sending data to a sql data base is a pretty important feature. Awasu's sends data to a sql database and any sql capable application can access it and do what it wants with it and Awasu is done with and has done its job well.
2. Then SendEmail worked too, but I found that it really slowed down Awasu to the point that my updates were taking too long. We're talking about sending 300+ emails at a time. I even set up separate channels that didn't get updated with all the others and then I would update those channels manually but that took too much time as well. Those channels where generated from ohter channel reports. At some point I could have set my SendEmail Channels to run after the main update but you never really know how long those updates are going to take so effecient timing could be an issue. Either way sending out all those emails seemed to be taking a toll on the system.
3. Ftping worked quite well, and with scripts in place on the server I was able to parse, convert, and display whatever was in those files on my website. It was all completely automated and working well, and for those that know a little about php, html, etc. it could be a good solution.
The problem with all of the options above is that there are many many variables involved. One of them is, do you really want to do this to begin with because you are throwing tons of data onto you're website without knowing what is actually in it or what it is going to look like. And what about categorizing it, or attaching an Author to it, and even editing it, changing it's time stamp and whatever else you would want to do with your content. btw. I had separate channel reports for separate categories on my blog site and that way the data was tagged and catagorized appropriately.
Plus, you also need to think about the legalities of all of this. It is legal to post some one else's articles for the sake of review or commentary as long as you add some value to it but just to post their post on your site is not legal without their permission or the appropriate licensing. Also, if you are adding content to your site for the sake of having content for the search engines to find your not going to benefit from it much because search engines know and consider where the original content came from and will rank the originators way higher then yours as it should be. That is if you don't spin it or add some value to it. Content Spinning in Code Orange? (in the works)
So to benefit from automated content flowing into your website you have to add to it and/or modify it, and add some value to it.
So all that being said I needed to come up with my own solution and Code Orange was born.
I won't get into all of the details about Code Orange but you can take a look at it at http://www.leadingedgesoftware.net/category/products/code-orange/
but what is relevant to this conversation is the fact that...
Code Orange will import Awasu Channel Reports
, xml, atom, rss, plain text, html, etc from a local file or from a remote file available via ftp, http, html and either immediately send it to your blog or allow you to quickly look at each record review it, edit it, spin it, decide what blog it is supposed to go to, what categories and tags it belongs too, etc, etc.
***in our testing just yesterday Code Orange imported a channel report with 276 posts in it in less then a minute. Parsing maybe took 15 seconds, the rest was adding the records to the database.***
btw. Code Orange has a timer mechanism so that it can be set to do a lot of these things automatically.
Code Orange can also send out HTTP requests and process that data as well. So were watching the whole HTTP API thread.
So when it comes to the post processing side of this equation I think Code Orange will be the place to be. I am a huge fan of Awasu and all of the things you guys are trying to put into it, as well as the great support you guys offer, and that is why we are building many features into Code Orange specifically for Awasu.
If you have any questions post here if you wish or PM or email me.
Now back to work.