abwilson
Posts: 247
Joined: Sun Feb 09, 2003 12:36 am
Location: San Francisco, CA -- USA

Postby abwilson » Wed Nov 03, 2004 5:28 am

Perhaps there is a better way to do this, but...

I would like to run one or more (sequential or parallel) processes -- some potentially long running -- within a channel plug-in so that they can report status, summaries, etc. via RSS.

Could you create another "special" parameter (special like DownloadUrl) that would either be binary (plug-in times out according to global timeout setting or it doesn't) or perhaps a value: 0 seconds means no timeout, i.e., it can run as long as it needs to, while a positive integer would represent the number of seconds until the plug-in timed out and Awasu gave up on it.

Thanks

Allan

User avatar
support
Site Admin
Posts: 3021
Joined: Fri Feb 07, 2003 12:48 pm
Location: Melbourne, Australia
Contact:

Postby support » Wed Nov 03, 2004 5:40 am

abwilson wrote:I would like to run one or more (sequential or parallel) processes -- some potentially long running -- within a channel plug-in so that they can report status, summaries, etc. via RSS.


What exactly are you trying to run?

Allowing plugins to run indefinitely is not a good idea. Per-plugin timeouts would be easy and make more sense but it would be configurable by the *user*, not the plugin. Nevertheless, long-running plugin channels are not good - Awasu can have only a certain number of channels updating at once so if your plugin runs for a long time, you will be hogging one of those slots. Fill them all up and no-one else will be able to update.

Depending on exactly what you are trying to do, I would think about an architecture like this: when the plugin runs, it checks to see if a background daemon is running and if not, spawns it. This runs continously in the background doing whatever it needs to do. Plugins then simply communicate with this daemon and (quickly) retrieve whatever they needs to generate a feed. Or you could have the daemon running as a service :-)

abwilson
Posts: 247
Joined: Sun Feb 09, 2003 12:36 am
Location: San Francisco, CA -- USA

Postby abwilson » Wed Nov 03, 2004 6:23 am

Yes, I considered a daemon/service, but don't know if I have enough tasks overall to justify it right now.

I have thought in general about this before, but here it what triggered the latest:

I decided that until such time as Awasu implements enclosures (unless it already does and I missed it!) it would be interesting to have a plug-in do it. First I was going to do a channel hook, but then I would have to identify and attach the hook to each channel that might have enclosures. Then I decided a program/plug-in could simply look through all the .xml files in my logs directory and find all the feeds with enclosures. It could then download them and generate a report in RSS.

I wrote such a beast, but some enclosures are many megabytes in size and downloads take more time than the timeout my Awasu is currently configured for and I didn't want to change the global timeout value for this. Hence, the idea of letting a plug-in say "let me run until I finish".

abwilson
Posts: 247
Joined: Sun Feb 09, 2003 12:36 am
Location: San Francisco, CA -- USA

Postby abwilson » Wed Nov 03, 2004 6:34 am

Sorry, I still haven't made it completely clear that the user, not the plug-in, would set the timeout value. Earlier when I said "special" like DownloadUrl I meant it would be in the .plugin file so the user would set it when the channel is created.

Allan

User avatar
support
Site Admin
Posts: 3021
Joined: Fri Feb 07, 2003 12:48 pm
Location: Melbourne, Australia
Contact:

Postby support » Wed Nov 03, 2004 12:53 pm

abwilson wrote:I decided that until such time as Awasu implements enclosures (unless it already does and I missed it!) it would be interesting to have a plug-in do it.


Man, you *really* need to start pinging me before starting on these projects :-) I've literally just finished the first cut of enclosures (and more) and am starting full-on testing now. It'll be in the next alpha, say, end of next week.

abwilson wrote:I wrote such a beast, but some enclosures are many megabytes in size and downloads take more time than the timeout my Awasu is currently configured for and I didn't want to change the global timeout value for this. Hence, the idea of letting a plug-in say "let me run until I finish".


In this case, I would simply spawn off another process to do the download. It would take the URL and where to save it as parameters. The plugin wouldn't care when it finishes so it would just kick it off and forget about it. Easy :-)

abwilson
Posts: 247
Joined: Sun Feb 09, 2003 12:36 am
Location: San Francisco, CA -- USA

Postby abwilson » Wed Nov 03, 2004 6:23 pm

Man, you *really* need to start pinging me before starting on these projects icon_smile.gif I've literally just finished the first cut of enclosures (and more) and am starting full-on testing now. It'll be in the next alpha, say, end of next week.


Well, it wouldn't have really helped -- I only happened to notice a reference to a feed with an enclosure that I wanted to check out about this time yesterday. I put the plug-in together in about an hour (most of that time was spent figuring out a new XML/RSS parser I wanted to try). So, I created my first cut before you could have responded to a query about "official" enclosure support being imminent. :)

In this case, I would simply spawn off another process to do the download. It would take the URL and where to save it as parameters. The plugin wouldn't care when it finishes so it would just kick it off and forget about it. Easy


Easy, but not very satisfying:

1) Awasu just functions as an elaborate cron;

2) The plug-in could report via RSS the enclosures it *intended* to have downloaded before kicking off the separate process, but the *actual* download status/history would have to be kept in a separate log (and/or monitored in yet another plug-in channel).

So, forgetting the issue of how enclosures are handled, I still don't believe there is a generic framework for "background" processing and monitoring that would be satisfactory.

User avatar
support
Site Admin
Posts: 3021
Joined: Fri Feb 07, 2003 12:48 pm
Location: Melbourne, Australia
Contact:

Postby support » Thu Nov 04, 2004 12:34 pm

abwilson wrote:Easy, but not very satisfying:

2) The plug-in could report via RSS the enclosures it *intended* to have downloaded before kicking off the separate process, but the *actual* download status/history would have to be kept in a separate log (and/or monitored in yet another plug-in channel).


This is perfectly OK. How else were you going to do it? Even if you had a long-running plugin, every time it got invoked it would still report each enclosure as "still downloading" or "download complete".

abwilson wrote:So, forgetting the issue of how enclosures are handled, I still don't believe there is a generic framework for "background" processing and monitoring that would be satisfactory.


A plugin is invoked to *retrieve* the latest information from a given data source. Having a framework to manage processes that *do* the actual work being monitored is beyond the scope of what Awasu is about. What you are suggesting is in fact a cron framework :-)

abwilson
Posts: 247
Joined: Sun Feb 09, 2003 12:36 am
Location: San Francisco, CA -- USA

Postby abwilson » Sat Nov 06, 2004 1:27 am

This is perfectly OK. How else were you going to do it? Even if you had a long-running plugin, every time it got invoked it would still report each enclosure as "still downloading" or "download complete".


Perhaps I need to clarify: I wrote this with the intention that "all the action" be performed by a single, standard channel plug-in. The plug-in would look through my Awasu Logs directory and determine what enclosures were available to download by examining the RSS in the .xml files. It would then proceed to download those new enclosures that match plug-in parameters: max size enclosure to download, and types to include and exclude. It would download what it could, reporting download success/failure for each via RSS.

The only thing different about this plug-in and, say, my plug-in that monitors the Windows Event Log is how long each runs: grabbing data from the Event Log is fast enough that that plug-in completes in a "normal" time; downloading data over the Internet is much slower. Both plug-ins would gather data (one locally, one remotely) and both would report their results via RSS. I suggested a "special like DownloadUrl" variable, e.g., PluginTimeout, to allow the user to let Awasu know a particular plug-in needed more wall-clock time to complete (to avoid using the "global" Awasu channel timeout setting just to handle a special case).

I was planning on setting the channel to update once or twice per day. In this particular instance, "long running" is much less than 12 hours per day. :) In fact, so far the typical daily enclosure list might take 20-45 minutes to download.

Awasu would start the plug-in normally and it would be expected to run to completion. PluginTimeout would be set by the user to either 0 for no elapsed-time limit or a value in seconds; 3600 would obviously give it an hour, after which Awasu would shut it down if it hadn't finished.

A plugin is invoked to *retrieve* the latest information from a given data source. Having a framework to manage processes that *do* the actual work being monitored is beyond the scope of what Awasu is about. What you are suggesting is in fact a cron framework


Yes; exactly what I intended. Since you are building in enclosure support right now, my current plug-in toy isn't really necessary, but I built it precisely to retrieve the latest information (the enclosures) and report its (long running!) activities via RSS.

Having said/written the above, you can obviously take whatever path you deem appropriate for Awasu. I am still able to get everything done I need -- sometimes using Awasu, sometimes not. :)


Return to “Awasu - Feature Requests”

Who is online

Users browsing this forum: No registered users and 2 guests