Greg_Spain
Posts: 3
Joined: Tue Apr 14, 2009 12:22 pm

Postby Greg_Spain » Tue Apr 14, 2009 12:33 pm

Hello,

My company is looking to buy an Awasu license to keep track of industry news. I'm currently testing the program using Webscrape to generate feeds from websites without RSS feeds.

I have the personal edition of Awasu at home, and Webscrape runs beautifully. However, when I try to run the same files at work, Webscrape is unable to get past the proxy.

If I try to run Webscrape within Awasu, it brings up the message "Updated xxxx Ok", but no feed items appear. WebscrapeSettings, however, returns a 407 proxy authentication error.

Is there any way of getting Webscrape to overcome this problem? Any help would be appreciated.

Thanks!


Greg

User avatar
support
Site Admin
Posts: 3021
Joined: Fri Feb 07, 2003 12:48 pm
Location: Melbourne, Australia
Contact:

Postby support » Tue Apr 14, 2009 1:51 pm

Greg_Spain wrote:If I try to run Webscrape within Awasu, it brings up the message "Updated xxxx Ok", but no feed items appear. WebscrapeSettings, however, returns a 407 proxy authentication error

Yes, the way WebScrape and WebScrapeSettings download the page is different. WebScrape just downloads it every time it's run, WebScrapeSettings downloads it once and saves it away, then passes that file to WebScrape every time you want to test your configuration (the idea being that this will happen a lot, so you don't want to have to wait for the page to be downloaded every time).

WebScrape was written by one of our users and he's no longer involved with Awasu. He never released the source code which means we can't fix it to go through a proxy. I've been having a play with it and the only way I can think of to get it to work would be to write an intermediate plugin that downloaded the page, through a proxy if necessary, saved it to a temp file and then told WebScrape to process that.

It's a bit messy but it shouldn't be hard to do. Awasu passes through an INI file to the plugin containing all the information it needs to do it's thing. You would configure the channel to run this intermediate plugin (let's call it WebScrapeProxy.py) that read this INI file to figure out the URL to be scraped and downloaded it to a temp file. It would then tweak the INI file to point to the temp file (instead of the real URL), then call WebScrape.exe. WebScrape.exe would have no idea it was not being called by Awasu, it just does what it's told to do in the INI file so it would do it's thing, WebScrapeProxy.py deletes the temp file and control returns back to Awasu...

User avatar
support
Site Admin
Posts: 3021
Joined: Fri Feb 07, 2003 12:48 pm
Location: Melbourne, Australia
Contact:

Postby support » Tue Apr 14, 2009 2:02 pm

There also might be an alternate way. If a plugin defines a parameter called DownloadUrl, Awasu will download the specified URL on behalf of the plugin. This feature was added specifically to get around problems like this, and IIRC it was specifically for WebScrape.

WebScrape defines this parameter but I can't get it to use the downloaded file instead of trying to download the URL itself. The 1.30 documentation says that the only change was support for this parameter so I'm guessing it works, I'm just not using the right incantations. It's gettting late here but you might want to have a play with this and see if you can get it to work. There might also be something in the documentation :-)

At the very least, this makes writing WebScrapeProxy much easier since you can get Awasu to do the work of downloading the file, using the proxy authentication details it already has, then just fool WebScrape to use that instead.

Greg_Spain
Posts: 3
Joined: Tue Apr 14, 2009 12:22 pm

Postby Greg_Spain » Wed Apr 15, 2009 9:49 am

Thank you for your suggestions. I tried using the DownloadUrl parameter to no avail. As for the intermediate plugin, it seems like the best possible fix, given the circumstances. Unfortunately, I am not familiar with Python. I'll give it a go, but it's a large pill to swallow without any prior experience, so the outlook at the moment is rather bleak...

Thanks again!

User avatar
support
Site Admin
Posts: 3021
Joined: Fri Feb 07, 2003 12:48 pm
Location: Melbourne, Australia
Contact:

Postby support » Wed Apr 15, 2009 10:06 am

Greg_Spain wrote:Unfortunately, I am not familiar with Python. I'll give it a go, but it's a large pill to swallow without any prior experience

Well, it doesn't have to be written in Python, it can be in any language. It sounds like you can program in something, so just use whatever language you're most comfortable with. For example, someone's written something in VBScript here.

Greg_Spain
Posts: 3
Joined: Tue Apr 14, 2009 12:22 pm

Postby Greg_Spain » Wed Apr 15, 2009 10:14 am

Thank you for the link. I'm a lot more comfortable with Vb, and it's a good starting point.

User avatar
support
Site Admin
Posts: 3021
Joined: Fri Feb 07, 2003 12:48 pm
Location: Melbourne, Australia
Contact:

Postby support » Wed Apr 15, 2009 11:23 am

Greg_Spain wrote:Thank you for the link. I'm a lot more comfortable with Vb, and it's a good starting point.

Yikes, I just realized the thing I linked to isn't a channel plugin. Sorry :oops:

The documentation for writing plugins starts here and while most of the examples are in Python, it's a really easy language to understand if you're just reading it.

User avatar
kevotheclone
Posts: 239
Joined: Mon Sep 08, 2008 7:16 pm
Location: Elk Grove, California

Postby kevotheclone » Wed May 13, 2009 1:02 am

Hi Greg_Spain if you're still out there...
I've been meaning to post a reply to this thread for quite a while, but I've been busy with other things. I'm the guy who wrote the VBScript code Taka linked you to.

If you have any questions post a message and I'll try to help you. If you're having problems with the "download through a proxy" portion of the code, you should be able to use either XMLHTTP, ServerXMLHTTP, or WinHTTP objects. Another options is run() or execute() a command line utility like cURL http://curl.haxx.se/ or Wget http://www.gnu.org/software/wget/. Both utilities support proxy UserIDs and Passwords.


Return to “Awasu - Extensions”

Who is online

Users browsing this forum: No registered users and 1 guest