Awasu
Sunday 6th December 2015 11:11 PM [Awasu News]

Continuing on from the first alpha, this second alpha release adds more optimizations and make-Awasu-run-faster changes.

The change list is relatively short, but the changes involved were extensive and needed a lot of testing. In particular, the way channel summary pages are generated has been completely revamped, and is much smoother and snappier now. Some of you have been sending me crash reports from earlier versions of Awasu, and they’ve all been the same thing – a problem in how the channel summary pages were generated – so since this code has been replaced, this crash should no longer happen [1]This is, of course, not to say that I haven’t introduced another crashing bug, but that’s another issue 😐 .

   [ + ]

1. This is, of course, not to say that I haven’t introduced another crashing bug, but that’s another issue 😐
Sunday 25th October 2015 2:43 AM [Tutorial]

Just a quick follow up on my recent epic tome on setting up a Banana Pi as a file server. I mentioned that I configured my disks to use the ext3 file system, and while it’s generally fine, it does have one weakness: it is very slow deleting large files[1]Since I use my NAS for back ups, some of my files are well over 100GB.. Even worse, it locks up the file system, impacting other activity and causing stalls, which kinda sucks if you’re watching a movie at the time :(

depesz took a very detailed look at the problem and some possible solutions, the TL;DR being that it’s better to progressively shrink the file until it’s all gone rather than asking the operating system to delete it as a file.

For the benefit of anyone having problems with this, here’s a script that I wrote that implements this idea:

#!/bin/bash

# parse the command-line arguments
if [ $# -lt 1 ] ; then
    echo "$(basename $0) file1 file2 file3 ..."
    echo "  Delete file(s) slowly."
    exit 1
fi

chunk_size=100000000
for fname in "$@"; do
    # check if we were given a directory
    if [ -d "$fname" ]; then
        # yup - process each file, then remove the directory
        #echo "Deleting directory: $fname"
        find "$fname" -type f -exec $(readlink -e "$0") \{\} \;
        rm -rf "$fname" 
        continue
    fi
    # check if we were given a file
    if [ ! -f "$fname" ]; then
        # nope - ignore it
        continue ;
    fi
    # yup - delete the file slowly
    #echo "Deleting file: $fname"
    while true; do
        # get the current size of the file
        fsize=$(ls -l "$fname" | cut -d' ' -f 5)    
        if [ $fsize -lt $chunk_size ] ; then
            # the file is small enough to just delete
            #echo "- Deleting file."
            rm -f "$fname"
            break
        fi
        # truncate the file, then loop back
        #echo "- Truncating file: $fsize"
        truncate -c -s -$chunk_size "$fname" || exit
        sleep 0.25
    done
done

Pass in a list of files and/or directories and it will slow-delete them. It will take longer to run, but will have far less impact on the rest of the system.

Nothing to do with Awasu, but hopefully someone out there in Internet-land will find it useful… :)

   [ + ]

1. Since I use my NAS for back ups, some of my files are well over 100GB.
Thursday 8th October 2015 1:56 AM [Awasu News]

Awasu, of course, uses an embedded Internet Explorer browser to show web content, but it’s always bugged me that rendering never seemed to be quite the same as a standalone browser.

I’ve finally figured out what was going on :jig: [1]Although to be honest, it should’ve twigged a long time ago. I guess sometimes I’m just a bit slow… :roll: – while all the documentation says that installed version of IE will be used, what they invariably forget to tell you is that it’s in IE7 compatibility mode :bigshock:

My usual :wall: emoticon is clearly totally inadequate at this point, I need a little yellow man blowing his brains out with a shotgun, or something like that 👿

I’ll include a fix for this in the next release, but fortunately, the temporary fix for this is straight-forward: just download and run this file[2]It adds an entry to the Registry saying that awasu.exe should run IE11 for embedded browsers.. If you want test it, open the forums (from within Awasu) and check that the navbar across the top is displayed correctly. Sites using HTML5 and all that other new-fangled frippery should also work a bit better now. Sigh…

/taka toddles off to find a new smilies pack to install… :bah:

   [ + ]

1. Although to be honest, it should’ve twigged a long time ago. I guess sometimes I’m just a bit slow… :roll:
2. It adds an entry to the Registry saying that awasu.exe should run IE11 for embedded browsers.
Monday 5th October 2015 12:39 AM [Tutorial]

I’ve been a big fan of NAS‘s for many years, that is, a small file server that sits on my network and serves up music, movies, provides space for backups, etc. In the past, I’ve had Synology and QNAP units, and while they were both nice, they were both were relatively expensive, loaded with features I never used. They also both only lasted a few years, and rebuilding a NAS with 5-6 TB of data is a painfully long process :(

So for the next one, the plan was to grab an old laptop, load it up with FreeNAS, and then just hang a few disks off it. If and when the laptop dies, I can just set up a new one and the external disks, with all the data on them, should just plug straight in.

However, this is a bit of clunky solution, so when the Raspberry Pi came out, I got very interested in the idea of using that. Unfortunately, the rPi has one big drawback that makes it unsuitable for use as a file server: it only has 10/100 Mbps ethernet. All the computers on my network have gigabit ethernet, and since I’m moving 100’s of GB’s of data every night for backups, my file server also needs to have gigabit ethernet.

Enter the Banana Pi. Released in late 2014 by LeMaker in China, it’s slightly more expensive but significantly more powerful, notably with gigabit ethernet and a SATA port. Add in a case, and I’ll be able to build my own future-proof NAS for well under a hundred bucks, plus the cost of the disks.

There are quite a few tutorials floating around that explain how to set up a Banana Pi as a NAS, but they invariably only talk about how to set up the factory image of Open Media Vault[1]This is the successor to FreeNAS, written by one of the FreeNAS guys, that runs on Linux instead of FreeBSD. (which is relatively easy to do), but this series of tutorials will also talk about the many things you need to do after that to get a usable system.

   [ + ]

1. This is the successor to FreeNAS, written by one of the FreeNAS guys, that runs on Linux instead of FreeBSD.
Thursday 1st October 2015 5:29 PM [Awasu News]

It’s been an arduous slog to get this release finished, but Awasu 3.0.3.alpha1 is finally done, and it has probably one of the shortest change lists you’re ever likely to see:

  • Made Awasu run faster :)

Core components of Awasu’s underlying feed engine are running a whopping 3-5 times faster, so you should see Awasu running noticeably quicker. I’ve also done a lot of work making sure that things work in non-English environments.

These changes affected nearly every file in Awasu’s source code, and I was trawling through some code that hadn’t been touched in literally 20 years :bigshock: There was some seriously freaky stuff in there – they don’t call it code spelunking for nothing :roll: One consequence of this is that some things may have been broken, so please keep an eye out for any weirdness.

Monday 6th July 2015 9:14 AM [Awasu News,Tutorial]

A while back, I posted a tutorial that showed how easy it is to extend Awasu through the use of plugins and channel hooks, and continuing on from that, here’s another series that shows how you can control your Awasu via its API.

Whether you just want to find out what state your channels or reports are in, or if you want to programmatically create, update and delete them, the Python and PHP libraries available make it a breeze.

Have a play with them, hope you find them useful and, as always, feel free to ask questions in the forums.

Wednesday 22nd April 2015 12:49 AM [Awasu News]

Work on the next release of Awasu has been well underway for a while now, as the optimizing juggernaut plows onwards and upwards :) Part of doing this kind of work is to do before-and-after performance tests, to see how much better the new version is running, and I figured that since I was doing it already, I might as well do some tests on Awasu Server as well.

I’ve always known that Awasu Server runs much faster and more smoothly than the desktop version, but this is the first time I’ve collected hard data on how it performs, and the results are, well, impressive :jig:

You can check out the full report here, but to the right is the money shot: Awasu Server updates around 67% more channels per hour than Awasu Pro.

If that’s not insanely awesome, I don’t know what is ::-):

Awasu Server is in private beta, but if you have lots of channels [1]For example, some of our clients are running tens of thousands of channels., if you want to monitor huge amounts of information, then this is definitely the way to go.

And not only can Awasu bring in this amount of information at high-speed, it can also do something with it all, whether it be forward the information onto a database, or send out emails, or generate reports.

Drop us a line if you’d like to know more…

   [ + ]

1. For example, some of our clients are running tens of thousands of channels.
Tuesday 7th April 2015 2:05 AM [Awasu News,Tutorial]

As promised, the tutorial on writing Awasu plugin channels and channel hooks is now up here.

Part 1 talks about how to get set up with the new awasu_tools library, and how to generate a basic feed, then Part 2 shows how to convert a basic script into something that can be called by Awasu.

Part 3 explores some more of the features offered by the awasu_tools library, while Part 4 rounds things off by explaining how to compile your extension, ready for distribution.

Writing Awasu extensions has always been relatively easy, but there was a bit of a learning curve, mostly spent looking at the samples supplied with Awasu. Hopefully, this tutorial will make things a bit easier, and the new awasu_tools library really makes things a breeze ::-):

As an aside, I’ve been programming computers for around 30 years (professionally for 25), yet publishing my first bit of code on PyPI and GitHub somehow makes me suddenly feel like a Real Programmer :roll:

Still some life left in this old dog, methinks… :jig:

Wednesday 25th March 2015 12:32 PM [Awasu News]

Now that you’ve all had a bit of time to have a play with the 3.0.2 release, it’s time to explain in a bit more detail about one of its new cool features.

One of Awasu’s strongest features has always been its extensibility [1]If you’re not sick of me droning on about this by now, you will be soon :roll: , and the 3.0.2 release builds on this by adding support for a bunch of new plugin channels, that can retrieve content from a variety of sources:

Google

Twitter

Wikipedia

Email accounts

Web pages
(via scraping)

Pinterest

There is also one channel hook:

Save incoming content in an external database
(currently MySQL or SQL Server, Postgres coming soon)

These are all currently still in private beta, but if you’d like to take them for a spin, drop me a line and let me know:

  • What version of Awasu you’re running.
  • Which extensions you’re interested in.

and I’ll let you know when they’re ready.

You should be running Awasu Professional Edition v3.0.2, although support for them will be back-ported to v3.0 shortly.

Part of the work in building these was putting together a framework [2]This will be released soon, to make it easier for you to write your own extensions. that will let me easily churn them out, so you can expect a slew of new ones in the coming year [3]Currently on my to-do list: LinkedIn, YouTube, Vimeo, SoundCloud, GitHub, CrunchBase, Vine, Instagram, Spotify , Flickr, Picasa, Dropbox, Instapaper, FourSquare, Yelp. Sigh… :whip: . Let me know if there’s anything you’d like support for… [4]And yes, I know everyone will want one for Facebook, and I actually had one working, but Facebook changed their API and broke things :( , so I’m going to have to revisit this one. Double-sigh…

   [ + ]

1. If you’re not sick of me droning on about this by now, you will be soon :roll:
2. This will be released soon, to make it easier for you to write your own extensions.
3. Currently on my to-do list: LinkedIn, YouTube, Vimeo, SoundCloud, GitHub, CrunchBase, Vine, Instagram, Spotify , Flickr, Picasa, Dropbox, Instapaper, FourSquare, Yelp. Sigh… :whip:
4. And yes, I know everyone will want one for Facebook, and I actually had one working, but Facebook changed their API and broke things :( , so I’m going to have to revisit this one. Double-sigh…
Thursday 19th March 2015 9:21 AM [Awasu News]

It’s been a bit longer between releases than I normally like, but there’s been a lot of work done on this 3.0.2 release [1]I’ve also been remarkably busy building systems built around Awasu. As examples, one client is using a battery of web scraping and custom-built plugins to monitor web sites they’re interested in, while another is using their system to monitor many thousands of sites, downloading the content to an external database, with a custom search engine and alert system built on top. Drop us a line if you’re interested in having something built for yourself. , so I hope it’ll have been worth the wait :)

This release builds on the work done in 3.0.1, making Awasu run better, faster and more efficiently. I’ve been adding these optimizations incrementally, so the effects were less noticeable, but running this new version side-by-side the old one, the difference is very apparent :jig: This will be an on-going process throughout the current release cycle, but of course, there will be a whole bunch of new features as well.

In this release, on top of the many, many optimizations, we also have:

  • Numerous UI improvements.
  • A new email engine, that lets you use third-party services like Gmail and Hotmail.
  • Improvements in the search engine, to give better results.
  • User tools now have access to all of a feed’s metadata, making it easier to export content out, or integrate Awasu into another service.
  • Support for a new range of plugin channels and hooks.

This last one is a biggie, and I’ll be making an announcement about this shortly, but in the meantime, get yourself set up with Awasu 3.0.2 and see what you think.

   [ + ]

1. I’ve also been remarkably busy building systems built around Awasu. As examples, one client is using a battery of web scraping and custom-built plugins to monitor web sites they’re interested in, while another is using their system to monitor many thousands of sites, downloading the content to an external database, with a custom search engine and alert system built on top. Drop us a line if you’re interested in having something built for yourself.