Monday 11th December 2017 7:49 PM [Awasu News]

Just a quick alpha release to tie up some loose ends from the rather extensive work done in the previous release. Nothing earth-shattering to report, just more of the Awasu goodness we all know and love πŸ™‚ , and the 3.1.1 beta will be hot on its heels in the new year...

Thursday 30th November 2017 5:11 PM [General]

OMG, this is just so freaking cool!

I can't remember exactly how, but I stumbled on these a few days ago, and I simply had to order a bundle, and they just arrived today :jig:

If you don't understand what it means, vi is a text editor for developers that is, while extremely powerful, notoriously difficult to learn and use, and :wq is what you have to type in to save your file and exit - ":" puts you into "command" mode, "w" writes the file out, and then "q" to quit.

This is, of course, not particularly intuitive and it's a rite of passage for Linux newbies to get insanely frustrated trying to use Ctrl-C to exit the program, to no effect, to the point where the developer of a popular clone felt it would be worthwhile to show a hint about how to exit if the user presses Ctrl-C[1]Although even doing this doesn't always help πŸ˜€ .

The story of how these stickers came to be can be found here, and if you feel like getting a few for yourself, they're only a buck each! Even shipping to Australia was an extremely reasonable $2.20, so you really have no excuse, no matter where you are in the world.

I'm doing a lot of work right now for a client where I'm stuck on a back-end server, writing Python in vim[2]Lots of print() statements πŸ˜₯ I got a license for PyCharm, but that turned out to be a complete waste of time :wall: , and while I'm not one for putting stickers on my laptop, one these these will brighten up my day enormously.

Almost enough to overcome the ignominy of having to use a Mac[3]Almost. πŸ™

   [ + ]

1. Although even doing this doesn't always help πŸ˜€
2. Lots of print() statements πŸ˜₯ I got a license for PyCharm, but that turned out to be a complete waste of time :wall:
3. Almost.
Saturday 23rd September 2017 1:25 PM [Awasu News]

And the Awasu juggernaut just keeps on a-rollin' on... πŸ™‚

If we were ever to adopt the WordPress practice of naming releases after people, the 3.1.1.alpha1 release would surely be called "Jacek". He's a data analyst in Europe who has been pushing Awasu into territories far beyond where it had gone before, particularly wrt its search engine. To be honest, I was impressed that Awasu held up as well as it did, but even I would have to admit that it was a bit sluggish and crashy when under load, and so we've been working closely together over the past few months, sorting all these issues out.

A lot of the improvements relate to search, both in how the search index gets updated, as well as how search results are generated. In one case, I managed to speed up the code by 50 times, which made preparing search results significantly faster ::-): It's rare to get this kind of a win, so yes, I was definitely jumping around the room, pumping my fist in the air :jig: And since that bit of code is used elsewhere in Awasu, those operations will be noticeably snappier as well.

The other area that received attention was shutting down Awasu, which was prone to stalling or crashing. You might think that this is a funny thing to be focusing on, but startup and shutdown are often the fiddliest parts of a program. Large programs are split up into separate modules that do different things, and so while the program is starting and module A is coming up, it might need a service provided by module B, so you need to bring that up before module A. Except that module B needs a service provided by module A :wall: , so you need to partially bring up module A, just enough so that module B can start, then go back and finish bringing up module A. Sigh...

At least during startup, you're in a known state i.e. nothing. During shutdown, literally anything could be happening at that time e.g. the part of Awasu that updates channels finishes updating a feed at just the wrong time, and it wants to notify the database module that it needs to store some new content, except that the database module has already shutdown, or is half-way through shutdown, and Awasu gets confused. It's very easy to have problems with this kind of thing going on, and it's very difficult to debug since it's so dependent on what's happening at that exact moment πŸ™

But things are much better now - I haven't received a crash report for, oh, at least a few days now (joking!) - and Awasu shuts down much quicker and much more reliably now.

Not that you would ever want to shut Awasu down, of course... πŸ˜€

Tuesday 18th July 2017 7:45 PM [Awasu News]

The first translation of Awasu 3.1 has been baking for a while, and is now ready to step out. ClΓ‘udio Mantovani Vieira was kind enough to do the translation, and it looks great :clap: There's just something about seeing Awasu running in a different language that is way cool; you shoulda seen the Arabic one, that was seriously freaky πŸ™‚

If you'd like to translate Awasu into your favorite language, drop us a line, and if you happen to speak German or Russian, since we have translations for Awasu 3.0, the job's already half-done!

Thursday 13th July 2017 7:47 PM [Tutorial]

An Awasu user recently asked for some help in getting their Python code to work with Awasu, and while the problem turned out to be related to text encoding (which is not, strictly speaking, anything to do with Awasu), since this is such a common issue, I thought I'd write up some notes on how all it works.

The canonical introductory article on the subject is, of course, Joel Spolsky's The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)", but these notes will be a more practical guide, along the lines of "How do I get my code to *?@#'ing work?!?!"[1]Now, there's a meme for you πŸ™‚ .

Note that while this tutorial has separate sections on Python 2 and Python 3, even if you're only using one version of Python, you should read both sections if you want to really understand how things work.

This stuff is tricky to get your head around at first, but once you figure it out, it's actually not too bad. The problem is that even when you've got your code right, you start receiving content from elsewhere that is wrong, which breaks your code, so you change it to handle that content, but then your code breaks when you receive content from somewhere else that is doing things correctly[2]Or also doing things incorrectly, but in a different way :wall: , and you get stuck in a cycle where your code never works properly :wall: Hopefully, these notes will help you know when your code is right, and you can stick to your guns and start yelling at the other guy to fix their code... πŸ™‚

   [ + ]

1. Now, there's a meme for you πŸ™‚
2. Or also doing things incorrectly, but in a different way :wall:
Wednesday 26th April 2017 9:53 PM [Awasu News]

We have Pi Day, we have Square Root Day, there's even a Weed Day; well, today is Awasu Day.

Awasu listens on port 2604 for HTTP API requests, and while I realize y'all in the USA write your dates backwards, here in Australia, today is 26/04, and so it's insanely appropriate to announce the release of Awasu 3.1 here.

It's better, faster, sexier than ever before - check out the features here and here - and even better, no price increase! :clap:

So, grab it while it's hot, and I'm off to the pub... :jig:

Wednesday 22nd March 2017 2:39 PM [Awasu News]

Awasu 3.1.rc1 has been released here.

Most of the work for this release was for the installer, which will upgrade your existing installation to 3.1, if you're running 3.0, or any of the 3.0.x betas. If you're running a 2.x version, or (eek!) a 1.x version, please contact us for details on how to upgrade.

The documentation has also been updated, and is available here (a CHM version will be coming later).

Note to self: if anyone asks you to do installer work, run away! It's a horrible job, testing on different versions of Windows, with many different scenarios that need to be verified. But it's all done now, so I'll let this release bake before pushing out the final 3.1 general release in a few weeks. Woo hoo!

Thursday 16th February 2017 9:21 PM [Awasu News]

Awasu 3.0.4 has been released here.

A final round of optimizations, enhancements and bug fixes - this'll be the last one before the 3.1 general release, which shouldn't be too far away.

Tuesday 31st January 2017 9:27 PM [Awasu News]

Awasu 3.0.4.alpha3 has been released here, mostly minor bug fixes and enhancements. Only a few things left to do, and that'll close off this release cycle - woo hoo! :clap:

Wednesday 25th January 2017 6:05 PM [General]

Awasu has had many optimizations made to it over the past few releases, and one of them was a change to how it captures output from plugins. The old code used to set up temp files to capture stdout and stderr, configured the plugin process to send its output to them, then read the captured content when the plugin finished. This was changed in 3.0.4.alpha2 to use in-memory buffers, thus removing the need to use slow disk files.

This worked pretty well, except for one rare, bizarre problem: if Awasu crashed was forcibly terminated via Task Manager while plugins were running, when it was restarted, it was unable to open the server socket it normally listens on for API requests, because another process had it open. The plugins would also hang indefinitely. Awasu was recently changed to automatically find an alternative port, which is kinda cool, but (1) it also kinda sucks if you have scripts that have this port hard-coded, and (2) it's indicative of a serious problem, a zombie kinda problem πŸ™

When Awasu is started, it checks to see if another copy is already running, and transfers control to that if it is, so the old process was dead enough that the new copy of Awasu wasn't seeing it, but it was alive enough that it was keeping the old socket (and presumably other stuff) open.

I hate zombies.

I finally figured out what the problem was, and thought it might make a good blog post, since it was an interesting situation, and as a PSA to correct an omission in the MSDN example. However, when I went looking for the MSDN example, it did the necessary magic to avoid this problem :wall: but since it doesn't explain why it performs these incantations, it's maybe worth jotting down these notes for anyone else running into the same problem...

The problem

The old code[1]Error handling has been removed for clarity. created the pipe [2]Awasu captures both stdout and stderr, but stderr is not important for this example, and so is omitted. and launched the plugin process something like this:

// create a pipe to capture stdout
memset( &secAttrs , 0 , sizeof(secAttrs) ) ; 
secAttrs.nLength = sizeof(secAttrs) ; 
secAttrs.bInheritHandle = TRUE ; // the handles returned will be inheritable
HANDLE hStdoutRead , hStdoutWrite ; 
BOOL rc = CreatePipe( &hStdoutRead , &hStdoutWrite , &secAttrs , 0 ) ;

// start the child process
STARTUPINFO startupInfo ; 
memset( &startupInfo , 0 , sizeof(startupInfo) ) ; 
startupInfo.cb = sizeof(startupInfo) ; 
startupInfo.wShowWindow = SW_HIDE ;
startupInfo.hStdOutput = hStdoutWrite ;
BOOL rc = CreateProcess(
    NULL , pCmdLine ,
    NULL , NULL ,
    TRUE , // the child process will get our inheritable handles
    NULL , pWorkingDir ,
    &startupInfo , &processInfo
) ;

The idea is that we create a pipe to capture the output from the plugin process, with 2 handles to access it[3]There are some things you need to be careful about when using this technique, but they are (sorta) well-known. One (hStdoutWrite) is given to the plugin process so that it can write to the pipe[4]It's actually connected to the process' stdout, and it doesn't even know anything funky is going on., while the other (hStdoutRead) is used by Awasu to read from it.

Inheriting handles

But what does it mean to inherit handles, anyway?

When you open something like a file, or socket, or registry key, the Windows kernel creates an in-memory object to manage it, then gives you a handle to it, which is an alias for the underlying kernel object. The thing about these aliases is:

  • they're only valid within one process (typically the one that opened the underlying object)
  • there can be more than one of them, even across multiple processes, referring to the same underlying object

When the code above runs, Windows creates the pipe buffer and 2 objects to manage reading/writing it, and gives us handles (i.e. aliases) to those underlying kernel objects:

  • handle #1 (an alias for kernel object #12[5]These numbers are internal to the kernel, and have no meaning to user programs.
  • handle #2 (an alias for kernel object #47)

When the child process is created, it inherits these handles[6]Because CreateProcess() was told to do this, and these handles were created as inheritable. i.e. it gets its own copy of them, that refer to the same underlying kernel objects:

The child process needs to inherit these, since it needs the pipe write handle. When it outputs something, it goes to hStdoutWrite[7]Because we passed it in to CreateProcess() as the process's stdout., and ends up in the pipe buffer. From there, Awasu can read it via its hStdoutRead handle.

The solution

You would think that if the reading end of a pipe disappears, anyone trying to write to it would get a "broken pipe" error, but as described above, the plugins would hang indefinitely and the parent awasu.exe got stuck in zombie limbo πŸ™

If we update our diagram to reflect Awasu disappearing, we can see what's happening:

The child process also inherited a copy of the pipe's read handle (even though it doesn't actually use it), and since the process is still running, Windows thinks the underlying kernel object is still in use and so can't close it. So, since something is still connected to the reading end of the pipe, it's still alive, the plugin merrily writes its output to the pipe buffer, until it fills up, and then waits for someone to read the output, to free up some space so that it can write out more output. Except, of course, the only thing with a read handle to the pipe is the child process itself[8]Which it never reads from, since it doesn't need to, that's Awasu's job., so it waits forever, and all this is also apparently enough to stop Windows from fully cleaning up the dead-but-not-quite Awasu process. Sigh...

Once we understand what's going on, the fix is easy: don't inherit the hStdoutRead handle. We set things up like this:

Now, if Awasu disappears, there are no more handles to the pipe's read handle, so Windows will close it, realize the pipe is broken, then returns an error to the plugin the next time it tries to write to it, and in the grand Shakespearian tradition, everyone dies as they should.

   [ + ]

1. Error handling has been removed for clarity.
2. Awasu captures both stdout and stderr, but stderr is not important for this example, and so is omitted.
3. There are some things you need to be careful about when using this technique, but they are (sorta) well-known
4. It's actually connected to the process' stdout, and it doesn't even know anything funky is going on.
5. These numbers are internal to the kernel, and have no meaning to user programs.
6. Because CreateProcess() was told to do this, and these handles were created as inheritable.
7. Because we passed it in to CreateProcess() as the process's stdout.
8. Which it never reads from, since it doesn't need to, that's Awasu's job.