Awasu
Wednesday 22nd March 2017 2:39 PM [Awasu News]

Awasu 3.1.rc1 has been released here.

Most of the work for this release was for the installer, which will upgrade your existing installation to 3.1, if you're running 3.0, or any of the 3.0.x betas. If you're running a 2.x version, or (eek!) a 1.x version, please contact us for details on how to upgrade.

The documentation has also been updated, and is available here (a CHM version will be coming later).

Note to self: if anyone asks you to do installer work, run away! It's a horrible job, testing on different versions of Windows, with many different scenarios that need to be verified. But it's all done now, so I'll let this release bake before pushing out the final 3.1 general release in a few weeks. Woo hoo!

Thursday 16th February 2017 9:21 PM [Awasu News]

Awasu 3.0.4 has been released here.

A final round of optimizations, enhancements and bug fixes - this'll be the last one before the 3.1 general release, which shouldn't be too far away.

Tuesday 31st January 2017 9:27 PM [Awasu News]

Awasu 3.0.4.alpha3 has been released here, mostly minor bug fixes and enhancements. Only a few things left to do, and that'll close off this release cycle - woo hoo! :clap:

Wednesday 25th January 2017 6:05 PM [General]

Awasu has had many optimizations made to it over the past few releases, and one of them was a change to how it captures output from plugins. The old code used to set up temp files to capture stdout and stderr, configured the plugin process to send its output to them, then read the captured content when the plugin finished. This was changed in 3.0.4.alpha2 to use in-memory buffers, thus removing the need to use slow disk files.

This worked pretty well, except for one rare, bizarre problem: if Awasu crashed was forcibly terminated via Task Manager while plugins were running, when it was restarted, it was unable to open the server socket it normally listens on for API requests, because another process had it open. The plugins would also hang indefinitely. Awasu was recently changed to automatically find an alternative port, which is kinda cool, but (1) it also kinda sucks if you have scripts that have this port hard-coded, and (2) it's indicative of a serious problem, a zombie kinda problem 🙁

When Awasu is started, it checks to see if another copy is already running, and transfers control to that if it is, so the old process was dead enough that the new copy of Awasu wasn't seeing it, but it was alive enough that it was keeping the old socket (and presumably other stuff) open.

I hate zombies.

I finally figured out what the problem was, and thought it might make a good blog post, since it was an interesting situation, and as a PSA to correct an omission in the MSDN example. However, when I went looking for the MSDN example, it did the necessary magic to avoid this problem :wall: but since it doesn't explain why it performs these incantations, it's maybe worth jotting down these notes for anyone else running into the same problem...

The problem

The old code[1]Error handling has been removed for clarity. created the pipe [2]Awasu captures both stdout and stderr, but stderr is not important for this example, and so is omitted. and launched the plugin process something like this:

// create a pipe to capture stdout
SECURITY_ATTRIBUTES secAttrs ; 
memset( &secAttrs , 0 , sizeof(secAttrs) ) ; 
secAttrs.nLength = sizeof(secAttrs) ; 
secAttrs.bInheritHandle = TRUE ; // the handles returned will be inheritable
HANDLE hStdoutRead , hStdoutWrite ; 
BOOL rc = CreatePipe( &hStdoutRead , &hStdoutWrite , &secAttrs , 0 ) ;

// start the child process
STARTUPINFO startupInfo ; 
memset( &startupInfo , 0 , sizeof(startupInfo) ) ; 
startupInfo.cb = sizeof(startupInfo) ; 
startupInfo.dwFlags = STARTF_USESHOWWINDOW | STARTF_USESTDHANDLES ;
startupInfo.wShowWindow = SW_HIDE ;
startupInfo.hStdOutput = hStdoutWrite ;
PROCESS_INFORMATION processInfo ;
BOOL rc = CreateProcess(
    NULL , pCmdLine ,
    NULL , NULL ,
    TRUE , // the child process will get our inheritable handles
    CREATE_NO_WINDOW ,
    NULL , pWorkingDir ,
    &startupInfo , &processInfo
) ;

The idea is that we create a pipe to capture the output from the plugin process, with 2 handles to access it[3]There are some things you need to be careful about when using this technique, but they are (sorta) well-known. One (hStdoutWrite) is given to the plugin process so that it can write to the pipe[4]It's actually connected to the process' stdout, and it doesn't even know anything funky is going on., while the other (hStdoutRead) is used by Awasu to read from it.

Inheriting handles

But what does it mean to inherit handles, anyway?

When you open something like a file, or socket, or registry key, the Windows kernel creates an in-memory object to manage it, then gives you a handle to it, which is an alias for the underlying kernel object. The thing about these aliases is:

  • they're only valid within one process (typically the one that opened the underlying object)
  • there can be more than one of them, even across multiple processes, referring to the same underlying object

When the code above runs, Windows creates the pipe buffer and 2 objects to manage reading/writing it, and gives us handles (i.e. aliases) to those underlying kernel objects:

  • handle #1 (an alias for kernel object #12[5]These numbers are internal to the kernel, and have no meaning to user programs.
  • handle #2 (an alias for kernel object #47)


When the child process is created, it inherits these handles[6]Because CreateProcess() was told to do this, and these handles were created as inheritable. i.e. it gets its own copy of them, that refer to the same underlying kernel objects:

The child process needs to inherit these, since it needs the pipe write handle. When it outputs something, it goes to hStdoutWrite[7]Because we passed it in to CreateProcess() as the process's stdout., and ends up in the pipe buffer. From there, Awasu can read it via its hStdoutRead handle.


The solution

You would think that if the reading end of a pipe disappears, anyone trying to write to it would get a "broken pipe" error, but as described above, the plugins would hang indefinitely and the parent awasu.exe got stuck in zombie limbo 🙁

If we update our diagram to reflect Awasu disappearing, we can see what's happening:

The child process also inherited a copy of the pipe's read handle (even though it doesn't actually use it), and since the process is still running, Windows thinks the underlying kernel object is still in use and so can't close it. So, since something is still connected to the reading end of the pipe, it's still alive, the plugin merrily writes its output to the pipe buffer, until it fills up, and then waits for someone to read the output, to free up some space so that it can write out more output. Except, of course, the only thing with a read handle to the pipe is the child process itself[8]Which it never reads from, since it doesn't need to, that's Awasu's job., so it waits forever, and all this is also apparently enough to stop Windows from fully cleaning up the dead-but-not-quite Awasu process. Sigh...


Once we understand what's going on, the fix is easy: don't inherit the hStdoutRead handle. We set things up like this:


Now, if Awasu disappears, there are no more handles to the pipe's read handle, so Windows will close it, realize the pipe is broken, then returns an error to the plugin the next time it tries to write to it, and in the grand Shakespearian tradition, everyone dies as they should.



   [ + ]

1. Error handling has been removed for clarity.
2. Awasu captures both stdout and stderr, but stderr is not important for this example, and so is omitted.
3. There are some things you need to be careful about when using this technique, but they are (sorta) well-known
4. It's actually connected to the process' stdout, and it doesn't even know anything funky is going on.
5. These numbers are internal to the kernel, and have no meaning to user programs.
6. Because CreateProcess() was told to do this, and these handles were created as inheritable.
7. Because we passed it in to CreateProcess() as the process's stdout.
8. Which it never reads from, since it doesn't need to, that's Awasu's job.
Tuesday 3rd January 2017 9:07 PM [General]

I'm not a big one for browser plugins and add-ons, but one thing I absolutely love is this simple, but very cool, bookmarklet. In the author's own words:

I hate sticky headers. I want to kill sticky headers.

So I made this bookmarklet.

It removes those unbelievably annoying full-page overlays, header bars, chat boxes, anything that stays in a fixed position, from a web page[1]Technobabble: it looks for anything with a CSS position of "fixed" and removes it from the DOM.. I so want this in Awasu, so I was mulling over how best to implement it, when it occurred to me that you could already do it :wall:

Simply create a new user tool[2]Choose "Organize user tools..." from the Tools menu. called, say, "Kill stickies", with the following Javascript:
javascript:(function(){(function(){var i,elements=document.querySelectorAll('body *');for(i=0;i<elements.length;i++){if(getComputedStyle(elements[i]).position==='fixed'){elements[i].parentNode.removeChild(elements[i]);}}})()})()

Then, when you browse to a page that has these pesky things on it, just run the user tool and they'll be removed from the page! 😎

Wishing y'all a great and sticky-free 2017 ::-):

   [ + ]

1. Technobabble: it looks for anything with a CSS position of "fixed" and removes it from the DOM.
2. Choose "Organize user tools..." from the Tools menu.
Tuesday 29th November 2016 1:43 PM [Awasu News]

Awasu 3.0.4.alpha2 has been released here.

Lots of bug-fixes and optimizations, and while the change list might look a little short, some of the issues fixed were insanely difficult to track down. When something happens only after Awasu has been running under continuous heavy load for a week, it takes time to figure out what the problem is 🙁 But I've had several instances of Awasu running under load for 2 straight months now, so I think it's solid... 🙂

There also a bunch of fixes and improvements for corporate users, when Awasu is used in a domain, and connects to the internet via a proxy.

Monday 20th June 2016 8:30 PM [Awasu News]

Awasu 3.0.4.alpha1 has been released here. This will probably be the last release cycle before 3.1, so it'll be lots of minor enhancements and bug fixes from here on.

The most useful one in this release is the blocking of web-bugs. These are invisible images that publishers embed in their feeds, so that they can track when you read an item. When I was testing this feature, I was horrified at how many of these things were in there, so I'm very happy Awasu now strips them out.

Sunday 15th May 2016 7:24 PM [General]

Those of you who have been here for a while might remember that I used to run a music bar in Thailand a while back. One night, I was doing my one-man show, when a guy in a cowboy hat wandered in and sat down to listen. During the break, I went over and had a chat with him, and found out that, he too, was a musician.

I didn't know it at the time, but he was David LaMotte, and is one of the bigger names in the U.S. folk scene. He was in town to give a show at one of the universities, and was staying at a hotel around the corner.

We got on pretty well, one beer led to another, and we somehow ended up with him performing a special show at the bar, and it was definitely one of the most memorable shows we ever had. I did a lot of recording at the bar, and while this show[1]There is also video available here, although the video quality is less than stellar 😐 was one of the earlier ones I did, and so the sound engineering is less sophisticated than if I did it again today, it's not bad and (hopefully) catches the warmth of his performance. Folk is not something that I'm really into, but his songs are very accessible, and have a warmth and honesty that's quite amazing.

He has been on a musical hiatus for the past ten years, instead doing things like helping build and fund schools in Guatemala, travelling the world, speaking on how each of us can effect change, and helping select Nobel Peace Prize candidates (!)

However, the time has come and he's back in the studio, putting down a new CD in what sounds like a fascinating project, with a menagerie of musicians from every continent.

Being the cynical, grumpy old fart I am these days, it's not often I see someone who I would call inspirational, but this guy is definitely one of them. Check out his music, then put a few dineros in his jar to help make the CD happen. I have.

   [ + ]

1. There is also video available here, although the video quality is less than stellar 😐
Saturday 12th March 2016 4:22 PM [Tutorial]

A while back, I posted a massive tutorial on how to set up a NAS file server on a Banana Pi, and it was so much fun, I've written another even massiver one on how to set up an internet gateway on a Banana Pi.

A gateway lets you isolate computers in your home network from the internet. To reach the internet, a computer has to through the gateway, which means that if you put a firewall or virus checker or ad-blocker here, all your computers will benefit from them.

As before, this series of tutorials will walk you through the whole process of setting up a gateway, including a lot of not-essential-but-nice-to-have stuff. We start off by setting up a bare-bones gateway:

Once this is up and running, we then take a look at installing some useful extra services:

These days, a firewall is pretty much a necessity, and it's quite eye-opening to watch the logs and see the constant stream of attacks, as people try to break into your computer. And even if you run an ad-blocker like uBlock or AdBlock, a DNS-based ad-blocker can be run along-side it, without affecting browser performance at all[1]Browser plugins tend to slow the browser down noticeably, and can use huge amounts of memory..

   [ + ]

1. Browser plugins tend to slow the browser down noticeably, and can use huge amounts of memory.
Friday 4th March 2016 2:50 PM [Awasu News]

Awasu 3.0.3 has been released here. This is a enhancements and bug fixes releases, and while the change list is relatively short, there's been a great deal of work making Awasu run faster and more smoothly, which should make a noticeable difference.

Have fun, and as always, more Awasu goodness will be forthcoming shortly...