Awasu has had many optimizations made to it over the past few releases, and one of them was a change to how it captures output from plugins. The old code used to set up temp files to capture stdout and stderr, configured the plugin process to send its output to them, then read the captured content when the plugin finished. This was changed in 3.0.4.alpha2 to use in-memory buffers, thus removing the need to use slow disk files.
This worked pretty well, except for one rare, bizarre problem: if Awasu
crashed was forcibly terminated via Task Manager while plugins were running, when it was restarted, it was unable to open the server socket it normally listens on for API requests, because another process had it open. The plugins would also hang indefinitely. Awasu was recently changed to automatically find an alternative port, which is kinda cool, but (1) it also kinda sucks if you have scripts that have this port hard-coded, and (2) it's indicative of a serious problem, a zombie kinda problem 🙁
When Awasu is started, it checks to see if another copy is already running, and transfers control to that if it is, so the old process was dead enough that the new copy of Awasu wasn't seeing it, but it was alive enough that it was keeping the old socket (and presumably other stuff) open.
I hate zombies.
I finally figured out what the problem was, and thought it might make a good blog post, since it was an interesting situation, and as a PSA to correct an omission in the MSDN example. However, when I went looking for the MSDN example, it did the necessary magic to avoid this problem but since it doesn't explain why it performs these incantations, it's maybe worth jotting down these notes for anyone else running into the same problem...
The old code created the pipe and launched the plugin process something like this:
// create a pipe to capture stdout
SECURITY_ATTRIBUTES secAttrs ;
memset( &secAttrs , 0 , sizeof(secAttrs) ) ;
secAttrs.nLength = sizeof(secAttrs) ;
secAttrs.bInheritHandle = TRUE ; // the handles returned will be inheritable
HANDLE hStdoutRead , hStdoutWrite ;
BOOL rc = CreatePipe( &hStdoutRead , &hStdoutWrite , &secAttrs , 0 ) ;
// start the child process
STARTUPINFO startupInfo ;
memset( &startupInfo , 0 , sizeof(startupInfo) ) ;
startupInfo.cb = sizeof(startupInfo) ;
startupInfo.dwFlags = STARTF_USESHOWWINDOW | STARTF_USESTDHANDLES ;
startupInfo.wShowWindow = SW_HIDE ;
startupInfo.hStdOutput = hStdoutWrite ;
PROCESS_INFORMATION processInfo ;
BOOL rc = CreateProcess(
NULL , pCmdLine ,
NULL , NULL ,
TRUE , // the child process will get our inheritable handles
NULL , pWorkingDir ,
&startupInfo , &processInfo
The idea is that we create a pipe to capture the output from the plugin process, with 2 handles to access it. One (hStdoutWrite) is given to the plugin process so that it can write to the pipe, while the other (hStdoutRead) is used by Awasu to read from it.
But what does it mean to inherit handles, anyway?
When you open something like a file, or socket, or registry key, the Windows kernel creates an in-memory object to manage it, then gives you a handle to it, which is an alias for the underlying kernel object. The thing about these aliases is:
- they're only valid within one process (typically the one that opened the underlying object)
- there can be more than one of them, even across multiple processes, referring to the same underlying object
When the code above runs, Windows creates the pipe buffer and 2 objects to manage reading/writing it, and gives us handles (i.e. aliases) to those underlying kernel objects:
- handle #1 (an alias for kernel object #12
- handle #2 (an alias for kernel object #47)
When the child process is created, it inherits these handles i.e. it gets its own copy of them, that refer to the same underlying kernel objects:
The child process needs to inherit these, since it needs the pipe write handle. When it outputs something, it goes to hStdoutWrite, and ends up in the pipe buffer. From there, Awasu can read it via its hStdoutRead handle.
You would think that if the reading end of a pipe disappears, anyone trying to write to it would get a "broken pipe" error, but as described above, the plugins would hang indefinitely and the parent awasu.exe got stuck in zombie limbo 🙁
If we update our diagram to reflect Awasu disappearing, we can see what's happening:
The child process also inherited a copy of the pipe's read handle (even though it doesn't actually use it), and since the process is still running, Windows thinks the underlying kernel object is still in use and so can't close it. So, since something is still connected to the reading end of the pipe, it's still alive, the plugin merrily writes its output to the pipe buffer, until it fills up, and then waits for someone to read the output, to free up some space so that it can write out more output. Except, of course, the only thing with a read handle to the pipe is the child process itself, so it waits forever, and all this is also apparently enough to stop Windows from fully cleaning up the dead-but-not-quite Awasu process. Sigh...
Once we understand what's going on, the fix is easy: don't inherit the hStdoutRead handle. We set things up like this:
Now, if Awasu disappears, there are no more handles to the pipe's read handle, so Windows will close it, realize the pipe is broken, then returns an error to the plugin the next time it tries to write to it, and in the grand Shakespearian tradition, everyone dies as they should.