Ian Bicking: the old part of his blog


While it's probably a little annoying, it seems straight-forward enough to deal with: fork instead of using threads. You'll have to pre-partition the workload, or use some sort of multi-process queue (a database can be an easy way to do this communication, though I'm sure there are other good techniques). Results can be passed back on a socket, via shared memory (maybe POSH), or through files. Maybe use Pyro or Twisted Broker. If you don't use shared memory then you can probably scale this to separate machines fairly easily.

If I was implementing it I'd probably use Pyro, exposing some manager object that handed out jobs and collected results. Then you just need some process manager that can spawn new processes and set them on task (bonus points if it can spawn processes on other machines).
Comment on GIL of Doom!
by Ian Bicking