Ian Bicking: the old part of his blog

Re: Gil of doom comment 000

The GIL may not be the worst thing in the world but why are you defending it? It doesn't deserve it.

Ironically, the GIL is functionally almost equivalent to the Linux kernel's BKL (Big Kernel Lock). Now that it's virtually gone, no one would put it back right now. Few would claim that removing it was a bad idea. Speed gains were concrete. Considering that Python is anecdotally 30 times slower than straight C, if we can get a 10% speed increase, isn't that worthwhile?

I have had issues with the GIL in real projects. The most vexing was having my GUI fight with my XML processor over the GIL. Both shouldn't affect each other. It's not good, and it's not defensible. I ended up solving it by forking, but it generated over 400 lines of very touchy code that I have to maintain. Is it that unreasonable that I don't like Python getting in my way? That's why I use it in the first place.

Apart from my app, how many real projects need to put up a GUI and chew lots of XML? How about a web browser? Is that not a "real" project? I don't know what kind of projects you deal with on a daily basis. I do know that you are a proponent of WSGI. Can you really stand behind providing a world-class web standards kit that can't run in highly parallel applications? Isn't that what makes web-applications fly?

This is one of the major issues still remaining in Python.

While I'm at it, a lack of good memory management is another. I don't mean garbage collection SUX0RZ, I mean hogging megabytes of extra memory per process because Python doesn't like giving it back (and shared memory support too please). This is another problem that WSGI inadvertently addresses and that can complicate the "Use The Fork, Luke," suggestion above. It's difficult to run many Python processes on because they don't give memory back readily. Try cyclically generating a 250MB report once per day. You'll find that your server process will inflate and never deflate.

At any rate, having structure-granular ILs (or even memory-region-granular ILs) would be a godsend for some of us.

Comment on Gil of doom comment 000
by Jayson Vantuyl

Comments:

Returning memory to the OS during runtime is very rare among UNIX programs in general. I would support the creation of some function (perhaps in the sys module) that would cause Python to do this (perhaps sys.free()? or sys.sbrk(-1) or something like that).

However, the saner answer for a daemon process that's about to generate a 250MB report (or otherwise use a tremendous amount of memory) would be to fork(), generate the report, and exit(). The memory allocations will all happen in the child and it's memory will be freed back to the OS when the child process dies.

JimD

# Jim Dennis

> The GIL may not be the worst thing in the world but why are you defending it?

It has a cost, but it also has benefits -- there are an awful lot of threading-related bugs that just don't happen because of the GIL. If the cost isn't so very high (and maybe even if it is), then that is a good tradeoff.

> Apart from my app, how many real projects need to put up a GUI and chew lots of XML?

If the XML is a chokepoint, it might be worth using an extension module -- which can release the GIL.

> This is one of the major issues still remaining in Python.

Brett Cannon's thesis will work on sandboxing, so that you can run multiple interpreters. Since they'll have separate object spaces, the GIL contention should be greatly reduced.

> hogging megabytes of extra memory per process because Python doesn't like giving it back

I believe 2.5 includes a patch to change this. I don't know whether or not it is the default.

# JimJJewett

.... why are you defending [the GIL]?

It has a cost, but it also has benefits -- there are an awful lot of threading-related bugs that just don't happen because of the GIL. If the cost isn't so very high (and maybe even if it is), then that is a good tradeoff.

.... many real projects need to put up a GUI and chew lots of XML

If the XML is a chokepoint, it might be worth using an extension module -- which can release the GIL.

Also note that Brett Cannon's thesis will work on sandboxing, so that you can run multiple interpreters. Since they'll have separate object spaces, the GIL contention should be greatly reduced.

... [and also] hogging megabytes of extra memory per process because Python doesn't like giving it back

I believe 2.5 includes a patch to change this. I don't know whether or not it is the default.

# JimJJewett