There seems to be some confusion about Twisted's asynchronous programming, and the use of threads and blocking code.
As a quick primer, everything in Twisted runs in a single process. Which means when one piece of code is running the entire server process is dedicated to that code, and no other request can be handled. Twisted handles this with a sort of event-driven programming. When you have long-running code, you chop it up into pieces where each piece is short, then you let the event processor call those pieces in turn (the "reactor" in Twisted terminology). These use something called a "Deferred" and look something like:
def blocking_code():
val = init_val() # <-- not blocking
val = inner_blocking_code(val)
return val + 10
def non_blocking_code():
val = init_val()
d = inner_non_blocking_code(val)
d.addCallback(non_blocking_code_adder)
return d
def non_blocking_code_adder(val):
return val + 10
inner_non_block_code is a modified version of
inner_blocking_code, which uses events and returns a Twisted
Deferred object. Someday that deferred will produce a
value; when it does non_blocking_code_adder will be called. If
someone wants to use the "return" value of
non_blocking_code they will add another callback, and so on.
The call stack we've gotten used to is gone -- instead a series of
callbacks replaces it. (I'm only a Twisted tourist, so feel free to
correct me in comments if I've got this wrong)
This is a little weird, but there are advantages that I won't talk about here (mostly performance and some people find concurrent programming easier/safer with this style).
A big part of this example was the inner_blocking_code to inner_non_blocking_code rewrite. That rewrite may very well be hard to do. Or maybe that blocking code exists in a library you don't control, and most Python libraries are blocking. Once this occurs to people, a lot of people think Twisted is unusable in a lot of situations -- and if you don't handle blocking code, your Twisted server will be broken. At this point people sometimes dismiss Twisted as being too limited. I was most recently reminded of this by this post, but the confusion is very common.
Let's say you can't refactor inner_blocking_code; then you can do:
from twisted.internet import threads
def inner_non_blocking_code(val):
return threads.deferToThread(inner_blocking_code, val)
Twisted will then run inner_blocking_code in a different
thread, and when the function returns it will trigger the Deferred that
threads.deferToThread returns.
Threads in Twisted aren't great -- you've gone to all that trouble to factor your program into an asynchronous style, but you still have to run threads (which you may have been trying to avoid) -- but at least it's possible. Twisted doesn't put up walls, even if it might put up some hurdles.
I didn't say that twisted was too limited - actually I think it's too powerfull in some regards ;-) - but that a switch of PyDS to twisted as core would require too much changes to PyDS. Of course, I still could push longrunning code into threads with twisted - but that wouldn't gain me much over the current medusa+threads situation. I already could refactor PyDS to be more eventdriven by hooking into medusa. Of course this would require writing some stuff that's already there with twisted.
As I wrote already, had I known twisted when I started PyDS, I might even have used twisted as the base for PyDS and not medusa. But as it is the switch of PyDS from medusa+threads to twisted-threads would be problematic (because of parts of the system that can't be easily broken into smaller chunks, like the rendering stuff) and the change to twisted+threads would be much work without that much gain for PyDS (as I could already reach many of the goals of such a transition with it's current core).
The point with Twisted threads is that you can start the threading (and enter your section of blocking code) at any level you want. If you really want, you can start processing in a thread almost immediately, turning Twisted (against its will ;) into a threaded application server. You don't have to adopt the Twisted style throughout.
But you also had another point -- if you eschew the Twisted style it becomes questionable that Twisted is worth the effort.# Ian Bicking
Is there difference between a Twisted "Deferred" object and a "Future" as used in various asynchronous frameworks and concurrent languages?
Nevermind - I did a little reading and it looks like you can attach callbacks to a "Deferred", making it more akin to a Proactor.
Ian, thanks. This is a terrific explanation of how Twisted's threading model works. Also a terrific explanation of the Twisted culture's resistance to, but not denial of, the necessity for threads in some situations. I loved the "hurdles but no walls" comment.
Avdi, yes, Deferreds are conceptually more like a Future than a Proactor. A Proactor typically manages multiple streams of callbacks, whereas a Deferred represents an asynchronous control flow for a single call.
Turning the calculation inside out can often be accomplished by generators. Placing yields throughout your calculation code gives you a simple way to cut a procedure into separate steps, and the generator's .next makes a pretty good callback.# Scott