Ian Bicking: the old part of his blog

Dealing with Context

I'm writing some code where every action has a context. Mostly this is to track who does what in a multi-user system, as well as some logging and preferences.

Usually when you have a bunch of state that gets passed around, that state is "self", a sort of implicit first argument to all your functions. This works well in the scope of a class, but what about when you have many classes, and many methods that need to share some state?

Currently I'm passing a context object around as the first argument to many of these methods. More than would seem immediately necessary, as it's easier to pass the context around now than realize that some deep method needs the context, and a whole chain of method signatures have to be changed to pass the context in.

In the end I don't really like any of it. This argument feels like a wart on every method. Are there other ways to pass around context?

Zope 2 does this through Acquisition, which takes the form of a wrapper around nearly every object you deal with, and the wrapper adds the contextual information. Acquisition is evil for other reasons, but it works fairly well here.

Other systems use global variables for this context. That works well in a single-process system. What about a threaded system? I could key the context off the thread name, creating a kind of system-local/thread-global value. I've never felt confident about doing that, though.

Other techniques? I don't really know what Zope 3 is doing for this -- I feel like adaptation is somehow involved, but there's got to be more to it than that.

Created 02 Aug '04
Modified 14 Dec '04

Comments:

Ug, I feel your pain. I think that the cflow AspectJ construct provides some hint of what I would like to see in a solution.

See http://dev.eclipse.org/viewcvs/indextech.cgi/~checkout~/aspectj-home/doc/progguide/semantics-pointcuts.html for some detail on cflow.

Best of luck,
John
# John D. Heintz

Rather than passing it to every function, you can make a global "context server" that any function that needs the context can access. You could make a module that handles this (which will automatically be global) and have methods on it. For multi-threaded access, the module can handle locking if you need that, so clients can block while you're making changes to the context. (Does this make sense, or am I missing something?)

The point is that if something needs to be passed to *many* different functions, objects, or methods, it's probably an aspect of the *system* that everyone should have easy access to, not something that should be shuffled along the call stack.
# Keith

By multithreaded, I don't just mean threadsafe, but that each thread has its own context. So I can't just put it in a global data structure, unless that structure is somehow parameterized by thread. Which is pretty doable -- threading.currentThread().getName() seems to return a good key -- but I'm not sure I feel entirely comfortable with that. Maybe I shouldn't worry and it'll be fine, but since it will be a central component I don't want to find later that it was a really bad idea ;)
# Ian Bicking

Re: aspects; I would be a little fearful to use aspects for something like this, as it's almost global to the application, and would mean all the code would get touched. Since aspect-oriented programming seems to work on a source-level (actually transforming the source), that could mean all sorts of crazy results. Most aspect-oriented use cases are things that strike me as a response to an inflexible language, and usually not necessary for Python, so hopefully that level of effort isn't necessary.
# Ian Bicking

ian: the stuff with using currentThread as a key is exactly what I do with TooFPy to put stuff into a thread-bound context. This isn't a guarantee that this is the best idea one can have, but at least you won't be alone to feel the pain if things blow up in our faces ;-)
# Georg Bauer

(I have nothing to do with it but) Zope 3 has a nicely abstracted thread globals approach, actually. I think I remember hearing it was the same sort of key approach. It's in a C extension, and has some docs and unit tests. Check in http://svn.zope.org/Zope3/trunk/src/zope/thread/ if you want to browse around. Pretty much everything in 'zope' that's not 'zope.app' is intended to be used outside of the zope app server. Dependencies are specified in DEPENDENCIES.cfg in each package--I don't think the thread code has any.
# Gary Poster

Just talked with Jim Fulton about the Zope 3 code--it's actually slated to be in Python 2.4. Most of the work was done Armin Rigo...see http://aspn.activestate.com/ASPN/Mail/Message/c++-sig/2106853 for mailing list thread if you want more.
# Gary Poster

Three issues about the threading:

1) threading.currentThread() returns a new-style object, so for simple needs, you can just add attr's to it.

2) If you're going to roll your own mapping keyed off of currentThread(), you might as well skip all of the threading-module overhead and just use thread.get_ident() directly.

3) I've had serious issues with e.g. mod_python reusing thread id's in concurrently running threads (!). So be careful. One of these days, I need to track that buglet down... :(
# Robert Brewer

I'm a big Nevow fan (I also wrote some recipes about Nevow in the Cookbook). By design Nevow passes 2 arguments to its methods:

a context, which represents the current object space state

a data argument, which contains the return value from the data_* method.

Using the context you can pretty much do anything, also remembering objects with a precise interface that can be retrieved later, here is a little example of how this would work:

class Page(rend.Page):
def render_stuff(self, context, data):
context.remember(data, ISomeData)

Which can be retrieved later with:

class SecondPage(rend.Page):
def render_otherStuff(self, context, data):
my_data = context.locate(ISomeData)

In the context is stored every information about the connection, like the request object:

request = context.locate(IRequest)

Also the Session is there:

session = context.locate(ISession)

and the current processed tag from the template:

context.tag

I think this is the most complete implementation of 'passing a context around'.
# Valentino Volonghi

This adaptation thing might be getting out of hand ;) Really, context.locate(ISession) is just like attribute access, except the attributes themselves have namespaces (i.e., ISession belongs to a namespace, where 'ISession' wouldn't). I'm not sure how I feel about that. (And why context.locate(ISession) instead of ISession(context)?) I thought that adaptation and traversal overlap considerably, though I'm still not sure what it means or how it helps. Why context.remember(data, ISomeData) instead of context.someData = data? Except maybe the namespace thing, are namespaces really that important?

Anyway, I'm glad to see that thread-local data is going to be built in to Python. That way at least I can implement a simpler thread-local system, knowing that even if the particular techniques I use aren't perfect there will be a well-supported, analogous technique in the future. (I don't want to deal with a C extension at the moment.)
# Ian Bicking

ISession and other various interfaces are in a separate module, I just directly wrote the interface name for verbosity needs ;)

In this specific case the interface is used just like a name.

def remember(self, adapter, interface=None):
"""Remember an object that implements some interfaces.
Later, calls to .locate which are passed an interface implemented
by this object will return this object.

If the 'interface' argument is supplied, this object will only
be remembered for this interface, and not any of
the other interfaces it implements.
"""
if interface is None:
interfaceList = megaGetInterfaces(adapter)
if not interfaceList:
interfaceList = [dataqual]
else:
interfaceList = [qual(interface)]
if self._remembrances is None:
self._remembrances = {}
for interface in interfaceList:
self._remembrances[interface] = adapter
return self

This is the specific context.remember() implementation inside Nevow. So Adaptation doesn't come into play with this example although it may seems at first glance. This is also why ISession(context) is not a way of having the session object from the context, since providing ISession with context would require context providing also IRequest, IHand and a lot more (user needed also) interfaces, which can't be done, also, you would end up with a huge context class that provides a method for every method of a request/session/so_on. As you can see, instead, interfaces are rather a key inside a dict.

context.someData = data

could have been used actually, but since context.someData is not namespaced you can have some collisions (remember that using context.remember() you give the user the possibility to remember all the objects he needs to, so I can have some collisions).

HTH, I should post more on my blog actually... :P
# Valentino Volonghi

argh... sorry about that code, I should have known :)

....def remember(self, adapter, interface=None):
........"""Remember an object that implements some interfaces.
........Later, calls to .locate which are passed an interface implemented
........by this object will return this object.
........
........If the 'interface' argument is supplied, this object will only
........be remembered for this interface, and not any of
........the other interfaces it implements.
........"""
........if interface is None:
............interfaceList = megaGetInterfaces(adapter)
............if not interfaceList:
................interfaceList = [dataqual]
........else:
............interfaceList = [qual(interface)]
........if self._remembrances is None:
............self._remembrances = {}
........for interface in interfaceList:
............self._remembrances[interface] = adapter
........return self
# Valentino Volonghi

Ian, you are right on target with your observations.

First, interfaces were chosen instead of strings (for example, implementing __getattr__ on Context) because they are namespaced.

Second, ISession(ctx) should absolutely be used instead of ctx.locate(ISession), and I plan on implementing that soon in my copious spare time.

Finally, context objects in nevow are chained in order to keep track of state across asynchronous callbacks. context.remember remembers an object for an interface at that particular context, and context.locate (or eventually IFoo(ctx)) searches through this parent chain if that particular interface isn't immediately available. This allows me to do things like:

- reuse context objects across multiple page renders (in the case where a template is precompiled and certain things happen once but other things happen every page render)

- implement locate() in different ways in different places; the normal Context objects used to represent a DOM node implement locate as described above, but other IContext implementors implement it by directly returning some object or by being a factory for interface implementors

- implement very explicit acquisition, should anyone choose to use it. Before writing Woven and Nevow I wrote a few very large applications in Zope and was struck by the power and simplicity of in-a inheritance, but burned a few times by it's implicitness. Thus, I wanted to build in a way to pass information from parent to children pages into nevow. The context chain is used to accomplish this.

Also, twisted.python.context is relevant to this discussion. It is used to implicitly pass state during the execution of a function, and it uses some form of thread identity checking to provide a unique context to each thread. (Read the implementation, it is short.) It is used like this:


from twisted.python import context

context.call({'any keys': 'any values'}, someFunction, someArgument, someKeyword='argument')


Then later, during the execution of someFunction or any function called by someFunction:


from twisted.python import context

context.get('any keys')


Thanks for the interesting blog posts as always!

dp
# Donovan Preston

Uuuhhh... I feel happy now that I know I was wrong about adaptation in remembrance. I'm looking forward the day I will be able to write ISession(context).
# Valentino Volonghi

I think there is often a good case for explicitly passing around a context object,
just because it's explicit. I've seen enough problems with acquisition contexts
in Zope 2 to appreciate this -- Zope 2 doesn't hide it perfectly, which means you have to
understand contexts anyway to use it, and the trickery only becomes more voodoo-like ('__of__',
'aq_inner', etc). An explicit context has the important benefit in that it is *simple*.

You can sometimes reduce the amount of context being passed around by making context an
instance variable and centralizing instance construction so that the context is passed
through upon creation. I know that's vague, but it's a pattern I sometimes see.
# Martijn Faassen

In java the Aspects can work at bytecode level too but i don't know if this is posible in python.
For more examples see http://aspectwerkz.codehaus.org/.
# oier

I agree with Martijn Faassen. Either keep it explicit, or make it an attribute. If only a few methods need it, leave it as a parameter to the methods. Otherwise, make it a parameter to the constructor. If you're just passing context along to some other dependent objects, maybe consider making it an attrobute of those objects instead.

Another way of looking at it is like this: Say you started with a bunch of ordinary functions (no objects). You realized after awhile that all of these functions took at least the same three parameters. So, you make a class to encapsulate those three parameters, and thus make the "context" of those functions (now methods) implicit. Now, time goes by, and you realize that you're passing yet another parameter to all (or almost all) of these functions. This additional parameter, unfortunately named "context" as well, looks like it's part of this class's encapsulated context as well. So, move it into the class, along with the first three. Granted, this is a very mechanical view of OO, but it often works for me.

I don't know your specific situation, so this is all based on an abstract notion I have of your real problem, but I'd be skeptical of any metaprogramming solution to--what seems to me--too simple of a problem to warrant it.
# Dave Benjamin