Ian Bicking: the old part of his blog

Events and SQLObject

I've been thinking about events for the 0.8 series of SQLObject. I'm thinking about using PyDispatcher, but there's still some open questions. Jim Fulton actually got me thinking about events a while ago, in lieu of "hooks"... PyCon 2003, I guess. He made a presentation at PyCon 2005 too, but it didn't resonate in the same way, it just seemed too low level; PyDispatcher isn't very complex, but it's a bit higher level. I think Jim was really proposing that higher level systems be built on the low-level systems, but I dunno... it's too abstract then to mean much to me.

Anyway, that's what I'm thinking about. The events, at least to start, would be at a fairly low level -- covering things like creating classes and instances, maybe creates and deletes.

One concern I have is that anything can be listening for an event, so it's hard to know what all will be effected. When that event is raised it's open season, anything could react. Especially when there's feedback from the events (which I'm planning on) I can imagine confusing ensuing. So... I dunno. Logging?

Created 06 Oct '05

Comments:

A possible solution: use the RuleDispatch package's generic functions for your operations. Define the default method to do whatever the to-be-monitored operation does, and then you get before/after/around methods for free, or you can define custom method combination policies if you like. To subscribe, just register methods on the functions, specifying the target class(es) of parameters. Plus, you can test arbitrary conditions on the parameter values, their attributes, and call arbitrary methods/functions as well to determine whether a method applies. "around" methods and normal methods can choose not to call the "next" method, or to call it with altered parameters or fudge its return value in some way.

The only thing this doesn't do that PyDispatcher does, is that PyDispatcher allows you to remove subscribers, and RuleDispatch doesn't (yet) allow that. RuleDispatch also isn't as efficient if you are subscribing to instances instead of classes. However, if I understand your intended use case, it seems unlikely that you need either of those features for SQLObject, but being able to offer your users an efficient trigger system operating on complex conditions would likely be a big plus. Can you say, "complete business rules system"? :)

If you give it a try, I'd be interested in your feedback.

# Phillip J. Eby

"""RuleDispatch also isn't as efficient if you are subscribing to instances instead of classes"""

Actually, I spoke too briefly. RuleDispatch is actually more efficient if you're subscribing to instances, as long as the set of instances involved is relatively static. It's just that if you are adding new instances to subscribe to all the time, then it can cost a lot to keep rebuilding the dispatch tree. However, if you just add a bunch of rules at application startup and then call them, RuleDispatch can be more efficient than PyDispatcher, if the subscribers have overlapping conditional tests taking place. (Because RuleDispatcher evaluates each conditional subexpression at most once, whereas PyDispatcher subscribers can't take advantage of tests evaluated by other subscribers.) This is particularly impactful if any of the tests involve query execution.

# Phillip J. Eby

I had thought about generic functions along the way, but I guess I was too nervous to try them for what seemed like a nominal benefit (over something more conventional like PyDispatcher). There's issues like the fact it introduces C code into a system that has no C code (except, I guess, the database drivers -- but those are commonly packaged). But even more I'm worried that I'd be a drag on an 0.8 release -- as it is, I feel I was a drag on 0.7. So I don't want to introduce new things that I don't entirely understand, unless they are at least limited. So PyDispatcher's more limited rules are an advantage there. Events are also a less complicated concept for other contributors to understand, and since I really want this feature to help open up SQLObject to other developers the comprehensibility is important.

If I was going to implement an adaptation system, I would think very seriously about generic functions. In part because conventional adapation systems like Zope Interfaces already use C code. But also because I think adaptation is not easy as it is, and so generic functions aren't such a new and fancy notion. Or, at least, they are no more new and fancy than conventional adaptation -- a concept which has gained little traction outside of Zope 3 (which actually surprises me). TurboGears people are planning to implement a JSON adaptation process... maybe I should suggest something there. Relatedly, there's some good room for adaptation-like processes with FormEncode (and with SQLObject and web framework integration thereof), and that might be an interesting place for experimentation. All the more because I think it's a hard problem that needs new ideas and new programming techniques, so generic functions may very well make it successful where traditional techniques would not. When going down a road strewn with the corpses of previous projects, one must consider why the next project will be different. Forms are a place where exceptions are common, and generic functions are uniquely capable of handling exceptional behavior.

# Ian Bicking

"""it introduces C code into a system that has no C code"""

Not true; RuleDispatch can be built --without-speedups if you want. It does make some difference to the performance, sure, but if I recall correctly the difference is measured in a handful of microseconds for most things, mainly because I haven't written a lot of C code for it yet anyway. And for Windows -- the main platform where lack of a compiler is usually an issue -- I distribute a binary egg.

"""Events are also a less complicated concept for other contributors to understand"""

  • Event type == generic function
  • Event data == function parameters
  • Subscribing a callback == adding a method

What's so complicated about that? It's true that you have to make some policy decisions about what kinds of subscribers you'll have - i.e., before/after/around/custom - but once you've made them that's that.

Further, nothing says you have to expose the full power of the system. You can always wrap it in a shortcut API that implements common use cases. Just say that the generic functions are event objects, and have subscribe function(s) that take an event, a callback, and a filter. Voila! It's an event system, and nobody needs to know any part of RuleDispatch that you don't want to expose.

# Phillip J. Eby

Oh those bloody spammers!

Ian,

what about support for the most widely used commercial dbs (oracle, mssql, db2)? That could also be a reason why python isn't taken seriously in webapp dev.

# Michael