Ian Bicking: the old part of his blog

Re: TurboGears and Pylons (a technical comparison)

Good article. Linked on the Pylons wiki. The differences really come down to what Ben thinks is cool vs what Kevin thinks is cool, and to their different philosophical emphases. Historically, Kevin wanted "one way to do it" throughout the whole stack for newbies, and he really likes how CherryPy maps URL component directly to a function and query parameters to arguments. Ben seems to focus more on flexibility, thus choosing Routes and Paste, which can be configured in every imaginable way under the sun.

However, practicality is forcing both frameworks to accomodate libraries that weren't originally intended (Buffet for both, SQLAlchemy for TG, ToscaWidgets for Pylons, and presumably Routes for TG). Since users are clamoring for access to the same set of Python libraries in both cases, this is pushing both frameworks toward each other.

Perhaps this shows the biggest mistake of the original TG vision: the days are gone when a developer could impose an arbitrary choice of peripheral libraries on users. To some extent this is because the libraries themselves have been evolving out from under them: Kid has speed problems which are corrected in Genshi, Myghty has too many non-template features which are absent in Mako, SQLAlchemy is more multi-layered than SQLObject. (How's SQLObject 2 coming along, Ian?) All this in one short year. Who knows what next year holds. Another factor is the real world: some users have a large number of existing templates or custom libraries which they can't just rewrite at the drop of a hat.

You're right that nothing is preventing Pylons from becoming more newbie-friendly or TG from becoming more backend-flexible: these areas have just not received much developer attention. They're evolving into KDE vs GNOME: more compatible but still distinct. Perhaps the codebases could be merged at some point, but then what happens to the two APIs? Either one disappears or is frozen, or they both continue to diverge. TG has looked into replacing CherryPy with RhubarbTart (a WSGI compatibility library for CP's dispatching model and global variables). Could a Paste- or Pylons-ified version of RhubarbTart be written that TurboGears could sit on top of?

Comparing CherryPy to Paste (the "sensitive point" link in Ian's original post) is a little unfair. Paste is lower-level than CherryPy and is not aimed at end users (except the hacker sort). Something like RhubarbTart-on-Paste could hide the WSGI dictionary that is so offensive to some people. In spite of Paste's more "cumbersome" and "non-Pythonic" syntax, I found it easier to trace the Paste code: how Pylons launches an application. I tried to do that with CherryPy (as TG uses it) and got totally lost in the thread pool code. Too many layers of do-nothing superclasses, and I couldn't follow a thread's lifetime. That in itself scared me away from CherryPy: I didn't want to depend on code I couldn't understand, especially when I started getting random errors and unhelpful tracebacks from various TG components (this was a year ago, before CherryPy 3).

So if we assume that hiding Paste's "cumbersome" WSGI dict in a super-framework level is acceptable, the incompatibilities come down to the config file format and command-line tools. Making a config format is hard: both TG and (I think) Paste have switched back and forth between INI format and Python format (or the hybrid Python-with-section-headers format). Each has some advantages and disadvantages. I wish I could declare ints and dicts directly in a Pylons config file. But the opposite position sees too many quotes and braces. Perhaps Paste and Pylons could one day support the hybrid format as an alternative. Of course this means type conversions would have to be pushed to a lower level when using Pylons INI format: right now they're done in the application code, and this would fail if the values were already converted. Neither format supports nested sections, if you ever want that.

Both Pylons and TurboGears need a CMS to fully compete with Django and Zope/Plone. This is a big task, and one which both communities could work on and perhaps create a common package both could use. One issue is whether this could be implemented purely as middleware or whether it would have to interact too closely with the application. As middleware it would use the WSGI protocol. As an application-level library it would have to sidestep the differences between TG and Pylons, meaning the frameworks would have to accommodate it either via custom code or a Buffet-like plugin system.

Comment on TurboGears and Pylons (a technical comparison)
by Mike Orr

Comments:

TG has looked into replacing CherryPy with RhubarbTart (a WSGI compatibility library for CP's dispatching model and global variables). Could a Paste- or Pylons-ified version of RhubarbTart be written that TurboGears could sit on top of?

RhubarbTart is very Pasty (I helped write parts of it, so that was inevitable ;). It also provides a surprisingly good story for CherryPy 2 compatibility -- some of it involves some messy monkeypatching, but it works. (During the sprint at last year's PyCon where we tried to use RhubartTart instead of CherryPy, we tried to avoid changing TurboGears too much, and TurboGears applications at all -- instead extending RhubartTart.)

Some of these hacks could be maintained or ported to Pylons. There are some other differences; for example, RT uses a request object modeled very directly on CP's. However, the Paste/Pylons request object does not require exclusive access to the environment, and multiple request objects with different APIs can coexist. There's also the dispatching system. Right now there's not really alternate dispatching systems for Pylons, but there could be, and RhubarbTart's could be ported over. Especially if you are comfortable depending on @expose (a different implementation than the current TG one with the same interface), that could be a very cleanly implemented alternative to Routes (RhubarbTart uses CherryPy's mechanisms, which rely on a more limited interface than WSGI).

Something like RhubarbTart-on-Paste could hide the WSGI dictionary that is so offensive to some people.

paste.wsgiwrappers.WSGIRequest already pretty much does that, so it's fine. They just chose to put of the strawman of the low-level functions in paste.request to argue against.

Making a config format is hard: both TG and (I think) Paste have switched back and forth between INI format and Python format (or the hybrid Python-with-section-headers format). Each has some advantages and disadvantages.

Early Paste used Python files for configuration. IMHO this worked quite poorly. Using Python expressions is much more reasonable (which is what CherryPy and TurboGears do), without the full generality of Python statements, conditionals, etc. Generally if you need really complicated configuration, I suspect you are doing your configuration wrong -- configuration should be a deployment concern, not something where you implement complex glue. The glue in a typical Paste applications still goes in Python files (e.g., middleware.py in a Pylons project). If that glue process needs to be informed by configuration values, you have the configuration and can glue stuff together based on that -- but providing a higher-level and domain-specific set of configuration values that effect that.

I wish I could declare ints and dicts directly in a Pylons config file. But the opposite position sees too many quotes and braces. Perhaps Paste and Pylons could one day support the hybrid format as an alternative. Of course this means type conversions would have to be pushed to a lower level when using Pylons INI format: right now they're done in the application code, and this would fail if the values were already converted.

At least most Paste-style conversions accept non-string values. It's really that when you do get a string value, you should treat it as though it's a config value. So 'false' is a false value (where in Python it would normally be true, since it is a non-empty string). So if you wanted to build a config format that accepted those values (and still used the Paste Deploy APIs, which are fairly config neutral), it would work. Of course, if you depend on that in your application you will be incompatible with simpler APIs.

It would be very reasonable to build another config file format that used Paste Deploy's APIs, but not its exact config format. Something more path-oriented (that natively supported urlmap) would be quite feasible, for instance.

Neither format supports nested sections, if you ever want that.

They both support nested keys, like component.key = value. Paste doesn't specially interpret those values (it's up to the consumer), while CherryPy does. When an application requires more structure in its configuration I usually use expressions in the keys.

Both Pylons and TurboGears need a CMS to fully compete with Django and Zope/Plone. This is a big task, and one which both communities could work on and perhaps create a common package both could use. One issue is whether this could be implemented purely as middleware or whether it would have to interact too closely with the application.

A CMS is a very vague thing. It means different things to different people, and any large website feels a lot like a CMS.

I think relying on HTTP, REST, and middleware, you could very possibly make CMS construction easy. For instance, AuthKit could be expanded with a full user system. You could make a pluggable authorization middleware where you could restrict and manage user's access to areas of the site. An event notification system could be constructed where you could notify a central component about the existence or deletion of pages, thus allowing a navigation system to keep track of what's on the website. Things like paste.cascade can be used to fold a CMS in with dynamic applications (like a shopping cart), while programmatically-driven layout (again using the Paste Deploy APIs, but not config format) could allow local "plugins" to be added (like a checkout, forum, etc).

But focusing on the big "CMS" will make this hard, because it's too big and too vague. Focusing on specific problems you want to solve and using some of the concepts of reusability that Paste encourages (like WSGI Framework Components) will build up the pieces you can use later.

# Ian Bicking

By CMS I specifically mean the workflow of one person editing a document over several sessions, others possibly previewing it and correcting it, and then clicking something to make it visible to the public. Enhancements include a rich Javascript editor, talkbacks, gathering articles into magazine issues, adding article metadata, etc. We have the parts for all this, but some users rightly want it to work out of the box, and will skip over TG/Pylons to Plone if they don't have it. Unfortunately Plone is in its own little world and depends on Zope 2, making it more difficult down the road when you want to combine the site with non-Plone tools.

Security is a relatively straightforward problem. Articles have roles (author, editor). There will have to be some map of users to roles ("editor for all articles", "author for this article"). Then the authorization code just has to enforce it.

# Mike Orr

It's true webservice will grow spread our posibilities. But it makes things more breakable. I'm not sure how to prevent the internet from getting instable...

# illy