Ian Bicking: the old part of his blog

What is Paste, yet again

The universal feedback I've received on paste attempts to define Paste is that no one got it. My fault. Anyway, since then I've started thinking differently about the composition of Paste and split it up in a different way. So, yet again, I try to explain what Paste is:

Paste is just a name

OK, that's a dodge. Paste is stuff I think is important and underdeveloped elsewhere. But I can be more specific about the pieces...

Paste Script

Paste Script is a pluggable command-line script (called paster). Using plugins you can add new commands like paster my-command to it, either adding them globally or framework-local. "Framework" doesn't necessarily mean web framework, it can mean anything, including ad hoc commands specific to a project.

Paste Script provides two commands by default:

Paste Deploy

Paste Deploy is a pluggable configuration-file loader for web applications and servers.

The "plugins" that you provide to Paste Deploy are actually factories for applications and servers. The configuration file will indicate what factory you want to use, and then the arguments to send to the factory. This is to say, the application and its configuration.

Factories themselves can be other configuration files, or Python Egg entry points. That's part of setuptools, not Paste, so I won't describe it here. But it's something good to understand. Ben Bangert had a post describing setuptools in the context of Paste, which might be helpful.

Paste Deploy is pretty simple when you get down to it -- it's just factories, passing in configuration data, and returning a WSGI app. What's a WSGI app? Your framework should be able to give you one of those -- if you are just writing an application, you shouldn't have to know about the details of WSGI. If your framework doesn't support WSGI, then you should ask the framework authors to fix that.

Paste Deploy does implicitly encourage some specific practices. One is that you keep your configuration separate from your application. Hopefully the benefits are obvious, but in practice this is not the norm in a lot of current frameworks. Also, there are other tools (in Paste core which I talk about next) which work better if your application is well encapsulated and obeys the WSGI spec and uses WSGI semantics (one common issue is respecting and properly adjusting SCRIPT_NAME and PATH_INFO, though other ones come up). These again are framework issues, but ones that several frameworks do not currently get right.

Another feature of a Paste-Deploy-enabled application is that it will have a simple factory that takes specific arguments. So while it makes the application configurable, it also makes it really easy to invoke the application programmatically. This means that it should be easy to create applications that embed each other, by delegating some incoming requests to the embedded application.

Maybe I'll do a screencast on Paste Deploy sometime soon. It's best feature, IMHO, is that it's actually quite simple, and it maps to simple Python function invocation.

Paste (the rest)

So there's another package called just "Paste". It has a bunch of tools. These tools are "dead", in that they don't magically appear in your environment, you aren't forced to use them, and they mostly aren't required in order to work with each other. The full list is described on the features page.

You don't have to use any of these. Most of them are of more interest to a framework developer than an application programmer, and they are best used when adopted into a framework. If you have an ad hoc framework then you might be interested; if you aren't using some other framework, you've likely built an ad hoc framework whether you meant to or not. It's hard to program directly against CGI, mod_python, or some other low-level system without somehow creating a framework.

All the Paste parts use WSGI, because they are all intended to be framework neutral, and there is no "framework neutral" in Python web programming for much of this functionality except for WSGI.

A number of the tools are "WSGI middleware", which means they wrap WSGI applications. What's a WSGI application? Remember: if you aren't developing a framework you shouldn't need to know, except to know that your framework creates a WSGI application for you. But, back to middleware -- it means that a request goes through the middleware (which can modify it) and the response goes out through the middleware (which can also modify the response). This is how things like exception catching are implemented:

But, this is a framework concern. If you like the exception catching middleware (because it's cool), then your framework should adopt it, or at least allow for it, and it should pass through configuration to that code. It's probably not something that a modest application developer will concern themselves with. Of course this is open source, so everyone can be a framework contributor. In the end, if you use lots of these Paste tools you can write a framework from scratch with very little code. Your framework will be all about creating an aesthetically cohesive wrapper around the tools; Paste isn't a framework, WSGI isn't a framework, they are just agreements, and tools that utilize those agreements.

There are also some tools in Paste that are useful with Paste Deploy, like something to map requests to subapplications based on URL prefixes. It should be possible to apply these to a well-encapsulated WSGI application without problem, and so these are "deployer" concerns, not application author or framework developer concerns.

Conclusion

So that's what Paste is. Maybe it makes more sense this time.

One thing I'll note is that I think Paste Deploy is the most important of these pieces. The other's are convenient, they might make your development easier, but if they don't then you shouldn't use them. But making your application available through Paste Deploy makes life easier for everything that wants to use your application. If, say, CatWalk was available through Paste Deploy, then I could embed it into any application I write, even if I'm not using TurboGears. If PyFileServer was Paste Deploy enabled, then I could include a WebDAV server with any app, etc. Paste Deploy is what starts to address Python's flaws compared to PHP.

Created 09 Nov '05

Comments:

So is Paste Deploy like a bootstrapper for a multiple service setup? It will import modules, instantiate classes, get multiple servers up and running and connect them to each other? Then the benefit over using native configuration and startup for individual servers would be - one unified way to do it. The servers still have to be able to talk to each other.

# Shalabh

Actually all the processes live in one process, so there's only one server. When a request comes in, it is dispatched to the appropriate application. Well, it works with forking servers too, but the applications still share a pool of processes in that case.

It would be cool to be able to spawn applications in subprocesses, and I could actually see a lot of utility in that, but it's not something that is possible now.

Paste Deploy does import modules for both application and server, and then plug them together. Well... the actual importing is typically done through Eggs (and entry points), but that's the basic idea.

# Ian Bicking

Ok - so it's like the QLime configuration module I wrote earlier (no longer available). Would Paste also instantiate any class I want and attach it anywhere? Both Quixote and CherryPy publish a tree of objects. In Quixote (don't know about CherryPy) this 'tree' is build by Python code. Do you think it would be useful if I could (via configuration) specify how to build the published tree? For example, instantiate class someusermanager.User and attach it at root.user.

# Shalabh

It doesn't use a tree of objects, so it's not exactly like that. egg:Paste#urlmap matches URL prefixes and redirects them to applications (which are in turn configured in another section or file). It doesn't use any kind of object publisher, so it doesn't construct any set of objects. Internally the application is free to parse the rest of the URL using an object publisher. I find the composition of distinct applications through assignment to be... well, it makes the applications no longer distinct. Monkeypatching objects to make the resolver happy is... well, not so hot either.

Typically an application will provide a simple callable, that in turn will instantiate the class. This can be provided by your framework, or embedded in the application itself. Since some frameworks don't have a root class, this concept isn't built in. (e.g., some frameworks simply have a root directory)

# Ian Bicking

I think of Paste Deploy as building a "WSGI Stack", Paste Script as a way to call Paste Deploy from the command line and create new projects and Paste as a collection of WSGI middleware and utility functions. Are people confused by Paste or WSGI servers/middleware/apps in general?

In any case, as the Paste stuff becomes more concrete and there are more examples out there I think everyone will get it.

# Sean Cazzell

Could you clarify the current status with CherryPy and its children Subway and TurboGears?

# Bill Seitz