Ian Bicking: the old part of his blog

Zope to LocalFS

So at my work we're in this kind of weird limbo, where a lot (but not all) of our newer projects are in Webware, and we have a lot of stuff left in Zope. And while some of it could really use a rewrite anyway -- especially the stuff from the Bad Old Days of pure DTML applications -- rewriting that stuff doesn't really accomplish anything. If it ain't broke...

Anyway, some new little app came up, and I figured I'd stick to Zope because it was just too small to do in Webware, and we haven't completely figured out how we're doing Webware deployment -- the time invested is worth it for large projects, but this wasn't a large project. Making small projects in Webware more feasible is definitely important to me, but this project was starting to get overdue, so I couldn't figure out too much.

But even then, I decided I'd do a little experimentation in Zope, and see if I could put the whole application into LocalFS -- which means all the files would be on the disk. I could have written it as a Zope Product, but I don't see Zope as our future and I didn't feel like investing the effort. But these legacy applications -- written in DTML, ZPT, ZSQL, and Python Scripts, stored in the ZODB, edited through the web -- are going to be around for a while. And I really want them under source control for all sorts of reasons -- not just history, but ease of deployment, centralized indexing of our applications, synchronizing development in branches, and deploying applications.

LocalFS, however, doesn't support ZSQL or Python Scripts. So I thought I'd try to fix that. In the end it wasn't as hard as I expected; it really is just Python code. The inheritance is out of control (with many base classes), and it has some funny idioms, and LocalFS is probably always going to be a little wonky in Zope (e.g., it doesn't participate in transactions), but to accomplish what we needed wasn't too hard. The code is accessible to humans. The only real catch was debugging in the presence of Acquisition (for a longer description of acquisition see this). I am convinced that Zope would have been way, way, way better if they'd never used Acquisition -- it's a clever hack that ruins the entire system. Thankfully Zope 3 has gotten rid of Acquisition. The inheritance situation in Zope 2 is difficult, but ultimately it's something you can get your head around if you read enough code (and probably you can avoid for small changes). But Acquisition simply ruins everything and makes the entire system unpredictable.

I had thought about using FSDump, which dumps part of the ZODB out to disk, specifically for use with version control. But it's one-way, just for writing out. And our use of the ZODB is actually really conservative, with all our valuable code really in those four types, plus some configuration in Properties and database connections, so LocalFS allows us to keep the canonical version in the filesystem and avoid both the ZODB and ZMI entirely. And since all our volatile data is in a database, I think LocalFS is safe enough (I might be worried otherwise).

I'll try to separate out the changes to LocalFS into a couple patches sometime soon, but for now I just thought I'd note the experience.

Created 21 Feb '05

Comments:

http://www.infrae.com/download/FileSystemSite might be what you are searching for

This is a repackaging of the CMF's FileSystem Directory Views such
that it is independent of the CMF.

The major use of this is to develop view code on the file system
such that it can be edited with normal development utilties and
checked into a source control management system.
# d2m

You might want to look at Ape which is a really cool abstract storage layer for the ZODB (actually, it's meant to be / trying to be a standalone abstract persistence system). It has file system representation with knowledge of Z SQL Methods, Python Scripts, Page Templates, etc. I've used it before to move items to and from disk with a Zope copy/paste operation.
# Jeff Shell

Also - Ape does a really good job of preserving Properties in an easy to read and edit format. It's file system representation is really really good - and it was a few revisions ago that I used it.

# Jeff Shell

No Ape is not a good product, it breaks zexp export and undo fonctions. It also stores metadatas in a .properties file but with binary in it.

# Alexdb

Another option is ZFolder2Product, located at http://sourceforge.net/projects/zfolder2product

ZFolder2Product takes the contents of an existing Zope2 folder and creates a custom Zope Product, which re-creates the entire folder contents when installed. The Product and folder contents are saved in external files usable within a version control system like CVS.

# rhys

Hi, I am trying to tweak localfs to support Python script and zsql methods. Where can I find the fixes that you have made, so that i can avoid reinventing the wheel. Can you please provide the patched version.

# Ratish