Ian Bicking: a bloghttps://ianbicking.org/2007-08-23T11:10:00-05:00About2007-07-31T15:53:00-05:002007-07-31T15:53:00-05:00Admintag:ianbicking.org,2007-07-31:/blog/2007/07/about.html<p>Hi, I’m <a class="reference external" href="https://ianbicking.org">Ian Bicking</a>. I work at <a class="reference external" href="http://topp.openplans.org">The Open Planning Project</a>. I do lots of programming in Python. I keep a <a class="reference external" href="https://ianbicking.org/cv.html#open-source-projects">list of projects I participate in</a>, though I don’t always update it. Lots of stuff in <a class="reference external" href="http://svn.colorstudy.com">svn.colorstudy.com</a> and <a class="reference external" href="http://svn.pythonpaste.org">svn.pythonpaste.org</a> is written by me …</p><p>Hi, I’m <a class="reference external" href="https://ianbicking.org">Ian Bicking</a>. I work at <a class="reference external" href="http://topp.openplans.org">The Open Planning Project</a>. I do lots of programming in Python. I keep a <a class="reference external" href="https://ianbicking.org/cv.html#open-source-projects">list of projects I participate in</a>, though I don’t always update it. Lots of stuff in <a class="reference external" href="http://svn.colorstudy.com">svn.colorstudy.com</a> and <a class="reference external" href="http://svn.pythonpaste.org">svn.pythonpaste.org</a> is written by me.</p>
New Blog Software (Previous)2007-08-01T12:02:00-05:002007-08-01T12:02:00-05:00Ian Bickingtag:ianbicking.org,2007-08-01:/blog/2007/08/new-blog-software-previous.html<!-- -*- mode: rst -*- -->
<p>I’ve switched my software over to WordPress. This was long overdue, as anyone who ever wanted to read anything at all on this site probably knows. Sometime I should really write an article reflecting on the failures of my previous blog software. Lets just say that flat files aren …</p><!-- -*- mode: rst -*- -->
<p>I’ve switched my software over to WordPress. This was long overdue, as anyone who ever wanted to read anything at all on this site probably knows. Sometime I should really write an article reflecting on the failures of my previous blog software. Lets just say that flat files aren’t so hot either.</p>
<p>Now that my software doesn’t suck, I have lots of posts I have been embarrassed to write because every new post potentially introduced new people to my crappy site.</p>
<p>Hopefully everything is setup correctly, redirects, archives, and the new feed.</p>
<p>My one worry is WordPress comments, which suck a bit. They shouldn’t collect the horrible quantity of spam that the old site has, so that’s good, but I hate disconnected streams of comments. I’ve tried to modify the theme on this site to be more roomy, with less of the excessive whitespace that has become the norm. I hope this whitespace kick goes the way of Creating Killer Websites Using Table Based Layout. I.e., it’ll soon look dated and everyone will move on. So I hope you’ll have more than two inches of width to comment in. Honestly I wonder if I should just ditch WordPress comments and use something else entirely, like some kind of forum software and rig in some way of including the comments in the theme. I wanted to install <a class="reference external" href="http://meidell.dk/archives/2004/09/04/nested-comments/">threaded comments</a>, but the installation process is rather invasive.</p>
<p>For editing I turned TinyMCE off (ugh), and installed <a class="reference external" href="http://goldenspud.com/rotr/index.php/2006/12/15/using-restructuredtext-with-wordpress/">a restructured text plugin</a>. It took a while to figure out, since I have to include <tt class="docutils literal">.. <span class="pre">-*-</span> mode: rst <span class="pre">-*-</span></tt> in the header of each post. Oh well, a minor inconvenience. I used <a class="reference external" href="http://wordpress.org/extend/plugins/text-control/">Text Control</a> to enable Markdown in comments, but I had to replace the actual <tt class="docutils literal">markdown.php</tt> it used, which was broken.</p>
Old Archives2007-08-01T21:46:00-05:002007-08-01T21:46:00-05:00Ian Bickingtag:ianbicking.org,2007-08-01:/blog/2007/08/old-archives.htmlAtom Models2007-08-02T10:30:00-05:002007-08-02T10:30:00-05:00Ian Bickingtag:ianbicking.org,2007-08-02:/blog/2007/08/atom-models.html<p>I’ve been doing a bit more with Atom lately.</p>
<p>First, I started writing a library to manipulate Atom feeds and entries. For the moment this is located in <a class="reference external" href="http://svn.colorstudy.com/home/ianb/atom.py">atom.py</a>. It uses <a class="reference external" href="http://codespeak.net/lxml/">lxml</a>, as does everything markup related I do these days.</p>
<p>I came upon a revelation of sorts …</p><p>I’ve been doing a bit more with Atom lately.</p>
<p>First, I started writing a library to manipulate Atom feeds and entries. For the moment this is located in <a class="reference external" href="http://svn.colorstudy.com/home/ianb/atom.py">atom.py</a>. It uses <a class="reference external" href="http://codespeak.net/lxml/">lxml</a>, as does everything markup related I do these days.</p>
<p>I came upon a revelation of sorts when I was writing the library. I first started with a library that looked like this:</p>
<pre class="literal-block">
class Feed(object):
def __init__(self, title, ...):
self.title = title
..
@classmethod
def parse(cls, xml):
if isinstance(xml, basestring):
xml = etree.XML(xml)
title = xml.xpath('//title').text
...
return cls(title, ...)
def serialize(self):
el = etree.Element('{%s}feed' % atom_ns)
title = etree.Element('{%s}title' % atom_ns)
title.text = self.title
el.append(title)
...
return el
</pre>
<p>Obviously there’s ways to improve this and make it less verbose, and I went down that path for a while. But then I decided the whole path was wrong. Atom <em>is</em> <span class="caps">XML</span>. It’s not the representation of some object I’m creating. If I have something that can’t be represented in <span class="caps">XML</span>, it isn’t Atom, and it doesn’t belong in my Atom-related objects.</p>
<p>So instead I started making lxml more convenient when using Atom. I don’t keep any information except what is in the markup, I just make it more convenient to access that information.</p>
<p>I used lots of <a class="reference external" href="http://users.rcn.com/python/download/Descriptor.htm">descriptors</a> to do this, as the same patterns happened over and over. For instance, the Feed object is fairly simple:</p>
<pre class="literal-block">
class Feed(AtomElement):
entries = _findall_property('entry')
author = _element_property('author')
</pre>
<p>Which basically means that <tt class="docutils literal">feed.entries</tt> returns all <tt class="docutils literal"><entry></tt> elements, and <tt class="docutils literal">feed.author</tt> returns the single <tt class="docutils literal">author</tt> element.</p>
<p>There’s also accessors for text elements (like <tt class="docutils literal"><id></tt>) and date containing elements (like <tt class="docutils literal"><updated></tt>) and just to access <span class="caps">XML</span> attributes as Python attributes.</p>
<p>There’s a number of advantages:</p>
<ul class="simple">
<li>No hidden state.</li>
<li>No deferred errors, since everything is always represented in the <a class="reference external" href="http://en.wikipedia.org/wiki/XML_Information_Set"><span class="caps">XML</span> infoset</a>.</li>
<li>All <span class="caps">XML</span> extensions work, even though my classes don’t know anything in particular about them. There’s a full <span class="caps">API</span> for manipulating the <span class="caps">XML</span> that you can use, you don’t have to use my APIs.</li>
<li>Even more obscure kinds of extensions work fine, like a custom attribute on an element. There’s absolutely zero normalization that happens.</li>
<li>I only have to write the parts where the normal <span class="caps">XML</span> (lxml) APIs are inconvenient, so the implementation stays simple.</li>
<li>There’s no confusion over which object I might be talking about in my code. There’s no distinction between the <span class="caps">XML</span> object and the domain object.</li>
</ul>
<p>Since then I’ve been working on a <a class="reference external" href="https://svn.openplans.org/svn/TaggerClient/trunk/javascript/atom.js">Javascript library</a> for handling Atom. It’s not as elegant. I am trying to keep to this same principle, but of course I can’t actually extend the <span class="caps">DOM</span> and so I can’t add convenience methods. So instead I’m making a class that lightly wraps the <span class="caps">DOM</span> objects, with explicit getters and setters that simply read and modify those <span class="caps">DOM</span> objects.</p>
<p>One thing that I have found very useful in my development on the Javascript side is doctest-style testing. You can see the <a class="reference external" href="https://svn.openplans.org/svn/TaggerClient/trunk/javascript/tests/test.html">test</a>, but to run it you have to check it out (it uses some <tt class="docutils literal">svn:externals</tt> which you don’t get through the direct svn access). After using that testing some more and being pleased with the result, I decided to package the Javascript doctest runner a bit better. I removed the framework dependencies, did a bit of renaming (now it is <em>doctestjs</em> or <tt class="docutils literal">doctest.js</tt> instead of <em>jsdoctest</em>), wrote up <a class="reference external" href="http://svn.colorstudy.com/doctestjs/trunk/docs/index.html">fairly comprehensive docs</a>, and uploaded it to <a class="reference external" href="http://openjsan.org/doc/i/ia/ianb/doctestjs/0.9/"><span class="caps">JSAN</span></a> (though at the moment the trunk from svn is probably better to use). I think it’s an excellent way of doing unit testing in Javascript, much better than any of the alternatives I’ve seen. It even has some notable advantages over <a class="reference external" href="http://python.org/doc/current/lib/module-doctest.html">Python’s doctest</a>, like if you are using <a class="reference external" href="http://www.getfirebug.com/">Firebug</a> (which you must if you do Javascript development) you get a console session that runs in the same namespace as your tests, so you can easily do inspection of the objects if there’s a failure.</p>
<p>I’m not sure about <span class="caps">JSAN</span>. It’s nice to have an index. But I think they copy stuff from <span class="caps">CPAN</span> a bit too much. Why should you have a text <span class="caps">README</span> file? That’s just silly; of course Javascript documentation should be <span class="caps">HTML</span>. They batch processing. Processing one package a day
on the fly shouldn’t be overwhelming. They want a <span class="caps">MANIFEST</span> file. The standard metadata file is <span class="caps">YAML</span>, not <span class="caps">JSON</span>. This should all be a little more Javascripty in my opinion. But they also accept any kind of upload, so there’s nothing stopping you from ignoring what you don’t
care about. I’ll probably improve the packaging of doctestjs a bit in the future, and still ignore the parts I think are silly.</p>
Environmental Theater2007-08-02T19:38:00-05:002007-08-02T19:38:00-05:00Ian Bickingtag:ianbicking.org,2007-08-02:/blog/2007/08/environmental-theater.html<p>If you read <a class="reference external" href="http://www.schneier.com/blog/">Bruce Schneier</a>, as any good geek should, you probably are familiar with the term <a class="reference external" href="http://en.wikipedia.org/wiki/Security_theater">“security theater”</a>: measures that provide the feeling of security while doing little or nothing to actually provide security.</p>
<p><span class="caps">OK</span>, digression. We had this recycling program in Chicago where we put our recyclables in …</p><p>If you read <a class="reference external" href="http://www.schneier.com/blog/">Bruce Schneier</a>, as any good geek should, you probably are familiar with the term <a class="reference external" href="http://en.wikipedia.org/wiki/Security_theater">“security theater”</a>: measures that provide the feeling of security while doing little or nothing to actually provide security.</p>
<p><span class="caps">OK</span>, digression. We had this recycling program in Chicago where we put our recyclables in blue bags into the trash, and they pick the blue bags out of the trash. One imagines fancy computerized systems. In reality I think there’s just some people who watch trash go by on a
conveyor belt.</p>
<p>This all seemed fishy, but I hate waste on principle so I would dutifully recycled my trash, washed out containers, all that stuff. You’d sometimes hear an environmentalist criticize the program because there was little perceived benefit, and so people didn’t actually recycle much. The system seemed a little improbable to me too, but then I also realized that recycling is a balance and it’s easy to put more effort into recycling programs than is saved through the recycling itself. So maybe this was efficient, all things considered.</p>
<p>Then I learned that actually only <a class="reference external" href="http://www.chicagotribune.com/news/nationworld/chi-0610250063oct25,1,4280486.story">8% of recycling in blue bags is recovered</a>. 92% of the time when I clean things out and put them carefully in their own container, I might as well have just thrown them away. This really pissed me off, because it made it obvious that there never was an honest attempt to reduce waste through recycling. Blue bags were just what they would give people to make them stop complaining about recycling.</p>
<p>The irony is that the environmentalists didn’t complain about the recovery rates (which always were estimated at a low amount). They complained about how many people were recycling. Of course with a recovery rate that low it didn’t <em>matter</em> how many people were recycling. The entire program was a total farce. Now that the program is going away there doesn’t seem to be much anger about how deceptive the program was, and I don’t know if anyone is paying attention to the actual environmental impact of the new program.</p>
<p>Even if they recover the recycling it might still just be a game. Recycling is filled with farce. Metal recycling is great. That’s why there’s trucks that roam the alleys around Chicago looking for scrap metal. There’s a market and someone is willing to pay for the results. There’s not much of a market for anything else; maybe some glass, maybe a little plastic.</p>
<p>People actually get <em>angry</em> when recycling programs restrict the plastics they will take. It doesn’t occur to them that some plastics are simply garbage. They are worthless, and moving them around in special recycling containers just wastes everyone’s time. They are angry because they want to pretend they aren’t being wasteful. They aren’t getting enough environmental theater.</p>
<p>A more concerning kind of environmental theater is <a class="reference external" href="http://www.theoildrum.com/node/2354">ethanol</a>. With an <a class="reference external" href="http://en.wikipedia.org/wiki/EROI"><span class="caps">EROI</span></a> (energy invested vs. energy produced) that hovers just above one, it’s not helping the environment. Biofuels on the whole seem quite questionable. Brazil has more efficient ethanol, but it’s paired with deforestation. A similar thing happens when trees for palm oil replace natural forests. And of course in all these cases, if plants weren’t grown for fuel then plants would be grown for some other purpose. So I can’t really see any advantage in terms of <span class="caps">CO2</span> emissions — and when you consider the relative inefficiency compared to attaining fossil fuels, the net effect of biofuels is <a class="reference external" href="http://www.ecoworld.com/blog/2007/02/12/biofuel-is-not-carbon-neutral/">probably worse</a>.</p>
<p>Now that environmental concern is mainstream I think we need to be on the watch for environmental theater. Many of the people who play their parts in this theater are well meaning, which can make it awkward. These are people who believe that The Important Thing Is To Raise Awareness. But awareness has been raised, so the time for that kind of bullshit is past. Lying about solutions, exaggerating specific problems, being fuzzy about facts — that’s always been bullshit, and I’ve never found it acceptable. But it’s unfortunately become the norm among advocates of all sorts in these times. The irony is that the advocacy has been done, the case has been made, enough people are convinced, but it may be hard to move beyond the theater to meaningful action. Especially as the well-meaning people are replaced with cynics out to make money.</p>
Pronouncing “Django”2007-08-02T19:53:00-05:002007-08-02T19:53:00-05:00Ian Bickingtag:ianbicking.org,2007-08-02:/blog/2007/08/pronouncing-django.html<p>I’m not saying this to anyone in particular, but I’ve heard people pronounce Django incorrectly <em>way</em> too often. The “dj” in Django is a hard J, like in the word “jury” or “jolly”. You don’t pronounce the D.</p>
<p><strong>Update</strong>: Alex Limi tells me <em>I’m</em> wrong too …</p><p>I’m not saying this to anyone in particular, but I’ve heard people pronounce Django incorrectly <em>way</em> too often. The “dj” in Django is a hard J, like in the word “jury” or “jolly”. You don’t pronounce the D.</p>
<p><strong>Update</strong>: Alex Limi tells me <em>I’m</em> wrong too, and it’s a soft J, like… damn, I can’t think of a word that uses a soft J in English.</p>
<p>I’m not sure I can use that pronunciation, I’m afraid I’ll sound all Frenchy and weird. I’ll give it a go. Zhango zhango zhango… hmm…</p>
<p><strong>Another update</strong>: confirming my original pronunciation, Adrian says it <em>is</em> a hard J. Alex is just too European for his own good. Does the debate rage on? Hopefully not.</p>
Fast CGI that isn’t FastCGI2007-08-03T15:33:00-05:002007-08-03T15:33:00-05:00Ian Bickingtag:ianbicking.org,2007-08-03:/blog/2007/08/fast-cgi-that-isnt-fastcgi.html<p>There’s a bunch of techniques for doing deployments of long-running processes (Zope, Python server, Rails, etc). A pretty good technique is to do <span class="caps">HTTP</span> proxying. There’s some details and conventions I’d like to <a class="reference external" href="https://ianbicking.org/http-proxying-questions.html">see for <span class="caps">HTTP</span></a>, but that’s not my concern here.</p>
<p><span class="caps">HTTP</span> proxying isn’t …</p><p>There’s a bunch of techniques for doing deployments of long-running processes (Zope, Python server, Rails, etc). A pretty good technique is to do <span class="caps">HTTP</span> proxying. There’s some details and conventions I’d like to <a class="reference external" href="https://ianbicking.org/http-proxying-questions.html">see for <span class="caps">HTTP</span></a>, but that’s not my concern here.</p>
<p><span class="caps">HTTP</span> proxying isn’t great for commodity hosting. Mostly you need to set up a new long-running process, and commodity hosts don’t make that easy or reliable. <a class="reference external" href="http://www.fastcgi.com/">FastCGI</a> offers one solution to that, essentially putting the process management into Apache or whatever web server you are using.</p>
<p>The problem with FastCGI is that it is finicky. There’s lots of configuration parameters, lots of parts don’t work right, and there seems to be a golden path where things actually work but it’s hard to know exactly what that is.</p>
<p>Another technique that has been used in the past instead of FastCGI is a very small <span class="caps">CGI</span> script. One example in <a class="reference external" href="http://python.ca/scgi/"><span class="caps">SCGI</span></a> is called <a class="reference external" href="http://quixote.python.ca/scgi.dev/cgi2scgi.c">cgi2scgi</a>. This small script is fast to run (it compiles to 12kb), and all it does is take the <span class="caps">CGI</span> request and turn it into a <span class="caps">SCGI</span> request to a long-running server.</p>
<p>This is a nice start, and easy to deploy, except it doesn’t handle long-running processes. A great feature to add to something like this would be simple process management. I imagine something where if the socket (named or a port) that the <tt class="docutils literal">cgi2scgi</tt> script connects to isn’t up or working, it runs a script that will start the server. If another request comes in while the server is starting up, it shouldn’t try to start the server twice. If the server is randomly killed (as is common on commodity hosters) then the next request will try to bring the server up.</p>
<p>Unlike FastCGI, this won’t try to handle different process models or anything fancy. It’s up to the startup script to set everything up properly, start multiple worker processes if necessary, etc. There’s probably some tricky details I haven’t thought of, and it’s slightly annoying to write all this in C (but necessary, since it’s part of the <span class="caps">CGI</span> script, which must be small). But I think it can be done better than existing in-the-wild FastCGI implementations.</p>
<p>And when we’re done, I think we could have something that would be a really good basis for commodity hosting of a whole bunch of non-<span class="caps">PHP</span> frameworks. You can distribute the Linux binaries, as all the Commodity Hosts That Matter can run those (even the <span class="caps">BSD</span> ones should be fine). Easy application installation practically falls right out of that.</p>
Zonbu & S32007-08-04T13:42:00-05:002007-08-04T13:42:00-05:00Ian Bickingtag:ianbicking.org,2007-08-04:/blog/2007/08/zonbu-s3.html<p>I read <a class="reference external" href="http://times.usefulinc.com/2007/08/04-zonbu">Edd Dumbill’s post on the Zonbu computer</a> with interest. The <a class="reference external" href="http://zonbu.com/">Zonbu</a> is a small and inexpensive computer, reminiscent of the Mac Mini but running Linux. The disk is fairly small (4Gb flash) and is intended to serve more as a cache for your network storage than as …</p><p>I read <a class="reference external" href="http://times.usefulinc.com/2007/08/04-zonbu">Edd Dumbill’s post on the Zonbu computer</a> with interest. The <a class="reference external" href="http://zonbu.com/">Zonbu</a> is a small and inexpensive computer, reminiscent of the Mac Mini but running Linux. The disk is fairly small (4Gb flash) and is intended to serve more as a cache for your network storage than as your primary store.</p>
<p>The network store is a frontend on <a class="reference external" href="http://www.amazon.com/gp/browse.html?node=16427261">Amazon S3</a>. This is interesting but confusing, because Zonbu is selling the computer at a price of $99 if you agree to a two year contract for storage at $12.95 a month (about $300 over two years).</p>
<p>The underlying S3 storage is pretty cheap: $0.15 per Gb-month, and $0.10/$0.18 per Gb-upload/download (discounts for higher quantities, which probably Zonbu can get but an individual user couldn’t). So if you are storing, say, 10Gb of data, and retrieving about 10Gb per month (including all the syncing, cache misses, etc), that comes to about $3 per month. Zonbu costs between $0.50 and $0.20 per Gb-month, depending on the plan, and you pay for capacity, not what you actually use (S3 only charges for what you really use). I assume there are bandwidth limits but they aren’t published.</p>
<p>As an aside, I was looking for backup systems for my dad a few months ago, and looked at some of the backup systems that included network storage. They were often in the range of $10-20 per month, and weren’t very high capacity. I came upon <a class="reference external" href="http://www.maluke.com/s3man/">S3 Backup</a>, which is a fairly simple Windows program to upload to S3. The price of S3 is way better than any of the other commercial solutions. The billing and account setup isn’t as simple as other systems (since it’s not intended to be), but this seems like something that should be fixed. There should be a consumer version of S3. It could make it easier for software developers to make services for people without actually having to maintain infrastructure. Or maybe more accurately, it would make this possible for <em>open source developers</em>, since we have no interest in being the intermediary for anything as that’s all liability with no payoff. (Or maybe it’s the opposite — only by being an intermediary can you get payoff? The economics of open source get confusing.)</p>
<p>Zonbu, as a device and company, appeals to me. But I can’t help but feel frustrated about the network storage pricing, even though those prices are completely reasonable (and it seems without draconian cancellation fees like mobile phones). Still there’s something about the equation that I just <em>hate</em> — loss leaders, unnecessarily intermediated transactions, hidden costs, and a price structure that depends on people not fully utilizing what they pay for. And I <em>really like</em> the S3 pricing — you pay for what you use and the pricing is completely transparent. What I like about it is that at no point is Amazon expecting you to act irrationally, and for Amazon to profit from your irrational choices. They aren’t expecting you to reserve more than you need. They aren’t going to punish you if you don’t reserve enough.</p>
<p>Another part of why I like S3’s structure is that Amazon (well, Amazon Web Services) owns this particular space in terms of services, and it’s not because of advertising or because they cornered the market or used proprietary anything to restrict choices or made secret business deals with anyone. They simply are providing a service with enough quality and efficiency that no one else can compete (at least at the moment). When quality and efficiency drives market choices it makes me feel all fuzzy and capitalist. This happens infrequently enough that perhaps I get a little overly excitable about resellers with different price structures.</p>
Atompub & OpenID2007-08-06T11:38:00-05:002007-08-06T11:38:00-05:00Ian Bickingtag:ianbicking.org,2007-08-06:/blog/2007/08/atompub-openid.html<p>One of the thinmgs I would like to do is to interact with <a class="reference external" href="http://atompub.org/rfc4287.html">Atompub (aka Atom Publishing Protocol)</a> stores in Javascript through the browser. Since this effectively the browser itself interacting with the Atompub server, browser-like authentication methods would be nice. But services like Atompub don’t work nicely with …</p><p>One of the thinmgs I would like to do is to interact with <a class="reference external" href="http://atompub.org/rfc4287.html">Atompub (aka Atom Publishing Protocol)</a> stores in Javascript through the browser. Since this effectively the browser itself interacting with the Atompub server, browser-like authentication methods would be nice. But services like Atompub don’t work nicely with the kinds of authentication methods that normal websites use. One of these is <a class="reference external" href="http://openid.net/">OpenID</a>, which is particularly browser-focused.</p>
<p>From the perspective of a client, OpenID basically works like this:</p>
<ul class="simple">
<li>You need to login. You tell the original server what your OpenID <span class="caps">URL</span> is, somehow.</li>
<li>The original server does some redirects, maybe some popups, etc.</li>
<li>Your OpenID server (attached to your OpenID <span class="caps">URL</span>) authenticates you in some fashion, and then tells the original server.</li>
<li>The original server probably sets a signed cookie so that in subsequent requests you stay logged in. You cannot do this little redirection dance for every request, since it’s actually quite intrusive.</li>
</ul>
<p>So what happens when I have an XMLHttpRequest that needs to be authenticated? Neither the XMLHttpRequest nor Javascript generally can do the authentication. Only the browser can, with the user’s interaction.</p>
<p>One thought I have is a 401 Unauthorized response, with a header like:</p>
<pre class="literal-block">
WWW-Authenticate: Cookie location="http://original.server/login.html"
</pre>
<p>Which means I need to open up <tt class="docutils literal"><span class="pre">http://original.server/login.html</span></tt> and have the user log in, and the final result is that a cookie will be set. XMLHttpRequest sends cookies automatically I believe, so once the browser has the cookie then all the Javascript requests get the same cookie and hence authentication.</p>
<p>One problem, though, is that you have to wait around for a while for the login to succede, then continue on your way. A typical situation is that you have to return to the original page you were requesting, and people often do something like <tt class="docutils literal"><span class="pre">/login?redirect_to=original_url</span></tt>. In this case we might want something like <tt class="docutils literal"><span class="pre">/login?opener_call=reattempt_request</span></tt>, where when the login process is over we call <tt class="docutils literal">window.opener.reattempt_request()</tt> in Javascript.</p>
<p>Maybe it would make sense for that <tt class="docutils literal">location</tt> variable to be a <a class="reference external" href="http://bitworking.org/projects/URI-Templates/draft-gregorio-uritemplate-01.html"><span class="caps">URI</span> Template</a>, with some predefined variables, like opener, back, etc.</p>
<p>For general backward compatibility, would it be reasonable to send 307 Temporary Redirect plus <span class="caps">WWW</span>-Authenticate, and let XMLHttpRequests or other service clients sort it out, while normal browser requests do the normal login redirect?</p>
<p><strong>Update:</strong> Another question/thought: is it okay to send multiple <span class="caps">WWW</span>-Authenticate headers, to give the client options for how it wants to do authentication? It seems vaguely okay, according to <a class="reference external" href="http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.47"><span class="caps">RFC</span> 2616 14.47</a>.</p>
Tempita2007-08-06T16:50:00-05:002007-08-06T16:50:00-05:00Ian Bickingtag:ianbicking.org,2007-08-06:/blog/2007/08/tempita.html<p>I mentioned a templating language I put into Paste <a class="reference external" href="https://ianbicking.org/what-im-up-to-jun07.html">a while ago</a>, but since then I extracted it into a separate package called <a class="reference external" href="http://pythonpaste.org/tempita/">Tempita</a>. I think the documentation is fairly complete (it’s a small language), but I’ll describe it shortly here.</p>
<p>I wanted a text-substitution language, because I …</p><p>I mentioned a templating language I put into Paste <a class="reference external" href="https://ianbicking.org/what-im-up-to-jun07.html">a while ago</a>, but since then I extracted it into a separate package called <a class="reference external" href="http://pythonpaste.org/tempita/">Tempita</a>. I think the documentation is fairly complete (it’s a small language), but I’ll describe it shortly here.</p>
<p>I wanted a text-substitution language, because I wanted something to be used to generate Python files, config files, etc. I also didn’t want a complex <span class="caps">API</span>, with search paths and components or something that interacts with import machinery, or any of that. <a class="reference external" href="http://python.org/doc/current/lib/node40.html">string.Template</a> is <em>almost</em> good enough, but not quite.</p>
<p>I started with the idea of something vaguely like <a class="reference external" href="http://www.djangoproject.com/documentation/templates/">Django Templates</a>, though since I didn’t care about more advanced templating features like blocks that didn’t apply to my use cases. You do variable substitution with <tt class="docutils literal">{{var|filter}}</tt>, and there’s no escape character, and that’s about where the similarity ends.</p>
<p>I realized there was no real reason to use anything but <tt class="docutils literal"><span class="pre">{{...}}</span></tt>, so it’s just <tt class="docutils literal">{{if expr}}</tt>, <tt class="docutils literal">{{endif}}</tt>, etc. There’s an escape for arbitrary Python, similar to how Kid does it — you can have blocks of Python code, but the Python code can only prepare variables and functions, it can’t write anything. I think this gives a nice escape for complex logic (for times when you can’t put the logic in a <tt class="docutils literal">.py</tt> file), without the jumbled mish-mash of languages like <span class="caps">PHP</span> where you can trully mix functions and output.</p>
<p>Because it allows Python expressions everywhere, special tags don’t seem so necessary. Instead you can just provide functions to do whatever you need. I wrote <a class="reference external" href="http://pythonpaste.org/tempita/#bunch-and-looper">a couple little ones</a> as a start. There’s a few things that are awkward still, because there’s no way to define a block of template as a function, or pass the output of a block to a function. I haven’t actually needed these yet, but I can <em>imagine</em> needing this (e.g., when creating nested structures).</p>
<p>I wouldn’t suggest using this templating language in a web application, but I think it can be quite helpful for all the cases where you have to generate text and you <em>aren’t</em> writing a web application (e.g., a <a class="reference external" href="http://www.groovie.org/articles/2007/02/11/wsgi-middleware-isnt-middleware-time-for-better-language">Framework Component</a>). In my experience the web templating languages tend to be complex to invoke and understand in these contexts (and <a class="reference external" href="http://projects.dowski.com/projects/buffet">Buffet</a> unfortunately doesn’t help in my mind, as it’s loading system is so vague).</p>
XO B42007-08-07T22:19:00-05:002007-08-07T22:19:00-05:00Ian Bickingtag:ianbicking.org,2007-08-07:/blog/2007/08/xo-b4.html<a href='https://ianbicking.org/wp-content/uploads/2007/08/front-view.jpg' title='Front view of the XO'><img class="alignright" src='https://ianbicking.org/wp-content/uploads/2007/08/front-view.thumbnail.jpg' alt='Front view of the XO' /></a><p>I recently received a Beta-4 <a class="reference external" href="http://en.wikipedia.org/wiki/XO-1_(laptop)"><span class="caps">XO</span> laptop</a>. I won’t describe the hardware on the whole, but probably a number of readers here have seen the B2 laptops so I thought I’d write up a quick description of the changes I’ve noticed. If you haven’t seen the …</p><a href='https://ianbicking.org/wp-content/uploads/2007/08/front-view.jpg' title='Front view of the XO'><img class="alignright" src='https://ianbicking.org/wp-content/uploads/2007/08/front-view.thumbnail.jpg' alt='Front view of the XO' /></a><p>I recently received a Beta-4 <a class="reference external" href="http://en.wikipedia.org/wiki/XO-1_(laptop)"><span class="caps">XO</span> laptop</a>. I won’t describe the hardware on the whole, but probably a number of readers here have seen the B2 laptops so I thought I’d write up a quick description of the changes I’ve noticed. If you haven’t seen the <span class="caps">XO</span> in person, then the minutia of this post may be boring.</p>
<p>First and most substantially, the <span class="caps">CPU</span>, memory, and disk have all been upgraded. It now has <span class="caps">256MB</span> <span class="caps">RAM</span>, <span class="caps">1GB</span> of flash disk, and a 433MHz Geode processor. This makes a very significant impact on the speed.</p>
<a href='https://ianbicking.org/wp-content/uploads/2007/08/back-cover.jpg' title='Back cover'><img class="alignleft" src='https://ianbicking.org/wp-content/uploads/2007/08/back-cover.thumbnail.jpg' alt='Back cover' /></a><p>It features a big colored <span class="caps">XO</span> on the back. Laptops will get different random combinations of X and O colors, so you can tell one laptop from another. I’m a little disappointed to have coincidentally received an X with the same color as the laptop’s green.</p>
<a href='https://ianbicking.org/wp-content/uploads/2007/08/tilt.jpg' title='The screen tilt'><img src='https://ianbicking.org/wp-content/uploads/2007/08/tilt.thumbnail.jpg' class="alignright" alt='The screen tilt' /></a><p>The screen now tilts back a bit further than it used to. It’s now comfortable to have it on a table or my lap, where before I liked to have it higher up. Putting the B2 and B4 side-by-side the change in tilt doesn’t seem significant, but using them it’s quite noticeable.</p>
<a href='https://ianbicking.org/wp-content/uploads/2007/08/rubber-antenna.jpg' title='Rubber antenna'><img class="alignleft" src='https://ianbicking.org/wp-content/uploads/2007/08/rubber-antenna.thumbnail.jpg' alt='Rubber antenna' /></a><p>The antenna (“ears”) are now rubber. This is intended to increase its durability when dropped (apparently it can sustain a <a class="reference external" href="http://lists.laptop.org/pipermail/community-news/2007-July/000069.html">1.5 meter drop onto its antenna</a>). Unfortunately along the way the latching mechanism became stiffer, so I don’t let people puzzle out how to open it anymore, it’s requires too much forcing to guess.</p>
<a href='https://ianbicking.org/wp-content/uploads/2007/08/bumps.jpg' title='Handle texture'><img class="alignright" src='https://ianbicking.org/wp-content/uploads/2007/08/bumps.thumbnail.jpg' alt='Handle texture' /></a><p>The handle is now textured. I never had any problem keeping a grip on it before, but the dots look nice. A cute detail is that around the edge the dots turn into X’s, making little <span class="caps">XO</span> figures.</p>
<a href='https://ianbicking.org/wp-content/uploads/2007/08/top-keys.jpg' title='Top of keyboard'><img class="alignleft" src='https://ianbicking.org/wp-content/uploads/2007/08/top-keys.thumbnail.jpg' alt='Top of keyboard' /></a><p>The keyboard has had a few changes. Instead of a slider for the backlight and another slider for the volume, they have been combined into one key with four sensors. The slider that had been used for the backlight is now free to be used by applications. The chat button changed appearances a bit, and it looks like the camera/voice button has been turned into a zoom button. The mouse buttons now have an X on the left button and an O on the right button, to make it easier to refer to them in instructions. The keyboard also is generally more responsive; the spacebar doesn’t seem to have any dead spots anymore, and the keys are more reliable when tapped. It’s still a <em>very</em> small keyboard if you try to touch type, but it’s not impossible (at some point I seem to have lost the ability to hunt and peck, but I can get by).</p>
<p>There are now small white LEDs under the plastic for both the microphone and camera. Whenever these are in use, the light turns on. This is done in hardware as a security measure, so malicious software can’t surreptitiously record things. The plastic around the screen is also now a light color of gray instead of white; from what I understand to make the screen seem higher contrast, I suppose because the white of the plastic could otherwise overpower the white of the screen.</p>
<p>The laptop also came with an <a class="reference external" href="http://en.wikipedia.org/wiki/Lithium_iron_phosphate_battery">LiFePO4</a> battery, which is lighter and higher capacity than the NiMH batteries used before. The total difference in weight isn’t very noticeable. (Li-Ion batteries haven’t been an option in the <span class="caps">XO</span> because of safety concerns.)</p>
<p>The software has had more changes, but that’s an entirely different topic.</p>
Opening Python Classes2007-08-08T14:03:00-05:002007-08-08T14:03:00-05:00Ian Bickingtag:ianbicking.org,2007-08-08:/blog/2007/08/opening-python-classes.html<p>So, I was reading through comments to despam my old posts before archiving them, and came upon <a class="reference external" href="http://dan.chokola.com/journal/?user=dan&entry=2007-01-27.105426">this old reply</a> to <a class="reference external" href="https://ianbicking.org/re-ruby-and-python-compared.html">this old post of mine</a> which was a reply to <a class="reference external" href="http://web.archive.org/web/20070329162213/http://www.rexx.com/~oinkoink/Ruby_v_Python.html">this much older post</a>.</p>
<p>I won’t reply to that post much, because it’s mostly… well, not useful …</p><p>So, I was reading through comments to despam my old posts before archiving them, and came upon <a class="reference external" href="http://dan.chokola.com/journal/?user=dan&entry=2007-01-27.105426">this old reply</a> to <a class="reference external" href="https://ianbicking.org/re-ruby-and-python-compared.html">this old post of mine</a> which was a reply to <a class="reference external" href="http://web.archive.org/web/20070329162213/http://www.rexx.com/~oinkoink/Ruby_v_Python.html">this much older post</a>.</p>
<p>I won’t reply to that post much, because it’s mostly… well, not useful to respond to. But people often talk about the wonders of Open Classes in Ruby. For Python people who aren’t familiar with what that means, you can do:</p>
<pre class="literal-block">
# Somehow acquire SomeClassThatAlreadyExists
class SomeClassThatAlreadyExists
def some_method(blahblahblah)
stuff
end
end
</pre>
<p>And <tt class="docutils literal">SomeClassThatAlreadyExists</tt> has a <tt class="docutils literal">some_method</tt> added to it (or if that method already exists, then the method is replaced with the new implementation).</p>
<p>In Python when you do this, you’ve defined an entirely new class that just happens to have the name <tt class="docutils literal">SomeClassThatAlreadyExists</tt>. It doesn’t actually effect the original class, and probably will leave you confused because of the two very different classes with the same name. In Ruby when you define a class that already exists, you are extending the class in-place.</p>
<p>You can change Python classes in-place, but there’s no special syntax for it, so people either think you can’t do it, or don’t realize that you are doing the same thing as in Ruby but without the syntactic help. I guess this will be easier with <a class="reference external" href="http://www.python.org/dev/peps/pep-3129/">class decorators</a>, but some time ago I also wrote a recipe using normal decorators that looks like this:</p>
<pre class="literal-block">
@magic_set(SomeClassThatAlreadyExists)
def some_method(self, blahblahblah):
stuff
</pre>
<p>The only thing that is even slightly magic about the setting is that I look at the first argument of the function to determine if you are adding an instance, class, or static method to an object, and let you add it to classes or instances. It’s really not that magic, even if it is called <a class="reference external" href="http://svn.colorstudy.com/home/ianb/recipes/magicset.py">magicset</a>.</p>
<p>I think with class decorators you could do this:</p>
<pre class="literal-block">
@extend(SomeClassThatAlreadyExists)
class SomeClassThatAlreadyExists:
def some_method(self, blahblahblah):
stuff
</pre>
<p>Implemented like this:</p>
<pre class="literal-block">
def extend(class_to_extend):
def decorator(extending_class):
class_to_extend.__dict__.update(extending_class.__dict__)
return class_to_extend
return decorator
</pre>
Defaults & Inheritance2007-08-10T17:45:00-05:002007-08-10T17:45:00-05:00Ian Bickingtag:ianbicking.org,2007-08-10:/blog/2007/08/defaults-inheritance.html<p>I thought I’d note a way I try to make classes reasonably customizable without creating lots of classes, but letting other people create classes if they want.</p>
<p>Here’s a common technique; I’m going to use a class from <a class="reference external" href="http://pythonpaste.org/wsgiproxy/">WSGIProxy</a> as an example, because that’s where I …</p><p>I thought I’d note a way I try to make classes reasonably customizable without creating lots of classes, but letting other people create classes if they want.</p>
<p>Here’s a common technique; I’m going to use a class from <a class="reference external" href="http://pythonpaste.org/wsgiproxy/">WSGIProxy</a> as an example, because that’s where I was about to use this technique when I thought it might make an okay post.</p>
<p>In this example there’s a <span class="caps">WSGI</span> application that forwards requests to another <span class="caps">HTTP</span> server. There’s different ways to forward requests, depending on what kind of data you want to give the remote server about the original request. One example is Zope’s VirtualHostMonster, which takes requests like <tt class="docutils literal">/VirtualHostBase/http/example.org:80/rootdir/VirtualHostBase/path</tt> — the idea is that the server can then realize that the original request was for <tt class="docutils literal"><span class="pre">http://example.org/path</span></tt> (and should ignore any Host headers), and that Zope is supposed to serve that from the internal path <tt class="docutils literal">/rootdir/path</tt>.</p>
<p>There’s a problem with this particular pattern, because there’s no way to mount, say, <tt class="docutils literal">/blog</tt> onto some Zope <tt class="docutils literal"><span class="pre">/sitename/blog-application</span></tt> path, because there’s no concept like in <span class="caps">WSGI</span> or <span class="caps">CGI</span> of SCRIPT_NAME — the base <em>path</em> of the request. It only handles the base <em>host</em>. So I didn’t just want to settle on that.</p>
<p>I’m kind of inclined to prefer headers, like <tt class="docutils literal"><span class="pre">X-Script-Name:</span> /blog</tt>, <tt class="docutils literal"><span class="pre">X-Forwarded-Server:</span> example.org</tt>, etc. But I want to support both forms.</p>
<p>The common way to do this is:</p>
<pre class="literal-block">
class WSGIProxyApp(object):
def __init__(self, host): ...
def __call__(self, environ, start_response):
# actual application interface...
# Constructs the base request:
request = self.construct_request(environ)
# Uses one of these conventions:
self.update_headers(environ, request)
... do stuff with request ...
def update_headers(self, orig_environ, request):
raise NotImplementedError
class VirtualHostMonsterApp(WSGIProxyApp):
def update_headers(self, orig_environ, request):
request.environ['SCRIPT_NAME'] = (
'/VirtualHostRoot/%(wsgi.scheme)s/%(HTTP_HOST)s/VirtualHostRoot/'
% orig_environ)
class HeaderSetterApp(WSGIProxyApp):
def update_headers(self, orig_environ, request):
request.environ['HTTP_X_SCRIPT_NAME'] = orig_environ['SCRIPT_NAME']
# and so on...
</pre>
<p>Then you use one of the subclasses depending on your needs. Personally I think this really sucks. For one thing, you may have to determine which class to use based on some configuration parameter, which can get awkward. And you might want to subclass the class to change the functionality some yourself, but you have to subclass <em>both</em> of them. There’s patterns to handle this, with policies and factories and other crap; but it’s <em>not a hard problem</em>, and those patterns are hard solutions to a problem that <em>shouldn’t</em> be hard.</p>
<p>Also, it’s harder to inform people about the options available to them, and somewhat harder to use these classes. So I tend to do something like:</p>
<pre class="literal-block">
class WSGIProxyApp(object):
default_forwarding_style = 'headers'
def __init__(self, host, forwarding_style=None):
...
if forwarding_style is None:
forwarding_style = self.default_forwarding_style
self.forwarding_style = forwarding_style
def __call__(self, environ, start_response):
...
method = self.forwarding_style
if isinstance(method, str):
method = getattr(self, 'forward_'+self.forwarding_style)
method(environ, request)
...
def forward_headers(self, orig_environ, request): ...
def forward_virtual_host_monster(self, orig_environ, request): ...
</pre>
<p>This way it’s just a simple parameter to change the style. You can pass in your own function, or use one of the named methods already available. The <tt class="docutils literal">default_forwarding_style</tt> class variable lets you change the default in subclasses. If the default was in the function signature it would be much more awkard to change it, because you’d have to override the method and its signature with just that one change, then delegate back to the superclass method.</p>
Atom Publishing Protocol: Atompub2007-08-12T13:50:00-05:002007-08-12T13:50:00-05:00Ian Bickingtag:ianbicking.org,2007-08-12:/blog/2007/08/atom-publishing-protocol-atompub.html<p>Doing stuff with the Atom Publishing Protocol, I’ve noticed that it goes by two (shortened) names: <span class="caps">APP</span> and Atompub. I’d become used to calling it <span class="caps">APP</span>, but I’ve decided to make a conscious effort to call it Atompub from now on, and I encourage you all to …</p><p>Doing stuff with the Atom Publishing Protocol, I’ve noticed that it goes by two (shortened) names: <span class="caps">APP</span> and Atompub. I’d become used to calling it <span class="caps">APP</span>, but I’ve decided to make a conscious effort to call it Atompub from now on, and I encourage you all to do the same. You cannot usefully search for “<span class="caps">APP</span>”, and it’s pronunciation is ambiguous. Atompub is a much better name.</p>
<p>And as long as we’re <a class="reference external" href="https://ianbicking.org/2007/08/02/pronouncing-django/">talking about names</a>, I’ll note that the <a class="reference external" href="http://mail.python.org/pipermail/catalog-sig/2007-July/001460.html">Cheese Shop is now called PyPI again</a>. I <em>think</em> we are supposed to pronounce it <em>pih-pee</em>, distinct from <a class="reference external" href="http://codespeak.net/pypy/">PyPy</a> which is pie-pie. (Blast, PyPI is down; the Zope guys have been making a static stripped-down mirror for use with Setuptools, <a class="reference external" href="http://download.zope.org/ppix/">over here</a>)</p>
Of Microformats and the Semantic Web2007-08-14T11:52:00-05:002007-08-14T11:52:00-05:00Ian Bickingtag:ianbicking.org,2007-08-14:/blog/2007/08/of-microformats-and-the-semantic-web.html<p>I was talking a little with <a class="reference external" href="http://eikeon.com/">Daniel Krech</a> (author of <a class="reference external" href="http://rdflib.net/">rdflib</a>) about Semantic Web stuff and <a class="reference external" href="http://microformats.org/">microformats</a> and what they all mean. And he was saying that microformats were nice, because you could <em>do</em> something with them, but it would be nice to see that generalized.</p>
<p>By “generalized” I think …</p><p>I was talking a little with <a class="reference external" href="http://eikeon.com/">Daniel Krech</a> (author of <a class="reference external" href="http://rdflib.net/">rdflib</a>) about Semantic Web stuff and <a class="reference external" href="http://microformats.org/">microformats</a> and what they all mean. And he was saying that microformats were nice, because you could <em>do</em> something with them, but it would be nice to see that generalized.</p>
<p>By “generalized” I think he meant a general way of expressing arbitrary relationships. As an example, in <a class="reference external" href="http://microformats.org/wiki/hcard">hCard</a> you can do:</p>
<pre class="code html literal-block">
<span class="p"><</span><span class="nt">span</span> <span class="na">class</span><span class="o">=</span><span class="s">"tel"</span><span class="p">></span>
<span class="p"><</span><span class="nt">span</span> <span class="na">class</span><span class="o">=</span><span class="s">"type"</span><span class="p">></span>home<span class="p"></</span><span class="nt">span</span><span class="p">></span>:
<span class="p"><</span><span class="nt">span</span> <span class="na">class</span><span class="o">=</span><span class="s">"value"</span><span class="p">></span>773-555-3821<span class="p"></</span><span class="nt">span</span><span class="p">></span>
<span class="p"></</span><span class="nt">span</span><span class="p">></span>
</pre>
<p>The hCard specification (itself leaning heavily on vCard) defines <tt class="docutils literal">tel</tt>, <tt class="docutils literal">type</tt>, and there’s a general pattern of what <tt class="docutils literal">value</tt> means. But if you want to describe some new kind of structure, there’s no way to do that really; there’s no marital status format, for instance (which would be useful for a singles search engine, as an example).</p>
<p>So I started thinking: can you really generalize it? And I started to think about <a class="reference external" href="http://bitworking.org/news/193/Do-we-need-WADL">Joe Gregorio’s attack of <span class="caps">WADL</span></a>:</p>
<blockquote>
<p>Here is the very <a class="reference external" href="http://bitworking.org/projects/pastebin/main.cgi/bin/7">first example in the <span class="caps">WADL</span> specification</a>.</p>
<p>That <span class="caps">WADL</span> file is a description of a search interface. But <a class="reference external" href="http://bitworking.org/projects/pastebin/main.cgi/bin/8">here is how you should really do it</a>. That’s an <a class="reference external" href="http://www.opensearch.org/Home">OpenSearch</a> document, that also describes a search interface.</p>
<p><strong>Q</strong>: What’s the difference?</p>
<p><strong>A</strong>: A mime-type.</p>
<p><strong>Q</strong>: That doesn’t seem like much, does it make a difference?</p>
<p><strong>A</strong>: Yes, it makes a big difference. When you get an OpenSearch document there is a whole data model and a set of interactions you know are possible because you read the OpenSearch specification. By reading that spec you know how to construct search queries. When I get a <span class="caps">WADL</span> document it might describe anything, from how to construct a search, to the <a class="reference external" href="http://bitworking.org/projects/atom/"><span class="caps">APP</span></a>, to <a class="reference external" href="http://bitworking.org/news/JEP"><span class="caps">JEP</span></a>, to <span class="caps">XML</span>-<span class="caps">RPC</span>.</p>
<p>…</p>
<p>So when I say the difference is a ‘mime-type’, what I mean is that there is an entire spec somewhere which describes what that document means, and that meaning may include hypertext functionality, ala (X)<span class="caps">HTML</span>, XForms, and OpenSearch.</p>
</blockquote>
<p>This made me think of shared understanding more than explicit descriptions. OpenSearch, <span class="caps">APP</span>, and Atom are very well described, but I think that’s only half of it: they are useful when they describe something that many people already understand.</p>
<p>Digressing slightly, one “semantic markup” ideal that still bugs me is <tt class="docutils literal"><strong></tt> and <tt class="docutils literal"><em></tt> vs. <tt class="docutils literal"><b></tt> and <tt class="docutils literal"><i></tt>. When I compose text I choose to make some words bold and some italic. I have no idea what “strong” and “emphasis” are even supposed to mean. When I’m composing text, I don’t actually know <em>why</em> I choose one or the other. If I sat down and thought about it I’m sure I could come up with a set of rules that describe when bold is appropriate and when italic is appropriate. But that is reflecting on my choice, it is not describing my choice. There is no intermediate semantic meaning between what I am saying and bold and italic. I <em>think</em> in bold and italic. Readers in turn find meaning in the text itself; they do not parse my writing into semantic markup in their brain.</p>
<p>I think there’s some connection between this and the shared understanding that microformats represents, and a more generalized <span class="caps">RDF</span> model does not represent. I know what hCard means; not just in an intellectual way, but I can imagine a dozen functional uses of it without hardly trying, and of course I am entirely clear on what contact information <em>means</em>. Moreover, I know what it means without actually figuring out what it means; if you asked me to articulate what contact information means I’d have to think a little, and I’m sure many people would come up with bad answers or be stumped. And yet they all actually understand what it means.</p>
<p>Bringing this back to Joe’s post, if I write something that produces or consumes Atom, Atompub, or OpenSearch, I understand the <em>why</em> of my code. With both <span class="caps">WADL</span> and <span class="caps">RDF</span> my code is divorced of the why. This isn’t about my personal understanding either; explaining it to <em>me</em> doesn’t serve any purpose, because with any exchange format it has to make sense to many many people to be useful. Even an education campaign won’t fix this: education by description is far inferior to education by doing, and there’s no “doing” to <span class="caps">WADL</span> and <span class="caps">RDF</span> right now.</p>
<p>That said, what is sufficiently obvious in the future may not be obvious now. Maybe we’ll all get smarter. Maybe someone will pioneer this stuff in a way that is really useful (Facebook?), and grow the public’s intuition about describing relationships in an abstract way. But <em>until then</em> I think microformats are going about this the right way, describing those things that are most easily describable.</p>
Reflection and Description Of Meaning2007-08-14T14:18:00-05:002007-08-14T14:18:00-05:00Ian Bickingtag:ianbicking.org,2007-08-14:/blog/2007/08/reflection-and-description-of-meaning.html<p>After writing my <a class="reference external" href="https://ianbicking.org/2007/08/14/of-microformats-and-the-semantic-web/">last post</a> I thought I might follow up with a bit of cognitive speculation. Since the <a class="reference external" href="https://ianbicking.org/2007/08/14/of-microformats-and-the-semantic-web/#comment-148">first comment</a> was exactly about the issue I was thinking about writing on, I might as well follow up quickly.</p>
<p>Jeff Snell replied:</p>
<blockquote>
You parse semantic markup in rich text all …</blockquote><p>After writing my <a class="reference external" href="https://ianbicking.org/2007/08/14/of-microformats-and-the-semantic-web/">last post</a> I thought I might follow up with a bit of cognitive speculation. Since the <a class="reference external" href="https://ianbicking.org/2007/08/14/of-microformats-and-the-semantic-web/#comment-148">first comment</a> was exactly about the issue I was thinking about writing on, I might as well follow up quickly.</p>
<p>Jeff Snell replied:</p>
<blockquote>
You parse semantic markup in rich text all the time. When formatting changes, you apply a reason. <span class="caps">RFC</span>’s don’t capitalize <span class="caps">MUST</span> and <span class="caps">SHOULD</span> because the author is thinking in upper-case versus lower-case. They’re putting a strong emphasis on those words. As a reader, you take special notice of those words being formatted that way and immediately recognize that they contain a special importance. So I think that readers <strong>do parse writing into semantic markup inside their brains.</strong></blockquote>
<p>Emphasis not added. Wait, bold isn’t emphasis, it’s strong! So sorry, <span class="caps">STRONG</span> not added.</p>
<p>I think the reasoning here is flawed, in that it supposes that <em>reflection</em> on how we think is an accurate way of <em>describing</em> how we think.</p>
<p>A few years ago I got interested in cognition for a while and particularly some of the new theories on consciousness. One of the parts that really stuck with me was the difference in how we think about thinking, and how thinking really works (as revealed with timing experiments). That is, our conscious thought (the thinking-about-thinking) happened after the actual thought; we make up reasons for our actions when we’re challenged, but if we aren’t challenged to explain our actions there’s no consciousness at all (of course, you can challenge yourself to explain your reasoning — but you usually won’t). And then we revise history so that our reasoning precedes our decision, but that’s not always very accurate. This gets around the infinite-loop problem, where either there’s always another level of meta-consciousness reasoning about the lower level of consciousness, or there’s a potentially infinite sequence of <em>whys</em> that have to be answered for every decision. And of course sometimes we really <em>do</em> make rational decisions and there are several levels of <em>why</em> answered before we commit. But this is not the most common case, and there’s always a limit to how much reflection we can do. There are always decisions made without conscious consideration — if only to free ourselves to focus on the important decisions.</p>
<p>And so as both a reader and a writer, I <em>think</em> in terms of italic and bold. As a reader and a writer there is of course translation from one form to another. There’s some idea inside of me that I want to get out in my writing, there’s some idea outside of me that I want to understand as a reader. But just because I <em>can</em> describe some intermediate form of semantic meaning, it doesn’t mean that that meaning is actually there. Instead I <em>invent</em> things like “strong” and “emphasis” when I’m asked to decide why I chose a particular text style. But the real decision is intuitive — I map directly from my ideas to words on the page, or vice versa for reading.</p>
<p>Obviously this is not true for all markup. But my intuition as both a reader and a writer about bold and italic is strong enough that I feel confident there’s no intermediary representation. This is not unlike the fact I don’t consider the phonetics of most words (though admittedly I did when trying to spell “phonetics”); common words are opaque tokens that I read in their entirety without consideration of their component letters. And a good reader reads text words without consideration of their vocal equivalents (though as a writer I read my own writing out loud… is that typical? I’m guessing it is). A good reader can of course vocalize if asked, but that doesn’t mean the vocalization is an accurate representation of their original reading experience.</p>
<p>Though it’s kind of an aside, I think the use of <span class="caps">MUST</span> and <span class="caps">SHOULD</span> in RFCs fits with this theory. By using all caps they emphasize the word over the prose, they make the reader see the words as tokens unique from “must” and “should”, with special meanings that are related to but also much more strict than their usual English meaning. The caps are a way of disturbing our natural way of determining meaning because they need a more exact language.</p>
DictMixin2007-08-17T00:02:00-05:002007-08-17T00:02:00-05:00Ian Bickingtag:ianbicking.org,2007-08-17:/blog/2007/08/dictmixin.html<p>Quite some time ago I gave a little presentation on <a class="reference external" href="http://python.org/doc/current/lib/module-UserDict.html">DictMixin</a> at <a class="reference external" href="http://chipy.org">ChiPy</a>. If you haven’t used DictMixin before, it’s a class that implements all the derivative methods of dictionaries so you only have to implement the most minimal set: <tt class="docutils literal">__getitem__</tt>, <tt class="docutils literal">__setitem__</tt>, <tt class="docutils literal">__delitem__</tt>, and <tt class="docutils literal">keys</tt>. It’s a …</p><p>Quite some time ago I gave a little presentation on <a class="reference external" href="http://python.org/doc/current/lib/module-UserDict.html">DictMixin</a> at <a class="reference external" href="http://chipy.org">ChiPy</a>. If you haven’t used DictMixin before, it’s a class that implements all the derivative methods of dictionaries so you only have to implement the most minimal set: <tt class="docutils literal">__getitem__</tt>, <tt class="docutils literal">__setitem__</tt>, <tt class="docutils literal">__delitem__</tt>, and <tt class="docutils literal">keys</tt>. It’s a lot better than subclassing <tt class="docutils literal">dict</tt> directly, as you have to implement a lot more, and <tt class="docutils literal">dict</tt> implies a specific kind of storage. With DictMixin you can get the information from anywhere.</p>
<p>I thought of a couple examples, and wrote some <a class="reference external" href="http://python.org/doc/current/lib/module-doctest.html">doctests</a> for them; I thought satisfying the doctests would itself be the presentation. I’m not sure how it worked; it was a fairly experienced crowd, but the switch from code to test can be disorienting.</p>
<p>One of the examples I used was a filesystem access layer. Representing a filesystem as a dictionary is nothing new, but the simplicity of the representation worked well. Here’s how it works:</p>
<ul class="simple">
<li>An <tt class="docutils literal">FSDict</tt> represents one directory.</li>
<li>The keys are the filenames in the directory.</li>
<li>The values are the contents of the files (strings).</li>
<li>When there is a subdirectory, it is another FSDict instance.</li>
<li>When you assign a dictionary-like object to a key, it creates a FSDict from that object.</li>
</ul>
<p>Dictionaries have lots of methods, like <tt class="docutils literal">items()</tt>, <tt class="docutils literal">update()</tt>, etc. But using DictMixin you just implement the four methods. First, the setup:</p>
<pre class="literal-block">
class FSDict(DictMixin):
def __init__(self, path):
self.path = path
</pre>
<p><em>Creation</em> of a dictionary is not part of the dictionary interface. This seems a little strange at first, but the <tt class="docutils literal">dict</tt> <em>class</em> interface isn’t the same as the dictionary <em>instance</em> interface. So <tt class="docutils literal">FSDict.__init__</tt> doesn’t bear any particular relation to <tt class="docutils literal">dict.__init__</tt>.</p>
<p>Now the other methods… in each case, strings and dictionaries (files and directories) are treated differently.</p>
<pre class="literal-block">
def __getitem__(self, item):
fn = os.path.join(self.path, item)
if not os.path.exists(fn):
raise KeyError("File %s does not exist" % fn)
if os.path.isdir(fn):
return self.__class__(fn)
f = open(fn, 'rb')
c = f.read()
f.close()
return c
</pre>
<p>Note the use of <tt class="docutils literal">self.__class__(fn)</tt> instead of <tt class="docutils literal">FSDict(fn)</tt>. This makes the class subclassable if you retain the <tt class="docutils literal">FSDict.__init__</tt> signature. This way subclasses will create new instances using the subclass. Note also that <tt class="docutils literal">KeyError</tt> is part of the dictionary interface (an important part!), so we can’t raise <tt class="docutils literal">IOError</tt>.</p>
<p>Now, assignment…</p>
<pre class="literal-block">
def __setitem__(self, item, value):
if item in self:
del self[item]
fn = os.path.join(self.path, item)
if isinstance(value, str):
f = open(fn, 'wb')
f.write(value)
f.close()
else:
# Assume it is a dictionary
os.mkdir(fn)
f = self[item]
f.update(value)
</pre>
<p>Note that with subdirectories (represented as nested dictionaries) we let <tt class="docutils literal">DictMixin.update</tt> do all the hard work, and just create an empty directory to be filled.</p>
<p>Deletion…</p>
<pre class="literal-block">
def __delitem__(self, item):
fn = os.path.join(self.path, item)
if not os.path.exists(fn):
raise KeyError("File %s does not exist" % fn)
if os.path.isdir(fn):
## one way...
self[item].clear()
os.rmdir(fn)
## another way...
#shutil.rmtree(fn)
else:
os.unlink(fn)
</pre>
<p>Enumeration…</p>
<pre class="literal-block">
def keys(self):
return os.listdir(self.path)
</pre>
<p>So, to recursively copy <tt class="docutils literal">'/foo/bar'</tt> to <tt class="docutils literal">'/dest/path/bar'</tt> you do:</p>
<pre class="literal-block">
FSDict('/dest/path')['bar'] = FSDict('/foo')['bar']
</pre>
<p>It doesn’t really matter if <tt class="docutils literal">'/foo/bar'</tt> is a directory or file. There’s a number of other clever things that come out of this. I think it’s an example of the power of <a class="reference external" href="http://en.wikipedia.org/wiki/Closure_%28mathematics%29">a closed set</a> — dictionaries are expressable from these four operations, and all the other methods can be derived from there. If you find this interesting, you might want to read the <a class="reference external" href="http://svn.python.org/view/python/trunk/Lib/UserDict.py?view=markup">source for DictMixin</a>; it’s only about 95 lines.</p>
<p>My article <a class="reference external" href="https://ianbicking.org/templating-via-dict-wrappers.html">templating via dict wrappers</a> has some other similar dict tricks.</p>
WebOb2007-08-18T19:37:00-05:002007-08-18T19:37:00-05:00Ian Bickingtag:ianbicking.org,2007-08-18:/blog/2007/08/webob.html<p>I’ve have it in my head to extract/rewrite parts of <a class="reference external" href="http://pythonpaste.org">Paste</a> lately. <a class="reference external" href="https://ianbicking.org/2007/08/06/tempita/">Tempita</a> was one example.</p>
<p>The request and response functions in Paste grew very organically. I wasn’t trying to create a framework, so I studiously avoided anything that might look like a request or response object …</p><p>I’ve have it in my head to extract/rewrite parts of <a class="reference external" href="http://pythonpaste.org">Paste</a> lately. <a class="reference external" href="https://ianbicking.org/2007/08/06/tempita/">Tempita</a> was one example.</p>
<p>The request and response functions in Paste grew very organically. I wasn’t trying to create a framework, so I studiously avoided anything that might look like a request or response object. I felt that would be stepping on toes or something. Eventually, though, <a class="reference external" href="http://groovie.org/">Ben Bangert</a> really wanted a request object for <a class="reference external" href="http://pylonshq.com/">Pylons</a>, and it went in <a class="reference external" href="http://pythonpaste.org/module-paste.wsgiwrappers.html">paste.wsgiwrappers</a>. And at a certain point I decided that the class-based access was really just fine, and doing lots of <tt class="docutils literal">function(environ, <span class="pre">...)</span></tt> was no better than <tt class="docutils literal"><span class="pre">Request(environ).function(...)</span></tt>.</p>
<p>So I started <a class="reference external" href="http://pythonpaste.org/webob/">WebOb</a>. WebOb has Request, Response, and some exceptions, incorporating the functionality of Paste’s <tt class="docutils literal">paste.request</tt>, <tt class="docutils literal">paste.response</tt>, <tt class="docutils literal">paste.wsgilib</tt>, <tt class="docutils literal">paste.httpexceptions</tt>, and <tt class="docutils literal">paste.httpheaders</tt>. And some extra stuff.</p>
<p>I’ve included a comparison with <a class="reference external" href="http://pythonpaste.org/webob/differences.html">a few other framework request/response objects</a>. What this doesn’t note, though, is that WebOb has a much larger <a class="reference external" href="http://pythonpaste.org/webob/class-webob.Request.html">Request</a> and <a class="reference external" href="http://pythonpaste.org/webob/class-webob.Response.html">Response</a> objects. I’ve taken almost all the <a class="reference external" href="http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html"><span class="caps">HTTP</span> headers</a> and mapped them to parsed attributes. So <tt class="docutils literal">req.if_modified_since</tt> returns a <tt class="docutils literal">datetime</tt> object, and <tt class="docutils literal">req.if_none_match</tt> returns a somewhat set-like object, as a few examples. I created a lot of view-like objects for this, representing the canonical form of the information in several other forms (the <span class="caps">WSGI</span> request environment, and the status/headers/body of the response).</p>
<p>It’s fairly well tested and includes almost everything I think it should include, but I reserve the right to change the <span class="caps">API</span> any way I want until 1.0; this means if you have <em>any</em> opinion on the <span class="caps">API</span> I have nothing to stop me from taking your opinions into account.</p>
<p>Oh, and it has <a class="reference external" href="http://pythonpaste.org/webob/">docs</a>, really. They may not be the best docs, but they mention most everything and are automatically tested for accuracy. If you just want a sense of the feel, maybe the <a class="reference external" href="http://pythonpaste.org/webob/file-example.html">file-serving example</a> would be a good place to start (though really you’ll only read about the Response object there).</p>
The Shrinking Python Web Framework World2007-08-21T23:25:00-05:002007-08-21T23:25:00-05:00Ian Bickingtag:ianbicking.org,2007-08-21:/blog/2007/08/the-shrinking-python-web-framework-world.html<p>When I was writing the summary of <a class="reference external" href="http://pythonpaste.org/webob/differences.html">differences between WebOb and other request objects</a>, to remind myself of web frameworks I might have forgotten I went to the <a class="reference external" href="http://wiki.python.org/moin/WebFrameworks">WebFrameworks page</a> on the Python wiki.</p>
<p>Looking through that page I’m reminded how many framework options there have been. And I …</p><p>When I was writing the summary of <a class="reference external" href="http://pythonpaste.org/webob/differences.html">differences between WebOb and other request objects</a>, to remind myself of web frameworks I might have forgotten I went to the <a class="reference external" href="http://wiki.python.org/moin/WebFrameworks">WebFrameworks page</a> on the Python wiki.</p>
<p>Looking through that page I’m reminded how many framework options there have been. And I was further reminded of how few relevant options there are now. From all this, there have emerged just a few options: Django, Pylons, TurboGears, Zope. No offense to anyone left out of that list — I know there’s some other actively developed frameworks out there. But frankly they aren’t serious choices; they might be fine internal tools, or interesting experiments, but they are clearly on a different tier (and they all have questionable futures).</p>
<p>And now that <a class="reference external" href="http://groups.google.com/group/turbogears/browse_thread/thread/d1d2e416023e7033">TurboGears 2 will be based on Pylons</a> the list looks smaller still.</p>
<p>For a long, long time (longer than most of those frameworks have existed) people have complained about the proliferation of web frameworks in Python. Those of us involved in developing web frameworks in Python haven’t been able to respond all that well. Complaining doesn’t magically lead to solutions, and you can’t just will there to be a single Python web framework. You can work towards that, but that’s what we’ve been doing… mostly people don’t seem to notice. It’s just not an easy thing to work towards; the problem space for a web framework isn’t well defined, its end goal is far more vague than most people immediately realize, and it involves <em>consensus</em>, which makes everything much harder. We said the market would decide, which is kind of a cop out (the market decides through the decisions of developers) but that’s the best answer we had.</p>
<p>But after all this time, it seems clear that we are getting much closer to that goal. If you squint <em>really</em> hard, you can almost imagine we <em>are</em> there. The <em>total</em> list of frameworks only gets longer over time — that’s how open source works — but the list of <em>choices</em> has become quite compact.</p>
<p>How we get to the next level is a little less clear. We’ve gotten this way largely through attrition, but that’s not going to get us any further. I’ll at least assure people that we are discussing this stuff — it’s slow going, but everyone is interested. And if anyone actually wants to do some leg work to move this forward, a lot of the work is actually technical, not political, so don’t be afraid to jump in.</p>
Doctest for Ruby2007-08-23T11:10:00-05:002007-08-23T11:10:00-05:00Ian Bickingtag:ianbicking.org,2007-08-23:/blog/2007/08/doctest-for-ruby.html<p>Finally, <a class="reference external" href="http://clintonforbes.blogspot.com/2007/08/doctest-for-ruby-and-rails.html">someone wrote a version of doctest for Ruby</a>.</p>
<p>Recently I’ve been writing most of my tests using <a class="reference external" href="http://python.org/doc/current/lib/doctest-simple-testfile.html">stand-alone doctest files</a>. It’s a great way to do <span class="caps">TDD</span> — mostly because the cognitive load is so low. Also, I write my examples but don’t write my output, then …</p><p>Finally, <a class="reference external" href="http://clintonforbes.blogspot.com/2007/08/doctest-for-ruby-and-rails.html">someone wrote a version of doctest for Ruby</a>.</p>
<p>Recently I’ve been writing most of my tests using <a class="reference external" href="http://python.org/doc/current/lib/doctest-simple-testfile.html">stand-alone doctest files</a>. It’s a great way to do <span class="caps">TDD</span> — mostly because the cognitive load is so low. Also, I write my examples but don’t write my output, then copy the output after visually confirming it is correct. So the basic pattern is:</p>
<ul class="simple">
<li>Figure out <em>what</em> I want to do</li>
<li>Figure out <em>how</em> I want to test it</li>
<li>Automate my conditions</li>
<li>Manually inspect whether the output is correct (i.e., implement and debug)</li>
<li>Copy the output so that in the future the manual process is automated (<a class="reference external" href="http://www.cis.upenn.edu/~edloper/projects/doctestmode/">doctest-mode</a> for Emacs makes this particularly easy)</li>
</ul>
<p>The result is a really good balance of manual and automated testing, I think giving you the benefit of both processes — the ease of manual testing, and the robustness of automated testing.</p>
<p>Another good thing about doctest is it doesn’t let you hide any boilerplate and setup. If it’s easy to use doctest, it’s probably easy to use the library.</p>
<p>There’s nothing Python-specific about doctest (e.g., <a class="reference external" href="http://svn.colorstudy.com/doctestjs/trunk/docs/index.html">doctestjs</a>), so it’s good to see it moving to other languages. Even if the language doesn’t have a <a class="reference external" href="http://en.wikipedia.org/wiki/REPL"><span class="caps">REPL</span></a>, <span class="caps">IMHO</span> it’s worth inventing it just for this.</p>