Ian Bicking: the old part of his blog

py.std

So, I was chatting about "helpers" with Ben Bangert on #webware, and thinking about adding stuff to __builtins__, like html_quote and the like. Hey, don't you judge me! Anyway, I think I like string.encode('html') more.

But anyway, I was thinking about how to make this stuff somewhat easier. And I was thinking about a discussion on Py-Dev about adding a dedent method to strings. The rationale was that inline strings that match up with surrounding whitespace need dedenting (ala textwrap.dedent). Why not just use textwrap.dedent? My opinion is that the value is too small for the overhead. And this kind of overhead happens often.

I also think the overhead shouldn't exist. Managing import statements is obnoxious. When they represent real dependencies or namespace operations that's fine -- I'd even say it is good. I like namespaces, really. But managing the imports is annoying.

In this case, I might be 100 lines into a module, and I'm writing a string. If I really want to use textwrap.dedent, I have to go up to the top of the file, add import textwrap and return to my previous location. And, since I know the standard library pretty well now and tend to use it, it is not uncommon for modules to start with 15 lines of imports. That just sucks!

Then I remember about the py-lib which has py.std. Using it I wouldn't have to import textwrap, I'd just use py.std.textwrap.dedent immediately. Does that string seem too long? I don't think so; it's not the length that is bothersome to me (it helps backtracking, and that's good), it's the disturbance of my workflow to add import statements, when I feel like I'm just feeding the compiler. There's good reasons to be explicit, but this is just as explicit as an import (maybe moreso). Unlike an import it is fully localized to the code fragment that uses the module.

Now putting std in __builtins__ is something I could stand behind.

Created 08 Dec '05

Comments:

"I have to go up to the top of the file"

No, you don't. If you want a local import, use a local import. Import != include; it's perfectly okay to import things only when and where you need them.

# Fredrik

Yeah, but... There is a certain feeling of sequentialness involved in importing, #include-ing, and <!--#include-->ing everything atop the file, so that you can tell, at a glance, exactly which files have to be visible at parse/compile time to make the the code worky-worky.

# Chris

Adding stuff to __builtins__ is bad. BoboPOS, Principia, Zope 1 and 2 all did that - not very much, just a couple of things (maybe even just 1) - like get_transaction(). Not really knowing when or where that statement comes from makes it awfully painful to debug when you've got something breaking there.

A pattern that Zope 3 uses, and which I've started to adopt for my own projects, is a concept of API modules. The main one for Zope 3 is zapi, and it houses many common functions. And it doesn't define them - it pulls them in from other packages. The nice trick that Zope 3 does with zapi and the packages it pulls from is that it uses zope.interface. There's an interface defined for all of the modules an IZAPI sub-interfaces all of them. The zope.interface package then brings other niceties - it's easy to generate an __all__ for zapi so that no extra bits are accidentally included in the zapi namespace; and it ties in to Zope 3's API documentation tool like all other registered interfaces, so it's easy to look up to see the documentation on any of the items in the API. The API doesn't define any new objects - as far as I've found there's no zapi.foo.bar or deeper nesting. It's a flat namespace. Many of the functions are things that are commonly used to query the greater registries of the component architecture.

And lastly - zapi pulls things into it. I can't start pushing my own things in there (well, I probably could without much clever hacking, but it would defeat the purpose). It's nice knowing that its behavior is the same everywhere.

Anyways, I've started implementing something like this in a toolkit that I've been building on top of Zope. I noticed there were some very common functions that I was using - a lot - so I started following the same model. It's a nice solution, I think - I only put items into the API that are proving to be very common. They're well documented. They're all kept in the packages that make sense to them and then collected together by myapi, and now I have easy access to myapi.flash(...) and myapi.getCacheForObject() (Zope 3's getCacheForObject didn't do enough for me, so I wrote my own). Something has to earn its spot in there. So far, it's working out quite nice

But other than that - I honestly don't mind managing the imports. VIM makes it so easy to jump to the top of the document and back to where I was editing; most of the time if I need to do a few imports I just split the frame so I can jump up to the imports in one frame while still seeing my code on the bottom (so I remember what I'm importing). I've also started using its "search for this word exactly" feature (# and * keys) to quickly scan through my imports when the list looks really big and I want to quickly check if I'm still using all the named items. In other editors too, if I need access to my import list, it's no big thing to just split the frame and always keep the imports on top.

Or I just program away and then run the code and go "d'oh!" for every name error that comes from me forgetting to import because I didn't want to interrupt my workflow. But that doesn't happen much.

Anyways - I think that import is one of the best things about Python. It's the first thing that I miss when I have to use just about any other language out there.

# Jeff Shell

I propose this as something that is entirely complimentary to importing, not in conflict with it. If you understand what std means, then it's also just as explicit as importing. It's really just another import mechanism which is more convenient in certain (fairly common) circumstances. One could argue (as the paragraph on py.std does) that this is actually more explicit because it never allows module-relative imports. I think it's only really useful if it is idiomatic, or even better built in (without hacking __builtins__), but it's a very simple idiom. The implementation in the py lib is actually just this:

import sys

class Std(object):
    def __init__(self):
        self.__dict__ = sys.modules

    def __getattr__(self, name):
        try:
            m = __import__(name)
        except ImportError:
            raise AttributeError("py.std: could not import %s" % name)
        return m

std = Std()
# Ian Bicking

+1

Aside: one of the things I hate most (there aren't many) coming to Python from Java is the ability to import a package and then be able to navigate to any subpackage using dot notation. Example:
>>> import PythonCard
>>> PythonCard.components.button
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
AttributeError: 'module' object has no attribute 'components'
I have a dream of being able to do this (does anyone else hate the from keyword?):
>>> import my.deeply.nested.package.function as func  # results in error
Which usually starts out as:
>>> import my.deeply.nested.package as package
>>> package.function(...)
And then mutates to this when we realize we're only using a single function in that package:
>>> from my.deeply.nested.package import function as func

And that requires me to rearrange the entire line, which is annoying. And as if that weren't enough trouble with Python's import mechanism, why in the heck couldn't they make the __import__ function "just work" like it's supposed to (i.e. col = __import__("sqlobject.col") should have the same effect as import sqlobject.col as col)?

Maybe the "std" object could be enhanced to fix that problem too. The only downside I can see here might be performance (try exploring std in Pyshell--it's a bit slow on my PIII 733 (then again, Eclipse is nearly useless on this machine)). On the positive side, it's much easier to explore the standard library this way, which is another thing I miss from Java. Eclipse allowed me to explore Java api's through the auto-complete feature, providing JavaDoc tooltips when available--very productive. But now I'm asking for a good IDE which is an entirely different subject. I guess I should check out PyDev again sometime.

Wow, I think you hit a sore spot there.

# Daniel