You write "don't create a situation where a dumb manager invalidates your entire implementation." This to me seems a very low priority. Macros did what mattered, which was to win in the market and get us bought by Yahoo. What the generic corporate programmers who inherited the code after we sold it to them did with it is of minuscule importance by comparison.
Your advice is equivalent to: write novels that Hollywood writers won't need to change to turn into screenplays.
If some Hollywood script doctor tacks a happy ending onto your novel, does that "invalidate" it?
I'm glad you guys are at least talking about macros though. Ten years ago the mainstream programming world wasn't even aware of the concept.
Extending your metaphor, your advice is: write good novels. But that's not a viable plan for working writers, anymore than it is for working programmers. Working writers write things like magazine articles and manuals and brochures, and Steven King doesn't get hired to write even really important brochures. So, you've got a plan to become a Steven King; but we all know that's not a practical goal for the industry.
Yahoo Stores showed that a small group of really good programmers, using powerful tools, could create something good, fast, and innovative all at once. Small groups of really good programmers should pay attention. But that applies to very, very few people -- including very few really good programmers! And so it isn't a very good showcase for Lisp either.
If we talk about creating powerful tools that allow us to extend the state of the art -- the "us" that includes the entire open source community, or the entire IT industry -- then we need to think a little more democratically about these things. We should have a system where programmers with a diversity of skills can all participate usefully in a project, even if at different levels of productivity.
I've made extensive use of Scheme macros on a project with many novice coders. The macros define a domain-specific language, and have been largely responsible for the success of the project to date (i.e., asking our coders to use a general purpose language for domain-specific problems would almost certainly have been a disaster).
We seriously evaluated using Python for this project, and reached two conclusions:
- Python was a far easier language to learn than Scheme.
- There was no good way to represent our problem domain in Python. Users would be forced to add gratuitous levels of class structure and indenting, and they're have to do it right every time. Unfortunately, some of the defining characteristics of novice programmers are that they don't reliably follow style-guides, they don't understand complex coding conventions, and they don't deal well with extra levels of indentation.
So we wrote ten pages of fairly complex macro code which massively reduced the complexity of hundreds of pages of domain code. Management knows how hard it will be to hire someone to maintain the macro code, and they still consider it an acceptable tradeoff--we gain so much elsewhere that a small amount of guru-level code is OK.
I've contributed to largish Python projects (including Zope) and these problems do occur elsewhere. The Zope security system, in particular, requires programmers to add security "declarations", which are actually fiddly little snippets of executable code, each of which needs to be in almost exactly the correct place, and few of which are meaningfully error-checked. If Python had a decent macro system--or even C#-style class and method adjectives--Zope's security system could be made completely declarative, which I think would be a win. Pythonistas may disagree, of course.
I'm not arguing that Python should include macros, not even simple and elegant ones (if such are possible). After all, language design is an art, and Guido needs to balance many competing factors to make a widely accessible language.
I am, however, arguing that on certain real-world projects, Python's lack of macros cripples it so badly that even as esoteric a language as Scheme can ultimately be more usable for novice programmers. Python is a general-purpose notation--perhaps the best I've ever seen--but a general-purpose notation can be significantly harder to apply than a good domain-specific notation.# EK
I think "cripple" is a little extreme. But I do agree that we need good metaprogramming constructs, and I'm really very interested in that, I'm just not that psyched about some particular techniques.
And there's a lot more opportunities to do metaprogramming now than there were a few years ago -- in part because of changes in the language (mostly Python 2.2), and in part because we're learning better techniques to use those features. There are a lot of avenues for exploration, most of which are probably a bad idea ;) And we have what I'm guessing is the same as method adjectives in Python, I believe at the bequest of Zope people, though it was too late for them to use them -- functions can have arbitrary attributes with arbitrary values (since Python 2.0 or 2.1?) And of course classes have always had this ability. Or maybe you are thinking of something else?