Yeah. What is up with me and Lisp? Maybe you are a Lisp programmer and you’re interested in why I choose to spend so much time in parentheses. Or maybe you’ve never programmed in Lisp and you’re interested in why someone might choose to.
Well, these days, the question of why a person might choose Lisp is somewhat difficult to definitively answer. Time was, Lisp could make the claim, along with cousins, like Smalltalk, to be among the most forward thinking of programming languages, but contemporary life is littered with languages just as clever, dynamic, ergonomic, or mathematically sound. So while the question is, in many ways, more difficult to answer than ever, it is also more salient than it has ever been1.
(Maybe it’s not the) Metaprogramming
It bears emphasis that Lisp isn’t a single language, but a family of dialects with widely varying semantics, and significantly varying syntaxes. This misperceived closeness among dialects is a somewhat unusual aspect of the Lisp family or the culture around the culture around Lisp1. For instance, Lua and Javascript are much more closely related than, say, Clojure and Common Lisp, yet we do not instantly group the former pair into a language family, say of, `table based dynamic programming languages`.
However, the one thing Lisp dialects tend to have in common is the ease with which one can metaprogram, and I think this is a major component of the Lisp experience and this at least seems to be related to their distinctive, fully parenthesized, s-expression based syntaxes. And I would like to advance that it is in the area of syntactic simplicity that Lisp’s true appeal lies, at least to this author.
So what, right? The observation that metaprogramming is the reason to use and/or like Lisp is hardly new. Sure. But what is true is that metaprogramming is now hardly the exclusive domain of Lisp. Contemporary programming languages all support aspects of metaprogramming in one form or another. In Ruby, for instance, blocks give you 70% of what you want out of metaprogramming for 10% of the price you pay in Lisp2, Python uses decorators to expose a limited set of metaprogrammatic techniques, C++ allows psychopaths3 to template metaprogram, language-level support for monads in Haskell constitutes a kind of metaprogramming in that it exposes an interface for the nature of binding in certain contexts4. And if these language features don’t strike you as rich enough to compete with Lisp’s `just mess with the code before the compiler gets it` approach, most languages have these sorts of facilities these days.
So what is it that you can’t get from other languages that you can get in Lisp?
What you get
At the outset, I want to say that I’ve had the luck to work in Common Lisp for the past two years and that, after an initial period of adjustment, I’ve come to see the language as extraordinarily well designed, if lacking, somewhat, in modern sensibilities. So if you choose to work in Common Lisp, you get a very pragmatic, extensively well thought out, extremely powerful base language to work with. As a matter of comparison, Common Lisp is more cleverly and extensibly object oriented than Ruby, with semantics carefully thought out so that most implementations can acheive significantly better performance. So if you have the expertise and the temperment, you can rely on a good Common Lisp implementation to get you where you need to to go.
But that isn’t really the point of this little essay. I’m not primarily interested in the best language for getting a project to completion. If that is your interest, lots of factors will influence your decision that don’t have anything to do with the fact that Lisp might feel like a really well broken in pair of jeans.
Ultimately, I think Lisp’s real strength is the unpretentiousness with which it exposes the underlying nature of the programming system. While the Lisp community might be accused of pretention from time to time, I think the language is decidedly straightforward: in most languages you go through the, in my opinion, silly exercise of denoting a program tree using some large set of syntactic conventions, and then the language runtime converts that into a syntax tree, which is then operated upon to produce executable instructions of some kind. In Lisp, you are able to pretend, at least5, that you are denoting that intermediate tree directly.
This is not profound. On the contrary, it is essentially pragmatic6, and it has the appeal of the essentially pragmatic.
One source of feelings of alienation, in my opinion, is to be expected or forced to act in ways which one knows are meaningless. It might be argued that your average algol-syntaxed language is asking you to do that which is alienating: denote what you know is simply a tree as something which requires a complex transformation to arrive there.
The benefits of this simplicity extend beyond vague feelings of empowerment. They enable, for instance, one to engage in code transformations without the conceptual overhead of source to source transformation. Of course, the runtime is performing a source-to-tree transformation, but it feels trivial, and your own code which operates on the parse tree can, most of the time, be written as though it is operating on the thing you typed, rather than the thing which is produced by the runtime. Once you encompass backquote, unquote, and unquote splice, denoting program transformations requires very little conceptual overhead7.
But the other benefit is the fact that your toolchain benefits in kind from the simplicity of the language. Tools like paredit or the excellent Dr Racket8 further close the gap between typing individual characters and denoting and manipulating, directly, the code tree.
The Anticlimax
So this isn’t exactly a resounding endorsement of the language, but that is exactly my point. I don’t think it is credible, in the contemporary programming language ecosystem, to claim that Lisp is significantly better as an existential matter, than a lot of languages out there. In many respects, languages like Haskell are much better considered intellectually, and a language like Javascript is much better positioned to actually deliver a product to someone.
But I do think that Lisp’s simplicity in the arena of “source to parse tree to executable code” gives it a distinctly pleasant, distinctly empowering “feel,” and that probably accounts for a significant number of its deep adherents.
And that is the deal with me and Lisp.
Footnotes
1: That is to say, the similarity is not itself a property of the languages, nor a property of the immediate Lisp community, who somewhat close to the languages in question, tend not to confuse them, but a property of the community of programmers `at large`.
2: Of course Smalltalk did blocks earlier and better.
3: No shortage of these, somehow.
4: A given monad can be thought of as specifying alternative semantics for the act of binding a value to a variable and handling the result of an expression evaluated in the context of that binding. A surprisingly large space of `mini-languages` can be reached by controlling exactly what these two operations constitute.
5: I say pretend because it turns out that your average Lisp reader is really doing a lot of work, and that the data structure you denote and its denotation are, as a consequence, not as close to one another as you might imagine. The key is that they are close enough that you can avoid thinking too hard about it most of the time.
6: Literally. The original intent was to denote Lisp programs using m-expressions, but it was an extra complication that never really caught on, since most programmers found it simpler to just use parenthesized notation.
7: Or so it seems. One shortcoming of Lisp, it might be argued, is that it makes thinking about program transformation seem easier than it actually is. Correct program transformations, in which the resulting code does what you expect it to do in all scenarios is a pretty hard problem. A hard problem which is not even generally conceived of in its entirety by those writing macros, as evidenced by the frequency with which one hears the false claim that `gensym` is sufficient to avoid unexpected behavior.
8: To be fair, Racket goes way beyond Lisp in general in this area.
Regarding the 70%/10%, what do you mean by “price”, and what did you measure to get those numbers?
The numbers appear by way of convenient idiom, but I can explain the observation behind the hand-waving. Ruby’s one big trick in the realm of metaprogramming is the block. From my personal experience as a dude writing a lot of metaprogramming code, that about 70-80% of the time the main thing one does while metaprogramming is control evaluation.
In most quotidian languages you can’t write your own if statement or for loop because the semantics make it prohibitive to indicate that the arguments to a user defined operator are only to be evaluated at the behest of the operator itself. That is, in most languages, function arguments are evaluated before the body of the function has anything to do with them.
In Lisp, one ropes in the entire universe of macro programming, special syntax for backquote, unquote and unquote splice, and compile time transformations, and therefore staged execution. In Lisps that want to “get it right” you get the relatively large complexity of macro hygiene, which involves representing and carrying around a lot of information about syntax. To get macros to behave, your macroexpander and macro writer need to know a bit about what the symbols mean, basically, at macroexpansion time.
It is complicated.
Ok, so in languages without this conceptual overhead you can still get a lot of work done in the realm of controlling the order and frequency of evaluation of forms if you have anonymous functions with lexical scope. If you’re programming in Javascript these days, you’ve seen this. Let’s just write a little
ifFun:function ifFun(cond, trueBranch, falseBranch){
if cond {
return trueBranch();
} else {
return falseBranch();
}
}
Which you use thusly:
var result = ifFun(true,
function(){ return "true";},
function(){return "false";});
This is a bit ugly. Smalltalk was smart enough to realize that the ugliness here isn’t fundamental to the approach, but just that the denotation for an anonymous function is too big and the language is a bit too into commas. Imagine if we could leave out commas in Javascript and if you could denote anonymous functions with no arguments with
[[]]. The above becomes and imagine further that the last expression in such an anonymous function is automatically “returned” to the caller:var result = ifFun(true
[["true"]]
[["false"]]);
This is how Smalltalk lets you cook up new special forms, basically. Blocks/anonymous functions have very terse denotations which encourages you to use them to control evaluation.
As I meanly intimated above, Ruby doesn’t even do as nice a job of this as Smalltalk. In Ruby a method can accept a single block argument, which is uncoincidentally placed after the traditional argument list inside
{}. Inside the body of the method you canyieldto execute the block as often as you’d like. This allows Rubyists to write a very large number of simple special forms that you can’t easily write, for instance, in C or even Python. What is more, they look just like special forms in curly brace languages! Super cool!So Ruby’s blocks expose a large amount of metaprogramming space without roping in the complexity of a serious macroexpansion system. It gets the low hanging fruit without bothering to grab the ladder for the higher stuff. There is a cleverness to that.
WordPress does not allow “pre” tags by default, and so the code in the comment above is not formatted well, but I think the point is still clear.
Also, I don’t mean to imply that the above is the only metaprogrammatic trick up Ruby’s sleeve. Only that it is the most visible and most commonly used one.
You write : « In Lisp, you are able to pretend, at least, that you are denoting that intermediate tree directly. » I am a newby in Lisp, and it’s the main reason why I like Lisp : directly writing trees.
I did love this essay and I wonder what should be your opinion about such a project : Yet Another Wiki, trying to follow John McCarthy’s idea : « An environment where the markup, styling and scripting is all s-expression based would be nice. »
So I find projects like alphawiki pretty neat, but on the other hand I feel like it is ahead of its time, perhaps so much so that it is bound to fail, for various definitions of the word “fail.” I think Lispers frequently and easily fall into the trap of building large systems which can, in principle, do everything great, but, in practice, do nothing well, while forgetting that, for the most part, people are interested in doing one specific thing fairly well right now.
Common Lisp has other issues, but arguably it falls into this trap itself. Certainly things like the MOP fall into this category. Yes, with the MOP you can do all sorts of amazing things if you spend time building the infrastructure. But really, the vast majority of the time people want to bundle up a bit of state, call it an object, and go off to the races, building a game or whatever. If you spend a lot of time engineering something like MOP and forget that the most common use case is 3% of its functionality, then that functionality most commonly used will be under or mal-engineered. Yes, they could fix that mismatch with MOP hacking or whatever, but most folks don’t care to spend time on that extra-engineering, not because they are dumb or incurious but because they are busy and concerned primarily with specific goals.
In concrete terms of productivity, its often better to just avoid most design problems (like how, exactly, your class system should behave) even when your language lets you produce an exactly optimal solution. Realistically, in most cases, the productivity differential between your ideal class system and whatever your programming language has lying around is going to be marginal compared to the total engineering requirements of your problem. So why bother? Indeed, if you need to get done on time and under budget, its dangerous to muck around in defining a whole new system of object orientation!
Hence, the question about alphawiki is not “Is this a better wiki?” The question is “Is this so amazingly better that it justifies the extra engineering and learning time spent compared to an off the shelf thingy?” Go to San Francisco and throw a rock. You will hit a PHP programmer, and the rock will bounce off her and hit a Javascript programmer. No one knows how to use alphawiki. Some projects have the extra budget to learn or create new technologies. Some projects have the engineering expertise and the technical requirements that make creating or using new technologies important. Lots don’t. It sucks!
Thanks for your so prompt response : « It sucks! ».
Sixteen minutes to have a look on this work and to decide it has no interest. OK !
This is what Ward Cunningham thinks about this work : «I am impressed with this work and understand its uniqueness better now. I have also rambled through other pages you have made, many as experiments I would guess, and have been summarized by you in this single work. Bravo. (mail_20140429/Ward Cunningham) »
Ciao
I don’t think alphawiki sucks at all! I just think its software which may not have enough of a clear userbase to reach wide use. Do not mistake me! I have written tons of stuff which lots of people will never use, and I’m happy with it and think some of it is good software. But many things require the network-effects of tons of users to make a big impact, and many people consider software that doesn’t get lots of users as “failing.” Anyway, I think alphawiki is super cool software!