What can Lisp teach us about the future of programming?
For a while, Slashdot was a favorite web hangout of mine, and I racked up probably close to 100 posts. My attention to that favored web site of geekdom was interrupted two years ago by my mother's illness and decline, as well as by my concentration on several dev projects that took up a lot of what had been my spare time.
Now that I am off my last project and have a little time, I have been visiting with Slashdot again. A visit to the front page of Slashdot the other day is leading to an examination by me of-- of all things-- Lisp. The story that led up to this was a link to an article written by Simeon Simeonov, who, I believe, was the author of the Cold Fusion web scripting language. He discussed how he thought that "Rich Internet Applications" were the way of the future, and how "Composite applications"-- mixing HTML with server-side code and database access-- is not the right programming model, but repeats the mistakes of the past. His discourse on Rich Internet Applications, and how Flash and Cold Fusion were tools for delivering such applications, was really just tooting the horn of Macromedia. However, his first paragraph seems to advocate the idea that it's a better programming model, not better tools, that is what's needed to deploy the next generation of internet applications. On this, I think he might be right on. The better programming model is more important than the tools, but the right tool for the job puts you that much further ahead.
A poster in the Slashdot discussion provided a link to an article written by Paul Graham on his web site, talking about how he believes that programmers benefit from having the most powerful programming tool they can get. The poster was trying to make the point that the better programming languages were more like Lisp. Graham talks about how he created the software for a successful ecommerce web site, Viaweb, in the early days of the web. It was written primarily in Lisp, while the rest of the software for their site was written in C. As Graham says, Viaweb was the first major end-user application of Lisp, which had been confined mostly to universities and research labs. When Graham and his business partner started Viaweb, he said that most other commercial web site owners were creating their site with Perl or C++, but that they decided to use Lisp because it's what they knew best. Because of their use of Lisp, Graham says that they were able to do things that made their site stand out from their competitors. He believes that not using the same programming tools that everyone else is using on the web can be a competitive advantage.
I vaguely recall from my high school and college days students who programmed with Lisp on the machines in the computer lab. Lisp was dismissed a long time ago by computer scientists as a serious tool for applications. They wanted, and needed, something more robust, which would handle larger numbers as well as strings. Other programming languages, such as FORTRAN, then COBOL, and finally, C, were available to do the job. This is why Lisp remained in the university and research labs and didn't catch on in the business world. Today's Common Lisp, however, is not your high school or college buddy's Lisp, that had to run on 48 kb or less of PC memory. Lisp has evolved with microcomputers, and the Common Lisp standard is a full-featured programming language that retains its extensibility. There is even support for objects through the Common Lisp Object System (CLOS), an extension of Lisp.
As it turns out, Graham wrote a few books on Lisp. I obtained a PDF copy of On Lisp from his web site, and I have been perusing it to find out why he feels Lisp is a useful programming language. In the preface, he says that his premise is two-fold:
...that Lisp is a natural language for programs written in the bottom-up style, and that the bottom-up style is a natural way to write Lisp programs. On Lisp will thus be of interest to two classes of readers. For people interested in writing extensible programs, this book will show what you can do if you have the right language. For Lisp programmers, this book offers a practical explanation of how to use Lisp to its best advantage...
The title is intended to stress the importance of bottom-up programming in Lisp. Instead of just writing your program in Lisp, you can write your own language on Lisp, and write your program in that.(p. vi)
Write my own language?!?! Why would I want to do that? I thought that having a 3rd or 4th generation programming language provided a level of abstraction above machine and assembler that allowed a programmer to give the machine the instructions it needed to do what needs to be done. This is how we avoid the tedium of messing with assembler. What about all the progamming languages that we have out there-- why can't I just use them? I am just playing devil's advocate here. Graham's point is that Lisp is extensible enough that you could write your own language if you wanted to. Lisp, after all, is just a functional language that really knows about how to process lists and numbers, but which he says can be used to create a software system whose capabilities can be extended to handle a lot of other things as well.
What grabs my attention, and should grab the attention of any serious programmer, is the notion that software development can-- or should-- be created from the bottom up. In chapter 1, Graham talks about why he thinks bottom-up programming is a better way to conceive of applications than the traditional top-down model:
...Whatever kind of program yours turns out to be, Lisp will, during the writing of it, have evolved into a language for writing that kind of program. If you’re not sure yet what kind of program you’re writing? To some ears that sentence has an odd ring to it. It is in jarring contrast with a certain model of doing things wherein you (1) carefully plan what you’re going to do, and then (2) do it. According to this model, if Lisp encourages you to start writing your program before you’ve decided how it should work, it merely encourages sloppy thinking. Well, it just ain’t so. The plan-and-implement method may have been a good way of building dams or launching invasions, but experience has not shown it to be as good a way of writing programs. Why? Perhaps it’s because computers are so exacting. Perhaps there is more variation between programs than there is between dams or invasions. Or perhaps the old methods don’t work because old concepts of redundancy have no analogue in software development: if a dam contains 30% too much concrete, that’s a margin for error, but if a program does 30% too much work, that is an error...Experienced programmers know that no matter how carefully you plan a program, when you write it the plans will turn out to be imperfect in some way. Sometimes the plans will be hopelessly wrong. Yet few of the victims of the plan-and-implement method question its basic soundness. Instead they blame human failings: if only the plans had been
made with more foresight, all this trouble could have been avoided. Since even the very best programmers run into problems when they turn to implementation, perhaps it’s too much to hope that people will ever have that much foresight. Perhaps the plan-and-implement method could be replaced with another approach
which better suits our limitations.(p. 2)
It is because of these limitations of foresight that Graham advocates Lisp as a way to write programs that avoid the pitfalls of the top-down model by replacing it with a bottom-up approach. In this idea, he anticipates the extreme programming model, but from Graham's point of view, the programming tools that are usually employed for extreme programming are inadequate to the task because they lack flexibility and extensibility. In addition, extreme programming, as it is often practiced, can't really be considered a bottom-up style of application development as much as it is a different style of programming within the context of waterfall systems development. It's because Lisp incorporated a lot of ideas that were ahead of their time that it should attract another look from developers, not as an application tool suitable for all our needs, but as a way of seeing what programming languages can become, and what systems development can become.
In Python, I see a lot of Lisp: how Python handles lists, how functions work, the presence of lambda, and in the syntactical similarities of the languages. One big difference with Python, though, is how it handles numbers: Python syntax for handling mathematics is more like how humans add, subtract, multiply and divide. Like Lisp, you can create functions in Python from the shell, too. There is a minimum of syntactical debris to trip over with Python; like Lisp, it lends itself to expressing ideas from natural language constructs. I think the poster on Slashdot was right when he described Python as "Lispy". I am not so sure about JavaScript being "Lisp in C's clothing"!
What really has me interested in Lisp, though, is its notion of macros. Graham talks about them in his article as something not yet incorporated in other languages; in On Lisp, he spends much of the book talking about macros. His succinct definition of macros is "a function that generates Lisp code-- a program that writes programs" (p. 82). The practical use of macros is to extend the language by generating Lisp code that the parser understands how to work with. This is also how a programmer can "write their own language" in Lisp, by using macros to make one thing represent something else. Essentially, this is what macros do in C and C++ -- they perform operations that allow one expression in our code to represent the operation, and it's more efficient because that work is done at compile time instead of with a function at run time. But we don't say that macros in C/C++ are "programs that write programs". So what is the basis of Graham making this claim?
Graham says that the difference between functions and macros in Lisp is that functions return a value, but that macros return expressions which can then be evaluated. He explains how Lisp macros are different from other programming languages by describing what happens at compile time:
...Lisp macros are singularly powerful. When a file of Lisp is compiled, a parser reads the source code and sends its output to the compiler. Here’s the stroke of genius: the output of the parser consists of lists of Lisp objects. With macros, we can manipulate the program while it’s in this intermediate form between parser and compiler. If necessary, these manipulations can be very extensive. A macro generating its expansion has at its disposition the full power of Lisp. Indeed, a macro is really a Lisp function— one which happens to return expressions... (a) macro might invoke a whole subprogram to generate its expansion. Being able to change what the compiler sees is almost like being able to rewrite it. We can add any construct to the language that we can define by transformation into existing constructs.(pp. 83-84.)
This differs from other programming languages, which have compilers that compile executable code and link to object code. So it is in the build process that Lisp is different from other programming languages, and the step of parsing before compiling is what gives Lisp added flexibility.
Python developers would argue that Python will also support metaprogramming, but through different means. Through constructions such as metaclasses, generic functions, and, as of Python 2.4, decorators, Python facilitates metaprogramming of the kind that Lisp does with macros. As other writers have also poined out, since Python is a scripting language, the definitions of classes and functions can be changed dynamically if necessary. Even Peter Norvig, a leading artificial intelligence expert, praises Python's suitability in a domain where Lisp has been the standard language:
Python is an excellent language for my intended use. It is easy to use (interactive with no compile-link-load-run cycle), which is important for my pedagogical purposes. While Python doesn't satisfy the prerequisite of being spelled J-A-V-A, Jython is close. Python seems to be easier to read than Lisp for someone with no experience in either language. The Python code I developed looks much more like the (independently developed) pseudo-code in the book than does the Lisp code. This is important, because some students were complaining that they had a hard time seeing how the pseudo-code in the book mapped into the online Lisp code (even though it seemed obvious to Lisp programmers).
Although Lisp might have more native power, and is faster when running compiled code, I think I will still stick with Python as my language of choice. Python syntax is simpler-- Lisp requires nesting everything in parentheses. Although Python is often criticized for using white space to mark off code blocks, I find it reasonably intuitive, and I think that it promotes readable code. Python is also the language with "batteries included"-- you can write GUIs, web scripts, system scripts, even emulate databases, all without stepping outside of the language. I think that Graham is on to something when he talks about Lisp and the power of its macros-- I think that this is where computer programming is heading, with applications that can perform metaprogramming. Lisp and Python have this capability now; it's up to developers to use it to best advantage.
Comments