Thus, programs must be written for people to read, and only incidentally for machines to execute. — SICP
I have come to feel that this mindset is mostly hogwash outside academia.
The principal utility of programs is their utility. This seems obvious but is overtly contradicted by the cliche above. The market for programs that cannot be executed (or are not primarily to be executed) is precisely the book market. There are no paper programmers. Programming is a profession because there is a desire for new software to execute.
There is something beautiful about arresting statements like the above. The deception of the quote is that it feeds software’s narcissism. The code is important—it’s what makes it go, and we spend all day in there reading it and writing it. We have to be able to understand it to extend it or debug it. But if I’m not able to communicate clearly to a human, it won’t stop the program from entering production—but the “incidental” detail of it being wrong will.
I put a lot of stock in Brooks’s quote, “Build the first one to throw away, because you will.” It isn’t obvious to management why this is true, but any working programming knows that it is because the first act of programming is discovery. In practice, usually the customer gets the “benefit” of the throwaway first one because budgets are constrained and we are bad at estimation. This means that the first one you deliver really represents your first guess as to how this problem might be solved. It’s often wildly off-base and wrong, and you hope against hope that the user will find it useful anyway.
This leads to the second material limitation of literate programming, which is that if you were doing literate first, you have either just written a book about the wrong approach to the problem, which incidentally is also the throwaway program, or you have expended twice the resources to produce a book when what was desired was a program. A third option, which I have seen in practice, is that you have produced a book of negligible value, because although the book-production toolchain was employed and literate code was written, almost no effort went into forming the book-as-literature—that effort went directly into the code anyway.
This doesn’t mean there are no literate programs I wish existed, which I would enjoy reading. I would deeply love to take in a complete implementation of ACCRETE. The original, if possible. But the ACCRETE I want to read is a simplified core; the ACCRETE I want to run is flush with features and functionality. Similarly, I would love to read a presentation of the core of Postgres, but I fear if I tried to read it as-is I would be snowed by the details and complexity. In other words, I’m not convinced that programs of didactic value are necessarily the same programs I want to execute.
The success of projects like iPython Notebook, org-babel and Mathematica seems to indicate that there is a desire for “live documents,” which is to say, rich documents with built-in calculations. Prior to these technologies, people used Word and Excel, possibly even using something like OLE to embed Excel calculations in Word documents, but the process is clunky and limiting. Mathematica I think innovated here first, and with Mathematica Player, people could share Mathematica documents, which work somewhat like lab reports. A New Kind of Science showed that you could do a large document this way, but that doesn’t seem to be the way people use it. fpcomplete’s blogging scheme shows that this style is good for essay-length documents. This raises the question, is there a place for book-length live documents? I’m inclined to say no, because I cannot imagine a book-length single calculation, and live documents often represent performing a single exemplary calculation.
When I imagine reading my fantasy ACCRETE book online, I picture something like a fractal document. At first, you get a one-sentence summary of the system. You can then zoom in and get one paragraph, then one page. Each section, you can expand, at first from a very high-level English description, to pseudocode elaborated with technical language, finally to the actual source code. But this sounds like a very labor-intensive production. You would begin with the completed code and then invest significant additional time into it.
I don’t know if you could create the same experience in a linear manner. I suppose what you would do is have an introduction which unfolds each layer of the high-level description up to the pseudocode. Then, each major subsystem becomes its own chapter, and you repeat the progression. But the linearity defeats the premise that you could jump around.
If you consider every programmer that has written a book about programming, that’s probably the market for literate programming. This is a tiny fraction of people coding for a living. Books do require code that works and literate programming represents tooling in support of that use case. Outside authors I’m not convinced there is much call for it.
Programs are written for humans to execute, and only incidentally for other programmers to read.
Edit 2016-02-11: Read John Shipman’s rebuttal of this article.