What makes Stack Overflow so successful? I was thinking about this the other day. What came before Stack Overflow? Mostly forums and mailing lists. And in a forum or a mailing list, what you get is a pile-up of data in basically chronological order.
This turns out to be a difficult structure for asking and answering questions. So what Stack Overflow really did was this:
- Constrain the domain of possibilities to questions & answers
- Elaborate the types of communication that happen in that context, their operators and semantics
You can talk about anything on a forum, or a mailing list. It’s not confined to technical questions and answers. But supposing that’s all you’re doing, the chronology gets in the way. You wind up helping people serially with the same problems over and over. Some idiot chimes in with unhelpful, unproductive advice. You have to read through it and so does the next guy coming through. In a long forum chain on a single topic, the software may be evolving, and the first ten or hundred pages of dialogue may not be relevant to the current release. The forum can’t order posts by helpfulness or they won’t make sense because of the implicit context.
Stack Overflow fixes these problems by adding constraints. You can’t have a free-form response to a question; you have to either Answer or leave a Comment. The semantics of comments is that if they are low quality, they can be hidden. The semantics of answers is that they can be reordered according to their utility. There are different operators for questions, answers and comments. And the whole system is built around various functions of questions, answers and comments.
How many other systems are there that suffer from chronological pile-ups due to lack of constraints, operators and semantics? One that comes to mind is bug/issue trackers like JIRA. Sure we have a lot of objects in the system—issues, milestones, components, etc. But at the end of the day, each ticket allows an unlimited chronologically-sorted pile-up. Is there a workaround in that pile? Maybe; grab a cup of coffee and start reading, buddy. How do you distinguish a request from the developer for more information from the response from the user from administrativia about what release it should go in? You read the comments, in order.
I’m not aware of a system that solves this by not allowing unconstrained “replies” to tickets, but I think that would be an interesting piece of software to use.
Thus, programs must be written for people to read, and only incidentally for machines to execute. — SICP
I have come to feel that this mindset is mostly hogwash outside academia.
The principal utility of programs is their utility. This seems obvious but is overtly contradicted by the cliche above. The market for programs that cannot be executed (or are not primarily to be executed) is precisely the book market. There are no paper programmers. Programming is a profession because there is a desire for new software to execute.
There is something beautiful about arresting statements like the above. The deception of the quote is that it feeds software’s narcissism. The code is important—it’s what makes it go, and we spend all day in there reading it and writing it. We have to be able to understand it to extend it or debug it. But if I’m not able to communicate clearly to a human, it won’t stop the program from entering production—but the “incidental” detail of it being wrong will.
I put a lot of stock in Brooks’s quote, “Build the first one to throw away, because you will.” It isn’t obvious to management why this is true, but any working programming knows that it is because the first act of programming is discovery. In practice, usually the customer gets the “benefit” of the throwaway first one because budgets are constrained and we are bad at estimation. This means that the first one you deliver really represents your first guess as to how this problem might be solved. It’s often wildly off-base and wrong, and you hope against hope that the user will find it useful anyway.
This leads to the second material limitation of literate programming, which is that if you were doing literate first, you have either just written a book about the wrong approach to the problem, which incidentally is also the throwaway program, or you have expended twice the resources to produce a book when what was desired was a program. A third option, which I have seen in practice, is that you have produced a book of negligible value, because although the book-production toolchain was employed and literate code was written, almost no effort went into forming the book-as-literature—that effort went directly into the code anyway.
This doesn’t mean there are no literate programs I wish existed, which I would enjoy reading. I would deeply love to take in a complete implementation of ACCRETE. The original, if possible. But the ACCRETE I want to read is a simplified core; the ACCRETE I want to run is flush with features and functionality. Similarly, I would love to read a presentation of the core of Postgres, but I fear if I tried to read it as-is I would be snowed by the details and complexity. In other words, I’m not convinced that programs of didactic value are necessarily the same programs I want to execute.
The success of projects like iPython Notebook, org-babel and Mathematica seems to indicate that there is a desire for “live documents,” which is to say, rich documents with built-in calculations. Prior to these technologies, people used Word and Excel, possibly even using something like OLE to embed Excel calculations in Word documents, but the process is clunky and limiting. Mathematica I think innovated here first, and with Mathematica Player, people could share Mathematica documents, which work somewhat like lab reports. A New Kind of Science showed that you could do a large document this way, but that doesn’t seem to be the way people use it. fpcomplete’s blogging scheme shows that this style is good for essay-length documents. This raises the question, is there a place for book-length live documents? I’m inclined to say no, because I cannot imagine a book-length single calculation, and live documents often represent performing a single exemplary calculation.
When I imagine reading my fantasy ACCRETE book online, I picture something like a fractal document. At first, you get a one-sentence summary of the system. You can then zoom in and get one paragraph, then one page. Each section, you can expand, at first from a very high-level English description, to pseudocode elaborated with technical language, finally to the actual source code. But this sounds like a very labor-intensive production. You would begin with the completed code and then invest significant additional time into it.
I don’t know if you could create the same experience in a linear manner. I suppose what you would do is have an introduction which unfolds each layer of the high-level description up to the pseudocode. Then, each major subsystem becomes its own chapter, and you repeat the progression. But the linearity defeats the premise that you could jump around.
If you consider every programmer that has written a book about programming, that’s probably the market for literate programming. This is a tiny fraction of people coding for a living. Books do require code that works and literate programming represents tooling in support of that use case. Outside authors I’m not convinced there is much call for it.
Programs are written for humans to execute, and only incidentally for other programmers to read.
Edit 2016-02-11: Read John Shipman’s rebuttal of this article.
Why would one bother to learn a made-up language? Whenever I talk about constructed languages this question seems to come up. The most obvious reason is common to all (non-artistic) constructed languages: they have a smaller vocabulary and a more regular grammar, so they are all faster to learn and easier to use than any natural language. This actually helps you twice: first, because you gain a second language much faster than if you study a language far from your native tongue, and again, because going from language #2 to language #3 is much easier than going from language #1 to language #2. (Someone will substantiate me with that study of people who spent two years studying Spanish versus people who spent one year studying Esperanto and one year studying Spanish, who got further than the all-Spanish control group.)
Now, each of the constructed languages has a particular appeal:
- Toki Pona has just 120 words. You can literally learn the whole thing in a week to a month.
- Lojban is radically different from other languages and very precise, despite having a total vocabulary of about 2000 words. You can quickly learn enough to say quite complex things!
- Interlingua can be comprehended by anyone who speaks a Romance language, even if they do not know Interlingua! It is probably the most actually useful of the languages on the list.
- Esperanto has the largest community of any constructed language, with its own culture and media. Knowing it gives access to couches to sleep on in many countries across the world.
Learning any language is hard! But these are somewhat less hard.
The other day I answered a question on Stack Overflow that had to do with traversing a maze. Somewhat more idiomatically, the code would have looked like this:
mazeCell(Maze, What, X@Y) :-
nth0(X, Maze, Row),
nth0(Y, Row, What).Before I explain what’s special about this, let’s reflect for a moment on Prolog. Paul Graham once observed:
You often hear that programming languages are good because they provide abstraction. I think what we really like is not abstraction per se but brevity. A way of expressing programs that was more abstract, but made your programs longer, would not be very enticing. (This is not just a hypothetical example. It happens in Prolog.)
Why might Prolog have a reputation for being expansive rather than brief? One reason that comes to mind is that Prolog doesn’t actually have block structure. Each basic block usually has to become its own predicate.
Before I cower in defeat, let’s think about this maze problem. If I were writing in an object-oriented language and I wanted to create this maze abstraction, I would create an object to represent the maze, called Maze. I would add a method for finding what’s in the maze at a particular coordinate. I’d want a method for iterating through the maze, and maybe another one to look up the coordinate for a particular value. That last one, I may need to write a couple ways, one that returns all the occurrences, one that returns just one occurrence. I’d probably also want a class for the coordinate pairs, so I could treat them atomically, and maybe some class to combine a point and a value for the iteration scheme. In the end, I would wind up with probably seven or so methods and probably two or three classes and interfaces. It would look something like this:
public class Point {
public int x, y;
}
public class Maze implements Iterable<MazeLocation> {
public class MazeLocation {
public Point location;
public char value;
}
public Point locationOf(char value) { /* ... */ }
public List<Point> locationsOf(char value) { /* ... */ }
public char valueAt(Point point) { /* ... */ }
public Iterator<MazeLocation> iterator() { /* ... */ }
...
}What’s interesting about mazeCell/3 is that it actually can do all of those things at the same time. mazeCell(+Maze, -Value, +Point) (i.e. calling with a ground maze and point) returns the value at that point, equivalent to Java #valueAt. Calling with just the maze (mazeCell(+Maze, -Value, -Point)) iterates the maze, like Java’s #iterator. Calling with the value and the maze (mazeCell(+Maze, +Value, -Point)) searches the maze for that value, like Java’s locationsOf and locationOf. No new types were needed to represent a combination of multiple things; Prolog has no problem “returning” multiple values. Defining a new operator like @ is trivial.
You may object that the type and internal structure of the maze is not being hidden, and that may be true, but mazeCell/3 doesn’t reveal bits of that structure directly. You could change your representation of mazes and just change mazeCell/3; if all your other interactions with the maze go through it, they will continue to work. There is some loss of encapsulation (though Prolog implementations typically have modules which make it possible to regain it) but the inner structure can be mostly ignored since most queries can just go through this one relation.
What’s really going on here is that we are not defining a “method” to iterate mazeCell, we are defining a relation between a maze, the locations in the maze and the contents of those locations. The relation is more abstract than a method; that’s why it encompasses so many different procedures in the procedural domain.
So I dispute Paul Graham’s assertion. The Prolog code for some procedure may be larger than equivalent code in other languages. But you actually buy something for the higher abstraction: much greater flexibility, and that winds up paying off harder. The code for mazeCell/3 may be long compared to locationOf in Java, but you have to write essentially the same code four or five times where in Prolog, you could write it once. The total savings are big.
I’m a big fan of Haskell. Really big. It’s my other favorite language besides Prolog. And there can be no question that a significant aspect of Haskell is its expressive and powerful type system. In a sense, it’s the face that launched a thousand type systems. There was recently a bit of a row about dynamic and static languages (in my opinion this is the winning post) and I won’t rehash it here. Instead, let me trash a few idiotic arguments for static typing that arise in web development.
Statically-Typed SQL Generation!
This one really grinds my gears. First of all, there is not a single (to my knowledge) complete statically-typed wrapper around all of SQL. The static framework du jour probably supports some “common subset” of SQL that MySQL, Postgres and SQLite can do. This is a really awful subset to work with, yoking the oxen of Postgres and SQLite to the ass that is MySQL, but this isn’t my point. My point is, how often have you seen an application of yours get into production, only to have the SQL fail because of compile-time detectable type errors?
The answer is zero. This has never happened to anybody. Sure, you’ve had SQL injections. And it’s quite likely that your framework prevents them—nevermind the categories of valid and safe SQL that are excluded from it in the process. But it has literally never happened that SQL that worked when it was committed stopped working in production because of some unforeseen type issue.
Why is that? Because you write SQL interactively and you don’t commit it until it actually works.
There are plenty of interesting frameworks out there for SQL. The most compelling ones I’ve seen lately are Postmodern for Common Lisp and SQL Alchemy for Python. I have a certain appreciation for myBatis. The rest are taking an elegant-albeit-verbose external DSL (SQL) and replacing it with an inelegant, limiting, frequently also verbose internal DSL.
This is a non-problem, folks! Don’t waste my time solving it.
Valid HTML Generation!
Whoopee, you solved the hardest problem of all! My HTML is now going to be valid!
Funny story: it almost does not matter at all if your HTML is valid. Being valid won’t make it render properly. Nobody can tell or care if your HTML is valid.
Similar to the above, what happens with your static generator when WHATWG releases new elements? Do I have to circumvent your system, or do I have to wait for the next release? This was a real pain with JSF and HTML5.
Other Considerations
Note that there are things you can do for me with your library that I may value. For instance, Postmodern is able to give me a level of compositionality that is kind of missing from SQL. That’s nice. But the only way for it to really support all of SQL is by having a generic process for converting s-exps into SQL. You don’t have to get a new Postmodern whenever a new version of Postgres comes out to use new features. That’s the flipside of the type checking argument. If you wrap a lower-level system that has worse type checking than your language, you have to basically code that other system from scratch with your types. What are you buying with all that effort?
The other irksome thing about wrapping both SQL and HTML, but especially SQL, is that you’re trying to treat something high-level as if it were low-level, in something that you believe is high-level, but is actually lower-level. You cannot really have a procedural wrapper around SQL and have it be more powerful than SQL, which is declarative. The best you can hope for is a procedural library for writing SQL statements. And that’s what Postmodern and SQLAlchemy do, and why they don’t get into as much trouble as Hibernate does.
By “augmenting human intellect” we mean increasing the capability of a man to approach a complex problem situation, to gain comprehension to suit his particular needs, and to derive solutions to problems. Increased capability in this respect is taken to mean a mixture of the following: more-rapid comprehension, better comprehension, the possibility of gaining a useful degree of comprehension in a situation that previously was too complex, speedier solutions, better solutions, and the possibility of finding solutions to problems that before seemed insoluble.
— Douglas Engelbart, Augmenting Human Intellect: A Conceptual Framework (1962)
I think we have a very weak notion of how to improve the life of a scientist. As user interfaces go, a programming language offers nearly the full capability of the machine at fairly high abstraction, with significant up-front comprehension costs. (Similar to compression: you say less, but it’s because both you and the machine have more complexity.) The GUI offers almost the opposite; a well-designed GUI is intended for beginners, to take them up to either an intermediate or advanced level of use, but wholly interactive.
I think this represents a nice divide into two “classes” of computer user: the “user” who is a permanent underclass, and the “developer” or “programmer” who is a permanent overlord. Business and government like this; the former because it’s hard to sell something open and modifiable without giving away your secret sauce; the latter because they don’t want clerks suddenly differentiating from each other in terms of productivity or finding and changing anything. This is why Smalltalk failed, in my opinion. It’s hard to convince businesses to deliver apps that are fundamentally open to modification. Intellectual property and all that.
(Incidentally, you now see that business has discovered it’s possible to make the developers the middle class under themselves and the law.)
I don’t think this is a necessary state of affairs. Unix, Smalltalk and Lisp all anticipated the existence of a whole spectrum between low-level developers through users-that-program, through power users, to complete grunts. The whole Unix proposition was that your local wizard would prepare you C programs on-demand and the user-operators would string them together with shell scripting on-the-fly. Smalltalk and Lisp both supplied “open worlds” where the user cooperated with the system in a similar fashion using the local metaphors: creating objects or functions or what-have-yous, but able to modify the whole system interactively. The user of a Smalltalk system is basically a programmer. The Liskov substitution principle is really just a way of respecting that relationship as a good actor.
I frankly think it’s a little insulting to system analysts to assume that because they’re not programmers, they cannot fathom some kind of inbetweener system that maybe isn’t as brutal as full programming but also is substantially more powerful than using a GUI. And the real problem with the GUI, IMO, is not what it does to the user experience, it’s what it does to the development time—“the Programmer’s Full Employment Act.” A feature like subarrays is really quite simple in the model. It’s the layer upon layer of goo that has to be created and maintained which makes it so expensive. And it will finally culminate in something like an insolent child; not really knowing what it is doing or why, in severe need of more discipline.
The scientists are similar to the analysts. It seems strange to me to make them crawl through page after page of forms for submitting their scheduling block, only to give them essentially Python plus some libraries in the form of CASA for reducing the data they get. They obviously are able to grasp programming constructs—why do we force them, at great internal expense, to tolerate a UI that coddles them wrongly and then gets in their way?
A friend of mine likes to use the word operator to describe this class: people who are neither wholly victims of the GUI/programmer’s whims nor fully liberated programmers themselves. They use software that may be have required some training to understand. The software works more like their servant or peer than their boss.
I think that is what Engelbart foresaw in software as a tool to augment intellect. We got very distracted by the idea of direct manipulation, although now I think we build GUIs out of more of a cargo-cult sensibility than principle. But even that was a distraction from simpler, earlier principles: help humans comprehend faster and better and speed them to better solutions to problems. Maybe if we accept a little ugly, we can get there faster and more intelligently with less work.