I have a project of 140,000 lines of Java + 3000 lines of JS in an Angular application. I have Maven build about 20 separate Java modules and then exec Grunt to build the Angular front-end. The build takes 2 minutes on a warm machine that has run the build before. One minute of that is Grunt. Fully half of my build time is taken up by the 2% of my codebase in Javascript. This is to “build” artifacts for a language that does not get compiled in any real sense on my end at all.
The reasons for this inefficiency are many: my 3000 line Angular program somehow drags in 300 MB of node_modules and 30 MB of bower_components; Grunt is supposedly less efficient than Gulp because it has to make a lot of disk copies; my template is probably too fancy, etc. But this is a 50x discrepancy. There must be a lot of negative multipliers combining here to lead to inefficiency like this. And that is what the software engineers of the world don’t like about Node: the whiff of amateur that pervades the system.
Maven is not a good program. It’s verbose (input and output), it inflicts XML on humans, its documentation is awful. But it has a much more appropriate level of flexibility. It has made the NRAO’s Java code a lot more maintainable. Anybody can check anything out and run mvn package to build it, mvn deploy to deploy it (well, usually). And this is what makes Maven a great program, even though it is also an awful program. It’s prescriptive and pervasive. It wouldn’t be great if it were optional.
People don’t like that Java is slow and plodding in terms of change. But let me point out that Maven was established long before Grunt was ever a thing, and off in Node.js land, they’ve already gone through Gulp and are on to Webpack and a few other build systems besides. This time next year, I’m sure webpack will be a memory and something else will have taken root. In five years, there probably will have been five or ten more Javascript “build” tools that all do less work than Maven less efficiently. And Maven will still be the de-facto build tool for Java and my packages that use it will still build the same way as they do today—with lots of unhelpful, pedantic logging; with the same bullshit in the XML file to convince Java that I don’t need compatibility with Java 1.3; with the same dependency hell that I have learned how to deal with. Because that’s what Java is like.
I don’t think Node or any other technology can be inherently bad. But, technologies can be immature, or they can foster a community that encourages amateurish (or bad, or insecure) behavior. Eventually, maybe someone falls in love with it and leads a big crusade to clean things up. Or maybe it becomes so popular that heartless buck-chasing programmers hammer on it until the old ways become unpleasant and new ways take root (this is what happened to PHP). But it feels like we’re dealing with Node.js in its awkward teenage years. A great Cambrian explosion of ideas, mostly bad ideas, playing out on npm. A lot of libraries with an air of perfunctoriness. “It connects, but there’s no error handling.” This kind of thing is upsetting to engineers, and we are mean.
The difference between Java and Javascript is very small. It’s basically that Java was designed to kidnap Unix and Windows developers and haul them off to Sun’s niche platform, whereas Javascript was designed to help high schoolers impress their long-distance girlfriend. Java has never become cool, despite squeezing out frameworks with sassy names like Play and Ninja. They’re like having a cool Dad who lets you drink a beer with him when you’re 17—it’s your senior year, after all.
Side note, a lot of the appeal of Node is that you have Javascript on the front-end and the back-end. You know, so you can pass JSON around. Is it hard to pass JSON to or from any language today? Not really. But you could have the same domain objects! But nobody really does that. The work that is done in the front-end and the back-end are different. You could make a big fat object and use it on both sides, but how are you going to avoid trying to send mail from the front-end or writing DOM nodes from the back-end? This is really about maintaining comfort. Teenage-you didn’t like hanging out with Dad’s friends. Awwwwkk-ward.
Maybe Javascript will grow up someday. Who knows? Sometimes high schoolers grow into excellent engineers. Only time will tell.
What makes Stack Overflow so successful? I was thinking about this the other day. What came before Stack Overflow? Mostly forums and mailing lists. And in a forum or a mailing list, what you get is a pile-up of data in basically chronological order.
This turns out to be a difficult structure for asking and answering questions. So what Stack Overflow really did was this:
- Constrain the domain of possibilities to questions & answers
- Elaborate the types of communication that happen in that context, their operators and semantics
You can talk about anything on a forum, or a mailing list. It’s not confined to technical questions and answers. But supposing that’s all you’re doing, the chronology gets in the way. You wind up helping people serially with the same problems over and over. Some idiot chimes in with unhelpful, unproductive advice. You have to read through it and so does the next guy coming through. In a long forum chain on a single topic, the software may be evolving, and the first ten or hundred pages of dialogue may not be relevant to the current release. The forum can’t order posts by helpfulness or they won’t make sense because of the implicit context.
Stack Overflow fixes these problems by adding constraints. You can’t have a free-form response to a question; you have to either Answer or leave a Comment. The semantics of comments is that if they are low quality, they can be hidden. The semantics of answers is that they can be reordered according to their utility. There are different operators for questions, answers and comments. And the whole system is built around various functions of questions, answers and comments.
How many other systems are there that suffer from chronological pile-ups due to lack of constraints, operators and semantics? One that comes to mind is bug/issue trackers like JIRA. Sure we have a lot of objects in the system—issues, milestones, components, etc. But at the end of the day, each ticket allows an unlimited chronologically-sorted pile-up. Is there a workaround in that pile? Maybe; grab a cup of coffee and start reading, buddy. How do you distinguish a request from the developer for more information from the response from the user from administrativia about what release it should go in? You read the comments, in order.
I’m not aware of a system that solves this by not allowing unconstrained “replies” to tickets, but I think that would be an interesting piece of software to use.
Thus, programs must be written for people to read, and only incidentally for machines to execute. — SICP
I have come to feel that this mindset is mostly hogwash outside academia.
The principal utility of programs is their utility. This seems obvious but is overtly contradicted by the cliche above. The market for programs that cannot be executed (or are not primarily to be executed) is precisely the book market. There are no paper programmers. Programming is a profession because there is a desire for new software to execute.
There is something beautiful about arresting statements like the above. The deception of the quote is that it feeds software’s narcissism. The code is important—it’s what makes it go, and we spend all day in there reading it and writing it. We have to be able to understand it to extend it or debug it. But if I’m not able to communicate clearly to a human, it won’t stop the program from entering production—but the “incidental” detail of it being wrong will.
I put a lot of stock in Brooks’s quote, “Build the first one to throw away, because you will.” It isn’t obvious to management why this is true, but any working programming knows that it is because the first act of programming is discovery. In practice, usually the customer gets the “benefit” of the throwaway first one because budgets are constrained and we are bad at estimation. This means that the first one you deliver really represents your first guess as to how this problem might be solved. It’s often wildly off-base and wrong, and you hope against hope that the user will find it useful anyway.
This leads to the second material limitation of literate programming, which is that if you were doing literate first, you have either just written a book about the wrong approach to the problem, which incidentally is also the throwaway program, or you have expended twice the resources to produce a book when what was desired was a program. A third option, which I have seen in practice, is that you have produced a book of negligible value, because although the book-production toolchain was employed and literate code was written, almost no effort went into forming the book-as-literature—that effort went directly into the code anyway.
This doesn’t mean there are no literate programs I wish existed, which I would enjoy reading. I would deeply love to take in a complete implementation of ACCRETE. The original, if possible. But the ACCRETE I want to read is a simplified core; the ACCRETE I want to run is flush with features and functionality. Similarly, I would love to read a presentation of the core of Postgres, but I fear if I tried to read it as-is I would be snowed by the details and complexity. In other words, I’m not convinced that programs of didactic value are necessarily the same programs I want to execute.
The success of projects like iPython Notebook, org-babel and Mathematica seems to indicate that there is a desire for “live documents,” which is to say, rich documents with built-in calculations. Prior to these technologies, people used Word and Excel, possibly even using something like OLE to embed Excel calculations in Word documents, but the process is clunky and limiting. Mathematica I think innovated here first, and with Mathematica Player, people could share Mathematica documents, which work somewhat like lab reports. A New Kind of Science showed that you could do a large document this way, but that doesn’t seem to be the way people use it. fpcomplete’s blogging scheme shows that this style is good for essay-length documents. This raises the question, is there a place for book-length live documents? I’m inclined to say no, because I cannot imagine a book-length single calculation, and live documents often represent performing a single exemplary calculation.
When I imagine reading my fantasy ACCRETE book online, I picture something like a fractal document. At first, you get a one-sentence summary of the system. You can then zoom in and get one paragraph, then one page. Each section, you can expand, at first from a very high-level English description, to pseudocode elaborated with technical language, finally to the actual source code. But this sounds like a very labor-intensive production. You would begin with the completed code and then invest significant additional time into it.
I don’t know if you could create the same experience in a linear manner. I suppose what you would do is have an introduction which unfolds each layer of the high-level description up to the pseudocode. Then, each major subsystem becomes its own chapter, and you repeat the progression. But the linearity defeats the premise that you could jump around.
If you consider every programmer that has written a book about programming, that’s probably the market for literate programming. This is a tiny fraction of people coding for a living. Books do require code that works and literate programming represents tooling in support of that use case. Outside authors I’m not convinced there is much call for it.
Programs are written for humans to execute, and only incidentally for other programmers to read.
Edit 2016-02-11: Read John Shipman’s rebuttal of this article.
Why would one bother to learn a made-up language? Whenever I talk about constructed languages this question seems to come up. The most obvious reason is common to all (non-artistic) constructed languages: they have a smaller vocabulary and a more regular grammar, so they are all faster to learn and easier to use than any natural language. This actually helps you twice: first, because you gain a second language much faster than if you study a language far from your native tongue, and again, because going from language #2 to language #3 is much easier than going from language #1 to language #2. (Someone will substantiate me with that study of people who spent two years studying Spanish versus people who spent one year studying Esperanto and one year studying Spanish, who got further than the all-Spanish control group.)
Now, each of the constructed languages has a particular appeal:
- Toki Pona has just 120 words. You can literally learn the whole thing in a week to a month.
- Lojban is radically different from other languages and very precise, despite having a total vocabulary of about 2000 words. You can quickly learn enough to say quite complex things!
- Interlingua can be comprehended by anyone who speaks a Romance language, even if they do not know Interlingua! It is probably the most actually useful of the languages on the list.
- Esperanto has the largest community of any constructed language, with its own culture and media. Knowing it gives access to couches to sleep on in many countries across the world.
Learning any language is hard! But these are somewhat less hard.
The other day I answered a question on Stack Overflow that had to do with traversing a maze. Somewhat more idiomatically, the code would have looked like this:
mazeCell(Maze, What, X@Y) :-
nth0(X, Maze, Row),
nth0(Y, Row, What).Before I explain what’s special about this, let’s reflect for a moment on Prolog. Paul Graham once observed:
You often hear that programming languages are good because they provide abstraction. I think what we really like is not abstraction per se but brevity. A way of expressing programs that was more abstract, but made your programs longer, would not be very enticing. (This is not just a hypothetical example. It happens in Prolog.)
Why might Prolog have a reputation for being expansive rather than brief? One reason that comes to mind is that Prolog doesn’t actually have block structure. Each basic block usually has to become its own predicate.
Before I cower in defeat, let’s think about this maze problem. If I were writing in an object-oriented language and I wanted to create this maze abstraction, I would create an object to represent the maze, called Maze. I would add a method for finding what’s in the maze at a particular coordinate. I’d want a method for iterating through the maze, and maybe another one to look up the coordinate for a particular value. That last one, I may need to write a couple ways, one that returns all the occurrences, one that returns just one occurrence. I’d probably also want a class for the coordinate pairs, so I could treat them atomically, and maybe some class to combine a point and a value for the iteration scheme. In the end, I would wind up with probably seven or so methods and probably two or three classes and interfaces. It would look something like this:
public class Point {
public int x, y;
}
public class Maze implements Iterable<MazeLocation> {
public class MazeLocation {
public Point location;
public char value;
}
public Point locationOf(char value) { /* ... */ }
public List<Point> locationsOf(char value) { /* ... */ }
public char valueAt(Point point) { /* ... */ }
public Iterator<MazeLocation> iterator() { /* ... */ }
...
}What’s interesting about mazeCell/3 is that it actually can do all of those things at the same time. mazeCell(+Maze, -Value, +Point) (i.e. calling with a ground maze and point) returns the value at that point, equivalent to Java #valueAt. Calling with just the maze (mazeCell(+Maze, -Value, -Point)) iterates the maze, like Java’s #iterator. Calling with the value and the maze (mazeCell(+Maze, +Value, -Point)) searches the maze for that value, like Java’s locationsOf and locationOf. No new types were needed to represent a combination of multiple things; Prolog has no problem “returning” multiple values. Defining a new operator like @ is trivial.
You may object that the type and internal structure of the maze is not being hidden, and that may be true, but mazeCell/3 doesn’t reveal bits of that structure directly. You could change your representation of mazes and just change mazeCell/3; if all your other interactions with the maze go through it, they will continue to work. There is some loss of encapsulation (though Prolog implementations typically have modules which make it possible to regain it) but the inner structure can be mostly ignored since most queries can just go through this one relation.
What’s really going on here is that we are not defining a “method” to iterate mazeCell, we are defining a relation between a maze, the locations in the maze and the contents of those locations. The relation is more abstract than a method; that’s why it encompasses so many different procedures in the procedural domain.
So I dispute Paul Graham’s assertion. The Prolog code for some procedure may be larger than equivalent code in other languages. But you actually buy something for the higher abstraction: much greater flexibility, and that winds up paying off harder. The code for mazeCell/3 may be long compared to locationOf in Java, but you have to write essentially the same code four or five times where in Prolog, you could write it once. The total savings are big.
I’m a big fan of Haskell. Really big. It’s my other favorite language besides Prolog. And there can be no question that a significant aspect of Haskell is its expressive and powerful type system. In a sense, it’s the face that launched a thousand type systems. There was recently a bit of a row about dynamic and static languages (in my opinion this is the winning post) and I won’t rehash it here. Instead, let me trash a few idiotic arguments for static typing that arise in web development.
Statically-Typed SQL Generation!
This one really grinds my gears. First of all, there is not a single (to my knowledge) complete statically-typed wrapper around all of SQL. The static framework du jour probably supports some “common subset” of SQL that MySQL, Postgres and SQLite can do. This is a really awful subset to work with, yoking the oxen of Postgres and SQLite to the ass that is MySQL, but this isn’t my point. My point is, how often have you seen an application of yours get into production, only to have the SQL fail because of compile-time detectable type errors?
The answer is zero. This has never happened to anybody. Sure, you’ve had SQL injections. And it’s quite likely that your framework prevents them—nevermind the categories of valid and safe SQL that are excluded from it in the process. But it has literally never happened that SQL that worked when it was committed stopped working in production because of some unforeseen type issue.
Why is that? Because you write SQL interactively and you don’t commit it until it actually works.
There are plenty of interesting frameworks out there for SQL. The most compelling ones I’ve seen lately are Postmodern for Common Lisp and SQL Alchemy for Python. I have a certain appreciation for myBatis. The rest are taking an elegant-albeit-verbose external DSL (SQL) and replacing it with an inelegant, limiting, frequently also verbose internal DSL.
This is a non-problem, folks! Don’t waste my time solving it.
Valid HTML Generation!
Whoopee, you solved the hardest problem of all! My HTML is now going to be valid!
Funny story: it almost does not matter at all if your HTML is valid. Being valid won’t make it render properly. Nobody can tell or care if your HTML is valid.
Similar to the above, what happens with your static generator when WHATWG releases new elements? Do I have to circumvent your system, or do I have to wait for the next release? This was a real pain with JSF and HTML5.
Other Considerations
Note that there are things you can do for me with your library that I may value. For instance, Postmodern is able to give me a level of compositionality that is kind of missing from SQL. That’s nice. But the only way for it to really support all of SQL is by having a generic process for converting s-exps into SQL. You don’t have to get a new Postmodern whenever a new version of Postgres comes out to use new features. That’s the flipside of the type checking argument. If you wrap a lower-level system that has worse type checking than your language, you have to basically code that other system from scratch with your types. What are you buying with all that effort?
The other irksome thing about wrapping both SQL and HTML, but especially SQL, is that you’re trying to treat something high-level as if it were low-level, in something that you believe is high-level, but is actually lower-level. You cannot really have a procedural wrapper around SQL and have it be more powerful than SQL, which is declarative. The best you can hope for is a procedural library for writing SQL statements. And that’s what Postmodern and SQLAlchemy do, and why they don’t get into as much trouble as Hibernate does.