If you’re like me, you’re always in search of new ways to fetishize your books. I recently started keeping track of my books at all—a few too many of my expensive computer science tomes have walked away without my really knowing who has them. This has to stop, but solving this problem is boring and easy. You just go install Delicious Monster or BookPedia or something.

If we want to make the task harder, and thus, more fun, we have to go further than that. We have to assign books call numbers. This is the only way to convert your freeloading friends into legitimate library patrons.

What Librarians Do

What do librarians do? I never really asked myself this question. I assumed they had things to do other than “ssh!”ing people but I never really probed further than that. It turns out one of their major problems is classifying the books. Perhaps you thought, as I did, that every book has a magic code on the inside flap with where it should be placed! Well, many do, but many do not, and aside from that, this just gets you the class of the book, not whatever additional information you might want to incorporate into the call number.

You may be surprised to discover, as I was, that every library (potentially!) uses different call numbers to identify books. The call number usually incorporates some auxilliary information about the book, like the author, title, or year of publication. There are three purposes for the call number: to make it possible to uniquely identify a book, to give your patrons a way to look for a book, and to give you a unique relative placement for the book on the shelves. The latter thing there implies sortabililty.

Methods

There are four ways to classify books, and they each have strengths and weaknesses. The Library of Congress classification (LoC) has a pleasant hideous formality, but the tables are quite enormous and it has a tendency to scatter related things into different corners. Dewey (DDC) has that childish air to it, but can be delivered in a single abbreviated volume and produces short and tidy call numbers. Universal Decimal Classification (UDC) adds some syntatic horror and can be had in a real electronic format. It is, unfortunately, almost totally unheard of here in America. And then there is the Colon Classification, which deserves a special, lengthy introduction.

Colon Classification

The primary problem as perceived by LoC and DDC is: where does this book go on the shelves? And that question raises design concerns: one wants books to be together with other books on the same topic rather than sorted by color or author’s favorite animal. So both spend most of their effort trying to figure out what the real subject of the book is, so they can produce a call number that puts things together helpfully.

This isn’t the only perspective though. Colon instead seems to be saying: let’s create a precise, formal statement about the subject of a book. That statement can then be encoded into a terse syntax, and we’ll use that syntax as the class portion of the call number.

All of the work done in other systems to produce the total classification is just part of the “personality” facet of the book under CC. A further four facets can be used to narrow the book down by time, space, noun and verb (called “matter” and “energy” but I prefer my words). Then, each class or subclass can define its own facets. History, for instance, using additional time and space facets to discuss the subject of the book, freeing up the other two for the origin of the book itself. Literature makes the author a facet. And then there are a great number of additional “devices” that can be used to go further: books can be promoted to “classic” status and made into their own category; anteriorizing and posteriorizing facets to describe the structure of the book. The language and form, whether it is a commentary on some other book, etc. can all be encoded. Further, the “personality” can be combined with other personalities with special codes to indicate, for instance, the book contains both, or the book is in one subject biased for people who know another, or the book compares or discusses the similarities or differences between two subjects. Facets can be connected with each other. And this whole process can be nested in a recursive fashion, leading to terms like “second round energy.”

While shockingly powerful as a way of encoding a statement about a book, it suffers greatly from the huge complexity. The sorting rules are not at all intuitive: some sections are to be treated as integers, others as decimals. There is an “octavizing digit” that changes the sorting order, so you count 1, 2, 3, … 8, 91, 92, 93, … 991, 992, etc., but each class of character has one (so, z for lower-case alphabetic characters). Additionally, much complexity is spent trying to save a character here or there: instead of writing 2010, you would write N1, or N14 for 2014. It’s not clear to me that the syntax would be all that parseable to a lay person after any amount of use.

APL

The whole thing reminds me a lot of APL. Reading the book A Programming Language is a fairly uncomfortable affair because of the strange order and missing information. One gets the sense that the language doesn’t really make a distinction between syntax and semantics. It jumps around strangely from low-level to high-level concerns and back. It’s kind of a poor spec.

Colon Classification, 6th Edition is a lot better, but still has that sort of feel to it. One of the early definitions specifies the character set for the call number as containing “some Greek letters.” Which ones? Read two more pages, and you’ll see them listed in their sort order. Similarly, what appears in what order in the call number? It says the collection number can go “above” the class and the book number can go “below” the class when printed vertically (two different rules) but for the normal left-to-right direction nothing is said explicitly. I infer that it would go collection-class-book from the order it would appear vertically.

In other ways, it reminds me of SGML, prior to XML. The “chronological device” that saves you a whole digit or two, for instance, is very reminiscent of not needing closing tags. With a class number like L,45;421:6;253:f.44’N5, how much worse is the two-character longer L,45;421:6;253:f.44’1950, really?

Faceted Search

The benefit of CC is the facets. But on your bookshelf, you don’t benefit much from the facets. Either your collection is so small, every category has one book, or your collection is so huge, only the first part matters anyway. The facets might help if you needed to do faceted search, but you’re either looking for one of your own books, or you’re doing an electronic search, where the facets don’t need to be called out explicitly.

It would probably be beneficial for searching to have a comprehensive statement of the subject matter of a book. And in that case, having a complex of facets like CC provides would probably be helpful. But if you aren’t using it as the call number, there’s no need for the syntax to be so murdersome. Choosing a restricted vocabulary and a sensible metadata format would be enough; whatever you chose, if it has an electronic representation that can be parsed by a machine, you’re done—there’s no need to create horrors like the above. Even a basic full-text search on any of the words in the sentence “Research in the cure of the tuberculosis of lungs by x-ray conducted in India in 1950s” would get you meaningful results.

Planes

Ranganathan structured CC with a separate “language” plane for the human expression and a “number” plane for constructing the call number. I’m not sure his syntax is worth so much it would be worth saving (Medicine,Lungs;Tuberculosis:Treatment;X-ray:Research.India’1950) instead of something else, but it raises the question (for me) of whether it would be meaningful or helpful to use CC to develop statements about the subject of books_, even if one were using DDC to file them on the shelf.

Improvements?

The detour through CC was intellectually refreshing. The system is quite interesting, but ultimately, seems to be more like two half-solutions to two problems than a complete solution to either a metadata problem or a call number problem. I could see someone else applying the system—particularly if they have a large collection of Indian religious texts, as that’s the only fully-elaborated example in the book. For someone like me, a computer science-y guy with a fondness for formality and a large collection of CS-related books, I’m not convinced you’d be able to implement the system faithfully anyway. I think you’d wind up having to create so much from scratch (and fail to properly apply so much of the many and complex rules) that you’d be no better off than trying to come up with something yourself out of whole cloth. For instance, these ideas occurred to me while reading Colon Classification, 6th Edition:

• Maybe I should co-opt another Greek letter for the Main Class for computer science. Perhaps Λ?
• I could devise a real collation around an actual character encoding (such as Unicode) if only I…
  • removed the Greek letters or sorted them to the end instead of “after the nearest similar Latin letter”
  • removed the “octave device”
• I should get rid of the “chronological device,” because it doesn’t save enough
• I should write a formal grammar for the syntax of this thing

And so forth.

Conclusion

Ultimately, I have decided to go forward with Dewey Decimal Classification. The abbreviated guide for one edition out-of-date can be had for $15 on Amazon. For a small collection like mine, this will be totally sufficient to the task of putting my books in order. DDC is also widely-used in sub-collegiate libraries. My wife won’t find it nearly as objectionable, and my son may benefit from being exposed to it. Also, you can find code like mine to query OCLC to look up DDC classes for books, and many books come with a DDC classification printed in them as part of the CIP data.

I would still like to think about the metadata problem. CC is very interesting and powerful, and might provide some inspiration here, but I think ultimately other modern systems are probably just as powerful and without the drawbacks.

For a better-written take on the same problem, please see this article by David Mundie.

March 27, 2014

John Carmack says VR is going to change the world. That ain’t gonna happen. Here’s why:

Zero mainstream non-gaming applications

The PC revolutionized the world because every office with a typewriter or a calculator eventually switched to a PC. Nearly every office function is augmented or replaced by a computerized version. A word processor is manifestly better than a typewriter. The calculator became a spreadsheet. The centerpiece of the meeting is a Powerpoint presentation; several participants are remote.

To suggest that something is going to be as big or bigger than the PC revolution, one must have in mind that whatever it is will augment or replace every PC. So what’s the lady at the DMV going to do with a VR headset? Immerse herself in the web-based registration database?

Casual gaming is worth more than dedicated gaming

The biggest phenomenons in gaming of the last ten years were Angry Birds and Farmville. While gamer geeks obsess over technical specifications and drool over technology like the Oculus Rift, the rest of the world is quietly Candy Crushing at the DMV. If I can’t play your game in a waiting room, I’m not going to play your game at all. This is the kind of fact that upsets the true believer, much like college students love to hate pop musicians for the sin of making fun, accessible music. The greater the love and technical finesse in a genre, the smaller the appeal, the higher the cost of bringing a new product to market, and the lower the revenue. Grand Theft Auto is a major outlier.

Looking like a tool = failure

Where have all those bluetooth headsets gone? Five years ago they were the unmistakable symbol of being serious business. Today they’re the unforgivable mark of being a tool. People who still use them keep them politely in their shirt pocket when not in use. I wish I could say this is in response to the unbelievable rudeness of being one second away from answering your phone while conversing in person on a ten second delay, but I know the truth. They’re just unfashionable. Using your touchscreen phone does not make you look like a tool.

We don’t want more bandwidth with our tools

Someone tried to sell me on this technology with this: “Imagine a world in which you can actually lose track of, or be mistaken about exactly which level of simulation you’re currently experiencing.” Imagine all that, minus all your understanding of how computers work, minus your operational knowledge. This is a horror movie scenario, not a technology someone will buy for themselves to experience. Programmers and power users may imagine they want a higher bandwidth interface to the machine, but muggles want that about as badly as they want a higher bandwidth interface to their washing machine. Take your discomfort and square it? No thanks. The harder a tool is to put down, the more specialized it is, the less mass appeal it has.

Imagine the possibilities, man

Asking for examples how use outside gaming, I got a few good specialized ideas but mostly a bunch of bad ones.

• Education. “Why would children watch a slideshow about ancient Egypt when they can walk in a digital recreation made by brilliant artists/game designers who worked with leading historians, archeologists, etc. I’ll warrant that experiencing things ‘first hand’ will make people learn faster/better.”

For one thing, comparing a musty, shitty old slideshow to a state-of-the-art, Hollywood-budgeted masterpiece isn’t a fair comparison. As it happens, we already have something like that though: IMAX movies. And guess what? It’s neither saving nor destroying education, and the idea that virtual reality might help it is assumed rather than proven.

To take a brief detour, everybody loves to complain about education, but where is the proof that what we have is a technical problem? I think technology is the one avenue to improving education that has actually been explored. A good half of teachers are complete technophobes, the other half are clearly tech-obsessed, yet the results speak for themselves. What is needed is individualized education from motivated educators not especially wedded to bureaucracy and non-educational goals (test results, raising costs, etc.) These are systemic problems that are totally orthogonal to technology. There is no technical problem with teaching, just systematized wrong incentives.

• Training simulations.

Sure, but again, niche. The whole DMV office might have a need for one or two VR helmets at that point. They also only have one or two photocopiers—is that the kind of “change the world” we’re talking about, with our big ol’ teary puppydog eyes?

• It’s the next step after 2D displays. “If you look at human history, we started with writing, then newspapers, then radio, followed by television. The way in which we consume media hasn’t changed from television/displays. Virtual reality is the next evolutionary step in media consumption. The final step being human-computer interfaces.”

Something can be the “next step” and still not be the next “big thing.” Writing is used for entertainment, but it’s not solely useful for entertainment. 3D might be better if it were free, but as long as it comes with a helmet tax or a glasses tax the improvement will have to offset the cost. How will 3D improve the news or the sitcom?

It’s helpful to compare to color. Color is a much bigger win than 3D, and it comes without imposing any nerd taxes. It isn’t invasive, in the sense that I can just walk away from a color TV. I don’t have to dismount or untangle myself or place anything in its charger. If there were a 3D TV I could just look at and see 3D, it would immediately destroy VR helmets and conventional 3D TVs. That would be a big advance, no question. But more than that, everything benefits from color: news, entertainment, information, emergency broadcasts, etc. The benefit of 3D is almost entirely limited to entertainment.

• “Supposedly the porn industry is one of the largest drivers of technology when it comes to media consumption.”

Invariably, somebody wants to bring up porn. To me, this is the least compelling capstone on the whole rickety arch. Your DVD player had a “Select Angle” button on it. This was added by the porn industry, for obvious reasons—let’s face it, it would be a very odd cinematographic decision to provide the viewer with such options in a real movie. But notice your Blu-ray player does not have this button, because this feature was never used in real life.

I’m sure there are people who will be very interested in this application of technology. However, I doubt this will be a major driver of technological adoption. Everybody had a DVD player with the Select Angle button, but very few got that DVD player just to watch porn. Products that have such limited uses tend to be bought sparingly so as to fit in the sock drawer. The internet has already accomplished everything that can be hoped for vis-a-vis pornography.

• It’s a cost issue

The technologists pimping VR have always laid out some kind of bizarre assumption that everybody would buy a VR helmet if only it were cheaper or more accessible or there were more somethings taking advantage of it. But this is like saying everybody would buy books with six-foot pages if only the cost were low enough. After all, such a book would be so much more immersive—you certainly couldn’t do anything else while reading it! Yet enormous books remain niche items, quietly resented by their owners for taking up too much shelf space and being too onerous for serious study on the toilet.

True technical innovation is mostly the drab story of accepting lower fidelity in exchange for secondary benefits. I’d much rather have a beautiful calligraphic script Bible written on the finest parchment than some movable type piece of junk with smeared ink and letters missing. Barring that, I’d much rather have a nice leather-bound edition on good quality acid-free paper with proper justification and footnotes, a beautiful font and proper ligatures. But apparently I’ll settle for one with no ligatures at all, rendered in dark grey Times New Roman on a light grey background—if I can keep it with a few thousand of my other favorite books on a Kindle or a Nook. (The recharge time on a paper book is utterly unbeatable, though the backlighting is, for the most part, atrocious.)

• But it’s so much more real and immersive!

Have you noticed soap operas feel weird and unnatural for some reason you can’t quite put your finger on? It’s because they’re recorded in a 60 frames per second video format, unlike theatrical releases, which are still shot in 24 frames per second. For whatever reason, 24 fps seems to put our minds in the right place for movies. Reviews of the new “Hobbit” movie were largely in agreement that the “high frame rate” edition was less pleasant, robbing the movie of its “cinematic look.” Technical virtuosity may be incompatible with an immersive cinematic experience. Looking “better” made the movie worse.

We should not be too bothered by the idea that we have reached the limits of human perception with a particular medium. Audio CDs are essentially optimal for music. They encode much more than we can hear and stereo seems to be as much information as we need to reconstruct a reasonable semblance of depth. Competing platforms like SuperAudio haven’t really gained any traction partly because nobody can hear a tangible difference. The bigger reason for their flop obviously being that they were superceded by a lower fidelity alternative with better secondary benefits: MP3s.

Right now the TV industry is trying to get us excited about the idea of 4K TVs which promise pixels so small we’d have to be inches from the screen to see them. At the proper viewing distance from a “normal” HD TV it would be impossible to see the pixels. Even streaming 720p to my 1080p TV I can see no discernible degradation of quality. Anyone trying to foist truly absurd levels of fidelity like 24/192 audio and 4K on us is missing the fact that people are watching movies on their phones and listening to music through bundled $10 earbuds, and perfectly happy about both. They will be using the next big thing, in home and office, working and playing.

A brief history of VR

Do you remember the video game Descent? It came out in 1994 and sported compatibility with several 3D systems of the day, like the iGlasses (no relation to Apple). We’re talking resolutions in the 320x240 range. The game was kind of a big deal, but the iGlasses? Not a big hit.

What about the Virtual Boy? It came out in 1995 and I got one, absolutely loved it: real 3D in bright red on black. The manual had some puzzling health remarks though: take a break every hour, for instance, and even the games mandated it (“auto pause”). It also mentioned that you might get headaches from playing with it or nausea, and you should really stop if you have a seizure. Monochrome plus headaches equals a very small fan base.

Suppose you have a technology. Only one gender is interested in maximizing fidelity of their entertainment, so you’ve already eliminated half the population. Now remove the 10% that lack depth perception for whatever reason, and another 10% that get nauseated or simply don’t like it. Cut it in half again to get the die-hard gamers. Now we’re talking about 10-20% of the population who might desperately want one of these things. Now you have to make it affordable and compelling. This is looking less and less like a mass-market hit. Niche? Sure. Sustain a company or two? Maybe, sure. A world-changing industry? Extremely doubtful.

Conclusion

I have no idea what the next revolution in computing is, but I can tell you what it won’t be: anything that “improves” something by raising its fidelity while making you look like a goober.

Read the original thread at HN.

October 22, 2013

The best tool we have in computing is the relation. We have exactly two ways of representing a relation. One is a function. The other is a table. What could be more clear than this:

The programming language Prolog is explicitly relational. We would represent the preceeding table in Prolog explicitly with the following code:

% cats(Name, Markings, Disposition)
cats(abby, black_and_white, skittish).
cats(happy, black_and_white, talkative).
cats(cookie, calico, warm).
cats(ben, black, paternal).

This is pretty close to the table, but with a little alignment we can get closer:

% Name         Markings         Disposition)
cats(abby,   black_and_white, skittish).
cats(happy,  black_and_white, talkative).
cats(cookie, calico,          warm).
cats(ben,    black,           paternal).

Or using commas to demarcate columns:

% Name         Markings         Disposition)
  cats(abby  , black_and_white, skittish).
  cats(happy , black_and_white, talkative).
  cats(cookie, calico         , warm).
  cats(ben   , black          , paternal).

A better approach would be to adopt elastic tabs, permitting a much more natural visual representation.

Another way of representing it:

I leave it to other developers to decide which of these is nicer to look at.

A clear side-benefit to using elastic tabs is that the alignment is not based on a stable uniform character width. This means you can use fonts intended for reading, rather than fonts made to waste space.

What’s the downside? Apart from gedit, no editor implements elastic tabstops. The technology remains a good idea without meaningful implementations in common editors.

Why is this unused? I’m not sure. Reading about Emacs’s “smart tabs” it sounds like Emacs itself doesn’t have a good way to do an absolute positioning. Calculating the distance in spaces from one tab to one absolute position is arithmetic, but laying out the page according to regions is not easy. Moreover, the code won’t render correctly in uninformed editors.

How important is correct rendering in other editors? Word documents don’t render correctly in Pages and vice-versa. The more complex the editor, the more likely it is that similar software won’t work identically. Yet programmers seem to hold beliefs that prevent the situation from getting better: we should be able to use any editor to edit the code, and the code should look the same in every editor.

Another thing that would greatly help: literate programming. Unfortunately, in this case, the elastic tabs have been simulated by an unbordered table. In order to export this properly to source code, the table would have to be eliminated in a syntactically acceptable fashion. Better would be writing literate tools that understand elastic tabs. Either way it’s more stuff to code, and for a tiny audience.

September 25, 2013

This is an implementation of Intermediate Challenge #125, “Halt! It’s simulation time!”

The crux of the problem comes down to a table of virtual machine instructions like this:

I came up with the virtual machine abstraction, a 5-tuple machine with values for code, instruction pointer, instruction count, register values and whether or not the machine is halted. Then I defined some functions like get_register/3, set_register/4, set_instruction_pointer/3 which take a machine and some other arguments (a register number, for instance) and then return a value or a new machine with the appropriate change having been made. This is defined in halt_machine.pl.

Then I wrote a parser for the assembler input. It’s quite simple, looking essentially like:

instruction(and(A,B))           --> "AND", whites, int(A), whites, int(B).
instruction(set(A,C))           --> "SET", whites, int(A), whites, int(C).
instruction(jump_if_zero(X, A)) --> "JZ",  whites, int(X), whites, int(A).

The entry point to the parser handles the rest:

instructions([Inst|Instructions]) -->
instruction(Inst),
blanks,
!,
instructions(Instructions).
instructions([]) --> [].
    
program(Program) --> integer(_), blanks, instructions(Program).

A sample program like this:

3
SET 1 15
AND 0 1
JZ  0 1

will thus be parsed into a “program” like so: [set(1,15), and(0,1), jump_if_zero(0,1)]. The assembler is defined in halt_parser.pl.

The table of operations is encoded directly using a custom operator:

:- op(700, xfy, :=).

spec(and(A,B))          :- m(A) := m(A) /\ m(B).
spec(set(A,C))          :- m(A) := C.
spec(jump_if_zero(X,A)) :- (m(A) = 0) -> ip := X ; advance.

Our evaluator is designed to first try to evaluate the instruction by loading and trying to evaluate the specification:

evaluate(M, X, V) :-
clause(spec(X), Body), evaluate(M, Body, V).

So, if I were to try evaluate(M, and(0,1), V) it will expand to evaluate(M, m(0) := m(0) /\ m(1), V). The rest of the rules for evaluate/3 handle the expansion in various ways:

evaluate(_, A,           A) :- number(A).
evaluate(M, m(A),        V) :- get_register(M, A, V).
evaluate(M, A /\ B,      V) :- evaluate(M, A, RA), evaluate(M, B, RB),
V is RA /\ RB.
evaluate(M, A -> B ; C,  V) :- evaluate(M, A, RA),
RA -> evaluate(M, B, V) ; evaluate(M, C, V).
evaluate(M, m(A) := B,   V) :- evaluate(M, B, RB), set_register(M, A, RB, V).

evaluate(M, advance,     V) :- advance(M, V).
evaluate(M, ip := X,    MA) :- set_instruction_pointer(M, X, MA).

The evaluation of m(0) := m(0) /\ m(1) will trigger the evaluation of m(0) /\ m(1), which will trigger the evaluation of m(0), then m(1), then A /\ B as it breaks the expression down into parts and evaluates each subtree, before evaluating m(0) := V to update the register. Special cases such as halted := true and ip := X are handled by calling special procedures to update other aspects of the machine.

The gain here is hard to overstate. Prolog doesn’t have arrays, for instance, but our sublanguage does.It’s easy to verify that the formal specification embedded in Prolog does the right thing, and it’s easy to verify that the sublanguage evaluator does the right thing. I think this technique will probably generalize nicely to other scenarios with Prolog. The rest of the code is visible in halt_assembler.pl.

This module doesn’t export the evaluation function directly; instead, callers are expected to provide the code and allow this module to handle executing it via execute/1 and execute/2, which run the code on a new machine, either discarding the final machine or returning it. The execution is also quite simple:

execute(Code, FinalMachine) :-
initialize(Code, InitialMachine),
execute_loop(InitialMachine, FinalMachine).

The loop is also pretty simple:

execute_loop(Machine0, MachineN) :-
execute_one(Machine0, Machine1),

%% if we're halted, display the message; otherwise continue
((is_halted(Machine1) ; instruction_count(Machine1, 10000))
-> do_halt(Machine1)
; execute_loop(Machine1, MachineN)).

The halt semantics there are as specified; either the machine just executed HALT or we’re at instruction 10,000. execute_one simply finds the next instruction and prepares to execute it. do_halt prints out whether the machine halted of its own accord or if it hit the 10,000 instruction barrier.

The main program is suitably simple:

process_file(File) :-
halt_parser:parse_file(File, Code),
halt_assembler:execute(Code).

May 30, 2013 prolog

First, Some Backstory

When I first started getting serious about Unix around the turn of the century I got really hard into Vim. In 2003 I had already had my first glimpse of RSI so I switched to the Dvorak keyboard layout and bought myself a beautiful Kinesis Ergo keyboard.

For some reason I decided this would be a good time to switch to Emacs, so I did. I never really achieved the fluence I had with Vim after switching from Emacs (though I did lose what I had with Vim). So I’ve slowly grown to resent Emacs over the years, and I made a point of trying out new editors pretty frequently.

Most recently this culminated in trying out Textadept.

Let me take a break here to make something really clear.

Textadept Rules

No, seriously. It does. Everyone should try it. It has a console mode, sensible modern keybindings, and it’s scripted in freaking Lua. It’s really awesome. Use it. Love it.

So it’s a real shame that I continue this post by telling you how I set up Emacs and how amazing it is, finally, after basically a decade of ignoring it, to actually use it.

Emacs Basics

My life with Emacs utterly sucked until I found out about the following settings. I’m going to explain them as I go.

If you don’t set this Emacs may think you’re some kind of ISO-9660-1 jackass.

(prefer-coding-system 'utf-8)

While learning Emacs, it’s going to beep at you a lot. This is infuriating. Disable the bell.

(setq visible-bell t)

Emacs by default thinks that you use the computer as if you emerged from some kind of 1970s typing class and your instructor took points off if you put a single space after the period. But it’s 2013, and nobody does this, and nobody should, because everything that matters either ignores the extra space (HTML) or typesets properly despite it (TeX). So fix it.

(setq sentence-end-double-space nil)

Impossibly, it will tell you the current line but not the current column by default. Fix it.

(setq column-number-mode t)

Emacs loves to tell you that it is Emacs and how proud you should feel for using it, so you actually have to modify bunches of settings to really make it clear to Emacs that you just don’t care about it or its feelings.

(setq inhibit-startup-message 't)
(setq initial-scratch-message "")
(setq inhibit-splash-screen t)

Emacs defaults, in a depressingly predictable manner, to Lisp mode, as if that’s something you give a shit about. Let’s make it sensible:

(setq default-major-mode 'text-mode)
(setq initial-major-mode 'text-mode)

Of course you could put a language mode in there if you mostly edit a particular language, but I find I usually have no idea what I’m up to when I start it up so text mode is more honest. Fundamental mode is even more useless than Lisp mode to me.

Now let’s disable those hideous backup files.

(setq make-backup-files nil)

Emacs by default is configured to be some kind of idiot savant about indentation, switching between tabs and spaces as necessary to achieve the layout of your dreams. In practice nobody wants this ever, so shut it down:

(setq indent-tabs-mode nil)
(setq tab-width 4)

Now global key bindings are a treacherous territory but I think this one is completely obvious. Every other editor on earth proudly advertises that it will indent properly by doing it after a newline, so Emacs should too:

(global-set-key (kbd "RET") 'newline-and-indent)

Finally, because Emacs has more keybindings than you can ever hope to master, let it tell you when it has a quicker way to do something than the M-x long-ass-name abomination you just used:

(setq suggest-key-bindings t)

A setup trick

So it happens that I have a fairly complex configuration now, but most of my settings are either host-specific or mode-specific. My ~/.emacs.d/init.el is actually very short:

;; load all my initialization files
(dolist (file (file-expand-wildcards "~/.emacs.d/init/*.el"))
  (load file))

;; load custom per-host files
(let ((host-file (format "~/.emacs.d/hosts/%s.el" system-name)))
  (setq custom-file host-file)
  (if (file-exists-p host-file)
      (load host-file)))

This file is really establishing two naming conventions: one for all my mode-specific crap and one for all my host-specific crap. The custom-file trick there ensures that if I make the mistake of using M-x customize the changes wind up cordoned off in the per-host file.

I’m sure other people have equally clever tricks, but this lets me use one repository for one Emacs configuration that varies slightly on each host. This is super handy because normally I’d make a small change to, say, the work configuration and then lose it on the way home or, possibly worse, go home and manually re-apply it elsewhere, slightly differently. Now by simply choosing the right file I can shared things that should be shared or I can sequester changes off in the per-host file that I don’t want shared.

Another quick tip

Hey! Refuse to use Emacs < 24.3. Just say no. It’s bad enough you’re learning Emacs, dealing with lots of different Emacses would be even worse.

Packages and Themes

Be sure to use the new package system for your own sanity:

(package-initialize)

This lets you install stuff directly from Emacs. Give it a shot:

M-x package-install RET auto-complete RET

Tricked you! You just installed magic Emacs tab completion! Now add this to your ~/.emacs.d/init/autocomp.el file, or whatever:

(add-hook 'after-init-hook
  (lambda ()
    (require 'auto-complete-config)
    (ac-config-default)))

Restart Emacs and enjoy the fact that you now have freakin’ tab completion in there.

Oh, and get you a proper theme: M-x load-theme RET misterioso. On the house. Or you could go get solarized with package-install.

mu4e

Your next question is going to be, but Dan, why should I use Emacs if it isn’t even a decent MUA?

Well my friend, it actually is a great MUA, using the elaborately-named mu4e. This mailer has the #1 most-needed feature of any mailer, which is that it is search-based. I could write at great length about how nice it has been, using mu4e, but you should just try it out yourself. It’s really nice. Tab completes addresses, ido-like folder choosing, and fast as hell. Emacs 24+ can connect directly to your SMTP host, so there’s no excuse there. Set up offlineimap to get your mail and get on with your day.

This can’t really be why you switched

No, and it isn’t. Unfortunately, nxml-mode is. It’s really fantastic at editing XML. I’m sure you didn’t want to hear that. I recently had need for editing some Docbook. Docbook is a ferocious XML format designed to kill your hands. However, Emacs makes it a lot more tolerable with nxml-mode, because while you’re working on your XML file it’s constantly telling you just how invalid your document is. And it has some shortcuts for inserting partial tags and completing tags. And since it understands RelaxNG, it also will make autocomplete suggestions for you.

It’s worth noting that it really does understand RelaxNG. You can design your own schema from scratch and it will give you context-sensitive completions.

Misc

I’m just starting to use org-mode so I can’t give you a proper endorsement there, but suffice to say, it’s really handy to be able to make links to email messages and files on your filesystem to embed in your todo list.

Conclusion

So there you have it.

Note that this is not an endorsement of Emacs. Please go on using Vim or Textadept as you please.

May 7, 2013

The old ORM chestnut is back and we’re seeing the usual mixture of defenders and aggressors. But why is there such a divide?

The argument is fundamentally about choosing what’s in charge of your system, the system being composed of your databases, your applications, and your supporting infrastructure (your scripts, your migrations, etc.) To relational database folk such as myself, the central authority is the database, and our principal interests are what ACID exists to provide: concurrent, isolated, atomic transactions that cannot be lost, on top of a well-defined schema with strong data validity guarantees. To us, what’s most important is the data, so everything else must serve that end: the data must always be valid and meaningful and flexible to query.

The side that argues for ORM has chosen the application, the codebase, to be in charge. The central authority is the code because all the data must ultimately enter or exit through the code, and the code has more flexible abstractions and better reuse characteristics.

It comes down to disagreement about what a database is about. To the OO programmer, strong validation is part of the behavior of the objects in a system: the objects are data and behavior, so they should know what makes them valid. So the OO perspective is that the objects are reality and the database is just the persistence mechanism. It doesn’t matter much to the programmer how the data is stored, it’s that the data is stored, and it just happens that nowadays we use relational databases. This is the perspective that sees SQL is an annoying middle layer between the storage and the objects.

To the relational database person, the database is what is real, and the objects are mostly irrelevant. We want the database to enforce validity because there will always be tools outside the OO library that need to access the database and we don’t want those tools to screw up the data. To us, screwing up the data is far worse than making development a little less convenient. We see SQL not as primarily a transport between the reality of the code and some kind of meaningless storage mechanism, but rather as a general purpose data restructuring tool. Most any page on most websites can be generated with just a small handful of queries if you know how to write them to properly filter, summarize and restructure the data. We see SQL as a tremendously powerful tool for everyday tasks—not as a burdensome way of inserting and retrieving records, and not as some kind of vehicle reserved for performance optimization.

At the end of the day, we need both perspectives. If the code is tedious and unpleasant to write, it won’t be written correctly. The code must be written—the database absolutely should not be running a web server and servicing clients directly. OOP is still the dominant programming methodology, and for good reasons, but data encapsulation stands at odds with proper database design. But people who ignore data validity are eventually bitten by consistency problems. OODBs have failed to take off for a variety of reasons, but one that can’t be easily discounted is that they are almost always tied to one or two languages, which makes it very hard to do the kind of scripting and reporting that invariably crops up with long-lived data. What starts out as application-specific data almost invariably becomes central to the organization with many clients written in many different languages and with many different frameworks.

ORM is destined to be hated, because the people who love databases don’t love the practical albeit leaky abstraction offered by ORM, and people who hate databases will resent how much effort the better ones require to be used properly.

Some thought experiments

Consider this hypothetical scenario. You run Facebook, and you have all the software and all the data. A catastrophe occurs and you lose everything, and due to a mistake in the way backups were made, you can choose to restore the software or the data, but not both. (Somehow, restoring the software will destroy the data, and restoring the data will destroy the software). Which one do you choose?

Of course, this scenario is unlikely, but it should serve to demonstrate that to the rest of the organization (the non-programmers), the code is secondary to the data it manages. You can rewrite the code, but you can’t always recreate the data. When you look at our database backup, migration and refactoring tools and compare them to what we have for source code management, it’s clear that we spend more time worrying about the code than the data. That’s not inappropriate (we work on code all day) but it can lead to a myopic view of the importance of the data.

Another thought experiment posed by jshen on HN points out that data validity is secondary to monetization, and that if the business finds a way to increase monetization while causing a low rate of data corruption, it may be worth sacrificing validity. This is a fair point, and I think this illustrates why NoSQL is a winning proposition for many companies. If scalability and speed are more valuable then they can represent a better choice—especially if the data is cheap or can be recreated without too much trouble (or on demand).

(This article was derived from this comment I left on HN).

May 15, 2012