Posted by Daniel Lyons
Fri, 02 Jan 2009 08:08:00 GMT
Nearly everyone in my life and several people not in my life replied to my last post. Several of my closest friends have contacted me personally on the side. I feel I owe the people who take the time to read what I have written a few words.
First of all, much of what was taken personally as indicative of some kind of internal depression was actually a failed attempt at humor. I’m out of the habit of writing blog posts at 4 AM. Consider it a misfire. Everything in my life is going surprisingly well. The exaggerations “jpc” detected were just meant to be funny.
Secondly, and mainly to Dan, I’m afraid you’ve interpreted the coincidence of a mention of Microsoft and an indictment of management in general as related. Microsoft is a complex situation. I was speaking in broad strokes. I’m amused at your mention of a change of perspective. Apparently it has escaped your attention that I am half owner of a company. However I think that management’s interest in the managed is a lot like the psychological discussion of altruism. A simpler explanation works in the best and the worst cases, especially in business. Every aspect of management you named as being relevant could be adequately motivated by ensuring better long term stability and hence, more money in total (over the long haul). But this is just splitting hairs on my part.
The question of conviction and “doing what you want to do” is really the crux, as that is exactly a restatement of the emotional problem. When it comes to passion, there isn’t much you can do to create it where it doesn’t exist. However, you cannot have hurt feelings without having feelings to begin with.
“Descartes’ Error presents a very simple thesis: You perform a surgical ablation on a piece of someone’s brain (say, to remove a tumor and the tissue around it) with the sole resulting effect of an inability to register emotions, nothing else (the IQ and every other faculty remain the same). What you have done is a controlled experiment to separate someone’s intelligence from his emotions. Now you have a purely rational human being unencumbered with feelings and emotions. Let’s watch: Damaslio reported that the purely unemotional man was incapable of making the simplest decision. He could not get out of bed in the morning, and frittered away his days fruitlessly weighing decisions. Shock! This flies in the face of everything one would have expected: One cannot make a decision without emotion… Psychologists call [emotions] ‘lubricants of reason.’”—Fooled by Randomness, Nassim Nicholas Taleb, p. 203.
I think everyone who can should read this book. The author doubts it will have long-lasting effects on the readership and this is probably true but one can always hope. I take some comfort in being the person taking the opposite advice of the book. Not that I am perfectly rational but I certainly seem to exhibit more of the flaws of thinking than the benefits. On the other hand, the book certainly diminishes the value of advice. None of us are doing a particularly adequate job of accounting randomness in our advice. However, I suspect that those like me in a state of less than stellar financial success are probably correctly attributing a portion of our condition to randomness. But I think others like Joel who are in a state of financial success should be sure to include a portion of randomness in that, yet they do not. This is more or less exactly what I said in my article on DHH last year:
“Lots of companies achieve success, make a blog, and rant about their awesome formula. Lots of fans arrive, read the blog, and start to copy the formula. You shouldn’t, though, because mixed in with “good” companies are bad companies with crap products. They’re just as successful. DHH believes that his formula is perfect and it’ll work for everyone, but he has no way to know which parts of his formula are brilliant and which parts are bullshit…Hating users is not a road to success. It’s an obstacle to success that, in DHH’s case, was apparently not enough of one to stop it from happening.”
In retrospect, at that point in time I did not understand 37 Signals’ history; it’s not really appropriate to heap that much on DHH himself. DHH is an employee of 37 Signals. Much of my criticism holds, I think, because the rest of the company has never decried this, usually agrees with it, and just isn’t as quotable (or as much of a cult phenomenon.)
I hope someday I can also be a perpetrator of this phenomenon rather than merely an agitator about it. But for the time being I am content with the role I have landed. I feel for the first time in four or so years, that I can pronounce the cure for my particular form of burnout is: rediscover an emotional basis and attempt to limit the available number of choices. I am not quite sure how to implement this. I suspect it is to finally cleave apart the previously linked aspects of my life: divorce my income source from my hobbies; divorce my artistic impulses from my business sense; in short, divorce my self-worth from programming and possibly from my income source. This isn’t a problem for anyone I admire.
The first step in the direction of change for me has been making a change to my /etc/hosts to redirect reddit.com and programming.reddit.com to 127.0.0.1, where I have put up a support page for myself with a couple encouraging words and a link to Wikipedia. Reddit, previously, has accounted for 2-4 hours of my time a day, reading and keeping up-to-date on things like the Ruby community and the Lisp community. Michael and Liz think this is self-destructive and I think they are right. Nassim Taleb notes in his book that he stopped allowing most forms of news into his life because it was a distraction and raises the signal-to-noise ratio of life. For the past two days I have been reading Wikipedia in my down time while I wait for things, time I normally would have gone to Reddit and then read a blog entry or started or continued a pointless debate about programming. I have three observations about it so far:
- Reading Wikipedia is more exhausting than reading a blog entry.
Actually, reading just about anything, takes more energy and attention than reading Reddit. Marshall McCluhan, according to what I remember of what Michael told me about his philosophy, would therefore categorize it as “cool media”: the kind of thing that sucks you in because it requires less of you. Consequently, doing anything other than reading Reddit is more like work than escape and more got done, even though I was sick, yesterday.
- I seem to be in a better mood.
Except I have been getting into nasty debates with my mother more than I was before. I think it’s too early to say whether or not this is just withdrawal. And yes, I am treating this as a kind of rehabilitation from a destructive habit.
My thinking initially was that after a week or so, I would have established new habits and could remove the block on reddit.com from my /etc/hosts file, but it doesn’t look like this week will have been structured enough to create a new habit.
I am also thinking about having a book with me for moments of downtime, or a slinky. I think news.google.com would be more of the same, with the possibly small benefit of not having a place to comment.
Michael asked me what I enjoy about programming and I had some trouble answering. I came up with database design and data migration as being intrinsically enjoyable. He told me about the joy of talking to his customers and hearing that they enjoy using his products. I reflected that I have been denied that pleasure for the past several years, owing to the fact that my clients have not been the ultimate users of my software. I think this alone might account for some of the problems I have experienced. Clients are negative about bugs, but a lack of them is merely expected. The economic incentive to produce a feature is less emotional than the desire to share a positive experience with one or more people that one has a positive, enabling relationship with. I wonder if contract/third-party development produces more burnout than other kinds for exactly this reason. This is a treatable problem.
So, thanks to all who wrote or commented on the previous post. Your thoughts and concerns have been on my mind. All is not lost. If you’ll tolerate a platitude, let me say that a feeling of being aimless at sea indicates a need for a new compass, not a new boat.
Tags emotion, programming | 6 comments
Posted by Daniel Lyons
Mon, 29 Dec 2008 08:58:00 GMT
This is reworked post from Reddit with some editing.
“Bob” posted on Joel’s forum, asking:
“I’m thinking of leaving the industry. What is a good industry to get into where your programming skills would put you at an advantage?”
To which Joel replies with a bunch of snarky words, basically telling Bob to shut up and eat his broccoli, closing with the tasteless remark “I don’t get the negativity in here. How did the Joel on Software discussion group turn into a mutual mope-fest for angsty emo girls.”
RoboticCoder on Reddit replies, offering the definition:
“It seems to me the industry is split into two groups, The programmers, who started out programming long before they ever thought of it as a career choice i.e. the bedroom coders, the geeks :) And then there are the monkeys, the people who fell into the computing field because it was seen to be a boom industry a few years ago, they learned some basic programming and got their degree or <insert other qualification here>.”
A guy named skinnybuddha replies to that, saying:
“I think monkeys are the ones who complain the most. I would not program for free, because I can find other interesting things to do that would pay more than 0. I find that programmers are treated pretty well by management, monkeys are seen as a commodity to be exploited.”
I think skinnybuddha basically got it backwards, and it’s all based on a couple of simple axioms:
- “Monkeys” as he calls them are optimizing for money. “Programmers” are optimizing for fun.
- “Programmers” spend a sizable portion of their free time learning more about programming.
- The Paradox of Choice is real.
Because of #1, “Monkeys” don’t investigate other languages and platforms, accumulating knowledge about alternatives that they are then prevented from putting into practice at work. They don’t need to do this to make more money.
“Monkeys” are more successful because it’s easier for them to succeed. This is because they happened to have selected a quantifiable value for success that is both easy to maximize and widely accepted as a definition of success: money. It’s a lot easier to tolerate a job that earns you a lot of money yet is boring if your primary goal is making money. Every company tells you what you’ll earn before you agree to start. Whether or not it is boring or interesting, a fact of greater importance to “programmers” than “monkeys,” can only be observed obliquely before accepting the job. Even defining success as something other than making a lot of money represents a choice. Every time you can’t do something because of a lack of money, the “programmer” will reconsider that choice and suffer some buyer’s remorse. A “monkey” will seldom experience buyer’s remorse for the one decision they did make, where to work, because if they simply selected the highest-paying job, there is nothing to clutter that question. Fun being a difficult concept to quantify, the “programmer” will be more susceptible to buyer’s remorse about the job they selected, compounded by their decision to optimize for fun rather than money.
Additionally, some “programmers” develop a distaste for popular platforms and languages, thus finding opportunities rarer. Rarer opportunities translates into higher compensation when danger is involved or the need is very high, but, generally speaking, programs can be written on any platform so denial of the popular platforms translates into additional requirements being imposed on the employer by the employee. Obscure platforms are a benefit to the employee, not a cost to the employer, so compensation goes down and the elusive fun factor (theoretically) goes up.
“Programmers” who actually enjoy programming the common platforms and languages are more likely to be in high demand and thus have more opportunities to distrust their own decisions about where to work. It’s easy to maximize compensation, and hard to maximize fun. You will always have buyer’s remorse for those other jobs you passed up on, whose enjoyability is hard to measure. And, whenever your job is unfun, you’ll be thinking about how you could instead be making ten times more money at that other unfun job you passed up.
As a thought experiment, consider a supermarket. When there are only two checkout lines to choose from, you don’t feel as bad when you happened to have gotten in the slow one as when there are ten open. When there are ten open, you will invariably compare your plight to the happy person in the fastest line. You’ll only consider the people in the average line if you happen to be in one of the 5 lanes that’s slower than average. But even if you’re in the second-fastest line, you’ll feel like a schmuck for not getting in the fastest lane.
Compound the problem by not being able to compare the speed of the lanes directly. If you care very deeply about the speed of the lane, but can’t measure it, you will be unable to escape the conclusion—rightly or wrongly—that every delay in this lane is of great importance. You will feel excruciatingly awful. The best way to solve this problem is simply to stop treating it as a problem, and stop optimizing this variable before you go crazy. Easy advice to administer, but hard to take.
Programming is also one of the most stressful occupations. Stress plus little money is a recipe for health problems. The best “programmers” are the most likely to have programming as an avocation as well as a vocation. While our “monkey” friends spend their weekends running marathons and lifting weights, we’re in front of computer screens getting and staying fat and unhealthy. The exercise they’re getting is making them live longer, be perceived as more attractive to mates, and getting them laid more. The extra money they’re making and their stability (i.e. boringness) probably helps in that department too. Meanwhile, we are learning new languages and platforms: adding choices to our life which we will either be unable to enact in the workplace, or which we will kill ourselves by agonizing over when we have the option. All of this will cause us to feel like artists, but won’t help us be perceived as artists. The suffering leads to more eating and depression and ultimately we die alone.
From a business perspective, remember that management also prefers “monkeys” over programmers, because management is taught to regard people as human resources: labor-producing machines with sets of skills. It’s much easier to replace a person with a common skill than one with an uncommon skill. Managers are also scrutinized by the quality of their decisions; no manager was ever fired for choosing (what’s perceived to be) the safe bet. The ready supply of X programmers gives them both the illusion that development is like any other skilled labor which people can be swapped into and out of, and the feeling that they’re choosing a safe choice.
The cool companies that let you use whatever technology you want tend to be run by programmers rather than business people. Companies that are good at business tend to grow and thus hire more middle management, who bring with them business cargo cults such as human-resources-as-a-commodity. Businesses that aren’t as successful don’t, and that’s the place “programmers” want to work. Such places also tend not to compromise on technology for business reasons, which stifles profitability—and exists in an environment in which business and profitability are seen as contrary to good programming. This translates to less money and fewer customers. “Programmers” are expected to endure this because of the sheer joy of programming, which becomes a factor in compensation. People doing fun jobs because they’re fun aren’t likely to make compromises that improve the bottom line.
A side-effect of this is that profitable companies have diminished abilities to differentiate “programmers” from “monkeys.” They’re just HR. There’s also a strong counter-incentive from above to make a distinction, since it would increase hiring costs and trouble, and would give more power to the “programmers,” who aren’t perceived as being particularly important to the profit equation.
I am sure that there exist exceptions to these principles, but they are exactly that and I would expect them to be correspondingly rare and short-lived.
Maybe this job just isn’t for everyone. But I’m starting to think this job is for precisely the people who shouldn’t be doing it in the first place, and not those who have a passion for it or a natural gift. I personally think I would enjoy it more if it went back to being my hobby. But I know so much now it would be ridiculous not to try and make money with it, and retooling myself for a different occupation sounds like a great way to give myself and even bigger helping of the paradox of choice than I already have, which is pretty significant.
If you’re a programmer and you’re happy, my advice to you right now, if you value your happiness, is to throw out the Pragmatic Programmer’s adage of learning a new language every year. Don’t do it, man; put down that language. It’ll only end in tears. Maybe not this year, maybe not next year, but eventually, it’s going to catch up to you. You’ll sit down to start a new project and ask yourself, what language should I do this in? And you’ll spend the next two or three hours trying to decide without making a conclusion, except that every language sucks.
As to Joel, he’s internet famous, which means almost nothing. Making it in New York doesn’t mean much to me; I consider him a schmuck who got lucky and a person who ought to read Fooled by Randomness a couple more times. His lack of empathy towards someone on his own board is disheartening.
Of course, it all adds up if you take it from a business perspective. it’s not in his business’s best interest to have a list of great career options for the exiting programmer on his board. He hires programmers, after all. It’s important to understand that Joel is a businessman first. Drama about his blog is just a fancy way of getting traffic to his website, to raise awareness of his software which is coincidentally marketed to developers. Unfortunately, knowing about marketing doesn’t seem to be enough to inoculate a person against it. Everyone should take any mention of Joel as a flashing red neon sign reading “BUY FOGBUGZ!!”
Tags industry, programming | 5 comments
Posted by Daniel Lyons
Wed, 24 Dec 2008 07:13:00 GMT
I don’t like to go into non-programming territory too much on this blog but I just finished reading a cool book, The Lost Messiah about the Jewish “messiah” Sabbatai Zevi.
It’s an interesting story, the brief summary being that in the mid-late 1600’s a Jew claimed to be the Jewish messiah, convinced a large portion of the Jewish world that this was the case and started a messianic branch of Judaism before his conversion to Islam.
Surprisingly (or not, if you study sociology of religion) Sabbatai’s conversion to Islam did not end his movement. In fact, he continued to lead the movement as a Muslim, asking some of his followers to convert and others to remain Jewish. A part of his message included the reversal of certain bans in Judaism so this was, strangely enough, justifiable with his interpretation of Kabbalistic mysticism. Conversion to Islam represented a descent into the qelippot to recover divine sparks from the very basest part of creation.
The followers who remained Jewish albeit messianic came to be known as Sabbateans. Orthodox rabbinic authorities came down pretty hard on them, so Sabbateanism was practiced secretly. Other followers who had embraced the “great tikkun (repair)” of conversion to Islam are now known as Selânikli (“from Thessaloniki”) Interestingly, this means that one religious figure actually created and separated his followers into different religions, intentionally.
There are a lot of interesting angles to this story. Most interesting to me is the similarity between the Selânikli community and the local Conversos, the Jews of Spain who were forced to convert to Catholicism before eventually settling here in New Mexico. Superficially, there are many similarities: they were both groups that were outwardly one religion and Jewish secretly, both groups were popularly known by a strongly negative name (the Conversos were called marranos, “pigs”; the Selânikli were called Dönmeh, “turncoats”).
From the perspective of religious sociology, it is interesting that these are groups which have always existed in high tension with their surrounding cultures. The Selânikli weren’t really quite Muslim enough for their neighbors, nor were they natively Turkish. Sabbateanism, due to its divergence from Judaism, held some fasts to be feasts, mentioned the name of the savior with some frequency and had other aspects which made them not really quite Jews in hiding either.
As time has marched on, the secret practices of both groups diminished. Most of the Conversos I have spoken with have just a few memories of things which their parents or grandparents said or did from time to time. Things like never eating pork. Indicative, but not much of a practice by itself. Modern Selânikli, according to the book, have very little knowledge about Sabbateanism or Sabbatai Zevi the person at all. Most are simply practicing Muslims, much like most Conversos are simply practicing Catholics.
Yet, due to their separateness, both groups retain a very strong self-identification as other. This identity has internal and external sources. Internally, they know they are different, as well as being excluded from things and perhaps even denigrated externally. Every secret is really three pieces of information: the secret itself, the reason for keeping it secret, and the knowledge of where the secret ends and public knowledge begins. In the case of the Conversos, knowledge of the reason for keeping the secret has been pretty well preserved. The secret itself, however, has been very poorly preserved. There is a lot of knowledge to practicing Judaism. With only oral traditions to preserve them, the practices have become quite fragmented.
It’s been quite interesting attending Sephardic services with my brother recently. His friends, the Conversos, struggle with the meaning of their identity despite their identity being quite certain. For Nathan and I (and most other converts), identity isn’t quite so solid. It’s predicated on action and behavior for us. Without behavior, there is no way to create memories which can later on add up to an identity. Without behavior, it’s just philosophy.
The Converso community is trying very hard to act, to gel itself and become a force of some kind in the greater Jewish community of New Mexico. They meet with a lot of resistance, and I think most of that resistance is internal to their own community. They have a Jewish identity, but many of them also have a Catholic identity. Catholicism and Judaism aren’t miscible. Each individual has their own limits and their own vision for what they would like to do. These visions aren’t necessarily shared, but every person in the community has the utmost respect for the other members of the community. Nobody wants to step on anyone’s toes.
I find it fascinating that there is an unrelated community in Turkey going through much of the same pain.
Tags conversos, judaism, religion, sabbateanism | no comments
Posted by Daniel Lyons
Mon, 22 Dec 2008 20:59:00 GMT
Today, the first day of Chanukah, it printed out the usual welcome screen with one small change:
i
. . . . I . . . i ooooo o ooooooo ooooo ooooo
I I I I I I I I I 8 8 8 8 8 o 8 8
I I \ `+' / I I 8 8 8 8 8 8
I \ `-+-' / I 8 8 8 ooooo 8oooo
\ `-__|__-' / 8 8 8 8 8
`--___|___--' 8 o 8 8 o 8 8
| ooooo 8oooooo ooo8ooo ooooo 8
--------+--------
Welcome to GNU CLISP 2.43 (2007-11-18) <http://clisp.cons.org/>
Copyright (c) Bruno Haible, Michael Stoll 1992, 1993
Copyright (c) Bruno Haible, Marcus Daniels 1994-1997
Copyright (c) Bruno Haible, Pierpaolo Bernardi, Sam Steingold 1998
Copyright (c) Bruno Haible, Sam Steingold 1999-2000
Copyright (c) Sam Steingold, Bruno Haible 2001-2007
At first I thought I was hallucinating, but I set my clock forward to tomorrow and it gave me this banner.
i
. . . . I . . i i ooooo o ooooooo ooooo ooooo
I I I I I I I I I 8 8 8 8 8 o 8 8
I I \ `+' / I I 8 8 8 8 8 8
I \ `-+-' / I 8 8 8 ooooo 8oooo
\ `-__|__-' / 8 8 8 8 8
`--___|___--' 8 o 8 8 o 8 8
| ooooo 8oooooo ooo8ooo ooooo 8
--------+--------
Welcome to GNU CLISP 2.43 (2007-11-18) <http://clisp.cons.org/>
Copyright (c) Bruno Haible, Michael Stoll 1992, 1993
Copyright (c) Bruno Haible, Marcus Daniels 1994-1997
Copyright (c) Bruno Haible, Pierpaolo Bernardi, Sam Steingold 1998
Copyright (c) Bruno Haible, Sam Steingold 1999-2000
Copyright (c) Sam Steingold, Bruno Haible 2001-2007
Two candles lit. The CLisp authors included the intimidating calculations necessary to determine whether or not you are in Chanukah and which night it is just to print out ASCII candles on the chanukiah.
Thank you, CLisp authors! What a wonderfully kind thing to do.
Tags chanukah, lisp | no comments
Posted by Daniel Lyons
Tue, 16 Dec 2008 06:20:00 GMT
Today I was lucky enough to see a video of Barry Schwartz talking about the paradox of choice. This really struck a nerve for me, because it really nicely explains the problems I’ve been having, more or less, since college.
You see, the problem is that once you have more than a certain number of choices, you are bewildered by the possibilities. Like trying to get in the line that goes the fastest at the supermarket. Your odds of picking the fastest line are actually 1/N, where N is the number of lanes. Your odds of picking the best anything are the same, presuming you have enough criteria to make a total ordering.
The result is what Josh used to call “analysis paralysis.” And suddenly a lot of things are making sense that didn’t used to. Especially the content management situation. Ordinary humans are only going to evaluate one or two. Knowing the ins and outs of a handful or more leads you to believe they’re all crap. Same story with programming languages. Once you know (to some degree or another) about thirty languages, it can be quite hard to pick the “right” one for a given problem.
These weren’t ever really big problems for Michael, and he’s much happier as a result. He didn’t wonder if he should use Lisp or Ruby or PHP or Python or Haskell or Erlang or something weirder for his website. He had PHP on the server, so he was going to learn and use PHP. He didn’t wonder if he should use PostgreSQL or MySQL; he had MySQL on the server and so did the clients. Michael just recently managed to underbid my company by an astonishing amount of money—not because he cheated or we overbid, but because he has three years of infrastructure built on his CMS Voltaire, and we don’t have any infrastructure built on anything so it would all have to be custom-written. We have spent so much time trying to decide, we have no reusable infrastructure.
People using PHP routinely spend, say, an hour or two doing something that would take me 10 minutes or so in a modern programming language like Lisp. But it doesn’t matter, because the other 7 hours of that day, they were making progress and I was reading Reddit. Trying to make sure I don’t make the wrong move, instead of moving in any direction at all.
The worst part is, even knowing this doesn’t seem to be enough to solve the problem. After listening to Barry’s talk, we sat there trying to decide what to do about it. I’m going to have buyer’s remorse no matter what. Our programming tools are fabulous now, so great we’re completely befuddled.
I still think unnecessary abstraction is a big part of the problem. All those layers of abstraction, they all have names, they all have different purposes and ultimately, they all add not just intellectual burden in terms of learning them, but tremendous intellectual burden in terms of comparing them to decide on one.
An often-heard complaint about programmers is that we take our tools too seriously; carpenters don’t develop preferences for saws or hammers. But this view is quite outdated when it comes to programming, just because, in programming, all the tools have the same power if they are Turing-complete. You simply cannot do the same things with a hammer and a saw. And hammers and saws don’t expand in capability over time. But programming languages and abstractions do expand over time.
I believe today that a major function of CMSes and programming language frameworks is simply to remove choice to enhance productivity. I doubt now that Merb will significantly affect Rails’s popularity, simply because Merb forces you to make more decisions than Rails does. ActiveRecord doesn’t need to be a great ORM, it just needs to be good enough to help you avoid learning SQL, to keep you from making another decision. Merb will be there for Rails the way the BSDs are there for Linux users: a better alternative that requires more thinking.
I am still torn on what my company is going to use for our major platform but there can’t be any doubt that a major platform is needed, if only to create an environment in which code sharing can occur so as to compete with Michael. My major choices are really PHP and Lisp. My fear in choosing PHP is that the language is too weak to express the things that I’ll be capable of thinking of and that programming in it will feel like drudgery. My fear in choosing Lisp is that tool support will be poor making it hard on Reid, that it will make it hard to sell, and, not insignificantly, that if I fail at it I will have to take more responsibility. But it seems clear that there will be no forward motion until there is a platform to be dedicated to (and stuck with), and only then will buyer’s remorse really begin to set in.
It would be nice if Barry had some suggestions on how to fix that but I don’t think he did. But this may be the root of my programming despair—and why Michael has succeeded and been much happier. The choice of tools actually may have been the problem.
Tags philosophy | 6 comments
Posted by Daniel Lyons
Wed, 10 Dec 2008 06:45:00 GMT
It’s really interesting to me to see these two blog entries on the same day: Consenting Adults and FFMpeg strikes, via Does FFmpeg Actually Suck?.
There’s a nice dichotomy at work here. On the one side, you have engineers complaining about a lack of release engineering leading to reduced code reliability, and on the other side you have forward-looking programmers belittling the stifling effects of stagnant APIs.
I’m not a Microsoft defender by any stretch of the imagination, but I am actually going to give some credit to them this time. I think we’re going to have to ask ourselves what kind of community we want to participate in. If we change APIs radically every year or so, it’ll be hard for specific applications to grow and mature, because the floorboards are changing underneath them. In other words, if my application is built on certain libraries and those libraries APIs become incompatible, what’s going to happen to my app? This has a strong negative effect on certain kinds of innovation.
I suspect the open source movement’s developers haven’t noticed this much because (obviously) few of them are trying to build large closed-source applications. If the whole environment is constantly changing, the right way to build is by cooperatively giving back all your improvements. That way, others become dependent on your APIs and your APIs are more likely to survive and become a stable part of the ecosystem. If your app is closed-source, the APIs you use don’t get counted (unless you’re very vocal).
In a closed-source world, backwards-compatibility is very important, because whatever you have had to develop was just for yourself. It’s your intellectual investment. If it breaks, you have to spend time fixing it. The time you spend fixing it eats into the profit of your investment.
Apart from any other concerns, I don’t see how you can maintain a closed-source investment on a open platform. Even if it doesn’t change particularly fast, it will change, and that change will slowly eat your investment. Microsoft has to spend a great deal of resources to ensure that the changes they make do not eat your investment, and it isn’t perfect. It’s sort of like the opposite of inflation: instead of the government printing money to steal yours, they’re spending lots of money to avoid destroying your intellectual investment.
For a long time open source developers have worked hard to make reliable software with meaningful version numbers, oftentimes more meaningful than the version numbers in commercial competitors. Some projects have release schedules based on the calendar, others based on completeness, but most have a release schedule based on meaning. Almost all use version control systems.
My biggest concern is change over time. As a species, we’re great at handling “right now” and we’re OK at handling last week. We’re also pretty good at handling dependencies. But we don’t seem to be very good about handling dependencies that change over time. Databases are also notoriously bad at storing and querying this kind of data (read the chapters on this in Celko’s “SQL for Smarties”). The fiasco I’m picturing is the one in which everyone is using GitHub and building software dependent on current versions of forks of other people’s unreleased software.
I actually believe this can be done, and work. But if it is, we will need to be religious about recording all dependencies and versioning, somehow. We will need to be religious about making a clear separation between versions with API changes. It’s going to have to be a lot like the game World of Goo, building connections between things that shift and trying to keep the connections solid.
I think the responsible thing for every developer to do is, really, just to document what they’re doing and make sure they’re not using unstable interfaces and making it clear which interfaces they provide are stable and which aren’t. Apart from that we have to work hard to keep our APIs small and tidy and try to get them right the first time. This is common sense. If we don’t work at these concerns, the speed of OSS development will only be matched by the pain as it all folds. Ask any Gentoo user from way back, and they’ll tell you just how bad things can get. Or anybody who’s familiar with RPM dependency hell. It’s hard to get this stuff right.
And even if you do get it right, there is still the money question. It’s pretty easy to get a bunch of people to pay $20 for a program (or $20 a month for a subscription to a program) and make a ton of money. Getting corporate sponsorship for your project is hard. Programming being such a specialized and difficult discipline, it’s going to be hard to keep the rank-and-file moving without financial incentive. Maybe I’m revealing too much about myself here, but my impression is that most programmers just aren’t making the crazy salaries people think we are, yet we have one of the most difficult and stressful jobs in existence. Open source development is detrimental to the most obvious revenue models for programming. Doing the right thing or the fun thing is great incentive to a point, but we all have to eat. Even if closed source programming is worse in every other way, it will be around forever because of this one reason. Unless someone can come up with a valid revenue model that works, even on the entrepreneurial scale.
2 comments
Posted by Daniel Lyons
Mon, 08 Dec 2008 08:37:00 GMT
Someone at the sprint (I think it was Bill) mentioned someone is writing a replacement for cfengine in Ruby called Puppet. He said it looks cool, but there’s a very important feature the developer has chosen not to implement, namely, the editing of files.
Now, I don’t want to shit on Puppet, because I haven’t used it. But I do have a problem with this mentality, and I think it’s very representative of Ruby in general. Don’t like something? Reimplement it in Ruby. But only half of it. Do the sexy part, make the sexy DSL. This pisses me off. Why can’t you have sexy, and still have functioning? Lately, Ruby is all about the 80/20 rule: 20% of the work, 80% of the street-cred.
I had a dream last night that I was on Reddit. I think that means I am on Reddit too much. Raise your hand if you’re surprised. That’s what I thought. But I have seen some real gem quotes on there. Here’s one from today: “I managed to get by as a coder for at least a decade before I ever heard the term ‘lexical scoping’. Not everyone needs or wants to be a language theorist.”—munificent. I’m usually the person who explains closures and I really agree with this sentiment. How much do you need to understand to get shit done? I think probably too much.
There’s a lot of people asking me and my company lately what content management system we use. I usually tell them “Apache” after rolling my eyes. But, let me credit Ramaze, a Ruby project, with stating something that I wish more people would come to terms with:
Before Rails, people wrote lots and lots of code when making a web app, and they did it over and over as they did web app after web app. With each new app, they would spend time “ramping up” with just the base application before they got into the real meat of the matter, the core functionality which varied from client to client, application to application. Rails and Merb and their ilk try to make your life easier by generating this “lots and lots of code” for you, so you don’t have to spend time repeating the motions of getting the skeleton of your application coded.
Ramaze takes a different approach. It asks: How about we dump the “lots and lots of code” part altogether? This way, you get the benefit of nearly nil ramp up time, without the gaping jaw effect that hits you after you run a code generator script and sit staring at the big filetree that just sprouted out of the ground (think Jack and the Beanstalk).
—
Ramaze by Example
I don’t think Ramaze is the solution to my woes, but there’s an excellent point here. Frameworks are larger abstractions than libraries. Abstractions leak, cost performance and take up mental resources. They also make things possible that aren’t before. I feel like I should have more language for addressing this problem, because I was about to type something nonsense like, we need thin, expansive abstractions. But that’s how I feel about it: abstractions constrain while setting you free, and striking a balance between freedom and safety is hard. Programming is almost death by too much freedom, at least by way of choice.
Michael told me about how developing his new product, TimeTagger, he noticed three things were bothering him, but they were all necessary: editing time slices, the interruption feature and something else. So he decided to try combining them, and that solved the problem: now there’s a single interface to do all three tasks and it’s quite nice. It reminds me of ZFS, which takes a bunch of lousy abstractions (the filesystem, RAID, volume management, disk compression and backups) and creates a single layer that attempts to solve all of those problems together. And it does a much nicer job than the situation you have on Linux with dozens of filesystems to choose from, LVM2 and EVMS conflating and contradicting each other and lousy snapshotting and compression choices. I think it was a case of Linux developers trying as hard as they could to solve the problem the same way as everyone else, arguing over the correct layers to synthesize, generalize and specialize. It’s not for lack of trying that they failed to hit on the winning combination (they’ve managed it in tons of other places). Plan 9, similarly, combines pretty much all networking, communication and security into a single concept, that of per-process namespaces of a filesystem.
What all of these things have in common is that they’re not afraid to look at a family of problems and say, yes, we have many words for many different aspects of these things, but they’re not really different. And these solutions are not afraid to have more moving parts than any one layer of the three that they’re replacing. The old filesystem fiasco has lots of layers and a great deal of potential for diversity. NetBSD, for example, would let you wire up any RAID combination you could imagine, and then throw it into the volume management system with which you could create a filesystem. Filesystems are free to evolve separately from the volume management system. The old three-layer way affords you tons more flexibility than the ZFS way. But, as it turns out, flexibility isn’t necessarily what we want. Right is better than many shitty options. Look at MySQL’s table types. PostgreSQL doesn’t have table types, it has one table type: a fast, consistent one. Tradeoffs, like preferences, are so often really just the software version of that other thing: abdication of personal responsibility.
MySQL looks at the problem of designing a good, high-performance, stable filesystem and says, well, stability and speed are really matters of taste. Users should be allowed to pick and choose whatever features they need and just use those. Not everyone needs ACID compliance! But these are implementation details, the very thing that the SQL standards are designed to divorce you from.
For years the debate has gone on about the proper filesystem to use with Linux. ReiserFS is great, XFS is great, JFS is great, ext3 is great, etc. A real protracted flamewar. And on Solaris, you get ZFS, and you’re glad that’s what you get. I’m tempted to say with OSS this is a recurring problem. When you can’t come to a consensus, you can usually at least come to a preference. Successful closed-source software doesn’t have nearly as many preferences. Especially if you look at, say, Apple. Look at the import settings window in iTunes. There’s three controls: kind, quality and whether or not to use error correction on the disks. cdparanoia, the premier Linux ripping tool, has 32 options and only gets you wave files from a CD. iTunes has one preference for that. cdparanoia has 10 output message types. iTunes tells you “I’m working” and lets you listen to music while it works. I haven’t even gotten to the encoder or library management problem.
All this is getting better with more modern Linux software like Banshee, and I’m not trying to rag on Linux here. I’m saying that creating a proper abstraction takes insight and a democracy isn’t really any better equipped to achieve it than a monarchy, and in some cases with OSS it seems like monarchy wins. Other times, democracy kicks the crap out of monarchy. SBCL generates tighter code than AllegroCL, which sells for thousands. But Allegro does better at memory management. What does that mean, memory sells, but is more boring to implement?
I still can’t seem to put my finger on what it is I’m trying to say. I guess I’m saying there’s a Tao here. If you have it, you make something like ZFS. If you don’t, you make EVMS. If you have it, you make a hundred line script to upload a file over FTP. If you don’t have it, you buy enterprise software to do it. If you don’t have it, you install Drupal. If you do have it, you invent Plan 9.
But you can have it one minute and lose it the next. The Plan 9 guys hated Emacs, loved C, and by extension probably hated Lisp. They have the perfect system for networking and sharing information, and foist the obnoxious parsing problem on everyone by only having flat files. Flat files beat the shit out of, say, XML or SOAP, but why not embrace Lisp and bring us to a place where structured data can be made and shared trivially all over the network? From where I’m sitting, it looks like their mouse orientation meant no Emacs, which meant no Lisp—which meant no simple, uniform data interchange, either, and everyone stuck in the hell of many parsers and DSLs. (Sadly, I think everyone is going to embrace JSON as the simple data exchange format of the future. Better than XML doesn’t imply greatness.)
I mentioned on Reddit somewhere that I think one reason why the software quality on the Mac is so much higher than Windows is simply because Windows is the bigger market. By which I mean, people motivated primarily by money, and not love or aesthetics, are on Windows. But if the situation were reversed, those people would be on the Mac making shitty software. Then, the people using Windows would actually be using it because they prefer it and, well, love it. I bet if that were to happen, Windows would actually gain a cult following due to high quality software. It sounds absurd, but that was basically Apple’s position in the 80’s: shitty OS but widely loved by users, some of whom became developers. Why this seesaw? Is this just a permanent aspect of the human condition?
Tags abstraction, ennui, languages, programming | 1 comment
Posted by Daniel Lyons
Mon, 08 Dec 2008 07:01:00 GMT
Today was Day 2 of the first UpgradeNM sprint. We really got a lot done!
Our beneficiary, First Mile, got a really nice new web design courtesy Reid. Michael Bernstein handled the Plone integration and Emily Lewis handled the content and accessibility. I wasn’t there for the work that was done yesterday, but what I saw today was superb; it’s beautiful and amazingly (to me) degrades just about perfectly in the absence of JavaScript or images.
Gabe Ortiz and I implemented the UpgradeNM site itself, designed by Reid again with assistance from Emily, and Jack Moffitt threw his weight behind testing and security, also with a bit of help from Bill Weiss. Markie Casias was present yesterday but I wasn’t there, it being Shabbat and all, so I’m not exactly sure what he helped with—but I know he did. :)
This sprint was much more relaxed and fun than the last one. It was a real pleasure to work with such talented and hard-working people—I really felt like the slacker. :) And I’m much more impressed with our output. I think this is a great group and we’re going to have a lot of fun working on other non-profits.
I wish I had more context to give you than that, but this is a very fresh group and much more stable than the last one.
If you program, design, copy edit, check usability, improve security, or really do anything at all related to web development, you should definitely sign up at the UpgradeNM site and participate with us. I think our next sprint will be in February.
We’re taking suggestions right now at the same site, so if you know of a non-profit that you think could use some web love, you should enter it on there and tell your friends to vote for it.
Tags upgradenm | no comments
Posted by Daniel Lyons
Thu, 04 Dec 2008 21:10:00 GMT
There’s an article going around, called Perl 5 is Dying (and its twin Perl 5 Programmers are Dying.) Both articles are talking about the decline of Perl, and are being hotly disputed on various forums including the one I read, Reddit.
I think it’s worth talking about in the abstract, because it doesn’t really matter if Perl is dying right now or not, because eventually it will die. But programming language death doesn’t mean what it used to mean. The success of the open source world means that even really goofy languages can be sustained if they have an ardent following. Or even just one ardent follower. SNOBOL4, which I’ve learned enough of to write joke programs, has a quite decent portable C implementation. REBOL is the work of one man, primarily. There are enough programmers now to sustain any number of subcultures centered around any language.
Lisp died in the 80’s. Lots of people stopped using it. But a small, ardent community still existed. Some of the people who left went to Smalltalk, others to C and then C++, others probably stopped programming altogether. Eventually most of those who went to Smalltalk moved on too (largely to Java, from what I understand), but enough were left that Smalltalk also has a small, ardent community. It doesn’t matter a lot, because the people who use Lisp now use it because they love it. I’m rather against advocacy of programming languages, because I don’t really want Lisp to become large enough and popular enough that it will be overburdened by fashionable morons the way certain other languages have been lately.
I think you should look forward to your language’s death. If it isn’t cool to use your language, then only the people who really love it will be using it. Everyone worries that if they don’t spread the word about their language, people who would love it won’t find it and love it. The reality is, they would find it and love it without your marketing. But only with marketing can you make it so sexy to use your language that stupid people wind up using it for stupid reasons. That is fashion in a nutshell, isn’t it?
On Reddit, user Calp points out an interesting possibility:
Actually, there are lots of cases where changing your programming language does actually solve all your (threading+concurrency) problems.
Python’s future competitors (for threading) will probably be languages like Erlang, Clojure and Concurrent/Parallel Haskell.
My informal, not valid “research” indicates to me that the trends around these languages are quite interesting. Clojure has certainly captured the hearts and minds of the Lisp-using world, but Lisp and Lisp-like languages are widely resisted because of their syntax. It might be growing like mad now, while it’s new and cool, eating up converts from people who already like Lisp, but once every Lisp user has used it and decided whether or not they like it, will it be able to win over people from the Java camp or from Ruby or Python? My gut instinct is that it probably won’t, because people have emotional reasons for not liking the syntax.
I see a great dichotomy between different syntaxes. Basically, you have your C-like languages. These are designed to make everything you’re doing explicit and explicitly different. They’re a compromise between easy to parse for the computer and easy to parse for the human.
Then you have languages that are extremely optimized for one or the other, the human or the machine. The advantage of human-oriented syntaxes like Ruby, Python and Haskell is that they have a lower learning curve (I’m strictly talking about syntax here) and they have high non-professional readability marks.
The advantage of languages that are extremely optimized for the machine is their flexibility. I’m talking about Lisp here. I find the syntax pleasing because it’s so consistent. It’s easy for the machine to help me edit it. And macros make it easy for me to have the machine help me prepare its own input. To get there, you have to get over the initial revulsion, and it’s not for everyone. If you want designers reading your code or non-programmers reading your tests, you’re not going to like Lisp.
Perl and Java are sort of at opposite ends of the spectrum. You can do a lot with Perl, and you can rewire a lot of Perl, and it has a very informal, ad-hoc feel to it. Java is the limit of formality and restriction on every axis. Yet they both are C-like curly-brace languages.
I don’t have a lot of love in my heart for Perl, but I at least admire its spirit. People switched to it initially, I believe, because it was faster and more powerful than using the usual suite of Unix tools to accomplish Unix administration tasks. CGI came along and gave Perl a major boost: it was the de-facto server-side language for years. But lots of people eventually switched to PHP and Ruby because they were sexier. Perl is never going to reclaim the popularity it once had because computing is much more ubiquitous than it was when Unix administration was young and Perl had a corner on the web development market. Those are monopoly conditions. How many of those users were using Perl because they loved Perl?
On Erlang, it seems to me (again, no real numbers) that the growth curve is quite flat. Erlang isn’t very suitable to web development. Lisp isn’t either, and that isn’t stopping me nor has it stopped many people, but it does mean that the barrier to entry is a little bit higher. That little bit makes it very hard to be the fad language. And everything about Erlang is just a little bit harder than you’d expect. Ctrl-C doesn’t exit (you have to do Ctrl-G q ). So I think people are talking about Erlang but nobody really wants to use it. Plus it has a strange variation on Prolog’s syntax. People fear different syntax. I’m not sure why they do, but they do, and it’s a factor: they’d rather have Erlang’s power in their own language, whether or not it’s possible. It’s quite a buzzword though.
Haskell, on the other hand, actually has pretty good growth potential, for (what I consider) surprising reasons. Haskell doesn’t have one feature that trumps all others. But it does have a fierce collection of small but excellent features. Nobody uses Haskell for just one reason. People use it because they like the Pythonic syntax, because they enjoy lazy evaluation, for the powerful yet largely hands-free type system, because they want the parallelism for free, because they enjoy the mathematical rigor, or any of a dozen other reasons. It’s quite a combination, having the state-of-the-art research going into the compiler. It’s also a lot of work to learn. The syntax is deceptive: it looks easy, and you see a lot of really compelling examples of people writing vastly complex algorithms in just a few easy-to-read lines. But doing that is hard, and learning how to use the machinery you’ve been given is extremely time consuming.
Don’t get me wrong, I like Haskell a lot, but it’s not for me. I’m not quite sure what a Haskell fad would look like, because I have a lot of trouble imagining that average programmers have the chops for it. I hope that the community they’ve built is not fad-based, because if it isn’t, it’s quite a powerhouse. Perhaps they’ll all move to Epigram or Disciple when they’re more complete.
These fads happen because people want more power, for free. But they also don’t want to have to learn much. We’re a weird hybrid. We want to get more done, more efficiently, but we also want to be able to type something when we have no idea whatsoever what we’re doing. I think this is part of the reason for the continued stagnation of some of these languages. People gush over Ruby because it’s terse, but it still has quite a bit of boilerplate (end; end; end; } end). Some of these languages have so little boilerplate, it’s hard to write any code at all if you don’t know what you’re doing. How’s that going to appeal to a Java programmer?
So congratulations Perl! May reports of your death be as exaggerated as Lisp’s, and may you prosper in obscurity while the rest of the world goes chasing after the latest magical solution to all its problems. :)
I do wonder if having all these subcultures could somehow be used as a weapon against management. The success of Java is really built around computing having a lingua franca. Management wants this because management wants to believe that programmers are like any other kind of employee, and we can be inserted into any job in which we know the language and we’ll do the work. This mentality is destructive to our creativity and independence and denies the reality of programming. Programming is not engineering, and efforts to make it such are dehumanizing. It’s the same situation with CMSes and web development. People want to believe that they aren’t being locked in, so they impose these constraints on you.
And these constraints are themselves just more of this search for better magic. As long as all my developers are building in Java on top of Struts, I can just hire another firm that does Java and Struts. Or PHP and Drupal. In reality, I don’t think the benefits are worth the drawbacks.
Tags philosophy, programming | no comments
Posted by Daniel Lyons
Mon, 24 Nov 2008 17:53:00 GMT
Cross posted from a mailing list I’m on.
I had the honor of being flamed by _why last week. In fact, my name was above Zed’s and DHH’s.
Like any community, it’s a complex beast. I think a lot of the BS that’s happened in the Ruby community is because the community idolizes _why so much and _why has basically shown again and again that if you’re cute enough, you can say or do anything you want. Have fun and don’t take anything too seriously. And this flower has blossomed into a culture where blogs are documentation, releases don’t exist, code gets abandoned, everybody has to clone whatever anyone else does because it looks fun rather than contributing back to fix the original. And DHH reminds us that doing all of this while swearing and lying about each other makes it even more fun.
And predators like Fowler, Thomas and Pollack see that there’s something going on here and there’s a way to make a quick buck. There’s definitely an air of twisted childishness about Ruby and Rails and I’m quite nauseated by it. I’m to a point where if a blog looks too good, I don’t trust it.
In linguistics there is a concept called the cryptotype, which is basically a category of consistency which isn’t made explicit in the grammar of the language. For example, you can add “up” only to certain verbs in English: you can mix it up, and you can start it up but you can’t bake it up or pour it up. And those of us who are disaffected with Ruby are those of us who have violated the cryptotypes of the Ruby community itself. We’re not supposed to notice that Chad is a werewolf, we’re not supposed to notice the real, endemic problems of Ruby itself, and we’re not supposed to dislike the trends of what’s happening with the software and the community in general. But we did and now we are anathema.
Tags rails, ruby | 4 comments
