The programmer of the night is at it again. This time he's claiming that software engineering is dead. I think he's got the wrong idea. But enough about that pale-skinned apparition; his post is merely the muse of today's post.

The thing I found most interesting was in the comments of his post. In particular, the article Coding is less science and more craft by someone named Joe Stump, written earlier this year. I think that article stumbles around a concept I've written about before, namely the difference between those with the engineering knack who are passionate about programming and Gehn programmers who see software development as just a job. Obviously there are more than two types of programmers, but those are the only two groups Joe's article touches on.

But a more subtle point that both Jeff and Joe mention is the standards and practices that are associated with the term "software engineering": things like project estimates, measuring deliverables, formalization of coding standards used at organizations, etc. They are concerned with treating software design and development as a manufacturing process. This is definitely true, but engineering isn't a manufacturing process either. It's unfair to claim that software can't be engineered simply because it resembles craftsmanship. One does not preclude the other.

Let's see what the Internet has to say about these terms, first:

craftsman - a professional whose work is consistently of high quality.
craftsman - a skilled worker who practices some trade or handicraft
engineering - the discipline dealing with the art or science of applying scientific knowledge to practical problems
engineering - the discipline and profession of applying technical, scientific and mathematical knowledge in order to use natural laws and physical resources to help design and implement materials, structures, machines, devices, systems, and processes that safely realize a desired objective

I see absolutely no reason why I can't be both an engineer and a craftsman. I can't think of any reason why one would want to shake off the title engineer from development at all. Certainly, in order to invent clever algorithms to solve problems, software designers need to apply a great deal of knowledge gained by experience or education. We analyze algorithms using sound approaches and prove they work for all possible inputs rigorously like any mathematician. We test our systems using approaches similar to the scientific method, proving complex systems work and debugging when our theories regarding how the world works (which is what code is, after all) end up being false. We are meticulous about detail because computers are too dumb to process problems at the same scales that we are capable of understanding. This is both craftsmanship and engineering.

With that out of the way, the other concept that Jeff brings up is the idea of control. Control is not engineering. Control is management of engineers and their products. It's important to make this distinction, I think, because engineering is good while I believe control is not. Tom DeMarco started the ball rolling with the first line of his book Controlling Software Projects: Management, Measurement, and Estimates: "You can't control what you can't measure." Basically, the idea is that software developers need to work toward small, measurable goals so that they can be controlled.

The primary argument given by Tom's recent article is that software design and development can't be controlled no matter what. I agree. Because software can't be contained within rules. Because it can't be broken down into measurable, controllable chunks. Because you can't say code is good solely because it conforms to a standard. Because really good programmers don't like control as a general rule.

You'd sooner control the process of composing a great symphony than you would the process of designing and creating software. Both require tremendous technical knowledge of how the different parts of the whole work at a very detailed level, and both require so much creativity and ingenuity that no one should hope to try to control it! Who would seek to control a composer's creative mind? No one but the Party, I should hope.

The best you can do is come up with reasonable, clear, high-level guidelines to drive the spirit of the creative process away from known issues. Many problems have been solved by many people over and over again. It would be foolish to disregard decades of problem solving and not learn from it. Guidelines that steer inexperienced developers away from traps can do wonders, as long as the reasoning behind the guidelines is explained properly. But that's not control at all. It's something more fundamental, simpler: it's teaching.

Standards can only do so much. Given the example of HTML, you can certainly write software that validates against any standard that still has no usability whatsoever, or that takes an hour to display a simple result, or that doesn't work when there is any minor deviation from valid input. Joe claims that we should have standards to keep bad developers from injuring good code. I claim that no matter what standards you have, bad developers are going to follow them to the letter rather than the spirit and end up mangling everything, standards notwithstanding. Additionally, good developers will find too many standards stifling. I know I do. I would rather work with guidelines and a wealth of known problem areas to avoid than a strict set of rules I must follow, and I know a great many fellow developers who feel the same way.

Another aspect that needs to be considered is the fact that programmers are good at programming. It should seem obvious: programmers are hired for and paid to write software. That's what we do. So why do you get them to do all kinds of other things that aren't programming? Except in rare situations, programmers aren't good at coming up with estimates, programmers aren't good at managing teams of people, programmers aren't introspective, programmers are programmers. Odds are if a programmer is good at doing these things he wasn't a programmer in the first place.

Management tends to want software development to be a rigorous process that can be monitored and controlled, so programmers are often interrupted to report on their progress, issues, etc. Good, productive programming can't be accomplished with interruptions. From the mind of Joel Spolsky:

We all know that knowledge workers work best by getting into "flow", also known as being "in the zone", where they are fully concentrated on their work and fully tuned out of their environment.

The trouble is, getting into "the zone" is not easy. When you try to measure it, it looks like it takes an average of 15 minutes to start working at maximum productivity.

The other trouble is that it's so easy to get knocked out of the zone. Noise, phone calls, going out for lunch, having to drive 5 minutes to Starbucks for coffee, and interruptions by coworkers -- especially interruptions by coworkers -- all knock you out of the zone.

I can't tell you how many entire days I've lost to this phenomenon. Days when I can do nothing but resolve the simplest surface-level bugs because of interruptions kicking me out of the groove. Occationally, at work, I get a few solid hours without anyone bothering me. I'm usually able to get work done that, with interruptions, would have normally taken me a week. It's fantastic, but it's also rare. It's even more rare now that I'm experienced and have more responsibilities. Nowadays I'm lucky if I can have half an hour without something kicking me out of flow.

My single most important piece of advice regarding working with programmers is leave them alone when they look busy. If you have a question, ask them when you see them walking around on break. But, to be honest, I do a lot of problem solving in my head when I'm wandering around, too.

In conclusion, engineering doesn't mean control. Engineering is good. Even some levels of control can be good. I think the problem is summarized best by a comment on Jeff's blog by residentoddball: "Process needs to improve the final product, not hinder it for the sake of itself." I'm fairly confident that is the guts of Jeff's, Tom's and Joe's argument anyway.