I’ve recently been following a discussion about Agile software development on LinkedIn. While there were some interesting points raised and also many pointless posts, there was one entry in particular that I found particularly interesting.

I am reproducing the post by Peter Wennerholm below with his permission, and have added my own commentary in between block quotes.

“Several contributors seem to see only two alternatives to how to produce software: Agile and Waterfall.

“That is not a fair view of the situation: waterfall was presented as a process that should NOT be used for software development. So far I have never heard of any shop that uses it, and that’s a good thing too.

“Let us chew on that for a while. The waterfall process was never intended to be used for software development. It was put forward as a process that should be avoided. So unless you really have found yourself in a pure waterfall team (and then you have my sympathies) please stop putting it forward as an alternative to Agile. The Agile “procedure” must be judged on its own merits, not as a contrast to an absurdity.”

Some interesting insight on the Waterfall model here! The paper [PDF] linked by Mr. Wennerholm, which dates back to 1970, describing what we now know as the waterfall model while clearly indicating (a) why it is not suitable for large software development projects, and (b) a number of improvements that make the Waterfall model a lot more reasonable to use. If you’re too lazy to read the paper itself, this blog post contains some pretty decent commentary about it.

“I have always made software in much the same way as how I cook: you add some ingredients, then taste, then add some more, taste again, etc. If you want to give it a fancy name, be my guest, but it seems more like common sense to me. Oh well… If someone can make money teaching common sense to programmers, who am I to deprive him of his living?”

I think that the cooking metaphor used by Mr. Wennerholm is pretty spot on. It is in the spirit of prototyping (which is at the heart of many agile practices) to create something small, see that you’re going in the right direction, and then continue with the next iteration, as opposed to building the whole thing and then realising you’re totally off track. The simplicity of the process is also worth noting. While software development may be as simple as a feedback loop, today there are many different methodologies involving piles of documents, meetings and ceremonies. While these are often useful if done right, it is not hard to realise that they are well hyped up by the people making a living from promoting them (e.g. by writing books) and by those who religiously think they’re a panacea.

“The problems seem to arise when people trained in managing non-programmers are put to manage programmers. The “common sense” is a double-edged sword: your life experience will dictate what is common sense to you, and it may differ very much from what is common sense to me. A non-software person tends to see a project as a collection of known problems with known solutions: it takes X seconds to lay one brick, the wall is Y long, so it will take Z days to finish the wall. A programmer knows that there are many unknown variables in his project: it is a collection of problems out of which some may be completely new, or handled by new tools, or in an unknown environment, or with unknown people. We cannot say how long it will take to finish the wall; when we start we do not even know if we should use bricks. Therefore, common sense dictates to make tiny waterfalls, or cycles, where each cycle takes us one step further towards completion and gives us more knowledge about the problems ahead of us. Agile and Scrum to you, perhaps.”

In this paragraph we recognise the fact that there are many unknowns in software development. When undertaking something new, we are often unfamiliar with the business, the technology, and many other aspects which can seriously affect the outcome of the project.

In my opinion it is pure madness to commit to strict deadlines on something that is very poorly understood. Therefore, the first thing that should be done is to make the effort to investigate the unknown factors so that they are at least fairly understood. This should be done as early as possible.

Aside from learning by communicating with the client, prototyping also helps a lot with understanding the problem domain. I believe this is what Mr. Wennerholm means when he mentions making tiny waterfalls. This is also reflected in the aforementioned paper, where Dr. Royce suggests creating the software twice: once as a small-scale throwaway prototype to capture the problem domain without wasting too much effort, and the second time as the actual deliverable.

“When we talk about what is the best way of programming, let us all be very humble. Our profession is so young that some of the very first programmers are still alive today. Look at house building: after thousands of years of experience we still have houses that burn or collapse or rot – we cannot claim to have all the answers on how to build software.”

I don’t think this paragraph requires much commentary: I think it is a pretty awesome message that we have so much to learn. Let’s try not to be religious about anything in software development; tomorrow someone may demonstrate that the awesome technique we touted so valiantly is actually a pretty bad practice. And when that happens, I hope we are open-minded enough to learn from our mistakes and grow in our beloved profession.