What is a Methodology?

>> Saturday, December 12, 2009

Since the beginning of my IT career some 25 years ago, I have entertained a sound reserve towards software development methodologies. I have always found it ironic that in our industry we label "methodology" a checklist of documents and a set of document templates that must be produced or filled in at different stages of a project. Such "methodologies" are very good at specifying what "deliverable" must be produced and when they should be deposited but usually very silent on how exactly do we produce the ultimate deliverable, i.e. the software system itself. It is true, that traditional methodologies also describe a series of software development phases and how should be doing what at each phase. But again, how exactly do we work within each phase was/is often left to the imagination.

For this reason, I was always more attracted by the literature that describes and discusses how to effectively and efficiently design software and how to write and manage code and systems. Strangely, this literature tended to shy away from the term methodology and preferred to use expressions such as "software engineering".

It was therefore refreshing for me when the Agile movement came into light. For once, I was hearing and reading about a "methodology" that corresponds to my own experience and inclinations. One that I could relate to. One that talked about practices, techniques, systems and tools. That puts the emphasize on the delivery of software instead of the delivery of documentation. An approach that prescribes the use of tests, of continuous refactoring, that talks about team organization, work habits, and human interactions. Those were the early days of the Agile movement with a lot of talk about extreme programming and unit testing.

However, methodologists at heart were not to be left on the sidelines. They woke up and smelled the coffee and had the intuition that there might be something for them in this new trend. Since it was difficult to sell the idea of a list of documents for each of the different stages of an agile project (agile is lean and iterative remember?) they instead invented a new religion with priests (scrum masters) and ceremonies (dailys, retros, demos etc). I am not arguing that all this is B.S. but I do see a dubious trend when people consider they are agile because they are holding SCRUM ceremonies. I may be naive, but to me, ceremonies are meant to be (or should be) a means to help teams and organizations increase their agility. If ceremonies become an obstacle to agility, then we should get rid of them and invent something else. 

On the same basis, we now find "agile" tools on the market that help organizations follow the SCRUM way of doing things. Considering yourself agile because you use such a tools is completely fallacious. I personally consider it much more important, for the sake of agility, to use automated testing tools and build systems then to use a SCRUM compliant project management tool.

So long,



Accumulating Technical Debt: A Costly Proposition

I just the passed couple of weeks doing some very technical work with Maven on a large (above 50K person-day) project. Essentially, our dedicated team of three engineers spent a total of approx 30 days to realign the build configuration and process for this  project. We already had a pretty good idea of what was wrong and how to fix it but what we found amazing is the extent and severity of the problems and anomalies that this project had been dragging for many many months and how little control over the whole build process the teams had. I won't get into the technical details here (maybe in a follow-up article), but imagine a project where daily builds take a few hours, where the result of the build is extremely messy on top of being somewhat unpredictable, where the deployed applications are plagued by failures due to conflicting jar versions and by different compile and runtime dependencies. In addition, the build process, although based on Maven, relied heavily on homegrown Maven plugins to do all sorts of "magic" (albeit unnecessary) tricks. I just described a recipe for problems and problems they had. The comforting news is that it was possible to turn the project around and realign it so we now have a controlled build and release configuration that relies almost exclusively on generic Maven plugins.

The real question is how do projects get into such a predicament and how come they tolerate the negative consequences for such a long time period? In this particular program, external audits were conducted on at least two occasions and each time, strong and concrete recommendations were made and warnings were issued but things continued as is. The direct result of this is that a lot of fuel was wasted on attempting to alleviate and compensate for the negative effects of the fundamentally flawed build and release process and configuration. I like to compare the situation to traveling in a luxury sedan with half deflated tires and a wobbly steering. You are burning too much fuel and rubber and you are a catastrophe in waiting but instead of taking the time to stop and fix (or replace) your slow leaking tires and steering you prefer to just add air from time to time and put up with the steering problems because your are in a hurry to complete your 5000 km journey. I mentioned a luxury sedan because the tool set for this project is top notch: Maven, Nexus, Hudson, Eclipse, Accurev. All the right tools, but most of them not being used properly.

Officially, this particular initiative is using the "Agile Methodology". In reality, it is applying the SCRUM approach to managing teams and projects. In many ways, the project is not compliant with the agile software development principles. Dragging for months and months a technical debt and paying interest at every detour is not, in my book, an agile strategy. In fact, the SCRUM discourse of focusing on client functionality and of managing a backlog of "User Stories" that are prioritized by the product owner and that must fit within the sprint can have  perverse effects that we must be aware of.  Organizations that adopt this approach should put some mechanism in places so that unjustified technical debt does not accumulate and reach crippling proportions.

Many methodologists like to present the so-called "Toyota approach" as a model to follow. I do not consider myself an expert in the subtleties of one method versus another but my understanding of the Toyota way is that if a problem in the assembly line jeopardizes product quality, we stop the assembly line immediately and fix the problem before we pursue. Even if that means that 20 or 30 people might be idle for a few minutes or a few hours. Software development is not an automobile assembly line but when the process is broken, taking the time to fix it always pays off.

Following our intervention and the extensive refactoring of build configurations, the builds are easier to manage, the build process is faster, lighter, transparent, predictable and flexible, they can be performed by different individuals (not just by one "specialist" know-it-all) the deployments are more stable and overall, teams are more productive. I am convinced that refusing to make this change earlier did not allow the project to deliver more functionality. Quite the contrary, it has delayed and slowed down the whole development process, it has poisoned the work atmosphere, it has distracted everybody from their core tasks and has entertained the impression of a semi-controlled chaos which most people and organizations do not appreciate.

So drive carefully and keep your vehicle in good repair,

Until next time,



  © Blogger template Webnolia by Ourblogtemplates.com 2009

Back to TOP