Accumulating Technical Debt: A Costly Proposition

>> Saturday, December 12, 2009

I just the passed couple of weeks doing some very technical work with Maven on a large (above 50K person-day) project. Essentially, our dedicated team of three engineers spent a total of approx 30 days to realign the build configuration and process for this  project. We already had a pretty good idea of what was wrong and how to fix it but what we found amazing is the extent and severity of the problems and anomalies that this project had been dragging for many many months and how little control over the whole build process the teams had. I won't get into the technical details here (maybe in a follow-up article), but imagine a project where daily builds take a few hours, where the result of the build is extremely messy on top of being somewhat unpredictable, where the deployed applications are plagued by failures due to conflicting jar versions and by different compile and runtime dependencies. In addition, the build process, although based on Maven, relied heavily on homegrown Maven plugins to do all sorts of "magic" (albeit unnecessary) tricks. I just described a recipe for problems and problems they had. The comforting news is that it was possible to turn the project around and realign it so we now have a controlled build and release configuration that relies almost exclusively on generic Maven plugins.

The real question is how do projects get into such a predicament and how come they tolerate the negative consequences for such a long time period? In this particular program, external audits were conducted on at least two occasions and each time, strong and concrete recommendations were made and warnings were issued but things continued as is. The direct result of this is that a lot of fuel was wasted on attempting to alleviate and compensate for the negative effects of the fundamentally flawed build and release process and configuration. I like to compare the situation to traveling in a luxury sedan with half deflated tires and a wobbly steering. You are burning too much fuel and rubber and you are a catastrophe in waiting but instead of taking the time to stop and fix (or replace) your slow leaking tires and steering you prefer to just add air from time to time and put up with the steering problems because your are in a hurry to complete your 5000 km journey. I mentioned a luxury sedan because the tool set for this project is top notch: Maven, Nexus, Hudson, Eclipse, Accurev. All the right tools, but most of them not being used properly.

Officially, this particular initiative is using the "Agile Methodology". In reality, it is applying the SCRUM approach to managing teams and projects. In many ways, the project is not compliant with the agile software development principles. Dragging for months and months a technical debt and paying interest at every detour is not, in my book, an agile strategy. In fact, the SCRUM discourse of focusing on client functionality and of managing a backlog of "User Stories" that are prioritized by the product owner and that must fit within the sprint can have  perverse effects that we must be aware of.  Organizations that adopt this approach should put some mechanism in places so that unjustified technical debt does not accumulate and reach crippling proportions.

Many methodologists like to present the so-called "Toyota approach" as a model to follow. I do not consider myself an expert in the subtleties of one method versus another but my understanding of the Toyota way is that if a problem in the assembly line jeopardizes product quality, we stop the assembly line immediately and fix the problem before we pursue. Even if that means that 20 or 30 people might be idle for a few minutes or a few hours. Software development is not an automobile assembly line but when the process is broken, taking the time to fix it always pays off.

Following our intervention and the extensive refactoring of build configurations, the builds are easier to manage, the build process is faster, lighter, transparent, predictable and flexible, they can be performed by different individuals (not just by one "specialist" know-it-all) the deployments are more stable and overall, teams are more productive. I am convinced that refusing to make this change earlier did not allow the project to deliver more functionality. Quite the contrary, it has delayed and slowed down the whole development process, it has poisoned the work atmosphere, it has distracted everybody from their core tasks and has entertained the impression of a semi-controlled chaos which most people and organizations do not appreciate.

So drive carefully and keep your vehicle in good repair,

Until next time,


0 commentaires:

Post a Comment

  © Blogger template Webnolia by 2009

Back to TOP