Avoiding Death By ComplexityPosted: August 28, 2008
Every developer knows that there are plenty of different ways to kill a software project: you can try to go too fast, you can change the requirements too often, you can build something no one wants, you can have the wrong team. But even the best-run, best-defined project can still run aground on my personal primary fear as a developer: complexity.
There are three especially scary things about complexity. First of all, it creeps up on you slowly. It’s entirely possible to get out a 1.0 or a 2.0 version of a product that’s overly complex, but at some point things just . . . stop. New features become harder and harder to add and bugs become harder to fix without introducing regressions. Pretty soon your team’s velocity has slowed to a crawl and your project just never seems to get to the quality level you need in order to release. I like to liken that endless bug-fix phase to playing whack-a-mole: every time you fix one bug, another bug pops up due to an unanticipated interaction or an incomplete understanding of the system. Both those conditions creep up slowly, and by the time they’re happening it’s often too late to really fix the underlying problems.
The second problem is that complexity-related problems are very, very hard to fix. They’re often caused by very deeply-rooted architectural issues that have been built upon for months or years before the problems became apparent, and fixing them can become incredibly high-risk. To matters even more difficult, it’s often hard to pinpoint a single cause of complexity, at least if it’s systemic. When the parts just don’t quite fit together right, it’s hard to tell which part is at fault or which part should be changed. And of course, when drastic changes are required, it’s always hard to be sure what the outcome will be, so fixing underlying architectural flaws is always something of a gamble.
The last problem is that complexity is, in the end, a force you can only hope to contain rather than one you can completely eliminate. Even the best developers with the most foresight, the best processes, and the clearest product definitions have upper limits on the amount of complexity they can handle, and trying to build something more complex than that upper limit is bound to fail. In my mind that complexity upper bound is up there with the halting problem or NP completeness as immovable forces you need to stay away from: if you’re trying to solve the halting problem, you’re barking up the wrong tree, and if you’re trying to make a product that’s too complex, you’d better find some way to simplify it.
How concerned you have to be about complexity will naturally depend on the scope of your application and how likely you are to come close to that upper bound: is it a large, complex application, a tiny script, or something in between? Is it a one-off, one-time project with a fairly contained scope, or is it something you’ll need to build on and extend for years? Of course, most large projects start out as small projects, so it’s dangerous to just assume that your small project will stay that way. That doesn’t mean you need to plan for complexity up front, but it does mean that if your formerly-small project starts growing up that you should worry about it before it’s too late or too expensive to fix.
Given all that, the question naturally becomes what to do about it. Dealing with complexity requires understanding the two different kinds of complexity: inherent and accidental. Inherent complexity is complexity that arises from the type of problem you’re trying to solve. Natural language processing, for example, has a lot of inherent complexity: there’s just only so much you can do to simplify the problem. Accidental complexity is complexity that arises from your choice of implementation: if you’re using a three-tier architecture and EJBs to power your blog, your blog software probably has a lot of accidental complexity.
The two kinds of complexity naturally have different strategies to mitigate them. Fixing accidental complexity is all about simplifying implementations using good software engineering practices: using higher-level languages and tools, properly decomposing code into the right kinds of re-usable building blocks, avoiding over-engineered solutions, etc. Fixing inherent complexity is all about constraining the problem space: cutting out overly-complicated features that don’t add enough value, not trying to do too much at once, and otherwise keeping requirements constrained. Somewhere in the middle is the idea of properly layering complexity, whereby you try to chop the application up into discreet components with a minimal, well-defined interface between them such that you can avoid worrying about anything besides the current layer you’re working on. Depending on how big the layers are, that can either serve to reduce the complexity of the implementation by reducing the maximum amount of complexity you have to deal with at one time, or it can also reduce inherent complexity by chopping the larger problem into smaller, more discreet sub-applications that can be specced out and implemented largely independently.
The real key to dealing with things is to always have a healthy respect for complexity and to be constantly looking for ways to reduce it. Good developers can handle dealing with more complexity than merely average ones, which means that a more talented team will be able to persevere longer in the face of rising complexity, but that also means that people sometimes take pride in being able to make really complicated things work, even if it’s ultimately a bad idea. The best developers, in my experience, tend to have a fear of complexity that leads them to back off and avoid those situations whenever possible, both by simplifying implementations and by working to make sure that the project scope stays controlled.