The Birth of the Requirements WikiPosted: August 22, 2008
As a development organization we’re by no means perfect, though we’re constantly looking for ways to improve, and one of the ways in which we’ve historically had a lot of room for improvement is around internal documentation of requirements and feature specifications. We’ve come up with what we hope will be a much better long-term solution, but before I describe what that solution is, I’d like to rewind and tell the story of how we ended up where we are right now.
Imagine that your company is writing a policy administration system (such a randomly chosen example, I know), and you’d like to know the answer to the question “How do policy renewals need to work?” How would you go about trying to answer it?
Historically, we’ve done our requirements documentation and feature specification in a fairly old-school manner: product management would write up a document describing the requirements for a new feature, development would up a design topic on the wiki for anything that requires some more thought and discussion, and development would write up more of a full specification afterwards describing how things actually work. The end result is that the “requirements” for a given piece of functionality tend to be difficult to discern after the fact: they’re scattered across a bunch of early-stage PM documents that are generally 1) deltas against each other, 2) don’t always resemble what was actually built, 3) tend to have a lot of ambiguities (since they’re written as normal prose), and 4) don’t capture any of the little things discussed and agreed upon by dev, PM, and QA over the course of actually building a feature. The development specs (when they’re actually up to date) tend to describe how things actually are rather than what the underlying business requirements are, are also written as deltas, are written at a semi-arbitrary level of detail, and aren’t written for things like UI functionality, while the design topics serve more to explain why things were implemented as they were. So if you want to know how policy renewals work your best bet is just to ask someone; the information is so scattered, out of date, and incomplete that it’s impossible to piece it back together.
We do a lot of test-driven development, so you might ask “But what about the tests?” The agile philosophy is that the tests can often serve as the documentation, and that’s kind of true of well-written, complete unit tests (I still don’t think that’s 100% true, but that’s a different argument). But the problem is that unit tests themselves are at too low a level to be useful for answering higher-level questions like “How does a policy renewal work?” Even questions like “What happens when I click the ‘Add Vehicle’ button?” are difficult to document via tests because they require an entirely different level of tests than “unit” tests. They require end-to-end tests, and those tests tend to be harder to write and harder to read; they’re also much more difficult to ensure completeness for, since you can’t measure test coverage using a tool or even match up the set of methods against your set of tests. In addition, for infrastructure work the tests tend to help describe the implementation, not the high level requirements.
The other problem with using tests is, unfortunately, that they tend to get deleted when they break too badly; at some point it’s inevitable that some refactoring or other major change will break enough unit tests that you just don’t have the energy or inclination to fix them all right away, so you rewrite and fix what you can and just comment out or delete the ones you can’t. That’s also true of tests that are written against the actual implementation rather than at some higher level of abstraction; if you change the implementation, all those tests are simply irrelevant, so you have no choice but to kill them. That might not be ideal, but practically speaking that’s what actually happens in the real world where real people write real tests, and as such tests are a bit shaky to rely on as the sole source of documentation about business requirements.
“What about story cards?” you might ask. Well, one unfortunate fact is that the policy team that I work on hasn’t used story cards in the past (we will be in the next release cycle). One of our other teams does drive everything off of story cards, but even then I think there are some problems. First of all, stories are inherently deltas, and over the course of a release or over many releases the same functionality is often continuously changed, making it difficult to piece together an answer to “How does policy renewal work?” because doing so requires assembling all the stories relating to renewals over the course of several releases in chronological order so that the appropriate deltas are applied in order. Ouch. Story cards are also inherently somewhat unorganized and can contain information relating to multiple different parts of the system, so just assembling that set of cards in the first place can be difficult. Story cards would still be lightyears ahead of where we were a year ago, so perhaps if we’d had them we wouldn’t have built the tools that we did, but since we didn’t have those cards we had to find a different way to do things.
So that was our situation a year ago: information about how things were supposed to work was largely in people’s heads, and we had scattered, generally untargeted end-to-end test coverage that touched many parts of the system.
That’s around the time we started to rework some major portions of our application, and before we started we thought the main risk we ran was that we’d break things without realizing it. In order to mitigate that, we wanted to fill out all (or at least a good number) of the tests for a given area of the application before we changed things. But how would we know we had “all” the tests and weren’t missing something? Without any obvious “units” to test we’d have no chance, so we decided to make our own units. They weren’t really stories in the traditional sense: they were statements like “The ‘Add Vehicle’ button takes you to the ‘New Vehicle’ page” and “The ‘Clone Vehicle’ button clones all selected vehicles, cloning all of their fields except for the VIN number.” Some of them could have been stories in the story card sense, but plenty of them were too fine-grained for story cards. For lack of a better term, we decided to call them “requirements” instead. Our process then became that we’d first attempt to reverse engineer the requirements for a page before we rewrote it, generally by reading any existing documentation and then by playing around with the page to see what it actually did. After we wrote those down, we’d try to have them reviewed by the product managers for accuracy and completeness, and then we’d use the requirements to drive a set of tests around the page. Ensuring a sufficient level of testing became much easier, because we could target the tests to the requirements just as you’d target unit tests to a method. Once we were done we were pretty sure we’d catch most of the breaks we might introduce, and we’d go ahead with whatever refactoring/rearchitecting needed to happen.
The idea was the right one, I think, but we had questions over how exactly we’d manage the requirements docs. What format would they be in? How would we organize them so people could find them? Hardest of all, how would we ensure they stayed up to date? We really wanted to measure coverage of the requirements as well, so how would we do that? To get the ball rolling we started out just using Google spreadsheets to track the requirements; the spreadsheet format ensured the requirements were relatively small and targeted (and hopefully unambiguous) line items instead of prose paragraphs describing things. I even wrote a way, using annotations and the Google SOAP API, to create some simple HTML reports about what requirements had tests. It was pretty clear that was a sub-optimal solution, but it was a start.
The question really became where to go with things: if we wanted to try to cover our whole application this way and really drive a lot of our automated end-to-end testing off of it, we’d really need everyone on the team to be on board with it, and doing that would probably require some much better tool for managing things. Thankfully we had an engineer who was fairly amazing at coming up with little tools to solve all sorts of development problems, and he agreed to take the lead on formalizing things and later driving adoption of the tool. The end result was basically an addition to MediaWiki that we called the “requirements wiki” and was eventually nicknamed “The Riki”. The modification added in some special tags for listing requirements, which would then (on a first update) assign them unique IDs. It also allows you to tag requirements with labels like “agreed” and “implemented,” along with several other clever things. The IDs can then be used as annotations for test methods to tie the test methods back to the requirements, and the Riki has a background process that periodically takes a build and processes all the annotations to link things up, resulting in the ability to display the current test methods inline with the requirements as well as coverage reports about what percentage of requirements have any tests at all as well as what percentage of the tests have actually been implemented. The latter statistic allows us to add in empty test methods as a way of sketching out a test plan without implementing them all immediately.
The riki is still pretty young in its life, so the jury is still out on our ability to really keep it up to date. So far, though, it’s proven useful as a way to coordinate dev, QA, and PM by giving everyone a shared, authoritative reference point about how things are supposed to work. I’m hopeful that by making it an indispensible part of our development process we’ll manage to overcome the inherent problems with keeping documentation up to date and that it’ll drive clearer, less ambigious requirements, better testing, better communication between dev, QA, and PM, and serve as an ongoing reference for anyone new to the team or to a particular area.