Quality Assurance in Guidewire’s Agile WorldPosted: January 25, 2008
The goal of quality software development is to release a product which will meet or exceed customer expectations. Sounds simple, but few software companies (especially those developing enterprise applications) ever achieve this goal and even fewer do so on a consistent basis. From it’s inception Guidewire adopted aspects of Agile Development and Extreme Programming with the objective of high quality and on-time, customer-relevant releases. My ambition with this blog entry is to share the experience of the QA team working in Guidewire’s Agile model with the hope of sharing lessons learned, how we work, and what we strive for.
There’s no QA in XP
QA and Agile/XP are not natural bedfellows. This issue has been discussed ad nauseum in various forums (see http://www.theserverside.com/news/thread.tss?thread_id=38785 for a relatively entertaining discourse on the matter). In synopsis, Kent Beck (the creator of Extreme Programming) did not see a role for a QA team or testers in general. In From Extreme Programming Explained Beck stated: “An XP tester is not a separate person, dedicated to breaking the system and humiliating the programmers.” Rather, in Agile/XP, programmers develop extensive unit tests and the end-customers decide whether the resulting product (or feature) is acceptable or not. In fact, the founders of Guidewire debated whether or not to have a QA team, based both on the ideals of XP in addition to negative experiences with the effectiveness of QA teams at prior companies. However, Guidewire has consistently maintained a 2-to-1 developer to QA Engineer ratio, a very high investment in QA as compared to the software industry as a whole. Why the disconnect? Is Guidewire really an Agile/XP company? Is Agile/XP too idealistic?
My 2 cents
My first assessment is that Guidewire indeed does try to emulate Agile/XP goals. My second assessment is that strict adherence to Agile/XP is simply unrealistic. Following are several reasons why Guidewire has required a dedicated QA effort:
- Unit testing, test-first development, and pair-programming fit into that category of ideas that most everyone agrees are worth the investment. However, the intense dedication involved to successfully and consistently implement XP practices means the reality of XP projects often fall short of ambitions.
- Unit tests generally fail to take into account both integration between features as well as the final packaged customer deliverable and the various environments the product will operate within. Without an effort to exercise the application in it’s end-customer state, including against all supported platforms, too many issues will be missed by unit testing alone.
- From a psychological point-of-view it’s difficult to objectively critique your own creation (i.e. your own code). An interesting case study would be to contrast the tests a developer implements versus tests defined by a QA Engineer. It’s likely the developer unit tests would cover the obvious use cases that the code was designed for, but how likely is it that the tests would exercise the uglier aspects that lurk in the boundaries?
- Automation (specifically unit tests) can not always be applied to all features or situations. Some amount of manual testing is realistically unavoidable.
- QA Engineers live in a world of higher abstraction than a typical developer and are exposed to a broader view of the product and it’s public requirements. Considerations that are obvious at the customer-level are often not made aware to a developer focused on a specific block of code.
- The customer feedback loop is not always ideal. Agile depends on customers willing to invest heavily in the product development effort, communicating to development their priorities and whether or not the product being built fits their needs. Many customers are unwilling or unequipped to make this investment. It’s arguable whether this is the best decision on the customer’s part, but the reality is that in Agile you’re asking customers to engage in what amounts to beta testing on steroids. As a result, some intermediary is necessary (namely QA) to determine if a feature meets it’s stated requirements.
- Finally, in the world of mission critical insurance applications it is simply unacceptable to rely on the end customer to discover product issues. Of course some bugs will exist in a release which will be caught by customers, but keeping the number of issues to a minimum is key to successful deployments and content customers.
Hmmm. We do need QA. Now, what is QA again?
So, given that QA has shown to be justified at Guidewire, what has proven to be it’s most effective application? First off, I feel a little guilty using “we” when describing the QA team. A separate QA organization truly only belongs in a waterfall development model where testing is a stand-alone stage and code is delivered wholesale from development to QA. In addition, a team expressly associated with ensuring quality inherently defeats the purpose of the Agile/XP process where testing/quality should be addressed by everyone in the organization and at all points of development. My belief is that implementation of QA in an Agile/XP environment should follow general Agile/XP tenets (when in Rome…), namely establishing and maintaining core Agile/XP ideals, specifically a passion for continual improvement. Following are some guiding principles that I feel are inherent to any successful QA effort within an Agile/XP environment:
- Never outsource QA. Luckily, this has never been an issue at Guidewire. The fact is that no salary differential (nor no communication device) will compensate for the collaboration that occurs when Engineers sit in the same room with no barriers to conversation.
- There is no substitute for a good Engineer. Hiring is key.
- Strive for tightly integrated development and QA teams. Ideally, QA Engineers sit next to their developer counterparts and the testing effort is shared and occurs in lockstep with code development. As well, the automated testing infrastructure should be common. At Guidewire we have what may be the ultimate automation solution. Tests developed by QA are run 24/7 by a harness which assigns broken tests to those who are responsible for their regression (usually development). Thus, a valid automated test is maintained ad infinitum. Simply checkin your test and walk away…
- Make holistic quality-related decisions. Involving Development and Product Management with decisions impacting testing resources allows for more effective use of limited time. QA should focus on areas known to be of high risk, for example new features where unit testing is known to be lacking or the code base is likely to exhibit buggy behavior based on inherent complexity. As well, the likely customer impact of a bug in a given area (knowledge usually unique to PM) is valuable in terms of determining whether or not that feature deserves special attention.
- Establish a model of continual training and leverage your knowledge base to keep Engineers up-to-date. At Guidewire we send all QA Engineers through training courses developed for Field Engineers. The expense is rather large (3 weeks of full-time training) but the payoff is Engineers exposed to the entire product and it’s customer-facing interface. Without this training it would likely take years for each Engineer to attain the same broad base of product knowledge.
- Develop good tests, whether manual or automated. A bad test (defined as either being redundant, poorly defined, trivial, or at worst deceiving) is expensive in terms of maintenance and misrepresented confidence levels.
- Automate, automate, automate. Guidewire strives for 100% automated test coverage. This is a brash goal and oftentimes the reality is far from ideal, but if you don’t shoot for the moon…
- Treat test code as production code. Follow good coding conventions, comment well, and refactor each test such that it remains relevant. This is another example of a pinnacle of testing that is difficult to reach.
Hopefully this is a decent primer on the often misunderstood and historically maligned area of software development called Quality Assurance, especially as applied to Agile. There are many topics that apply to quality software development I’d like to eventually delve into or expand upon. Test coverage is a fascinating pseudo-science that’s fun to debate. The evolution of the QA Engineer from a key-banging monkey to a fully-fledged Object-Oriented programmer is interesting, as well (especially from a staffing point-of-view where said QA programmers are as rare as an early Triassic mammal). I would also like to further explore whether it may in fact be a healthy goal of a development organization to reach a state where QA is superfluous. In addition I’d like to cover what perhaps is the greatest challenge of Agile development – scaling what works well on a small team to a much larger organization. Finally, it’s fun for me to reminisce on the history of the Guidewire QA team, from pure manual testing to a nascent Test Harness and limited test automation, to the world today where GScript tests and the ToolsHarness infrastructure allow for near limitless automation potential.