I’ve written before about similarities between Waterfall and Agile (Is Agile so Different), but I have recently been forced to conclude that I often struggle to present a balanced debate (hardened agilite that I am)! There are, of course, pros and cons to all approaches, so it’s probably time for a modicum of magnanimity (I think that’s a real word) and for me to consider more than just one side of the project coin. And perhaps the best way to structure my thoughts on this is to examine the “Agile Manifesto” to discover whether the behaviours that Agile rails against aren’t so bad after all.
1. Individuals and Interactions vs Processes and Tools
Agile suggests that the majority of conversations taking place during the development of a product or service should happen naturally. Thus, interactions between different teams in different departments are expected to occur organically as a natural result of the team taking joint responsibility for a great product, and so don’t need to be formalised by the project manager into a pre-programmed schedule. Essentially, Agile supports the concept of self-organising teams communicating, colluding, and creating by following their own values and ideas of how things should be done. Waterfall, on the other hand, offers a number of tried and tested tools (e.g. the stakeholder analysis, risks and issues log, and the work breakdown structure) and associated processes (e.g. Project Startup) to facilitate a project’s operation. Indeed, there is a large project management book of knowledge (PMBOK) that supports this approach, honed to completeness over many years of use. I think that the choice of methodology basically boils down to the way people work vs the way the organisation works. So, in a company such as Sony Computer Entertainment that promotes creativity (and accepts the resultant risk), Agile aligns beautifully with the culture and values of the people that work there. At HSBC, however, there is a legal and corporate responsibility to show compliance with established processes, so Waterfall is the way to go.
2. Working Software vs Comprehensive Documentation
One thing that Waterfall approaches do provide is templates for every facet of a project, for example the Project Brief, End Stage Report, and Quality Register. Traditional Project Managers have a point when they say that something isn’t known if it isn’t written down, and a formalised and documented approach can add consistency and control to an endeavour. Thus, under the auspices of Waterfall we would define project deliverables as being not only the products that the project will deliver but also the underlying documentation supporting the process. A key tenet of Agile development, however, is that only those activities that add (prioritised) business value should be undertaken. As Jim Highsmith (for the Agile Alliance) writes: “we embrace documentation, but not hundreds of pages of never-maintained and rarely-used tomes“. The most obvious project deliverable in an Agile project is working software (or product), so there is often a tendency to avoid creating any documentation when operating in this mode, instead concentrating solely on creating the final release. But this is a dangerous tactic, not least because the product that you create will be basically unsupportable. The trick, then, is to apply a “usefulness” test to judge whether documentation proposed from within (or without) the team is a must-have or just a waste of time and trees. My own “DO” pile generally includes artifacts such as training materials, test scripts, and admin guides, but into the “DON’T” category go Project Initiation Documents, Testing Strategies, and detailed “Solution Design” (whenever we are not sure what we are going to build and thus aren’t ready to design it). So when is comprehensive documentation acceptable? The easy answer: “when you really need it”.
3. Customer Collaboration vs Contract Negotiation
The classic Waterfall approach mandates a phase of detailed requirements-gathering during which all interested parties (“stakeholders”) are asked, with the help of a business analyst, to commit to paper exactly what it is that they want. This document then forms the contract between the project team and the wider stakeholder community, and provides the basis for all subsequent project planning, development, and testing. With a known set of requirements in hand, end-dates, resourcing, and costs can all be estimated and used to control project delivery. In this way, the unknown is reduced to the known. Any changes requested once the project is underway are then subjected to a change control process to ascertain the impact on plans and final delivery. Conversely, Agile relies upon the up-front collation of only broad requirements in the form of “user stories” captured on physical (or virtual) post-it notes or index cards in the format “as a… I want…. so that…”. One resultant user story may be, for example, “as a construction manager I want my team to log time on projects so that I can bill my clients accordingly”. Now it is true that, as it stands, this “requirement” would be extremely hard to deliver against, which is why we must remember that each user story is merely the promise of a future conversation. In Agile, the written word is deemed to be imprecise, so collaborating with the customer by maintaining constant dialogue is the only way to capture initial requirements and their subsequent evolution over time. So, which to use? Well, if pretty much everything is known about the product, for example when a construction company builds a “cookie-cutter” house using a plan that they have followed numerous times before, nearly all factors are known and understood: the work required; the underlying technolology; applicable regulations; material costs and quantities; the requisite skills mix; dependable suppliers; and even the effect of the vagaries of the weather. Would waterfall work in this scenario? Absolutely, it would*. But would the same approach work when attempting to build the same house in, say, China? Probably not, as (here’s that word again) cultures and regulations are different between countries just as policies and governance differ between organisations. Which approach you use, then, is basically dependent upon the extent to which you understand requirements and the underlying technology for delivery.
* Note: Agile can apply to mature industries too. Quadrant brought “standard customisation” to house building as early as the 1990’s, involving the customer throughout the build.
4. Responding to Change vs Following a Plan
Planning is central to running a successful, traditional project – “plan the work and work the plan” being a common refrain in the project managers’ break room! In fairness, it is extremely difficult to predict the future so it seems reasonable that we should expect great effort to be expended in understanding what is required in order to come up with predictable outcomes in terms of costs and outputs. Agile is different in that it only demands detailed plans for the next few weeks, or the “sprint”** in Scrum parlance. This allows the team to concentrate only on those user stories deemed important enough to be included in the current sprint while leaving decisions that pertain to future sprints until, well, future sprints. This is great as the team doesn’t have to worry about the future and change control is effected by merely prioritising items on the next sprint, but it can become hard to predict when the final product is to be completed. In essence, returning once again to Jim Highsmith: “we plan, but recognize the limits of planning in a turbulent environment”.
** Note: each sprint is like a level on your child’s computer game. It only “saves” at regular intervals (at the demonstration of product completed at the end of each sprint) so you must be sure to complete backlog items or risk losing all process since the last “save”. Don’t interrupt a team during a sprint by changing their backlog. That would be like telling your 9 year old to come to have her dinner when she is just a couple of minutes from the end of a level. Restarting activities is time-consuming!
Of course, choosing one approach “flavour” doesn’t preclude the use of elements of the other. Project methodologies are not gangs that insist you always display the right colours or political parties that disagree with each other for the sake of it. In my experience to date, I haven’t found Agile and Waterfall to be completely mutually exclusive; indeed, I use quite a few of the “tried and tested” traditional tools in my agile projects. So, in summary, I would say that if you know the route you’re about to travel then use Waterfall (Pirates Ahoy). If not, it’s time to Get Agile.