Far too many Agile development projects fail. It is hard to even accurately measure the number of failures because so many software development projects end up “complete” and shipped, even though:
- They took far too long to build
- The quality of what we built was poor
- What was built was not what the customer wanted
- The cost to develop was more than it should have been
Over the years, I’ve worked on many different Agile teams and have consulted for and trained many teams on Agile development. In this time, I have found 5 common problems, that if not resolved, are very likely to cause a project to be a failure.
1. Not having a product owner
Of all the things that can cause an Agile software project to fail, not having a person that is ultimately the decision maker for the product being developed is the quickest way to ensure its demise.
It doesn’t matter if you are following Scrum, doing your own Kanban style project or something else; if you want your project to succeed, you need someone who can set its direction and make decisions about the product being developed.
Think about remodeling a kitchen. If you hired a bunch of contractors to come in and do the various jobs, like: plumbing; carpentry; flooring; etc, but you don’t have someone deciding what the actual finished kitchen should look like, you are going to get a random mess.
A few skilled contractors will probably be smart enough to find the right person to ask about what should be done, but it takes more than just making arbitrary decisions about where the cabinets should go to design a kitchen. You need someone to actually come up with a real design and vision and be able to modify that vision appropriately as the project progresses and problems inevitably occur.
It seems pretty nuts to spend a huge amount of money remodeling your kitchen, but not want to invest any time or effort in either designing the finished product or hiring someone to do it.
Yet, day in and day out, I see this exact behavior in software projects. I see companies investing huge amounts of cash in Agile development, but not appointing anyone to be the true owner of what is being built and to set the vision for it.
2. Not iterating
One of the key values that Agile development brings to the software development world is iteration.
What is iteration though?
You may think it means breaking your project into 2 week sprints or iterations. While doing this can facilitate iterative development, it doesn’t mean you are actually iterating.
Confused? Let me explain.
The key to iterating is to develop a product a little bit at time. It would be accurate to describe the process of iterating on a product as evolution of a product.
Whether you believe in macroevolution or not, microevolution, or adaptation is a proven concept. The idea behind evolution is that things change gradually over time. Those small changes add up to result in a big change.
Imagine how silly it would be if evolution worked the way most “Agile” software is built.
Imagine, if you will, a fish that swims in the ocean and has some little fish babies that are born with fully functional legs. Then, those legged fish babies grow up and have fish babies that have wings.
Probably the legs and wings wouldn’t do that fish much good, nor would they be designed properly, because instead of adapting and changing over time, they just suddenly appeared.
Features should not be built in a single sprint or iteration. It is as silly as legs showing up on a fish in a single generation.
Instead, features should be evolved and grow over time. A feature shouldn’t be pushed into a single sprint or iteration and then be done. A feature should start out small and develop and evolve over time as feedback is received and customers or the product owner tries it out.
Far too many times, I see Agile development projects mistake having iterations with actually iterating the development of the product.
Don’t try to ship a completed feature at once, let it evolve over time.
3. Not breaking things down small enough
The main reason why this is so important is because it prevents procrastination. Procrastination usually occurs when either we dread some large task that will be difficult or we don’t know what to do next.
If you can break a big project up into small parts, it will seem easier to accomplish and will have clear steps of progression.
I often see software projects where the person creating the backlogs or work items is not considering the work enough before giving it to the team.
I coined a term for these kinds of backlogs: fatlogs. Fatlogs are backlogs that are not broken down small enough and often are very vague in what needs to be accomplished.
Fatlogs are notoriously difficult to estimate and waste time when trying to explain and understand them. It is critical that fatlogs be broken down into smaller actionable backlogs before being given to an Agile development team or a large amount of time will be wasted.
Many times, I have found that the creator of the fatlog could have easily broken down the work it represents into several smaller backlogs that would be easier to explain and understand, even without knowing much about software development.
I often recommend to Agile development teams that they outright reject fatlogs and send them back up the chain.
“If you can’t take enough time to clearly state what you want, it must not be that important.”
This doesn’t excuse development teams either. Development teams should break down any backlogs they get into small tasks as well.
4. Not setting done criteria
What is the first thing a waiter asks you when you order a steak?
That’s right, they ask you how you would like it done.
If the chef doesn’t know what the done criteria is for your steak, the chef will have to decide what he or she thinks the done criteria is.
You may get back a well-done steak, or a rare steak, or something in between, depending on the personal preferences of the person cooking your meat.
This isn’t a good way to cook steaks and it isn’t a good way to cook software either.
In many software projects I often encounter lots of steaks being cooked but no one defining when they are done. The backlogs most often end up being “done” when time runs out.
It is very important to have explicit unambiguous done criteria for any backlogs being worked on by an Agile development team.
This means that the product owner should be defining some level of acceptance testing. It doesn’t matter if the tests are manual tests or fully automated tests (BATs), what matters is that some criteria is defined by which the team can evaluate whether they have achieved their goal or not.
Lacking this criteria is like a parent responding with “enough” when their child asks “how many more peas do I have to eat?”
Not having done criteria is demotivating and results in the finger pointing game of why the correct thing was not built.
I have found that if you tell someone exactly what is expected of them and what criteria they will be judged by, you get much better results than just throwing them in the ring, slapping them on the back, and saying “do a good job.”
5. Not letting the team be a team
Teams are strange organisms, especially Agile teams.
There are many dynamics that affect how teams act and interact. There are many ways to foster a healthy team and many ways to create unhealthy teams.
A healthy motivated team has synergy, a unhealthy gaunt team can be less effective than all of its members are on their own.
The key to having a healthy motivated team, is letting them be mostly autonomous. Regardless of your political persuasion, you would probably agree that when a country invades another country and sets up an unelected government to oversee the people, there are going to be problems.
The same thing happens in Agile development.
I’m not saying teams shouldn’t have accountability. But if you want to run a software project in an Agile way, you have to let teams self organize and self manage to some degree.
When big brother is always watching and interfering, team dynamics become very different. Naturally, teams usually develop their own cadence, leadership and roles (this is called norms.) However, when outside forces directly interfere with how the team does things, and they don’t have the power to decide their own fate, team members begin to realize that they need to look out for themselves.
Additional agile development resources
I’ve found it to be pretty difficult to find good resources on these topics, but here are a few books I’ve found useful that address some of the issues I described above.
- Succeeding with Agile: Software Development Using Scrum – This book is focused on Scrum but it applies to many different kinds of Agile projects as well. Mike Cohn and I usually agree on most Agile subjects.
- Agile Retrospectives - I’ve found this book to be useful for getting good idea for team retrospectives that can really help to break down some barriers and help teams learn to resolve issues themselves.
- Scrumban and Kanban and Scrum – Both of these books have excellent information about combining Scrum and Kanban and have great solutions for dealing with some of the problems I described above.
What do you think? Have I missed any of the major Agile project killers? Any useful tips for dealing with them?
Let me know in the comments below.
I talked about how moving the finish line is one of the most destructive things to do to a Scrum team, but I wanted to talk about another variation of the same problem, which is scope creep on backlog items in progress.
Responding to change
Agile is all about responding to change. Don’t get me wrong, I am not for one second suggesting that you should specify your requirements upfront before working on the backlog item.
Agile is also about iterative development. We take a thing, get a rough idea of what we want to build, build it and then iterate on it, making it better as we progress and figure out what we truly want.
The specific problem I am after today is about knowing when to allow change and at what resolution to iterate.
I think these are very important ideas that need to be understood to really be successful at Agile software development.
To some people Agile means we allow change at any point in time. I think this view is a little too far on the extreme. There are certain points in an Agile process where change is welcomed and that depends on the process.
Scrum welcomes change up to the point the team has committed to the sprint. After that point changes to the scope of what will be worked on during the sprint are detrimental.
Kanban and Kanban like methodologies embrace change all the way up to the point that the backlog is in progress. At that point a significant change to the scope or focus of the backlog causes more harm than it provides business value. (In general)
What kind of changes are also important to define.
The kinds of changes greatly affect whether or not they can cause harm or be good. Changing the layout of a screen during the development of a backlog usually is not a big deal. Changing the way the screen functions usually is. Changes that add significant functionality also are harmful when the backlog is already in progress.
A little later on I’ll talk about some of the reasons for this. For now let’s look at…
There is a prevailing misunderstanding of what iteration means and how it is valuable in Agile software development.
Many Agile practitioners think iteration applies only to the software package itself and to iterate means to add features.
This resolution is far too large. If the smallest iteration of your software is a complete feature, you probably have fatlogs, and you are going to run into trouble.
You may not ship features to a customer until they are completely evolved, but if you try to complete the evolution of features all in one sprint or cycle, you are probably going to bite off too much and miss out on the true value of iteration.
Just as your software should be iterating by adding features, the features themselves should be iterating by adding functionality.
Think of your software as a tree and features as branches. If you were to watch that tree grow, it would be pretty bizarre if whole complete branches just started popping up on the tree everywhere.
It is much more natural for the branches themselves to grow from little stubs into a large branch that might even have branches of its own.
Tying it back together
Given a backlog is already in progress and you have a good understanding of when change should be allowed in your process, and what resolution iteration should happen, it should start to become obvious why adding new requirements to that backlog is a bad idea.
If you take an existing backlog in progress and add new requirements to it, you’re introducing change at the wrong point, and possibly trying to “poof” branches onto your software tree, instead of truly iterating.
It is much better to add a new backlog that will serve as an iteration of the existing feature than it is to try and tack on new requirements to the work already in progress.
Well, hopefully our discussion of change and iteration resolution has given you some idea of why, but there is much more depth to the question than that. Let’s discuss some of them.
There is a huge psychological portion of software development. We tend to be motivated by accomplishment and feelings of a “job well done.” We tend to be demotivated by goals changing out from underneath us and feeling like a job won’t ever really be done.
Let me give you a simple example. Imagine you tell your child that their chore for the day is to make their bed.
They go and do it, come back to you and tell you it is done, then ask for you to check it. When you check, you see that the bed is made, but you say “the floor also needs to have everything picked up.”
So they go and do that, it takes a little longer this time, because they have lost some motivation, but they finish and come back to you again and you see the floor is now cleaned up, but this time you say “the floor needs to be vacuumed.”
In this situation although each request is reasonable, the building of the requests on top of the original task has the effect of demotivating the individual and deprives them of their sense of accomplishment.
When you do this in software development, a similar thing happens. You also lose the trust of the team because the team starts to think that when the business says X they are really going to want X + Y + Z. The team starts to work much more slowly because it doesn’t really matter anyway. You do not want to operate in this mode! It is very bad!
Not letting evolution takes its place
If you try to cross a river by taking a huge leap across it, you are either going to get wet or you are going to make it. If the river is big, you will mostly get wet.
It is much better to cross a river leaping from stone to stone until you are all the way across.
Adding a requirement to a backlog in progress can be like making a big leap. By adding another backlog to bring you closer to the goal, you allow the current backlog to be a step, and the new backlog to be the next step.
It is a judgment call whether or not a change to an in progress backlog represents a new requirement which should be a new backlog or not. I would err on the side of making new backlogs rather than adding requirements though. I generally say if it will increase scope significantly it should be a new backlog. Either way the work is going to be done, so why not have your…
Estimating more accurate
Nothing kills historical estimations like scope creep. If you want accurate data, don’t add unaccounted for work. Squeezing new requirements into in-progress backlogs adds work to your historical data that is not properly accounted for.
It is much better to catalog the work and account for it than to tack it onto a place where you will not have a way to extract it back out when you are trying to analyze your data.
Consider also the loss in quality as a developer assigned to the expanding backlog starts rushing the work in order to get it done because they feel like they are going to be measured against the backlogs they completed, and work added to an existing backlog isn’t going to be counted in the metric.
Don’t forget prioritization here. If you tack a requirement onto a backlog in progress, you can’t prioritize that work. You may add that requirement at the expense of delaying other higher priority work in other backlogs. Make the new requirement into a backlog and you can prioritize it correctly.
If you are a product manager, product owner, or business person reading this, give it a try. You will be amazed how much more happy and productive your team is if the next time you are working on a backlog with them and realize it needs to account for X and Y also, you announce that you will create two more backlogs instead of tacking it right on.
If you are a developer on an Agile team, you may not have even realized this problem might be sapping your motivation. You might not be in a position to do much about it, but you can always suggest that the new requirement be a new backlog.
This book is a self-published book, which is really a collection of essays. Be prepared for that before you read the book, so that you know what to expect.
It is not a book that tells you how to exactly do Kanban or Scrumban, but it gives you ideas about Kanban and lean concepts and how to solve certain problems you might face in implementing a Kanban process.
Some parts of the book might be a bit difficult to understand without a background in lean methodology, but overall most of the common problems and solutions are understandable. This is definitely a book you could study and contemplate on as you read through the material and look at the charts.
- Excellent use of diagrams and pictures of real Kanban boards to illustrate points.
- Lots of different solutions for common Kanban problems.
- In depth talk about strategy and reason behind different possible solutions.
- Several unique perspectives on many of the concepts presented.
- Not exactly an easy read. It is a little rough and some of the concepts rely on knowledge outside of the context of the book.
- Some of the assumptions are a little broad and aren’t backed up by much real data.
What I learned:
This book really expanded my mind about different ways of implementing Kanban, and solutions for some of the common problems. Some of the solutions surprised me, but made sense.
I learned more about the reasoning behind limiting the WIP to force the system to self-correct. I had not considered that strictly limiting the WIP would force freed up resources to automatically move to help resolve the clog in the pipe since they will not be able to take in new work.
Corey presented a great technique on planning by having a just-in-time planning session when the number of items in development reaches a certain threshold. I had heard of this technique before but had not seen a practical implementation of it.
I would definitely recommend this book to expand your thinking on Kanban software development. If you are looking for instructions on exactly how to set up your Kanban process, this book probably won’t help you, but if you are looking to understand more about the why of Kanban and how to solve particular problem or pain points, get this book.
In our morning standup meeting one of my team members made a pretty good point about story sizing not being so important.
In Kanban we strive to have relatively same sized stories because it allows us certain benefits:
- More accurate metrics on estimated time in queue for any story to be delivered.
- Reduced dependence on estimation and planning.
- Continuous flow as opposed to slow… fast… fast… slow.
I thought you said the story sizing is not so important?
Right, exactly. It is important for them to be relatively same sized, but it is not so important that they be exactly the same size.
Basically, we don’t want to have mice and elephants thrown in together.
It is the difference between breaking a candy bar in half to share and measuring out prescription medication.
You can look at the candy bar and say “yeah, that looks about the same size,” but you’d better not do the same thing with medication.
It is a complete waste of time to go overboard with the splitting of the candy bar, getting it even to the exact milligram.
It is a complete waste to try and get stories estimated down to the hour.
Size ‘em up against each other
In that case how do you actually size the stories?
Just compare them against each other. When looking at a new story, consider the other stories you already have on the board. Is this new story like an elephant in comparison to a bunch of mice? Is it the other way around?
If so, adjust the story to make sense. Don’t trust your gut without comparing it to what is out there already, otherwise you’ll start to drift in your estimations.
Don’t worry too much if you’re wrong sometimes. As long as you are close, it will all balance out in the end.
Also don’t forget to get rid of all the Blue Whales. You don’t want any of those on your board, ever.
In my last post, I talked about some of the problems of Agile processes and suggested the introduction of a new process which takes some of the best parts of each of the popular processes and condenses them into the Kanban lean-style-thinking of limiting the work in progress.
For this post I want to clearly define what I am calling “Kanbanand” with two primary goals in mind.
- Allow someone who is interested in doing a Kanban like process to find enough information to be successful.
- Create a common thread for people who are already basically doing what I am suggesting.
Three Pronged Fork
I want to present Kanbanand as a three pronged fork. In my mind there are 3 distinct areas of a successful Agile process which must be addressed.
- Business and overall encompassing project process.
- Development practices, including standards and best practices.
- Infrastructure: build systems, deployment, source control.
We must know how to run our day to day process, how we physically construct our product, and how we actually build and deliver the product.
- Project is broken into stories which are small vertical slices through the system which are prioritized.
- A Kanban board is used to make the progress of a team visible and has limits to control the Work In Progress (WIP).
- At least once a day team members meet around the Kanban board in a stand up meeting to talk about the progress of the board.
- The team meets on a regular basis to have a retrospective to discuss how to better improve their process.
- Teams always pull in work, work is never “pushed” in.
- Once work has been approved as “done” it can never be “not done”.
- Find something to measure and measure it.
- The team is responsible for, and empowered to determine how the team accomplishes its goals.
- Roughly same sized stories over irregularly sized stories.
- Stories specifying what, over stories specifying how.
- Stories that are one sentence over stories that are one page.
- Talking about how to move the board forward over talking about what I or someone else did.
- Cross functional teams over specialized teams.
- All code is unit tested.
- Static code analysis tools are used to set code standards, all code must meet those standards.
- Any code that is done, is completely done. It is refactored, it is unit tested, it meets the quality standards.
- Code is never commented out. Anyone is allowed to delete any commented out code at any time.
- Determining code is meeting the quality standards must be done in an automated way.
- No one owns code, anyone can work on any code.
- Documentation is ONLY created when it will serve a known purpose and has an immediate value. (Code should document itself with good names and simple flow, unit tests document the low level requirements of the system. Automated tests document the behavior of the system.)
- Testing requirements come from the customer, not from the developer.
- Test driven development over writing unit tests after the code is done.
- Pair programming over working alone.
- Readable code over smaller sized code.
- Doing things right over doing things fast.
- Showing the customer over asking the customer.
- Impromptu discussions at white-boards over scheduled meetings.
- Building tools that will help make completing ten things faster, than completing one thing faster.
- Writing automated tests over manual testing
- Code is continuously built and integrated as soon as it is checked in.
- Builds build code, run unit tests, run quality checks, produce deployable bits.
- Builds fail if any unit test fails, or any quality check fails.
- Code is only deployed from the exact bits that were produced by the build machine.
- Code can be deployed to any environment with the push of a button.
- Production data never goes to any other environment.
- Developers can always run a local build which is the same as what will be built on the build server.
- Moving forward to fix problems over rolling back.
- Integrating soon over merging later.
- Quick builds over slow builds.
- Deploying on demand over deploying on a schedule.
That is my first crack at it. I have tried to keep the process as simple as possible, but still include as rules the things that I think are most important to be successful. I tried to keep the things that involved judgment calls as values. I believe this provides enough information to allow an organization to create a custom process that will work for them, but will still have enough controls to be consistent in the things that I feel make Agile practices valuable.
Let me know what you think or if you have any suggestions. I will update this post as this list evolves. I am sure there is something I missed or could state in a better way.
Like I said before, if you are doing some form of Kanban, you probably are already doing Kanbanand. If you are doing Scrum, your major change would be to drop the planning and instead try to get same sized stories while limiting your WIP.
I was originally going to call this post, “why I chose Kanban”, then I realized I didn’t choose Kanban.
What I chose was Kanban and most of the other “Agile” stuff that I consider good practices and find valuable. There really isn’t a good name for this and when people say they are doing Kanban, it really doesn’t say much. Hence, I am coining a new phrase (Kanbanand), which I hope will be much more specific than saying I am doing “Agile or Kanban development”, and not quite as specific as saying “I am doing Scrum”.
Agile is pretty vague
There is a major problem in the Agile community today. The problem is “Agile” is too broad and involves a large number of assumptions about what Agile involves. There is not much that defines Agile even though many types of best practices are considered to be Agile.
Which of these best practices are clearly defined in the Agile manifesto or Agile principles?
- Test driven development
- Paired programming
- Daily stand up meetings
- Progress board with swimlanes
- Having a product owner
- Stories or backlog items
- Continuous integration
- Automated testing
The correct answer is none of them. Don’t get me wrong here. I am not attacking Agile. I am just saying that when you are saying “I am doing Agile development”, what you’re saying means different things to different people, and most people don’t even realize what the Agile Manifesto actually says, or what the Principles Behind the Agile Manifesto says. Instead they have this general idea of what Agile is and it may include one, all, or none of the items I listed above. (Actually, I would venture to wager that most people think Scrum and Agile are synonymous.)
Scrum, XP and Kanban
Scrum and XP don’t really suffer from the problem that Agile in general does. They are not really vague at all. Actually, to be more accurate, they are not really vague about process at all. Scrum is a little vague about development practices. Either way, both of these processes suffer from a different problem than Agile itself. No one is actually doing them. Well, hardly anyone is.
Lots of people say they are doing Scrum. But what they are actually doing is what is known as Scrumbut. “We are doing Scrum, but… “. And no one that I know of is actually brave enough to claim they are doing XP.
The problem is these processes are just a little too restrictive, a little too prescriptive.
Kanban is out there on the other end of the spectrum. It is sitting closer to Agile. It is the least descriptive. Heck, Kanban doesn’t even have to be Agile at all! I am sure most people that are doing Kanban are doing it in an “Agile” way, but there really is nothing about Kanban that says anything more than you have a Kanban board and you limit your Work In Progress (WIP). If you have a whiteboard and you can put a number on that whiteboard, you are doing Kanban!
Kanbanand is good in the same way that Scrumbut is bad.
The goal of Kanbanand is to be a fully Agile process, adhearing to the Agile Manifesto and the Principles Behind the Agile Manifesto, as well as process that is not as specific as Scrum or XP, but specific enough to be distinct and recognizable. The basic idea of Kanbanand is to take the simple non-prescriptive, open framework of Kanban and formally add some of the best practices that are considered “Agile” as requirements and others as guidelines.
In the spirit of the 80/20 rule. I want someone saying they are doing Kanbanand to be able to convey with just those words, 80% of what their development process looks like.
In my next post I am going to attempt to define what I will call Kanbanand. I will proceed with the goal of making Kanbanand be much of exactly what most Kanban teams are already doing, just formalized in a way that is specific enough that someone doesn’t have to know the undocumented features of Agile development to understand it.
If there is enough support around this idea of creating a more formalized version of Kanban that more specifically states the process, and is still Agile, I intend to create a website and community dedicated to this idea. Contact me if you are interested in participating. I think we are at the point where the only thing holding Kanban back is that there isn’t a formal enough definition of exactly what it is. I believe once we have a real Kanban process that can be implemented and clearly identified many of the teams and organizations that intially thrived under Scrum, but are now held back by the strict time-boxing, will jump to our side of the fence and embrace the idea of limiting WIP.
In doing so, I had to really think about why exactly I think waterfall is bad.
The feedback loop
It really came back to one central thing for me, which is the length of the feedback loop. When we are using a waterfall process for software development, the customer doesn’t really have a chance for feedback until the end of the project. Even the testing is done at the end of the project, so that feedback is coming very late.
The biggest strength Agile gives over Waterfall, is the tightening of that feedback loop. The quicker you can get the feedback, the quicker you can respond to it and the less time you spend going in the wrong direction.
From a programmer’s perspective, think about how many of us used to write code before the tools came along to help us, and before we knew any better.
Write a bunch of code for 2 hours
- Run the compiler
- Fix an error
- Repeat 2-3 until all errors are fixed
That wasted a large amount of time. Especially when we had some syntax wrong that affected large portions of code.
Using a modern IDE, we are able to see syntax errors getting highlighted the second we type them. We can start to type a method name and auto-complete can take over and offer us suggestions. It it difficult to write bad syntax because we are getting constant and immediate feedback as we write out code. The feedback loop is so tight that we don’t even consider compiling to be a separate step. There are even some IDE plugins which will continuously run any unit tests which may be affected by your code change in the background as you are making the change.
If the modern IDE experience is Agile, then Waterfall is punch cards.
Taking the best of all
So, here is my prezi presentation. Please give feedback if you have some. Remember this presentation is designed to be a very basic beginner’s guide to why someone should choose Agile over Waterfall. It is not going to go into very detailed topics about Agile.
Also, at the end of the presentation you will notice that I suggest taking some of the process of Scrum, and the XP best practices, and running Kanban using those guiding principles. I currently believe that is the best mix of what Agile has to offer.
I have a prediction…
I’m becoming more and more confident of it everyday…
When I shake the magic 8 ball, it consistently says…
SCRUM WILL DIE
Now, don’t get me wrong here. I like Scrum. I think it is a good thing. I think it changed the way many organizations think about software development, but I also think it is not so good.
I remember when Scrum was first gaining traction. I thought to myself, is Scrum a fad? I knew Agile was around to stay because the amorphous thing that is Agile is a bundle of good practices that really make sense, but I often wondered about Scrum. I wondered if Scrum was over hyped by consultants trying to make money coaching people and trainers trying to make money training people to be certified ScrumMasters. I especially questioned it when I noticed that the only requirement of becoming a ScrumMaster was to breathe the air of a ScrumMaster trainer for two days. (I know there is a trivial test now.)
The failings of Scrum
Scrum addresses many problems of a development team, but it also tends to create problems. After being in Scrum environments for several years and talking to many other people in organizations doing Scrum, I’ve identified what I think are the main failings of Scrum.
Planning is a total waste of time. It provides no benefit to the bottom line. The supposed benefit of planning is to be able to identify how much work a team can do in a sprint and use those metrics to plan releases and schedules.
The concept seems solid. It seems to make sense, but the problem is we are devoting one day of a two week sprint to an activity that basically gathers metrics. That is a 10% overhead if your planning session doesn’t go over a single day. Essentially, planning’s primary purpose is to chart velocity and commit to some amount of work. Let’s break some of this down.
Velocity is the amount of work a team can get done on average based on how much work they have been able to get done in the past. What comes out of planning is based on estimates teams give to backlog items during planning.
Again, seems like a solid concept, but here are some of the problems. First of all, estimation is almost always wrong. The only way to get a real accurate estimation is to talk about the backlog item until you have squeezed all the requirements out of it up front. And if you do that, then your planning will probably take two days.
Even if you manage to get accurate estimates, if your team is not stable and your sprint length is not stable it won’t really matter, because you will be taking the average out of something that itself is not stable.
One central unspoken theme of Scrum is commitment. The team commits to get a certain amount of work done in a sprint. The business commits to not change the work the team is working on. These commitments are critical, because without them time boxing fails, velocity metrics fail, and trust is lost in both directions.
So, what is wrong with commitments? They cannot be followed. Everyone means well, at least I hope they do. The problem is, like a fat kid in a candy store, they just can’t help themselves. The business doesn’t want to change priorities, but a critical issue comes up. The development team wants to commit to the sprint, but the development team can’t make more code get done faster simply by wanting to. They can add more hours to the sprint, but then they are skewing the velocity. The only way the development team can realistically commit to the sprint is to ‘pad’, and that is a very bad word. Don’t ever say ‘pad’ in a development shop.
I’ve tried to fight it. I have closed my eyes and said “NO! NO! NO! There is a proper way to do tasks without copying and pasting,” but it’s simply not true. I have tried many different techniques, have asked many other Scrum teams, but everyone essentially copy and pastes the same set of tasks. The problem is, you don’t know the tasks for most backlog items until you have had some kind of design session to decide how you are going to implement the solution. I just cannot see a way around that simple plain fact. So, what we end up doing is making up templates and tasks that are generic.
Scrum says there should be one product owner. I agree with this point. In theory it works great. In practice it ends up putting a whole product’s life into the hands of one person. Which person? Well, if you make that person someone high up in the organization, they won’t be the product owner for the team, but rather the boss. The team will just do whatever the product owner says and stop being autonomous and self-managing. If you choose someone lower in the organization, you end up having someone who cannot make decisions, and now you have all these high up business people, who are basically worthless, trying to find ways to be worthwhile (which usually results in them abusing the poor product owner.) I like to call this the “product renter.”
These seem like a really good idea on the surface level. As a ScrumMaster, I tried to make these work correctly. 15 minute stand up meetings, you say what you did, what you will do, and what’s impeding you. The problem is, if you are working together doing pair programming, and “swarming” on backlogs, you already know what is going on and you are just repeating some rhetoric. If you are not working together on backlogs and are working in isolation, 15 minutes is not enough time. So what ends up happening is you either have a 1 hour long meeting where everyone talks about technical details of issues they are having, or you have a 15 minutes meeting where everyone repeats the same stuff everyone already knows.
What next then?
I don’t think Scrum is particularly bad. I actually like Scrum and think it can be a great process for a team that can help them to learn to use Agile techniques and vertically slice the system instead of trying to horizontally slice it. The best parts of Scrum are really Agile. Scrum is kind of forcing you to be Agile and do things like creating user stories, test driven development, having always releasable code, etc.
I wrote about the Kanban before, and I am still thinking that it is the next evolution of Scrum, but I can’t say because I haven’t actually done a project using Kanban. It seems to be the way that the Scrum project I am working on wants to evolve on its own.
So, why even use a process at all, or why not make our own? Rules and process are important because they keep us from arguing about how things should be done, and they keep us from wasting time reinventing the wheel. The trick is in having your process lightweight enough that it is not burdensome, but constraining enough that you can enforce good patterns and principles without having to argue about and debate them each time.
So my advice is to learn what you can from Scrum, but don’t think it is the “be all end all.” Scrum is a fad, it cannot last, at least not in the form it is now, because it has problems. But hold onto those Agile principles you learn from using Scrum; those principles will last much longer, if not indefinitely.
I am almost jumping on the Kanban bandwagon, but not quite. I don’t really like to be on any “bandwagon” because when you are you seem to get stuck in a rut and don’t see the next improvement that is coming along.
What is Kanban?
Let me give you the simple simple and you can look up what other people have to say about the details.
The simple simple is this: Kanban is a pull model where you are strictly controlling work in progress (WIP), by limiting the amount of work that can be in each phase of development. It eliminates the needs for time boxing and committed sprints and replaces it with just in time priorities and time in queue instead of velocity.
Is it still agile? In my opinion yes and even more so. Is it still Scrum? Only in some of the principles, but really it’s a different animal.
Here are some good links to people who have written about Kanban:
Excellent presentation here:
The reason I am starting to move in this direction and away from Scrum is mainly that very few people can actually get Scrum working. Let me list out some of the most important issues I have been seeing:
- Missing top level buy-in. (Scrum pretty much requires this since you have to have the business commit to an iteration and commit to what it means to be a chicken.)
- Planning meetings take up time and provide no real value. Estimation provides a value for planning releases and figuring out what work can get done but itself does not actually provide a tangible value.
- This is related to 1, but the business likes to be able to change priorities more frequently than 1 sprint. (In general I have seen this to be true)
In some ways I am starting to wonder if Scrum is like training wheels. It takes a large amount of dedication and attention to rules to make it work, and if you are missing the top level buy in, you can pretty much forget about it.
I am seeing Kanban as taking the good things that we learned doing Scrum and applying them in a less restrictive sense. I am really liking the idea of focusing on controlling work in progress versus controlling time in which work is done. There is a subtle difference here, but I think it is important. I am recognizing there are many things in Scrum that we are doing that are not really adding bottom line value and most of these issues really rely on the time boxing aspect of Scrum. If you take away the time box, but keep the idea of working on items in priority, limiting the number of items you are working on at a time, and the agile development methods which are used in Scrum you get pretty close to Kanban already.
Am I 100% sold on Kanban yet? No, but neither was I on Scrum. I really think the best approach may lie somewhere in between a Kanban + XP type of process. See my post on pair programming. Perhaps I’ll get a chance to try this out. I already bring a large amount of XP to my Scrum, because XP has very good software construction practices.
Kanban is also used in the manufacturing industry to determine what to produce, when to produce it, and how much to produce. The Kanban efforts are then synced up with a logistics company, such as LeSaint Logistics, to streamline the entire lean production process.