Make Your Software Better with Better Processes
You just released the latest version of your product. You and your team slogged through way more than the standard eight hours a day in order crank out the release on deadline. Like on all the projects before, that deadline was moved at least three times because you were playing whack-a-mole with bugs.
You’re drained. Your team is drained. You dread starting the next project because you know it will end the same way. You feel like you’re barely staying ahead of the bugs and deadlines. You know you can’t keep working in this chaotic style, but you don’t know how to fix it.
I’ve been there, and it’s awful. But there is a solution: use a better development process.
I used to think development processes were for chumps. When I first learned about development processes in college, I pooh-poohed them as unnecessary, restrictive nonsense that got in the way of writing actual code.
Processes felt like they forced developers to build software in unnaturally rigid ways and made them write a lot of documentation instead of code.
Since then, I’ve learned how useful processes actually are when they’re used the right way.
I’ve learned how using defined processes can be used to make better software, faster. These processes can make it easier to control changing requirements and manage expectations, and can help to ensure that what you build is what the customers actually want.
I’ve learned that instead of constraining me, processes can help free me to do my job better.
In short, using defined processes has helped me level up my coding. By improving my team’s processes, I’ve been able to control feature-creep, improve team morale and productivity, and reduce the number of bugs.
I’m going to show you how different processes, coupled with particular practices, can address specific problems and challenges you and your team face.
What is a Development Process Anyway?
A development process is like a recipe: it tells you what ingredients you need and how and when to combine them to make a “dish,” that is, a piece of software. Like recipes, there are different processes for the same dish and some have more details than others. Some specify the exact amounts of different ingredients while others just give general guidance.
The “ingredients” of development processes are practices. A soup recipe may specify two cloves of garlic, four cups of chicken broth, etc. A development process may specify that the team always does work in four-week iterations. A few more examples of practices are:
There are more practices than these, but this list is a good example of practices specified by some specific processes.
Like ingredients, practices need to be combined in certain ways. In a recipe, the steps define how and when to combine the ingredients. In software, the analogue to those steps are the phases of the Software Development Lifecycle (SDLC). These phases are:
- Project definition: figuring out what problem the software will solve
- Gathering and refining project requirements: determining the features the software needs as specifically as possible
- Designing the solution: defining the architecture of the software at both a high level, and lower, class level
- Implementation: writing the code
- Testing: testing the code, including unit, integration, system, and acceptance tests
- Releasing/delivery: getting the software into the hands of the customers
- Maintaining: fixing bugs and adding features in existing code
All development processes prescribe practices for some, or all, of the SDLC phases.
Older, traditional development processes follow models like waterfall and iterative and incremental development, which are more rigid processes. Like making Campbell’s soup in a factory, you follow them step-by-step and don’t move on to the next until you’ve completed the one you’re on.
In contrast, the most popular processes today are Agile processes. Agile processes are more like when your grandmother makes soup. She doesn’t rigidly follow a recipe and constantly modifies it as she goes.
Agile processes were created by developers who realized that processes should fit with how software is really made. For example, stakeholders often don’t know exactly what features they want or how the software should look. This uncertainty requires the development team to go back and forth to refine the software as both sides gain better understanding. Agile methods embrace this reality through flexibility.
More specifically, Agile processes differ from the traditional ones in that they:
- allow SDLC phases to occur in any order, and even simultaneously
- adapt to project changes instead of predicting them
- keep iterations short or remove them completely
- communicate with project stakeholders (anyone who’s involved with the project) often and in-person whenever possible
- make working software early and often
- minimize documentation and practices that don’t directly add value to the stakeholders
Scrum and Kanban focus on scheduling and managing what tasks a team works on.
Lean is more of a process philosophy that pushes teams to minimize tasks and practices that don’t add value to the stakeholders.
XP prescribes practices for collecting and defining requirements, coding, and testing.
Because none of these Agile processes specify practices for every phase of the SDLC, they can all be tailored with specific practices. It’s like when you add garlic to the soup because your family loves the taste.
By carefully choosing and combining processes and specific practices, you can make a great process for your team.
How to Choose the Right Process for Your Situation
Every project and situation is different, so choosing a process and related practices should be done carefully and explicitly. I would caution against choosing a process just because that’s the one you’re most familiar with, or because it worked for your last project, or because every other team uses the same one.
There are a lot of factors that can influence how well a process, and specific practices, will work for you. Some of those factors include whether you’re building new features or mostly maintaining existing code, if you’re building safety-critical software, if you’re working on strict deadlines, and how big and cross-functional your team is.
That said, there are some common situations where a development process can help. This is not an exhaustive or comprehensive set of scenarios, but is intended to help get you started.
I’m going to focus specifically on Agile methods and practices because they tend to be the most common these days. Their popularity is due to how Agile teams can deliver high-quality software that matches the stakeholders’ expectations faster than most traditional processes can.
Situation 1: You’re fighting with endless bugs
Constant bugs are an indication that you need to improve your code quality. The bugs may be due to multiple, or a combination of, factors.
For example, the management style your team works under can lead to bugs if your team is pressured to release code before it’s done or given deadlines that are unrealistically short.
Bugs can slip into production releases when your team doesn’t have enough tests, or when the tests you do have aren’t high-quality and don’t cover the important parts of the product.
Also, if the design of the product is fragile and not easily modified, you may get stuck in a whack-a-mole game of bug-bashing.
Even who’s on your team can affect how many defects are created. Teams with more inexperienced developers who have little or no guidance from more senior developers will likely create, and not catch, more defects.
If you believe that your challenges are partly due to how your team’s tasks are managed, then adopting a process that focuses on task management is a good start. Popular ones that I would recommend are Scrum and Kanban.
Both Scrum and Kanban prescribe practices that schedule and control what, and how many, tasks a team works on for a given timeframe.
Scrum uses the iterative and incremental development model and splits development into defined “sprints” of predefined length (canonically, four weeks). There’s a sprint planning meeting before each sprint where the team refines tasks (details and requirements), estimates the effort, prioritizes tasks and places them on a list of tasks (the sprint backlog) to be completed during the sprint.
Kanban, by contrast, is more fluid. It uses a continuous queue of work to feed the team with tasks. A new task can be inserted into the queue wherever the team and stakeholders believe it should go. Otherwise, the team controls the work in progress for each team member to minimize bottlenecks or downtime. In this way, Kanban keeps the team operating smoothly and efficiently.
While both of these processes will help tame your workflow, they won’t directly affect code quality caused by designing, coding, and testing practices, or developer inexperience. To address those challenges, the following practices can be used with the processes above:
- Test-driven development
- Behavior-driven development
- Continuous integration
- Inexperienced developers
- Pair Programming
- Code reviews
Obviously, a number of these practices affect multiple aspects of the implementation phase. Choosing practices that reinforce each other, like test-driven development and continuous integration, is an important aspect of making your process (and software) better.
Situation 2: Your requirements keep changing
Constantly changing requirements is, in my experience, unavoidable. It stems from the fact that most customers don’t truly know what they want until they see it. Often times, stakeholders have a general idea what they want, but not the specifics. So the developers, who are the software experts, need to help them think through the details of the solution.
Generally, then, changing requirements is something to accept as a task management challenge. Agile methods are built around the idea of an evolving project so using one is ideal.
Scrum is one of the most popular Agile methods currently. In my view, it is best for projects that are mostly developing new features, and not for those in maintenance mode.
Choosing your sprint length can be an experiment to see what gives the team and management the best balance between focus and handling new requirements. Originally, Scrum prescribed four-week sprints that were “sacred” and could not be interrupted by management. In my personal experience, I’ve had to change the sprints to one week in length to keep management happy and keep the team focused and productive.
Like I mentioned before, Kanban uses a queue of tasks instead of sprints with backlogs like Scrum. Because of this aspect, it works well for teams in both greenfield (building new software and features) and maintenance development. Kanban allows teams to add new tasks to any point in the queue and also (usually) allows for one or two “emergency” tasks, but no more.
Kanban, therefore, has less work that needs to be protected. Where Scrum requires the team to protect weeks of work, Kanban has to protect just what’s being worked on right now.
Furthermore, since Kanban’s task management is more fluid, it removes the need for planning meetings in the way that Scrum requires. These meetings can take a long time — one team I was on would take one or two days to complete sprint planning. It was agony.
Situation 3: You’re constantly missing deadlines
There can be many causes for missed deadlines; generally, they’re not the result of any one thing. Some of the probable causes are:
- Unrealistic deadlines
- Poorly defined and/or constantly changing requirements
- Poor estimates of how much work can be done
- No visibility into the team’s progress
These causes are usually interrelated. Unrealistic deadlines are often created because the team overestimates how much they can do and doesn’t track their progress so those estimates don’t get better.
Deadlines can also be set too soon because the requirements aren’t well defined upfront and the team (or company) doesn’t have a history of project tracking data to determine how past projects went.
In this situation, adopting Scrum or Kanban is a good start. Scrum tracks “velocity” (how much work a team can complete in a typical sprint) and Kanban tracks the average completion time for tasks.
Canonically, Scrum prescribes using story points to estimate and measure the level of effort tasks require. Story points are abstract units used to measure a task’s effort relative to other tasks. In my experience, traditional management doesn’t like them because they can’t say exactly how long a task will take. That doesn’t mean they’re not a good tool, however.
To track progress, Scrum has “burndown” charts that track the team’s velocity against the estimated amount of work remaining. These charts are helpful for determining whether you’ll hit your sprint goals during your sprint.
If more granular estimate and tracking detail is needed, you can do what I did on a team that had this problem: at the end of every work day, update how much time you’ve spent on each task and how much more time you’d estimate you will spend on it.
This practice makes the process a bit more heavyweight, but you get a much better idea of how good your estimates are and you can catch bad estimates earlier. You will also see, with higher precision, how far along you are and whether you’ll successfully complete your sprint.
You can also try planning poker. This practice is a more Agile-friendly version of the wideband delphi estimation method. If you’re working with deadlines and your team’s not estimating, planning poker is a good way to start.
Missing deadlines boils down to managing expectations. Unrealistic estimates and promises are almost always made because:
- there is no, or not enough, data to make historically based estimates.
- there is a desire, or pressure, to please the stakeholders.
- the team doesn’t want to appear incompetent.
- the team didn’t think the details of the task through enough.
I fall into this trap, too; it’s easy to do. I’ve even seen former-developer-turned-managers pressure dev teams into giving unrealistic, overly optimistic estimates. But the truth is, if you adopt a process like Scrum or Kanban and follow it, you can solve those problems that lead to poorly managed expectations.
For example, using a development process that helps track your team’s efforts can combat unrealistic estimates. Before I took a serious look at development processes, my “estimates” were no better than gut-feel guesses. I didn’t really care at the time either, because it didn’t seem like it had any impact. That was my mistake.
What I didn’t realize at the time was that making unrealistic, uniformed estimates creates many problems, starting with the stakeholders mistrusting the team. The team is then stressed because they feel they need to deliver what they said they would, when they said they would. This stress leads to burnout if the team goes through this cycle too many times, and it demoralizes the team.
Eventually, this cycle will make the team apathetic. If deadlines constantly come and go, and estimates aren’t taken seriously, then the team members will begin to care less and less about committing to anything definite.
These problems can seriously impact a team. I’ve been a part of a team that had these exact problems. It was not a fun experience.
To combat these problems, my team turned, once again, to our process. We started tracking more data on tasks and progress which led to more realistic sprint plans. Our realistic sprint plans led to better morale and productivity. Without our process, and our careful improvement of it, we wouldn’t have had the structure to collect meaningful data that we could use to manage stakeholders’ expectations.
When in Doubt, Start with a Lightweight Process
There are two different approaches you can take when choosing a process:
- start with a lightweight process and add practices as necessary.
- start with a heavyweight process and remove practices when you determine they’re not helping.
A lightweight process is one that doesn’t prescribe or enforce very many practices, while a heavyweight process tends to prescribe many practices. Agile methods are lightweight, but they’re not all the same “weight” — some Agile processes can be heavier than others. For example, Kanban is lighter than Scrum.
An example of a heavyweight process is the Rational Unified Process. It prescribes many roles for the team members, as well as many steps and phases that projects must go through.
I feel that in most cases, starting light is preferable because it will be easier to adopt at first, and small additions and changes will be easier to make.
Unless you’re working on safety-critical systems, starting with a heavyweight process is usually not necessary. Because of the number of practices you have to implement, it is more difficult to adopt and you’ll feel pressure to reduce or remove the most onerous aspects before you fully understand what benefits they provide.
Specifically, I would recommend Scrum if you need a bit more structure and are working on greenfield development and a lot of new features. A good book to introduce you to Scrum is Agile Software Development with Scrum.
I would recommend Kanban if you’re maintaining existing software more than building new features or if your team doesn’t need as much structure (e.g., no deadlines). To get started, I would read this book: Agile Project Management with Kanban.
In either case, heavy or light, you need to constantly reevaluate how your process is working for your team. Projects, teams, and situations change; revisiting and reevaluating your process is the only way to adapt and make sure your team continues producing great software quickly.
A good practice to use for those reevaluations is retrospectives. Scrum specifically prescribes sprint retrospectives, but retrospectives can be added to most processes.
In short, retrospectives are meetings at the end of iterations/sprints/set timeframes where the team discusses what went well, what didn’t, and how to address the challenges. You then take those ideas and implement a few in your process going forward and see how they work. View it as a continuing experiment to find the best process for your team.
A good book on how to do retrospectives in an Agile process is Agile Retrospectives: Making Good Teams Great.
Never Stop Improving
Changing your team’s development process, or modifying it, starts with stepping back and objectively identifying what’s working and what’s not. To get started, here are some questions you can ask yourself and your team:
- Do you constantly fight new defects?
- Do you consistently miss your delivery dates?
- Do you just have too many people telling you what to do?
- Do new features keep getting added as ‘must-haves’?
- Are you constantly waiting for other teams or companies to deliver their responsible parts?
Step back and evaluate where your team could improve. Do some research on what processes might solve, or minimize, those problems.
Process improvement is a continuous process. You and your team should regularly visit what’s working and what’s not. It’s very easy to get comfortable and complacent in the process you know, so building a routine around constantly improving is critical.
Remember, processes are designed to help you make better software. Different processes will help you achieve that in different ways. There is no one-size-fits-all process. Implementing an off-the-shelf process is only the beginning.
Your team’s combination of challenges are unique and will change over time. Your process and practices should change to address them. Making the best process for your current situation involves questioning your assumptions about what you can and should do, and taking action. Don’t be afraid to add or remove practices or change your process wholesale. These bold moves are what helped me make my software better, and I know they can help you too.