Category Archives: Agile

5 Ways Software Developers Can Become Better at Estimation

In my last post, I detailed four of the biggest reasons why software developers suck at estimation, but I didn’t talk about how to solve any of the problems I presented.

While estimation will always be inherently difficult for software developers, all hope is not lost.

In this post, I am going to give you five real tips you can utilize to become better at estimation–even for complex software development tasks.

Tip 1: Break Things Down Smaller

In my last post, I talked about how lengthy time periods, that are so common with software development projects, tend to make estimation very difficult and inaccurate.

If you are asked to estimate something that will take you five minutes, you are much more likely to be accurate than if you are asked to estimate something that will take you five months.

So, how can we solve this problem?

There is actually a relatively simple fix: Break things down into smaller chunks and estimate those smaller chunks.

smaller 5 Ways Software Developers Can Become Better at EstimationYes, I know this seems simple and obvious–and I know that this approach is often met with skepticism. There are plenty of excuses you can make about why you can’t break things down into smaller pieces, but the truth is, most things can be broken down–if you are willing to put forth the effort.

I’ve actually talked about why smaller is better and how to break down a backlog in the past, so I won’t rehash all the details again here.

The key point to realize is that you are never likely to get good at estimating large things. Well, let me rephrase that: The only way you are going to get good at estimating large things is to be learning how to break them down into many smaller things.

If you really need to accurately estimate something, it is well worth the effort to spend the time breaking down what you are estimating into much smaller pieces.

For example, suppose I was going to estimate how long it will take me to write a blog post. It’s not a very large task, but it’s big enough that estimates can be a bit inaccurate.

If I want to be more accurate, I can break down the task into smaller pieces.

Consider the difference between trying to estimate:

  • Write and publish a blog post

And:

  • Research blog post and brainstorm
  • Outline blog post
  • Write first draft of blog post
  • Add images, links and call-outs
  • Schedule post for publishing

By breaking things down into smaller pieces, I can more accurately estimate each piece. In fact–here is a little trick–when things are broken down this small, I can actually time-box certain parts of the process–which is effectively ensuring my estimate is accurate (but, we are jumping ahead, we’ll talk more about time-boxing in a little bit.)

The next time you are asked to implement some feature, instead of estimating how long you think it will take you to do it as a whole, try breaking down the task into very small pieces and estimating each piece individually. You can always add up the smaller estimates to give a more accurate estimate of the whole.

But wait! I know exactly what you are going to say is wrong with this kind of estimation. Sure, each individual piece’s estimation may be more accurate, but when you add them back together, in aggregate, you’ll still get the same level of error as you would from estimating one large thing.

All I can say to that argument is “try it.” To some degree you are right, the smaller errors in the smaller pieces will add up and cause the whole to be off by more in aggregate, but the smaller pieces also tend to average out. So, some take less time and some take more, which means that overall, you end up a lot more accurate than estimating one large thing with a large margin of error.

Tip 2: Taking time to research

Why do you suck at estimation?

Because you don’t know enough about what you are estimating.

In the previous post, I talked about how the unknown unknowns, that plague many software development projects, make estimation extremely difficult, but I didn’t really talk about how to deal with these things that we don’t know that we don’t know.

Again, the answer is really quite simple: research.

research 5 Ways Software Developers Can Become Better at Estimation

The best way to get rid of an unknown unknown is to know about it.

Whenever you are tasked with estimating something, your first instinct should be to want to do some research–to try and discover what it is that you don’t know that you don’t know yet.

Unfortunately, most software developers don’t immediately think about doing research when trying to estimate something. Instead, they rely on past experience. If they’ve done something in the past that they deem similar enough, they will confidently estimate it–ignoring the possible pitfalls of the unknown unknowns. If they’ve never done something similar to what they are being asked to estimate, they’ll assume there are unknown unknowns everywhere and come up with estimates full of padding.

Neither approach is good. Instead, you should first try and estimate how long it will take you to research a task before giving an estimate of how long the actual task will take. I’ve found that most software developers are pretty good at estimating how long it will take to research a task, even though they may be very bad at estimating how long it will take to complete the task itself.

Once you are armed with research, you should have fewer unknown unknowns to deal with. You may still have some unknowns, but at least you’ll know about them.

But, how does this look in reality?

How do you actually research tasks that you are supposed to be estimating?

Well, sometimes it involves pushing back and planning things out a bit ahead of time. I’ll give you an example of how this might work on a Scrum or Agile team.

Suppose you want to start improving your estimates by doing research before estimating tasks. The problem is that when you are working on an Agile project, you usually need to estimate the tasks in an iteration and don’t really have the time to research each and every task before you estimate it–especially the big ones.

I’ve found the best thing to do in this scenario is instead of estimating the big tasks right up front, to push the tasks back one iteration and instead estimate how long it will take to research each big tasks.

So, you might have in your iteration any number of small research tasks which only have the purpose of getting you enough information to have a more accurate estimate for the big task in the next iteration. During these research tasks, you can also break down large tasks into smaller ones as you know more about them.

Wait… wait.. wait… I know what you are thinking. I can’t just push a task into the next iteration. My boss and the business people will not like that. They want it done this iteration.

Right you are, so how do you deal with this problem?

Simple. You just start planning the bigger tasks one iteration in advance of when they need to be done. If you are working on an Agile team, you should adopt the habit of looking ahead and picking up research tasks for large tasks that will be coming up in future iterations.

By always looking forward and doing research before estimating anything substantial, you’ll get into the habit of producing much more accurate estimates.

This technique can also be applied to smaller tasks, by taking, sometimes, just five or ten minutes to do a minor amount of research on a task, before giving an estimation.

The next time you are trying to estimate a task, devote some time upfront to doing some research. You’ll be amazed at how much more accurate your estimates become.

Tip 3: Track your time

One of the big problems we have with estimating things is that we don’t have an accurate sense of time. My memory of how long past projects took tends to be skewed based on factors like how much I was enjoying the work and how hungry I was.

This skewed time in our heads can result in some pretty faulty estimations.

For this reason it is important to track that actual time things take you.

It is a very good idea to get into the habit of always tracking your time on whatever task you are doing. Right now, as I am writing this blog post, my Pomodoro timer is ticking down, tracking my time, so that I’ll have a better idea of how long blog posts take me to write. I’ll also have an idea if I am spending too much time on part of the process.

Once you get into the habit of tracking your time, you’ll have a better idea of how long things actually take you and where you are spending your time.

It’s crazy to think that you’ll be good at estimating things that haven’t happened yet, if you can’t even accurately say how long things that have happened took.

Seriously, think about that for a minute. No, really. I want you to think about how absurd it is to believe that you can be good at estimating anything when you don’t have an accurate idea of how long past things you have done have taken.

Many people argue that software development is unlike other work and it can’t be accurately estimated. While, I agree that software development is more difficult to estimate than installing carpets or re-roofing houses, I think that many software developer’s suck at estimation because they have no idea how long things actually take.

Do yourself a favor and start tracking your time. There are a ton of good tools for doing this, like:

If you are curious about how I track my time and plan my week, check out this video I did explaining the process I developed:

By the way, following this process has caused me to become extremely good at estimating. I can usually estimate an entire week worth of work within one-to-two hours of accuracy. And I know this for a fact, because I track it.

Tip 4: Time-box things

I said I’d get back to this one, and here it is.

One big secret to becoming a software developer who is better at estimating tasks is to time-box those tasks. It’s almost like cheating.

When you time-box a task, you ensure it will take exactly as long as you have planned for it to take.

You might think that most software development tasks can’t be time-boxed, but you are wrong. I use the technique very frequently, and I have found that many tasks we do tend to be quite variable in the time it takes us to do them.

I’ve found that if you give a certain amount of time to a task–and only that amount of time–you can work in a way to make sure the work gets done in that amount of time.

Consider the example of writing unit tests:

For most software developers, writing unit tests is a very subjective thing. Unless you are going for 100% code coverage, you usually just write unit tests until you feel that you have adequately tested the code you are trying to test. (If you do test driven development, TDD, that might not be true either.)

If you set a time-box for how long you are going to spend on writing unit tests, you can force yourself to work on the most important unit tests first and operate on the 80 / 20 principle to ensure you are getting the biggest bang for your buck.

For many tasks, you can end up spending hours of extra time working on minor details that don’t really make that much of a difference. Time-boxing forces you to work on what is important first and to avoid doing things like premature optimization or obsessively worrying about variable names.

Sure, sometimes, you’ll have to run over the time-box you set for a task, but many times, you’ll find that you actually got done what needed to be done and you can always come back and gold-plate things later if there is time for it.

Again, just like tracking your time, time-boxing is a habit you have to develop, but once you get used to it, you’ll be able to use it as a cheat to become more accurate at estimates than you ever imagined possible.

You may want to get yourself a Pomodoro or kitchen timer to help you track your time and time-box tasks. Sometimes it is nice to have a physical timer.

Tip 5: Revise your estimates

Here is a little secret: You don’t have to get it right on the first go. Instead, you can actually revise your estimates as you progress through a task.

revision 5 Ways Software Developers Can Become Better at Estimation

Yes, I know that your boss wants you to give an accurate estimate right now, not as you get closer to being done, but you can always give you best estimate right now and revise it as you progress through the task.

I can’t image any situation where giving more up-to-date information is not appreciated.

Use the other four tips to make sure your original estimate is as accurate as possible, but every so often, you should take a moment to reevaluate what the actual current estimate is.

Think about it this way: You know when you download a file and it tells you how long it will take? Would you prefer that it calculated that duration just at the beginning of the download process and never updated? Of course not. Instead, most download managers show a constantly updated estimate of how much time is left.

Just going through this process can make you better at estimations in general. When you constantly are updating and revising your estimates, you are forced to face the reasons why your original estimates were off.

What about you?

These are just a few of the most useful tips that I use to improve the accuracy of my estimates, but what about you? Is there something I am leaving out here? Let me know in the comments below.

Also, if you liked this post, join over 6,000 other software developers that are part of the Simple Programmer community. Just sign up here and start your journey to becoming a better, more fit, and productive software developer.

 

4 Biggest Reasons Why Software Developers Suck at Estimation

Estimation is difficult.

Most people aren’t good at it–even in mundane situations.

software development estimation 4 Biggest Reasons Why Software Developers Suck at Estimation

For example, when my wife asks me how much longer it will take me to fix some issue I’m working on or to head home, I almost always invariably reply “five minutes.”

I almost always honestly believe it will only take five minutes, but it never does. Most of the time my five minutes ends up being half an hour or more.

But, in comparison to the world of software development efforts, my five minutes estimation is usually fairly accurate–it’s only off by a factor of six or so.

It’s not unheard of to have software development estimations be off by as much as one-hundred-fold. I’ve literally had an hour long estimation turn into two weeks.

But, why are software development estimations usually off by so much?

Here are the four biggest reasons I have found:

Reason 1: The unknown unknowns

This phrase was first uttered by former Secretary of Defense of the United States, Donald Rumsfeld. It basically refers to those things that we don’t even know that we don’t know.

donald rumsfeld 4 Biggest Reasons Why Software Developers Suck at Estimation

You don’t know what you don’t know

By far, this is the biggest reason why software developers often flub at giving good estimations. It also happens to be the primary reason why I suck at telling my wife how long it will take me to get home–I don’t know about the distractions that I don’t know about yet.

Software development has a lot of unknowns. Some of the unknowns we know about.

When we first start a project, we might have a good idea that we need to store the data in a database, but we don’t know how we’ll do it. That is a known unknown. We know that we don’t know something that we’ll eventually need to know.

Estimating a known unknown is pretty difficult, but we can do a decent job of it if we can compare it to something similar we’ve already done before.

I don’t know how long it will take me to write this particular blog post, but I know about how long it has taken me to write other blog posts of a similar length.

What is really scary though, are the things that we don’t even know we don’t know yet.

These unknown unknowns sneak up on us, because we don’t even know they exist–by definition they are unknowable. It’s one thing to know that we have a gap in a bridge somewhere that we have to cross, it is a whole other thing to have to cross a bridge blindfolded and only find out about gaps when you fall through them.

Constantly, in software development, we are faced with situations where we have to encounter these unknown unknowns. There is no good way to estimate around them. The best we can do in these cases is give ourselves a lot of padding and a lot of rope so that we can climb out of any gaps in the bridge that we fall into.

Reason 2: Lengthy time periods

As if unknown unknowns weren’t bad enough, the deck is stacked against us even further.

Most software development estimations involve fairly long periods of time. This has changed somewhat with Agile development practices, but we still are often asked to estimate a week or more worth of work at a time. (Plus, let’s not forget those sneaky project managers who try to throw Agile projects into Microsoft Project anyway and say “yeah, I know this is Agile and all, but I’m just trying to get a rough idea of when we’ll be done with all of the features.”)

It’s fairly easy to estimate short periods of time–well, I guess unless it’s me telling my wife how long it will take for me to get off the computer. Most of us can pretty accurately predict how long it will take us to brush our teeth, write an email or eat dinner.

Long periods of time are much more difficult to estimate accurately. Estimating how long it will take you to clean out the garage, write a book, or even just to go grocery shopping is much more challenging.

grocery shopping 4 Biggest Reasons Why Software Developers Suck at Estimation

The longer the period of time you are trying to estimate, the more that small miscalculations and the effects of known unknowns can cause an initial estimate to be grossly off target.

In my experience, I’ve found that estimating anything that will take more than about two hours is where things really start to go off of the rails.

As a mental exercise, try and estimate things of varying lengths.

How long will it take you to:

  • do 10 pushups?
  • make a cup of coffee?
  • go to the convenience store and buy something?
  • write a one page letter?
  • read a 300 page novel?
  • get the oil changed in your car?

Notice how the things that can be done in under half an hour are very easy to estimate with a high level of confidence, but as you go out further in time it gets much more difficult.

Most of the time when we do software development estimates, we don’t try and estimate short things, like how long it will take to write a single unit test, instead we tend to estimate longer things, like how long it will take to complete a feature.

Reason 3: Overconfidence

I’m pretty presumptuous when it comes to estimations. I usually think I’m very accurate at making estimations. My wife would disagree–at least when it comes to making estimations of time things will take me. History would probably tend to vindicate her viewpoint.

As software developers, we can often become pretty convinced of our ability to accurately predict how long something will take. Often, if the programming task we are about to embark upon is one we feel confident about, we can be pretty aggressive with our estimates–sometimes to the point of absurdity.

overconfident 4 Biggest Reasons Why Software Developers Suck at Estimation

Oh ye of little faith, of course I can rewrite the application in half an hour

How long will it take you to get that feature done?

Oh, that? That’s easy. I can get that done in a few hours. I’ll have it done by tomorrow morning.

Are you sure? What about testing? What if something comes up?

Don’t worry, it’s easy. Shouldn’t be a problem at all. I just need to throw a few buttons on a page and hook up the backend code.

But, what happens when you actually sit down and try to implement the feature? Well, first you find out that it wasn’t quite as easy as you thought. You forgot to consider a few of the edge cases that have to be handled.

Pretty soon you find yourself taking the entire night to just get set up in order to actually start working on the problem. Hours turn into days, days into weeks and a month later, you’ve finally got some shippable code.

Now, this might be a bit of an exaggeration, but overconfidence can be a big problem in software development–especially when it comes to estimations.

Reason 4: Under-confidence

Under-confidence isn’t actually a word. I suppose that is because someone wasn’t confident enough to put it in the dictionary.

But, just as overconfidence can cause a software developer to under-estimate how long a programming task will take, under-confidence can cause that same software developer to overestimate a completely different task, which may even be much easier.

I don’t know about you, but I often find myself in situations where I am very unsure of how long something will take. I can turn a simple task that I don’t feel comfortable doing into a huge mountain that seems almost impassible.

We tend to view things that we’ve never done before as harder than they are and things that we have done before as easier than they are–it’s just human nature.

Although it may not seem like it, under-confidence can be just as deadly to estimations though. When we are under-confident, we are more likely to add large amounts of padding to our estimates. This padding might not seem all that bad, but work has a way of filling the time allotted for it. (This is known as Parkinson’s law.)

So, even though, when we are under-confident, it might appear that we are pretty accurate with our estimations, the truth is we may be wasting time by having work that might have been done in half the time fill the entire time that was allotted for it.

(By the way, if you are looking for a good book on Agile estimation, check out Mike Cohn’s book: Agile Estimating and Planning.)

What else?

Did I leave anything out? What do you think is the biggest reason why software development estimations are so difficult?

This post is the first in a series of posts about software development estimation. If you’d like to make sure you get the following post in this series talking about how to get better at estimation, make sure you subscribe here and you’ll get the post directly in your inbox.

Scrum Masters: What Makes a Good One?

Yes, that’s right. I am writing a blog post today about Scrum and Scrum Masters.

No, I haven’t lost my mind.

I just realized that out of everything I’ve written about Agile and Scrum, I never talked about what makes a good Scrum Master.

I’ve both been a Scrum Master and I’ve worked on a team with Scrum Masters and from both of those experiences I can tell you that there is much confusion about this particular role on a Scrum or Agile team.

Even the name Scrum Master has confusion around it, is it ScrumMaster or Scrum Master—you can tell, I prefer the latter.

So, let’s talk about why Scrum Masters exist in a Scrum team and what they actually should be doing.

scrummaster thumb Scrum Masters: What Makes a Good One?

Scrum Masters! What are they good for?

On some teams, unfortunately, it is absolutely nothing.

But, it doesn’t have to be that way. A Scrum Master actually serves a really important role on a properly functioning Scrum Team. (They bring the donuts or bagels in every morning, so the team can get actual work done.)

Ok, I am just kidding about that last part, but in reality it isn’t all that far from the truth. Let me explain.

A Scrum Master really is supposed to be the person who clears the path for the team so they can run as close to full speed as possible. The Scrum Master is sort of like the pit crew for a race car driver.

Without a Scrum Master, a Scrum team is slowed down by impediments which inevitably come up in any development project. It takes time and distracts the team to deal with these impediments, so the whole cadence of the team slows down unless someone external to the development process is moving the boulders out of the way.

So, really, the most important job of the Scrum Master is to remove impediments which may hamper a Scrum team from progressing on backlogs and getting their work done.

This isn’t the same thing as managing a project, because the Scrum Master isn’t deciding how and when things should be done. Instead, the Scrum Master is part of the team and the team as a whole is taking accountability for managing the project.

The Scrum Master also has the role of being the master of the Scrum process—hence the name. This is a tough spot to be in, but is a very important role that many teams neglect. The rules of Scrum are important to a successful Scrum team. One of the reasons why I started to write off Scrum as a process was simply because it was so difficult to get anyone to actually enforce the rules.

This is the job of the Scrum Master; he carries the big Scrum stick and he beats people over the head with it when they step out of line. He doesn’t do this because he is a big power-hungry bully. No, instead, he does this because he knows that the only way the team is going to produce their best work and not waste time arguing over process is if they all follow the process that was agreed upon from the start.

big stick scrum master thumb Scrum Masters: What Makes a Good One?

Scrum is intended to be more than just a way to develop software or organize teams, it is also a process that clearly defines what will happen, when it will happen and who will do what.

The Scrum Master is one of the most important roles on the team

It may seem, based on my previous description, that the role of a Scrum Master isn’t all that important to the overall performance of the team, but that is far from the truth.

In reality, the velocity of a team is more influenced by the Scrum Master than any other member of the team—with the exception of that lazy developer that breaks the build all the time and constantly falls asleep at meetings.

Even though the Scrum Master does not have direct control over the management of the team, the Scrum Master’s ability to both remove impediments and enforce the Scrum framework directly affects the team’s ability to get s!@# done.

A poor Scrum Master will let the team flounder and let outside influence distract the team from their work.

A poor Scrum Master will either be too timid or not care enough to force the team to obey the rules of Scrum, causing the whole platoon to go scampering off whatever direction they choose, rifles firing randomly in all directions.

I like to think of the Scrum Master as a guide who takes the team over rough terrain and shows them how to get water from tree leaves on their journey. Sure, the team could manage to bushwhack their way through the jungle without a guide, but it would take them a whole hell of a lot longer to do so—and they’d be much more likely to get eaten by a lion.

So, what should Scrum Masters actually do?

The answer is whatever needs to be done.

You know those gangster movies where some mob boss has a guy they call “the cleaner?” The guy that comes into a sticky situation and can hide a dead body, bribe the right cops, or just make someone disappear? If mobsters were following Scrum, that guy would be the Scrum Master.

The Scrum Master should be part of the team, but not part of the team. The Scrum Master should attend the standup meetings actively trying to spot impediments—especially the ones that aren’t mentioned by the team members, but exist beneath the surface of a problem.scrummaster taking a bullet thumb Scrum Masters: What Makes a Good One?

The Scrum Master should ensure that all the Scrum meetings and processes flow smoothly. He should make sure that standups are being used for their correct purpose. He should encourage the team to hold each other accountable and he, himself, should hold the team accountable to what they promised to deliver.

The Scrum Master should be the guy (or gal), who makes things happen. He should know the right people to talk to and know how to get things done. The team should focus on the work, the Scrum Master should focus on the politics. If the team is dealing with politics the Scrum Master has failed.

Most of all, the Scrum Master should be willing to lay it all on the line—to take the hits for the team. Even though the Scrum Master doesn’t control the team and get to boss them around, they are his team and his alone. A good Scrum Master isn’t afraid to take full responsibility for the actions and performance of the team and step in the way of that bullet and take one in the chest if he has to.

Like this post? Sign up for my weekly email and I’ll make sure posts, like this one, are delivered right to your inbox once a week. Also, you’ll get lots of content that I only share with my email subscribers.

Understanding The Problem Domain Is The Hardest Part Of Programming

What is the hardest thing about writing code?

There are many common answers to this question:

  • Learning a new technology
  • Naming things
  • Testing your code
  • Debugging
  • Fixing bugs
  • Making software maintainable

The list goes on and on.

But as I reflect back on my programming career, and I’ve conversed with many new programmers that are learning the craft, I’ve found the single hardest thing about programming is learning the problem domain.

A familiar problem

In a good portion of my Pluralsight courses I show the viewer how to build the “Protein Tracker” application.

I am often asked why I keep demonstrating how to build the same simple application over and over again in each of my courses.

The answer is “familiarity.”

When I first started using the Protein Tracker example in my Android course, I was just looking for a simple example of an application that could be easily understood and implemented.

The idea behind the Protein Tracker application is that it allows a user to set a goal for the amount of protein to consume in a day.  The user can add protein amounts which are added to a total protein count that is tracked for that user.

Protein Tracker 2013 07 03 16 44 41 thumb Understanding The Problem Domain Is The Hardest Part Of Programming

Very simple functionality, easily explainable, but most importantly, easily understood.

What I found with this simple application was that because it was so easy to understand, the focus was taken off of the problem domain and put instead on the technology.

Not only this, but as I reused this same exact application for teaching a variety of different technologies, it served as a reference problem domain that didn’t have to be re-learned, and provided a way to compare and contrast different technologies—I was getting this teaching mechanism for free if a viewer had already watched one of my other Pluralsight courses.

By creating a familiar problem domain, I found that both the tasks of me teaching a new technology and the viewer learning that technology were much easier, because it is very difficult to learn more than one thing at once.

So what am I trying to say here?

Simply that by taking away the problem domain, or making it so trivial that it is easily understood, I am able to make both teaching and learning easier.

Why problem domains are hard

Have you ever tried to put together a jigsaw puzzle that didn’t have any picture on it?  How about one like this one, that has a very similar pattern repeated on it and is double-sided?

81zQGlKs9oS. SL1500  Understanding The Problem Domain Is The Hardest Part Of Programming

The reason why puzzles like this one are so hard, is because you can’t really see what you are trying to build very clearly.  Normally when you put together a jigsaw puzzle you follow steps that might look something like this:

  1. Figure out what the major components of the picture are
  2. Sort the pieces by color or component
  3. Put together all the border pieces
  4. Put together each component of the picture from the piles you created

This all breaks down when you don’t have a picture with clear components that you can identify.

The same thing happens when writing code.  Writing code is a lot like putting together a jigsaw puzzle.  We put together code with the purpose of building components that we have taken out of the “bigger picture” of the problem domain.

The big issue is that many problem domains are like a puzzle with a blurry picture or no picture at all.

The real world is a messy place.  Many of the problem domains we face as programmers are difficult to understand and look completely different depending on your viewpoint.

As programmers, we also are often not given complete information about the problem domain, so we don’t even have the information we need to understand it.

Just try and read the famous Domain Driven Design book and you’ll quickly see how complicated and difficult problem domains can be.  (Great book by the way, although you may have to read it twice or three times—I certainly did.)

Programming is easy if you understand the problem domain

A long time ago, I worked for Hewlett Packard writing software for multi-function printers.

Most of the work at the time was basic waterfall development.  There wasn’t much Agile happening there—at least at the time I was there.Fotolia 39398078 XS thumb Understanding The Problem Domain Is The Hardest Part Of Programming

There was however something really interesting about the waterfall approach and the extreme amount of specification that was done before anything was built—it was very easy to write the code for a feature.

I remember writing a tab control for the user interface of a printer and having the complete pixel perfect specs handed to me before I began to write any code.  I was also given all the possible use cases and told exactly how it should function and what it should do under just about every circumstance.

Guess how easy it was to write the code to produce this tab control?  Super easy.

As much as I frown upon this approach for software development today, there is something interesting to think about here.

I was essentially given the entire problem domain in the form of a spec that was clear and unambiguous.  I was easily able to learn that problem domain and because of it, I was able to write the code very easily as well.

Perhaps you have had a similar experience, not necessarily working on a waterfall project where you were given the spec, but perhaps on an Agile project where you took the time to clearly understand the problem domain before writing any code.

I’ve spent days trying to implement a feature only to finally go back and talk to a product owner and hash out completely how something should work and why it should work in a particular way, only to go back to my desk and crank out the code in a matter of hours.

The more and more I write code, the more I learn that understanding the problem is the most critical piece to the equation. It is very difficult to solve a problem before you know the question.  It’s like buzzing in on Jeopardy before you hear the clue and shouting out random questions.

What can you do about it?

If understanding the problem domain is the hardest part of programming and you want to make programming easier, you can do one of two things:

  1. Make the problem domain easier
  2. Get better at understanding the problem domain

You can often make the problem domain easier by cutting out cases and narrowing your focus to a particular part of the problem.

What I mean by this is that it is often beneficial to take a part of the problem and fully understand that part before expanding the problem domain.

Games are really good at this.  Look at most games today and you’ll find that you start with a very small problem domain.  The first level is usually a tutorial that has a basic set of things you can do so that you don’t get overwhelmed.  But, as you advance through the levels, you usually find they get harder and introduce new concepts that build gradually on what you know, until you understand a pretty large problem domain.  (Starcraft is really good at this.)

The other choice is to become better at understanding problem domains.  As developers, we tend to think that sitting down and talking to customers or business people who know about the problem domain is a waste of time. It is easy to fall into the trap of thinking you understand enough of the problem to get started coding it.  Best to resist the temptation to “not waste anymore time talking” and make sure you understand a problem inside and out before you try and solve it with code.  It is much more expensive and time consuming to do things over than it is to do them right the first time.  I learn this lesson the hard way time and time again.

Quick update on my new product

I’m still not ready to unveil exactly what I am building, but I do have an active mailing list where you can sign up to find out when I release the product I’m working on to help developers get better career opportunities and market themselves.

So many developers don’t realize how much of an impact marketing themselves and branding can have on their opportunities.  I’m hoping to help developers learn not only how valuable marketing and branding is, but how to do it most effectively.

If you are interested, please sign up. (I won’t spam you.)

5 Things That Will Make Your Agile Development Project FAIL

Far too many Agile development projects fail. It is hard to even accurately measure the number of failures because so many software development projects end up “complete” and shipped, even though:

  • They took far too long to build
  • The quality of what we built was poor
  • What was built was not what the customer wanted
  • The cost to develop was more than it should have been

Over the years, I’ve worked on many different Agile teams and have consulted for and trained many teams on Agile development. In this time, I have found 5 common problems, that if not resolved, are very likely to cause a project to be a failure.

1. Not having a product owner

Of all the things that can cause an Agile software project to fail, not having a person that is ultimately the decision maker for the product being developed is the quickest way to ensure its demise.

It doesn’t matter if you are following Scrum, doing your own Kanban style project or something else; if you want your project to succeed, you need someone who can set its direction and make decisions about the product being developed.

Think about remodeling a kitchen. If you hired a bunch of contractors to come in and do the various jobs, like: plumbing; carpentry; flooring; etc, but you don’t have someone deciding what the actual finished kitchen should look like, you are going to get a random mess.

no product owner thumb 5 Things That Will Make Your Agile Development Project FAIL

A few skilled contractors will probably be smart enough to find the right person to ask about what should be done, but it takes more than just making arbitrary decisions about where the cabinets should go to design a kitchen. You need someone to actually come up with a real design and vision and be able to modify that vision appropriately as the project progresses and problems inevitably occur.

It seems pretty nuts to spend a huge amount of money remodeling your kitchen, but not want to invest any time or effort in either designing the finished product or hiring someone to do it.

Yet, day in and day out, I see this exact behavior in software projects. I see companies investing huge amounts of cash in Agile development, but not appointing anyone to be the true owner of what is being built and to set the vision for it.

2. Not iterating

One of the key values that Agile development brings to the software development world is iteration.

What is iteration though?

You may think it means breaking your project into 2 week sprints or iterations.  While doing this can facilitate iterative development, it doesn’t mean you are actually iterating.

Confused? Let me explain.

The key to iterating is to develop a product a little bit at time. It would be accurate to describe the process of iterating on a product as evolution of a product.

Whether you believe in macroevolution or not, microevolution, or adaptation is a proven concept.  The idea behind evolution is that things change gradually over time. Those small changes add up to result in a big change.

Imagine how silly it would be if evolution worked the way most “Agile” software is built.

Imagine, if you will, a fish that swims in the ocean and has some little fish babies that are born with fully functional legs. Then, those legged fish babies grow up and have fish babies that have wings.

evolution agile fish thumb 5 Things That Will Make Your Agile Development Project FAIL

Probably the legs and wings wouldn’t do that fish much good, nor would they be designed properly, because instead of adapting and changing over time, they just suddenly appeared.

Features should not be built in a single sprint or iteration. It is as silly as legs showing up on a fish in a single generation.

Instead, features should be evolved and grow over time. A feature shouldn’t be pushed into a single sprint or iteration and then be done. A feature should start out small and develop and evolve over time as feedback is received and customers or the product owner tries it out.

Far too many times, I see Agile development projects mistake having iterations with actually iterating the development of the product.

Don’t try to ship a completed feature at once, let it evolve over time.

3. Not breaking things down small enough

I am a big proponent of breaking things down into small, easily digestible pieces.

The main reason why this is so important is because it prevents procrastination. Procrastination usually occurs when either we dread some large task that will be difficult or we don’t know what to do next.

If you can break a big project up into small parts, it will seem easier to accomplish and will have clear steps of progression.

I often see software projects where the person creating the backlogs or work items is not considering the work enough before giving it to the team.

I coined a term for these kinds of backlogs: fatlogs. Fatlogs are backlogs that are not broken down small enough and often are very vague in what needs to be accomplished.

Fatlogs are notoriously difficult to estimate and waste time when trying to explain and understand them. It is critical that fatlogs be broken down into smaller actionable backlogs before being given to an Agile development team or a large amount of time will be wasted.

Many times, I have found that the creator of the fatlog could have easily broken down the work it represents into several smaller backlogs that would be easier to explain and understand, even without knowing much about software development.

I often recommend to Agile development teams that they outright reject fatlogs and send them back up the chain.

“If you can’t take enough time to clearly state what you want, it must not be that important.”

This doesn’t excuse development teams either. Development teams should break down any backlogs they get into small tasks as well.

4. Not setting done criteria

What is the first thing a waiter asks you when you order a steak?

That’s right, they ask you how you would like it done.

If the chef doesn’t know what the done criteria is for your steak, the chef will have to decide what he or she thinks the done criteria is.

done criteria agile steak thumb 5 Things That Will Make Your Agile Development Project FAIL

You may get back a well-done steak, or a rare steak, or something in between, depending on the personal preferences of the person cooking your meat.

This isn’t a good way to cook steaks and it isn’t a good way to cook software either.

In many software projects I often encounter lots of steaks being cooked but no one defining when they are done. The backlogs most often end up being “done” when time runs out.

It is very important to have explicit unambiguous done criteria for any backlogs being worked on by an Agile development team.

This means that the product owner should be defining some level of acceptance testing. It doesn’t matter if the tests are manual tests or fully automated tests (BATs), what matters is that some criteria is defined by which the team can evaluate whether they have achieved their goal or not.

Lacking this criteria is like a parent responding with “enough” when their child asks “how many more peas do I have to eat?”

Not having done criteria is demotivating and results in the finger pointing game of why the correct thing was not built.

I have found that if you tell someone exactly what is expected of them and what criteria they will be judged by, you get much better results than just throwing them in the ring, slapping them on the back, and saying “do a good job.”

5. Not letting the team be a team

Teams are strange organisms, especially Agile teams.

There are many dynamics that affect how teams act and interact. There are many ways to foster a healthy team and many ways to create unhealthy teams.

A healthy motivated team has synergy, a unhealthy gaunt team can be less effective than all of its members are on their own.

The key to having a healthy motivated team, is letting them be mostly autonomous. Regardless of your political persuasion, you would probably agree that when a country invades another country and sets up an unelected government to oversee the people, there are going to be problems.

The same thing happens in Agile development.

I’m not saying teams shouldn’t have accountability. But if you want to run a software project in an Agile way, you have to let teams self organize and self manage to some degree.

When big brother is always watching and interfering, team dynamics become very different. Naturally, teams usually develop their own cadence, leadership and roles (this is called norms.) However, when outside forces directly interfere with how the team does things, and they don’t have the power to decide their own fate, team members begin to realize that they need to look out for themselves.

Additional agile development resources

I’ve found it to be pretty difficult to find good resources on these topics, but here are a few books I’ve found useful that address some of the issues I described above.

  • Succeeding with Agile: Software Development Using Scrum – This book is focused on Scrum but it applies to many different kinds of Agile projects as well. Mike Cohn and I usually agree on most Agile subjects.
  • Agile Retrospectives - I’ve found this book to be useful for getting good idea for team retrospectives that can really help to break down some barriers and help teams learn to resolve issues themselves.
  • Scrumban and Kanban and Scrum – Both of these books have excellent information about combining Scrum and Kanban and have great solutions for dealing with some of the problems I described above.

What do you think? Have I missed any of the major Agile project killers? Any useful tips for dealing with them?

Let me know in the comments below.

Where Is Agile Now?

It seems just yesterday I was trying to push forward the idea of developing software in an Agile way, but somehow now it seems like that battle is over.

As if we won without a fight.

fight thumb Where Is Agile Now?

When I look around now, I don’t see software development shops doing big upfront design.  I don’t see consultants knocking down doors to certify you as a Scrum Master.

It seems that we have now entered a phase where Agile is accepted as the default and now instead of everyone trying to pitch the idea of Agile, everyone is trying to fix their broken Agile implementations.

The funny thing is, still no one even knows what Agile is

The big problem with Agile from the beginning has always been trying to define it.

Pretty early on, this problem was solved by calling it Scrum.  Scrum was something that was easily definable, and something you could get certified in.

Scrum was a set of rules you follow that makes you Agile.

At least that is how it was pitched too often.

I predicted that Scrum would die, and I am pretty ready to call that prediction as correct.

scrumdead thumb Where Is Agile Now?

Sure, there are plenty of development shops still using Scrum today, but it isn’t really growing and less and less organizations are following it strictly.  (I can’t back this up, just my feel.)

I am a pretty firm believer in most of the value of Scrum being that it contains a firm set of rules that doesn’t require debate or judgment calls.  If most organizations are just taking the idea of a 2 week sprint and having daily scrum meetings, they are not likely getting much of the value out of Scrum.

But, the problem is that Scrum itself was never Agile.  Scrum was a defined set of process, that if you followed, would give you the framework you needed to actually be Agile.

To me Agile has always meant stopping the BS about software development.

To me Agile meant stop building this plan that you know is going to fail and rules lawyering your customers to death to get them to pay for something they didn’t want, because that is what they agreed to.

To me Agile meant to instead try to develop software as honestly as possible.  To go in and find out exactly what the customer wanted at the moment, try to build that thing and as openly as possible and as quickly as possible get further feedback to refine and improve.  To focus on doing a good job and know that if you are doing that everything else will fall into place.  To me that is what Agile has always been.

So when I say where is Agile now, I am probably asking a different question than most people

I have to ask myself: are software development shops doing what I define as Agile?  Has that idea permeated the software development community as a whole?

I don’t think so, but I don’t think it has died, nor will it ever.

But, I have seen some things that make me hopeful.

I’ve seen a large amount of talk about MVP, Minimum Viable Product.

I’ve seen many start-ups launching MVPs and being successful doing so.  And I’ve seen awesome companies like Buffer using this idea to build a product that is exactly what I want, because their plan is completely based on the customer and it adapts to the customer.

Why am I saying all this?

Simple, I think that what the world thought was Agile was two things:

  1. Scrum
  2. Iterative development

For the most part the software development world has ditched Scrum, at least the only form of useful Scrum, which is strict by-the-book Scrum, and adopted Scrum meetings and iterative development.  Honestly, I could do without the Scrum meetings, because although they are a good idea, no one actually does them correctly.

So, in essence, we won the wrong battle and we did so with major concessions.  But, that is ok, because what the consultants packaged up, certified people in and sold as Agile, wasn’t really Agile at all. 

Instead the real Agile movement has been gaining traction and it isn’t being sold by consultants, it is being championed by small start-up companies that are producing products that are 100 times more weighted on the side of results in the results to employees ratio than traditional software development shops and they are calling it MVP.

customer thumb Where Is Agile Now?

This is where the true spirit and ideas of Agile live and thrive and as more and more of these companies become successful and more and more researchers dissect their results, they are going to find that these small software boutiques were the ones who were actually practicing Agile, because they were cutting through all the BS of software development and focusing and developing and building exactly what the customer wanted—tools, process, contracts, plans be damned!

Principles Are Timeless Best Practices Are Fads

There is a huge difference between a principle and a best practice.

Best practices are subjective and depend largely on context, while principles are eternal and universal.

time thumb Principles Are Timeless Best Practices Are Fads

After writing The More I Know The Less I Know, I received a few emails talking about how there are absolute best practices that should always be followed in software development.

I had already intended to write about principles, but that confusion made it clear to me that there should be a distinction made between best practices and principles.  We don’t want to throw the baby out with the bath water.

Looking at some examples of best practices

First let’s take a look at some software development best practices, then we’ll contrast them to principles to better get an idea of the difference.

One of the most common best practices today in software development is the idea of unit testing.  I’ve written about how I have my doubts about blindly following this best practice in the past, but whether or not we should follow it, is not what I am concerned with today.

Unit testing is extremely contextual.  What I mean by this, is that almost anyone would agree that there are a certain set of circumstances that makes unit testing have value.

If you work in an environment where the execution of unit tests takes a really long time, or you are developing your software in a waterfall approach where you have a big upfront design and detailed requirements, unit testing starts to lose value rapidly.

But rather than get trapped into the argument of when unit testing loses its value, it is better to address when it has the highest value—we are much more likely to agree there.

Unit testing has the highest value when we are working in agile environments where changes are being introduced into a software system rapidly and refactoring is taking place.  It also greatly increases in utility when you are able to execute and write the tests quickly, because that feedback loop makes it much easier to write the tests in a step by step approach, especially when doing TDD.

uml thumb Principles Are Timeless Best Practices Are FadsThere are plenty of other best practices that have fallen out of favor, like heavily commenting code and documenting requirements with UML diagrams, but context also greatly played a part in the value of these practices. 

When most developers wrote very short variable and method names, comments were really important.  Before Agile processes became prevalent, getting detailed requirements upfront was critical.

But, most best practices are good!

Yes, you are right, most best practices do apply pretty broadly and are generally helpful in a large number of different contexts.

For example, it is considered a best practice to use a source control system and it doesn’t seem like there are many situations where this wouldn’t be the case.

So doesn’t that make it a concrete rule or a principle?

No, it is still too specific to be generally applied in all cases and the act of putting your code in source control does nothing to improve the quality of your software or software product. 

If you were to blindly follow any best practice and not apply that best practice in a way that brings out the underlying principle, you would be very unlikely to actually receive any benefit.

You see, most best practices are actually derived from universally applicable principles that never change.  That is why most best practices are good.

The problem is applying the best practice itself in no way assures the benefit of its underlying principle.

To put it plainly, there is something greater at work that makes it a good idea to check your code into a source control system.  It is entirely possible to follow the action, but completely miss the spirit of the action.

More and more today, I see software development teams that are:

  • Writing unit tests
  • Using continuous integration systems
  • Using source control
  • Having Scrum meetings
  • Pair programming
  • Using IoC containers

Yet they are getting little to no benefit from it.  Just a bunch more pain and hoops to jump through.  The reason is simple…

It’s not the best practice that is effective, it is the principle behind the best practice

Principles are everywhere.  They apply in all aspects of our life.  You cannot go through the day without being affected by the results of 100s of different principles that have a constant influence on your life, just like the law of gravity does.gravity thumb Principles Are Timeless Best Practices Are Fads

Gravity is actually a great way to understand principles.  As far as we know, it is a universal force that is always in effect.  It is impossible to escape the law of gravity, wherever you go in the universe it affects you.

Principles are like laws of nature except bigger.  Principles are more like the laws of reality.  Even though you may not be able to describe them fully or understand how they work, they always work.

Take for instance, the law of the harvest.  Most people are familiar with this particular principle.  It basically goes like this.

You reap what you sow.

How universal is this truth?  How can anyone avoid it?  How many times have you found yourself subject to this inescapable law about how reality works?

Many software development best practices are actually based on this principle.  Think about best practices that have you make efforts to improve the quality of software early on in the process.

TDD or test driven development, is such a best practice.  The basis of TDD is to introduce quality into the software development process as early as possible, so that the finished product is better.

If you apply the practice of TDD without understanding this principle, you are just following the motions and you won’t actually gain the benefit of the practice.

If you can’t understand at some level that the point of doing TDD is to sow some good seeds in your software that you will harvest later on, you won’t be writing the right kind of tests.

There is nothing magical about writing tests before writing code, but there is something valuable in purposely investing in upfront quality with the end goal of getting a big yield on that investment in the right season.

By the way, that is why I like Bob Martin’s book Agile Principles, Patterns and Practices in C#; it discusses many principles of software development that are timeless.  Books like this one and the book I have mentioned probably 10 times in this blog, How to Win Friends and Influence People, are full of principles.

Also, check it out, you just learned what Agile really is.  With principles in mind, now read the Agile manifesto.  It was never designed to be a detailed process and set of best practices for developing software, it was always meant to be a recognition of a set of principles that guide software development.

So, just remember the next time you are arguing with someone over a best practice, or consider applying one to a project you are working on, if you don’t understand the underlying principle, no amount of ceremony and procedure will have the smallest amount of benefit.

If you like this post don’t forget to Follow @jsonmez or subscribe to my RSS feed.

The Why is More Important Than the What

The goal of software development is to solve problems.

At its heart, software development is really about solving problems through automation.

Many times we have the tendency to make software development about creating solutions.  It may seem that these are the same thing, but there is a subtle difference.

The difference is the focus

When we are trying to solve problems, we are obsessed with the question of why.  It is only by understanding why that we can know what to build.

mp900438755 thumb The Why is More Important Than the What

When we are trying to build a solution, we are obsessed with the question of what.  We want to know what to build so that we can build it.

This is fairly natural for software developers since the what is something we can control.  Any fairly skilled software developer can build any what you can describe to them.  Just like any skilled carpenter can build any type of wooden furniture you desire.

The difference is that software development is about more than just building things.  A carpenter doesn’t have to focus so much on the why, to build you a piece of furniture.  But a truly skilled carpenter will ask you why.  A skilled carpenter might make a house call to the project to custom build an entertainment center suited to your needs. 

A truly skilled craftsman of any craft, needs to know why.

It is odd that we constantly seem to neglect the why when building software (at least I do), primarily because we think we don’t have time for the why, when in the end solving the why is the only thing that really matters.

Do you really want a carpenter to build you a perfectly crafted entertainment center with plenty of custom shelves for all your music CDs when you don’t own any music CDs?

Why focusing on what doesn’t work

Consider for a moment what would happen if your GPS system stopped showing your route and the directions you were going to take ahead of time, but instead abruptly told you to turn right when you were supposed to turn.

One of the early GPS systems I used for navigation did exactly this.  It was very frustrating!  I would constantly shout at it that telling me what to do exactly when I am supposed to do it wasn’t any help.  I needed some forewarning so I could switch lanes and prepare to turn.

Also, have you ever asked your GPS the question “why are you taking me this way?”

Focusing only on the what results in split-second decisions and avoidable mistakes.  When you are focusing on the what you are not thinking, just doing.

Contrast this with a good GPS navigation system.

What makes a good navigation system good?

I’ve found that good systems will alert me far in advance to what my next move is.  This gives me time to switch lanes or to have a better understanding of the bigger picture of the trip.

mp900402136 thumb The Why is More Important Than the What

Now this allegory won’t carry us very far in the software development world.  You don’t really need to know why your GPS system is taking you a particular route in order to get there, but it does demonstrate how focusing only on the immediate what can lead you astray.

Perhaps a more direct analogy related to software development is outsourcing.

If you’ve ever worked on a software development project that has had a portion of its work outsourced, you may have felt the pain of what happens when oftentimes perfectly competent programmers focus completely on the what and might not even have the slightest clue about the why.

I’m not trying to knock outsourcing, and I’m not even specifically talking about outsourcing to different countries.  (The same problems exist in outsourcing whatever the conditions are.)

Oftentimes outsourced projects are treated like a set of blueprints that need to be built.  The poor developers working on the outsourced project tend to get thrown under the bus when they build something based off just the what and can’t anticipate the why.

Why focusing on why is better

Focusing on the why is focusing on providing holistic cures to software development problems rather than treating each symptom individually.

Just like you’d like to have a doctor set a broken leg and put it in a cast so that it can be wholly cured, rather than give you some pain medicine to manage the pain, give you crutches to help you walk, and send you to therapy where you can learn to live without the use of your leg, your customers would probably rather you built something that actually is focused on solving their problem, not something that makes their symptoms easier to live with.

If we start building software from the what that is given to us, versus the why, we are at a distinct disadvantage because we only have the ability to treat the symptoms of a problem that we don’t even understand.

Focusing on the why is vitally important because it helps us to better think about and design the what.

Reuse is also much more likely to be present on why focused solutions than what focused ones.

Which software component is more likely to be adopted for reuse: a component that addresses a specific set of account balancing steps or a component that addresses the general problem your customer has of doing accounting?

Why focused solutions are much more likely to be shared across many customers, while what focused solutions are often very specific to a single customer.

Let’s not forget one of the most important reasons why it is important to focus on the why.

Missing the why is costly!

You can build the what someone is asking for, and you can do it perfectly, but at the end of the day if what you built doesn’t solve the problem (the why,) it’s going to have to be redone or thrown out completely.

You may think you are saving time by starting with the what, and you may actually build your solution faster by not taking time for everyone on the project to understand the why, but chances are you’ll build the wrong thing, and when you do, you’ll lose any time benefit you may have accrued.

It is very hard for a team to focus on the what.  Focusing on the what tends towards segregating the responsibility of the team.  The team members end up only having ownership for their own parts of the solution, rather than solving the problem at whole.  So, when a team that focuses on the what ends up with a problem, finger pointing a shirking of responsibility inevitably ensue.

By instead focusing the team on the why, every member of that team becomes responsible for solving the problem rather than solving their part of the problem.

The most compelling reason to focus on the why?

For many problems, just having a thorough understanding of the problem takes you 90% of the way to the answer.  I have spent countless hours trying to design and architect a clever solution to a problem without having a really good grasp of the problem that I’m trying to solve, only to find that 20 minutes spent actually fully understanding the why of the problem resulted in the answer becoming completely apparent.

mh900370196 thumb The Why is More Important Than the What

How to know if your focusing on what

I’ve got a pretty good idea of how to know when I am focusing on what instead of why.  Not because of any special wisdom on my part.  No, it is because I fall down in this area quite a bit myself.

I’ve learned the hard way, and I am still learning how to identify when my focus needs to shift from what to why.

Here are some simple questions that you may find helpful to evaluate your own situation.  I’d be glad to hear any others that you can can think of or use routinely yourself.

  • Do you’re backlogs, stories or general work items state a solution instead of a problem?
  • Do you know how to actually use the thing you are building?  (Whoops, I am so guilty of this one so many times.)
  • Can you define in simple terms the problem you are trying to solve?  (Einstein said “If you can’t explain it simply, you don’t understand it well enough.”)
  • Can you clearly define why the what your are building is needed and what purpose it will solve?
  • Can you walk through the manual steps of the process you are automating, understanding each step and why it is necessary?

Taking the time to focus on the why instead of jumping into the what can be difficult to get accustomed to, but it is worth the effort.  You’ll find much more satisfaction in your work when you understand the purpose it is going to serve, the true problem you have solved.  In addition your customers will thank you!

Tie Your Shoes and Pull Up Your Pants

What slows down the development of software?

Think about this question for a bit.  Why is it that as most software evolves it gets harder and harder to add features and improve its structure?

Why is it that tasks that would have at one point been simple are now difficult and complex?

Why is it that teams that should be doing better over time seem to get worse?

mp900430685 thumb Tie Your Shoes and Pull Up Your Pants

Seeking answers

Don’t feel bad if you don’t have an immediate answer to those questions.  Most software practitioners don’t.  They are hard questions after all.

If we knew all the answers, we wouldn’t really have these problems to begin with.

Regardless though, you’ll find many managers, business owners, customers and even software developers themselves looking for the answers to these questions, but often looking in the wrong place.

Process is almost always the first to be blamed. It stands to reason that a degradation of process or problems with the software development process are slowing things down.

Often there is some merit to this proposition, but I’ve found that it is often not the root cause. If your team is not sitting idle and the work that is important is being prioritized, chances are your process is not slowing you down.

Now don’t get me wrong here.  I am not saying that these are the only two important aspects to judge a software development process, but I am saying that if generally your team is working hard on important stuff most of the time, you can’t magically improve process to the point of increasing productivity to any considerable order of magnitude.  (In most cases.)

Often questions are asked like:

  • Should we pair program or not pair program?
  • Should we be using Scrum instead of Kanban?
  • Should we be changing the way we define a backlog?
  • Should we use t-shirt sizes or story points or make all backlogs the same size?
  • Do we need more developers or more business analysts?
  • Do we need to organize the team differently?

Now these are all great questions that every software project should constantly evaluate and ask themselves, but I’ve found over and over again that there is often a bigger problem staring us in the face that often gets ignored.

The code!

mp900289113 thumb Tie Your Shoes and Pull Up Your Pants

Let’s do a little experiment.

Forget about process.  Forget about Scrum and backlogs and story points and everything else for a moment.

You are a developer.  You have a task to implement some feature in the code base.  No one else is around, there is no process, you just need to get this work done.

It might help to think about a feature you recently implemented or one that you are working on now.  The important thing with this experiment is that I want to take away all the other “stuff” that isn’t related directly to designing and implementing that feature in the code base.

You will likely come to one of these conclusions:

1. The feature is easy to implement, you can do it quickly and know where to go and what to modify.

Good!  That means you don’t really have a problem.

2. It is unclear what to do.  You aren’t sure exactly what you are supposed to implement and how it fits into the way the system will be used.

In this case, you may actually have somewhat of a process problem.  Your work needs to be more clearly defined before you begin on it.  It may be that you just need to ask more questions.  It may be that half baked ideas are ending up in your pipeline and someone needs to do a bit more thinking and legwork, before asking a developer to work on them.

3. Its hard to change the code.  You’ve got to really dig into multiple areas and ask many questions about how things are working or are intended to work before you can make any changes.

This is the most likely case.  Actually usually a combination of 2 and 3.  And they both share a common problem—the code and system do not have a design or have departed from that design.

I find time and time again with most software systems experiencing a slow down in feature development turnaround that the problem is the code itself and the system has lost touch with its original design.

You only find this problem in successful companies though, because…

Sometimes you need to run with your shoelaces untied

I’ve consulted for several startups that eventually failed.  There was one thing in common with those startups and many other startups in general—they had a well maintained and cared for codebase.

I’ve seen the best designs and best code in failed startups.

This seems a bit contradictory, I know, but let me explain.

The problem is that often these startups with pristine and well maintained code don’t make it to market fast enough.  They are basically making sure their shoes laces are nicely tied as they stroll down the block carefully judging each step before it is taken.

What happens is they have the best designed and most maintainable product, but it either doesn’t get out there fast enough and the competition comes in with some VB6 app that two caffeine fueled not-really-programmers-but-I-learned-a-bit-of-code developers wrote overnight or they don’t actually build what the customer wants, because they don’t iterate quick enough.

Now am I saying that you need to write crap code with no design and ship it or you will fail?

Am I saying that you can’t start a company with good software development practices and a clean well maintainable codebase and succeed?

No, but what I am saying is that a majority of companies that are successful are the ones that put the focus on customers and getting the product out there first and software second.

In other words if you look at 10 successful companies over 5 years old and look at their codebase, 9 of them might have some pretty crappy or non-existent architecture and a system that departed pretty far from the original design.

Didn’t you say something about pulling up your pants?

pants around ankles thumb Tie Your Shoes and Pull Up Your Pants

Ok, so where am I driving at with all this?

Time for an analogy.

So these companies that are winning and surviving past year 5, they are usually running.  They are running fast, but in the process of running their shoelaces come untied.

They might not even notice the shoelaces are untied until the first few times they step on one and trip.  Regardless they keep running.  And to some degree, this is good, this is what makes them succeed when some of their failed competitors do take the time to tie their shoelaces, but those competitors end up getting far behind in the race.

The problem comes pretty close to after that 5 year mark, when they want to take things to the next level.  All this time they have been running with those shoelaces untied and they have learned to do this kind of wobble run where they occasionally trip on a shoe lace, but they try to keep their legs far enough apart to not actually step on a shoelace.

It slows them down a bit, but they are still running.  Still adding those features fast and furious.

After some time though, their pants start to fall down.  They don’t really have time to stop running and pull up those pants, so as they are running those pants slip further down.

Now they are really running funny.  At this point they are putting forth the effort of running, but the shoelaces and pants are just too much, they are moving quite slow.  An old woman with ankle weights power walks past them, but they can’t stop now to tie the shoelaces and pull up those pants, because they have to make up for the time they lost earlier when the pants first fell down.

At this point they start looking for ways to fix the problem without slowing down and pulling up the pants.  At this point they try running different ways.  They try skipping.  Someone gets the idea that they need more legs.

I think you get the idea.

What they really need to do at this point though is…

Stop running, tie your shoes and pull up your pants!

Hopefully you’ve figured out that this analogy is what happens to a mature system’s code base and overall architecture.

Over time when you are running so fast, your system ends up getting its shoelaces undone, which slows you down a little.  Soon, your system’s pants start to fall down and then you really start to slow down.

It gets worse and worse until you are moving so slow you are actually moving backwards.

Unfortunately, I don’t have a magic answer.  If you’ve gotten the artificial speed boost you can gain from neglecting overall system design and architecture, you have to pay the piper and redesign that system and refactor it back into an architecture.

This might be a complete rewrite, it might be a concerted effort to get things back on track.  But, regardless it is going to require you to stop running.  (Have you ever tried to tie your shoelaces while running?)

Don’t feel bad, you didn’t do anything wrong.  You survived where others who were too careful failed.  Just don’t ignore the fact that your pants are at your ankles and you are tripping over every step, do something about it!

Switching Gears is Grinding Gears

Pay attention young programmers, this is the most important piece of programming advice you will ever hear.

Well perhaps not, but it might be the most important piece of programming advice you hear today.

“Switching gears is grinding gears.”

gear intro thumb Switching Gears is Grinding Gears

I’ve been doing this programming thing for quite a while now, and I have come to realize that the biggest thing that saps my productivity is switching gears.

What do I mean by switching gears?

Many different contexts

Switching gears can apply to a variety of different contexts, but it is basically whenever you are working on some task or technology and have to stop what you are doing and either do something else or change the type of thing you are doing.

This is really a bit different than multi-tasking.  Multi-tasking is really trying to do more than one thing at once.  It usually involves a large amount of rapid context switches, but for this post I am more concerned about the general idea of breaking rhythm.

I think it is easier to speak about this subject if we talk about some specific applications of the idea and why they are harmful.

Switching problems

In the Agile world of software development today, we often are required to switch from problem domain to problem domain as many of our iterations contain mixes of different features on different parts of the system and bugs to be fixed.

Have you ever been in the situation where you were just beginning to understand how some area of the code base worked, or how to work with a particular customer on a particular problem space and then had to switch to something else and start all over again?

Even when you switch back to the original context you were working with, if enough time goes by, it becomes new again.  You have to relearn much of what you had learned the last time and just when you begin to hit your groove, the cycle begins again.

This form of switching gears is at the very least frustrating, and at the most a complete demotivator and productivity sapper.

Unfortunately for most developers, this problem is out of your immediate control.  But, take note product owners and project managers, there is a reason why iterations should have a goal.

If you are in the precarious position of being a developer pinned against the ropes, try to make some noise and let your management and product owners know that your team will be much more efficient when you aren’t switching gears all the time.

In my experience, the results of replacing this constant context switching with the synergy of a common goal and related features in a segment of time has an astounding effect on productivity that is hard for any serious businessperson to ignore.  So speak up!

clss 065 thumb Switching Gears is Grinding Gears

Switching technology

This is the primary reason why I have a distaste for JavaScript.  It is why even in this abundance of web development, I would still rather program a client application.

Switching technology is painful!

This is one of those things that is unavoidable in todays development environment.  If you are a web developer, you are going to be working with HTML, JavaScript, probably some sort of server side language, and most likely some form of SQL.

It really isn’t worth trying to fight this because you are going to be going against the technology grain to do so.  Although, I do suppose this may be one of the reasons for the recent popularity of Node.js.

What we can try to do is to minimize the context switch as much as possible.  We do this by sticking with a particular way of doing things and not chasing after each new technology of JavaScript framework that comes out each week.

I’m not saying to not learn new things.  It is very important to always be learning.

What I am saying, is to try to find some kind of rhythm with the technology stack you are working with and try not to switch that up.

Technology specific training can really help here as well.  I for one, need to learn JQuery better.  The problem is that when I am working on a web based feature, I am not forced to learn JQuery because I am not in that context long enough.

So what do I do instead?

I waste time Googling for answers.  I know that I have a short coming here and I need to just bite the bullet and spend some dedicated time to really thoroughly learn JQuery, because by Googling for little pieces of it at a time, I am not really making much headway and the context switch ends up stealing what I do learn from my memory.

One more aspect of this is the idea of focused teams.  Many software development groups do not like to specialize their developers onto one focus area.  I agree whole-heartily with the idea of non-specialization. 

But!  There is huge benefit to be gained by keeping the same group of developers working on a specific set of technology or part of the code base for a set period of time.  I’ll talk about this a bit more, towards the end of this post, but the basic idea is that it takes time for people to find their groove.

I think it is optimal to break any large team up into smaller technology area focused teams that regularly rotate every 3 months or so.  The idea here is that you give those teams enough time to get good at what they are doing and actually commit what they have learned to memory, but you rotate them often enough that they don’t end up becoming specialists which are unable to see the whole picture.

Switching teams

This one typically isn’t an issue, but it can be depending on the environment that you are working in.

Teams need enough time to go through that forming, storming, and norming set of phases.  If you are constantly interrupting this process by switching around team members, you are never going to get the team to act as an entity of its own.

Teams are generally more efficient than individuals, because they benefit from synergy, when 1 + 1 = more than 2.

But just like a big truck taking some time to get up speed, a team takes time to get going.  Also like a big truck, a team can gain momentum that individual developers seldom can achieve.

A smaller aspect of this is pair programming.  I suppose that many have been successful switching up programming pairs once a day or so, but I suppose that more teams have been successful switching up pairs at longer intervals.

For me, this interval varies.  Sometimes, I feel like I need to stay paired with a teammate for longer than 2 weeks, which is our regular interval, sometimes 2 weeks is fine.  It depends on what you are working on and how much momentum you have.

They key here is to make sure that you are giving teams enough time to take up their own flag and stake their territory.  Teams can take some time to find their common goal.  Self-direction can really help with this.

The groove

Ever tried to turn a flat head screw with a screwdriver, but it doesn’t quite turn, because you can’t seem to get the head of the screwdriver into the groove of the screw?

screwdriver 1 thumb Switching Gears is Grinding Gears

You twist that screwdriver around a bit until finally it slips into the groove.  Then when you turn the screwdriver the screw turns with it.

As humans, we tend to be like screwdrivers, we have to find our groove.  We all have this adjustment period where we are fumbling through things. 

It is very important to make sure that we aren’t switching gears so often that we are not actually turning any screws once we are in the groove.

Regardless of what kind of context switching you are doing—whether it is problem domain, technology or teams—it is important to make sure that you are not spending 90% of your time finding the groove and only 10% of your time “in the groove.”

Depending on what you are doing and what the context switch is, the proper amount of time before switching contexts is going to vary, but I think it is very important be aware of this phenomenon and plan around it.  If you don’t you are going to end up spinning your wheels, which is unfulfilling to say the least.