There is a useful observation about the world that is often applied to software development called the Pareto principle or Pareto’s law.
This principle suggests that in many situations 80% of the results come from 20% of the causes.
For example, Pareto had realized that 80% of the land in Italy, during his time, was owned by 20% of the population.
Since then, many people, including Pareto himself, have applied the same observation of this 80-20 rule to many other areas of life from economics, to business and even software development.
But, I’ve found that we rely on Pareto’s law just a little bit too much.
The problem with generalizations
The biggest problem I have with Pareto’s law is that is applied way too often to too many situations. In many cases, especially in software development, Pareto’s law becomes a self-fulfilling prophecy—the more you look for Pareto’s law, the more you magically seem to find it.
None of this is to say that Pareto’s law isn’t a real thing—of course it is. If you go and take a look at real hard numbers about distributions of things, you’ll find Pareto’s law all over the place in your statistical data. But, at the same time, if you go and look for the number 13, you’ll find an alarming number of occurrences of that two digit number above all others as well.
It is very tempting to force things that don’t quite fit generalizations into those generalizations. How often do we use the phrases “always” and “never?” How often do we fudge the data just a little bit so that it fits into that nice 80-20 distribution? 82% is close enough to 80 right? And of course 17.5% is close enough to just call it 20 after all.
Not only can you take just about any piece of data and make it fit into Pareto’s law by changing what you are measuring a little bit and fudging the numbers just a little if they are close enough, but you can also take just about any problem domain and easily, unconsciously, find the data points which will fit nicely into Pareto’s law. There is a good chance you are doing this—we all are. I do it myself all the time, but most of the time I am not aware of it.
I’ve found myself spouting off generalizations about data involving Pareto’s law without really having enough evidence to back up what I am saying. It is really easy to assume that some data will fit into Pareto’s law, because deep down inside I know I can make it fit if I have to.
Seeing the world through Pareto colored glasses
You might think there is no harm in fudging numbers a bit and finding more and more places to apply Pareto’s law. But, looking at the world and blindly assuming all data falls into a distribution of 20 percent of the causes being responsible for 80 percent of the effects, is like walking around with blinders on your eyes—you are only seeing part of reality and even the reality you are seeing tends to be a bit distorted.
Again, this doesn’t mean that Pareto’s law isn’t correct a large amount of the time, but it means that when you are just assuming that any data that appears to obey this law will, or worse yet, that all data MUST obey this law, you are severely limiting your perspective and restricting your options to those that already fit your preconceived ideals.
Sometimes I wish I had never heard of Pareto’s law, so that I wouldn’t be subject to this bias.
Let me give you a bit of a more concrete example.
Suppose you blindly assume that 80% of your application’s performance bottleneck comes from 20% of your code. In that case, you might be right, but you might also be wrong. It is entirely feasible that there are some parts of your code that contribute more or less to the performance of the application. It is also pretty likely that there are some bottlenecks or portions of code that heavily impact the performance of your application. But, if you go in with the assumption that the ratio is 80-20, you may spend an inordinate amount of time looking for a magical 20% that doesn’t exist instead of applying a more practical method of looking for what the actual performance problems are and then fixing them in order of impact.
The same applies for bugs or testing. If we blindly assume that 20% of the code generates 80% of the bugs, or that 20% of our tests test 80% of our system, we are making pretty large conclusions about how our software works that may or may not be correct. What happens when you fix all the bugs caused by the 20% of code that generates 80% of them? Does a new section of code now magically produce 80% of the bugs? If 20% of your test cases test 80% of your code, can’t you just create those ones? Why create another 80% to only test another 20%? And if you did follow that advice, then wouldn’t you have the situation where 100% of your tests tested 80% of your code?
The problem is when you start applying and assuming that Pareto’s law applies blindly, you start making all kinds of incorrect assumptions about data and only see what you expect.
So, was Pareto wrong?
In short, no. He wasn’t wrong. Pareto’s principle is a thing. In general, in many cases, it is useful to observe that a small amount of causes are responsible for a majority of effects.
But, it is not useful to apply this pattern everywhere you can. The observation of the data should guide the conclusion and not the other way around.
I find it more useful, especially in software development, to ask the question “is it possible to find a small thing that will have a great effect?”
A good book on this very subject is The 4 Hour Chef. Although I don’t always agree with Tim Ferris, he is definitely the master of doing more with less and talks frequently about concepts like minimum effective dosages.
In other words, given a particular situation, can I find a small thing I can do, or change I can make, that will give me the biggest bang for my buck?
Sometimes the answer is actually “no.” Sometimes, no matter how hard we try, we just can’t find a minority that influences the majority. Sometimes the bugs are truly evenly distributed throughout the system. Sometimes the contributions of team members are fairly equal. One team member is not always responsible for 80% of the results.
And let’s not forget about synergy. Which basically is when 1 + 1 is equal to 3 or more. Sometimes the combination of things together makes the whole and separating out the parts at all greatly reduces the function.
For example: eggs, sugar, flour and butter can be used to make cake, and you could say that 80% of the “cakiness” comes from 20% of the ingredients, but if you leave one of those ingredients out, you’ll quickly find that 100% of them are necessary and it doesn’t even make sense to try and figure out which ingredient is most important, because alone each ingredient functions much differently than they do all together.
In software development this is especially true. Often in complex systems all kinds of interactions between different components of a system combine together to create a particular performance profile or bug distribution. Hunting for the magical 20% in many cases is as futile as saying eggs are responsible for 80% of the “cakiness” of cake.
Let’s just get right into it, shall we?
What is an abstraction?
Before we can talk about leaky abstractions, and why they are bad, let’s define what exactly an abstraction is.
An abstraction is when we take a complicated thing and we make it simpler by generalizing it based on the way we are using or talking about that thing.
We do this all the time. Our brains work by creating abstractions so that we can deal with vast amounts of information.
We don’t say “my Dell Inspiron 2676 with 2 GB of RAM and a 512 GB hard drive, serial number 12398ASD.” Instead we say simply “my computer.”
In programming, abstractions let us think at higher levels. We create abstractions when we give a bunch of lines of code that get the username and password out of a textbox and log the user into our system, a name; login. And we use abstractions when we utilize APIs or write to the “file system.”
Without abstractions we’d have a really hard time managing the complexity of any software system. Without abstractions, it would be pretty difficult to write any kind of non-trivial application.
If you want to feel what it is like to program with fewer abstractions, try serving up a web page with assembly code. If you want to feel what it is like to program with almost no abstractions, do the same exercise, but write your code in machine code… 0s and 1s.
How do abstractions “leak?”
This means the details of the lower level concept that are being abstracted away are leaking up through the higher level concept.
Ever get a cryptic error message when an application you are using crashes? That is a leaky abstraction.
“Memory location 00FFE133 could not be written to” is just bad news. It means that in order to understand what is going wrong with your application, you now have to understand the inner workings of the application. Your abstraction has effectively “leaked.”
Abstractions leak whenever you are forced to pull back the curtains and see the gnomes running in hamster wheels spinning the cogs.
Why “leaking” is bad
I’m often surprised when I hear someone say “I like how you didn’t hide the details of what is really going on” about some code or some technology.
Not being able to hide the details of what is really going on points to an inability to create the proper abstraction, which in my opinion, should be water tight.
It is not a good thing that we can see what is really going on. No, it is actually a very bad thing, because it forces us to think at the higher level some of the time, but have to switch gears to drop down to the lower level some of the time.
Leaky abstractions prevent us from being able to comfortably drive the kids to school; we have to constantly keep looking in our rear view mirror and checking to see if those rowdy kids who insist on sitting at the back of the bus are causing trouble again.
A good abstraction hides the details so well that we don’t ever have to think about them. A good abstraction is like a sturdy step on a ladder, it allows us to build another step above us and climb up it. A leaky abstraction is like a wobbly rotted wood rung; put too much weight on it and you’ll fall through—it traps you at the current step—you can’t go any higher.
Shouldn’t we understand how things really work?
No! Stop already! Enough with the guilt trips and unnecessary burdens. You are defeating the purpose of abstractions and stunting further growth.
You only “need” to know the details if you want or need to know the details.
I know that many people want to know how things really work, myself included for many things, but in many cases I don’t need to understand how things really work.
For example, I can buy and use a watch without understanding how the watch works. It really is better that way. I don’t need those details floating around in my head. I get no benefit from knowing what is going on inside my watch and frankly, I don’t want to know.
When I buy a car, I just want to drive it. I don’t want to remember when to rotate tires, check fluids, etc. Waste of my time. I don’t want to know how the engine works or anything about it besides I have to put gas in, turn a key, and press a few pedals to make the thing go. I know I could save money servicing my own car, but you know what? I’d rather pay money than have to do it myself. And since that is the case, as long as I can find someone I trust to work on my car, I don’t want that abstraction to leak.
The same with machine code and compilers. Yes, it is interesting to understand how memory mapping and CPUs and logic gates work and how my C# code gets compiled into MSIL which is interpreted by the CLR and so forth and so on, until it ends up as electrons operating on registers in my CPU. And I learned about how much of that works, because I was interested, but let’s be completely honest here. That abstraction is really good. It is so good and so air-tight, that I don’t need to know all of that crap in order to write a web app. If I did, I’d probably find another profession, because my brain would explode every time I tried to write the simplest piece of code.
By the way, if you do want to understand what is happening behind many of those abstractions, I highly recommend Charles Petzold’s book: “Code: The Hidden Language of Computer Hardware and Software”. (This is the kind of book you read for “fun,” not because you need to know this information.)
Leaking abstractions in technology
Too many technologies and frameworks today are leaky, and this is bad– really bad.
Want an example? Whatever happened to ORM (Object Relational Mapping?)
Nothing. Nothing happened. It was invented, oh, like thousands of years ago and while lots of other technologies and frameworks grew up and got jobs and left the nest, ORM is still a fat unemployed balding 30 year old man playing WOW all day in his mum and dad’s basement.
Don’t get me wrong. I use ORMS; I sometimes like to think I like ORMS. But, I can’t use them well because I have to understand SQL and what is going on in my relational databases anyway. What good is that?!
Seriously. This is a completely failed abstraction. The leakiness of it makes it actually more work to learn an ORM framework, because in learning one you must also learn hairy details of SQL and understand what N+1 means and so on. You can’t just fire up your ORM and put in the Konami code, you have to blow the cartridge out and fiddle with the reset button half a dozen times. Useless.
I could go on and on with examples of leaky abstractions in technology and how they are holding us back, but I think you get the point by now.
Leaky abstractions are bad. They don’t do anything good for us besides require us to know more and add more points where things can go wrong. We’ve got to seal up these abstractions, if we want to be able to keep climbing the software development ladder. You can’t keep building a tower if you have to worry about every single block beneath you. You eventually get to a point where you can’t keep it all straight and your tower comes crashing to the floor.
Leaky abstractions and you
So, what can you do about it?
You really have to make a choice. Am I going to hide the details so that no one needs to know about them, or am I going to expose the whole thing? Either make a really good abstraction that doesn’t require someone using your code to understand what is going on underneath that abstraction or don’t attempt to make an abstraction at all.
It takes more effort to create good abstractions. Sometimes the effort required is orders of magnitude greater. But, if other people are going to use your abstraction, to save the mental burden of understanding what is happening beneath it is going to go a long way in making up for your extra effort.
10 Steps to learn anything quickly course
By the way, I just released by 100% 10 Steps to Learn Anything course. If you want free access to it, just signup to my newsletter here. It is a totally free video course and optional workbook that reveals the process I use to quickly learn technologies.
I have another new course on Pluralsight, check out:
I am very excited to finally get this course out. Many viewers have been asking me to make a comprehensive course that actually shows you how to build a real automation framework, and I finally did it in this course.
I reveal all my tricks and tips here and tell you exactly what I have done in the past to build successful automation frameworks. It was a lot of fun to create this course and it brought me back to the glory days of creating automated tests. (This is some serious fun!)
Anyway, check out the course and let me know what you think.
Here is the official course description:
Learning how to use a tool like Selenium to create automated tests is not enough to be successful with an automation effort. You also need to know how to build an automation framework that can support creating tests that are not so fragile that they constantly break. This is the real key to success in any automation effort.
In this course, I will reveal every secret I know from creating several successful automation frameworks and consulting on the creation of others. I will show you exactly, step-by-step how to create your own automation framework and I will explain to you the reasoning behind everything we are doing, so you can apply what you learn to your own framework.
We’ll start off this course by going over the basics of automation and talking about why it is so important as well as discuss some of the common reasons for success and failure.
Then, I’ll take you into the architecture of an automation framework and show you why you need to pay careful attention to the structure of any framework you build and give you some of the underlying design principles I use when creating an automation framework.
After that we’ll be ready to start creating a framework. In the next few modules, I’ll show you how to create a real automation framework capable of automating the WordPress blogging platform administrative console. We’ll start off by creating smoke tests and using those smoke tests to build out our initial framework.
Then, we’ll expand the capabilities of our framework as we create more tests and learn how to use techniques like dummy data generators to make our tests as simple and easy to read as possible.
Finally, I’ll take you through some best practices and tips that cover topics like scaling out, working in Agile environments and other important issues you are likely to face. If you are responsible for an automation project for a web application or you want to start using automation, you’ll definitely want to check this course out.
Update: be sure to check out this comment chain to see a bit more clarification on what code katas really are. Thanks for Cory Foy for pointing out that I was unfairly calling all code katas as solving the same programming problem over and over again. Only some people treat code katas that way and this post is about why that kind of code kata is not effective. I also updated a few words in this post to reflect this point.
I don’t want to seem like I am bragging, but there is something I just have to get off my chest.
But, before I tell you what amazing skill I have mastered through countless hours of boring practice, let me tell you a bit about my training schedule.
Everyday I get up in the morning. I put on my shoes. I begin my practice.
Usually I start with a small warm-up, just to get the blood flowing on the way to my office at the end of the hall.
Later in the day, I’ll wander out of my office for lunch and practice more on the way to the kitchen.
Right now as I am typing out this blog post, I am on the treadmill honing my craft one painstaking repetitive step at a time.
That is right, you may have guessed it by now—I am a master walker.
All my life I have been practicing this humble skill. Each and every day I practice my secret craft. I see other fools riding around on mopeds and motorized scooters sheepishly talking of their perceived skill at bipedal transportation. But, although outwardly I acknowledge their words, inwardly I know that I am special, because I hone my skill to a craft.
Sounds ludicrous? That is what it sounds like to me when some people talk about doing code katas.
What is a Code Kata you may ask? Some people think a Code Kata is when you solve the same programming problem over and over again, thinking you are actually practicing something other than using shortcuts in your IDE. (To be fair, not all people view Code Katas that way.)
These kinds of Code Katas come from a failed attempt to forcibly simulate the same kind of practicing that a musician or athlete would perform in order to become better at their vocation.
Why Code Katas aren’t effective
Don’t get me wrong, the intention behind Code Katas seems to be well placed, and I am all for software craftsmanship (although not the elitism that sometimes accompanies it.) But, just as walking every single day doesn’t make you a master walker, and driving a car every day doesn’t make you a superior driver, solving the same sets of programming problems over and over again won’t make you a master programmer.
If you still have your doubts, consider old people. Old people are not master walkers or master drivers, yet they have been doing both of those activities day in and day out for many years. Not only do they not master these skills, the skills actually atrophy—they become worse at them, and not just because their muscles and reflexes become worse from age, it is because repeating the same thing over and over again without an increasing level of challenge actually results in the mind becoming complacent with the activity which leads to the eventual degradation of it. (At least, this appears to be the case to me.)
Don’t lose hope though, a little later on I will tell you a real way to practice your craft in software development—one that will give you results—but, first let’s dissect the cases of walking and driving a little bit further to see why repeated practice of those activities seems to have diminishing returns.
Doing the same thing over and over again doesn’t make us better at it
We first learn from walking and driving, because everything is new and challenging. Although walking seems like a repetitive process, it is actually the lack of repetition that provides the challenge our brain and body need to actually benefit from it.
As a toddler learning to walk or a teenager learning to drive, we are constantly bombarded with new scenarios and challenges that we haven’t faced before. What happens when the dog runs by and brushes our leg? How does this affect our balance and what do we have to do to adjust? What about this new surface? The carpet seems to absorb our feet and create more friction when we attempt to walk on it. How do we compensate to balance on this surface? Walking with socks on the floor gives us less grip, etc.
But, as an adult, or experienced walker or driver, we are not challenged in nearly the same way; we’ve already seen most of the scenarios. Our daily commute to work rarely presents us with an unfamiliar stimulus or problem to solve. We operate on autopilot, automatically walking and driving to our destinations without much effort or thought.
Repeatedly writing the code to solve the same programming problem over and over again is exactly the same thing, except unless you are introducing new challenges by adding constraints and twists, you are only benefiting from the exercise the first, and perhaps, the second time, you undertake the activity.
Now, not all programmers who do Code Katas repeat the exact same code over and over again. Some of them attempt to solve the same problem, but this time not using any loops or limited the line count to a certain number or some other similar constraint. This kind of practice is actual worthwhile practice which challenges the mind and introduces new scenarios that create new neural pathways in the brain and strengthen others. (Here is a great example of book that encourages you to do just that. I highly recommend working through the problems in this book.)
Simply typing the same code into the same IDE over and over again for 10 minutes a day may make you feel relaxed and calm or even like a concert violinist arduously practicing their scales, but at the very best it will make you faster at using your IDE and typing code; skills which can be easily practiced through the course of a normal day’s work. If you want to be really good at writing the code to sort lists of words and calculate prices for grocery store items in a cart, go ahead, be my guest.
How real mastery is achieved
If you are resisting what I am saying at this point, it is likely because you still mistakenly believe that walking and driving are in a different category than playing music, sports and programming. But, the truth of the matter is that walking, running, music, sports, and programming are in the same category, or rather, there are no categories.
It turns out there are actually master walker and master drivers. One sport that is not all that widely known goes by the name of Parkour or Freerunning. Don’t believe me, check out this video for an example. In this sport, the athlete becomes exceptionally good at traversing around everyday locations on their own two feet. It is amazing what some of these practitioners can do—just watch the video if you don’t believe me.
And as for driving, I probably don’t need to convince you that Nascar is “kind of a big deal.”
My point is that you are not going to become a Parkour expert or Nascar driver by just walking around your neighborhood or driving your car to and from work, even if you do tend to drive like a bat out of hell. To get to that level of skill in those seemingly mundane activities, you have to constantly practice at a higher level of difficulty and continually introduce new challenges.
In fact, when we actually look at what real musicians and sports athletes do, it is much of the same. I seriously doubt many musicians you hear on the radio or see in concert repeatedly played “bah bah black sheep” or “twinkle twinkle little star” on their instruments until one day they achieved mastery. And in the professional sports world, achievement only comes through repeatedly pushing beyond one’s limits day after day.
The point is this: if you want to get better at something, repeating practice alone is not enough. You must practice with increased difficulty and challenge.
How to improve your programming skills
So, how then do you become a master craftsman at the vocation of software development? If you don’t use Code Katas, how do you practice your craft?
Let me start off by giving you the best example I have ever seen.
This is Jennifer Dewalt’s blog; she decided to learn to code by building 180 websites in 180 days. While you are sitting at your keyboard typing the same 20 lines of code into your IDE over and over again, Jennifer is creating a new thing every single day and facing a brand new challenge.
Who do you think will improve their skills more after 180 days, you or Jennifer? I know who I’d hire without even thinking twice.
I get lots of emails asking me about the best way to learn programming or how someone can improve their skills. Recently, I’ve gotten a lot of emails asking about how to learn Android development. Do you know what I tell all these inquisitive minds?
Make Android apps.
But, umm, what book can I read?
No book, just come up with an idea and try to create it. Figure out what you need to do along the way. When you get stuck, go look for the answer or seek help.
But, I need to know what I am doing first.
Says who? Do and then learn. Learn and do at the same time, reading a book or repeating the same exercise over and over again won’t make you smarter or more skillful; constantly challenging your mind and current skills will.
Want to learn more?
I’ve got plenty more to say on the topic and many more techniques that are way more effective then Code Katas which you can use to hone your programming skills. I can’t fit it into a single post like this, but I’ll be putting it into the top secret project I am working on to help developers boost their careers and learn to market themselves.
If you’d like to be one of the first to be notified when it is available, sign up here. I’ll be including lots of practical valuable information that you probably won’t find anywhere else.
So, what do you think? Do you do Code Katas right now? If so, do you think they are actually useful? Let me know in the comments below.
Far too many Agile development projects fail. It is hard to even accurately measure the number of failures because so many software development projects end up “complete” and shipped, even though:
- They took far too long to build
- The quality of what we built was poor
- What was built was not what the customer wanted
- The cost to develop was more than it should have been
Over the years, I’ve worked on many different Agile teams and have consulted for and trained many teams on Agile development. In this time, I have found 5 common problems, that if not resolved, are very likely to cause a project to be a failure.
1. Not having a product owner
Of all the things that can cause an Agile software project to fail, not having a person that is ultimately the decision maker for the product being developed is the quickest way to ensure its demise.
It doesn’t matter if you are following Scrum, doing your own Kanban style project or something else; if you want your project to succeed, you need someone who can set its direction and make decisions about the product being developed.
Think about remodeling a kitchen. If you hired a bunch of contractors to come in and do the various jobs, like: plumbing; carpentry; flooring; etc, but you don’t have someone deciding what the actual finished kitchen should look like, you are going to get a random mess.
A few skilled contractors will probably be smart enough to find the right person to ask about what should be done, but it takes more than just making arbitrary decisions about where the cabinets should go to design a kitchen. You need someone to actually come up with a real design and vision and be able to modify that vision appropriately as the project progresses and problems inevitably occur.
It seems pretty nuts to spend a huge amount of money remodeling your kitchen, but not want to invest any time or effort in either designing the finished product or hiring someone to do it.
Yet, day in and day out, I see this exact behavior in software projects. I see companies investing huge amounts of cash in Agile development, but not appointing anyone to be the true owner of what is being built and to set the vision for it.
2. Not iterating
One of the key values that Agile development brings to the software development world is iteration.
What is iteration though?
You may think it means breaking your project into 2 week sprints or iterations. While doing this can facilitate iterative development, it doesn’t mean you are actually iterating.
Confused? Let me explain.
The key to iterating is to develop a product a little bit at time. It would be accurate to describe the process of iterating on a product as evolution of a product.
Whether you believe in macroevolution or not, microevolution, or adaptation is a proven concept. The idea behind evolution is that things change gradually over time. Those small changes add up to result in a big change.
Imagine how silly it would be if evolution worked the way most “Agile” software is built.
Imagine, if you will, a fish that swims in the ocean and has some little fish babies that are born with fully functional legs. Then, those legged fish babies grow up and have fish babies that have wings.
Probably the legs and wings wouldn’t do that fish much good, nor would they be designed properly, because instead of adapting and changing over time, they just suddenly appeared.
Features should not be built in a single sprint or iteration. It is as silly as legs showing up on a fish in a single generation.
Instead, features should be evolved and grow over time. A feature shouldn’t be pushed into a single sprint or iteration and then be done. A feature should start out small and develop and evolve over time as feedback is received and customers or the product owner tries it out.
Far too many times, I see Agile development projects mistake having iterations with actually iterating the development of the product.
Don’t try to ship a completed feature at once, let it evolve over time.
3. Not breaking things down small enough
The main reason why this is so important is because it prevents procrastination. Procrastination usually occurs when either we dread some large task that will be difficult or we don’t know what to do next.
If you can break a big project up into small parts, it will seem easier to accomplish and will have clear steps of progression.
I often see software projects where the person creating the backlogs or work items is not considering the work enough before giving it to the team.
I coined a term for these kinds of backlogs: fatlogs. Fatlogs are backlogs that are not broken down small enough and often are very vague in what needs to be accomplished.
Fatlogs are notoriously difficult to estimate and waste time when trying to explain and understand them. It is critical that fatlogs be broken down into smaller actionable backlogs before being given to an Agile development team or a large amount of time will be wasted.
Many times, I have found that the creator of the fatlog could have easily broken down the work it represents into several smaller backlogs that would be easier to explain and understand, even without knowing much about software development.
I often recommend to Agile development teams that they outright reject fatlogs and send them back up the chain.
“If you can’t take enough time to clearly state what you want, it must not be that important.”
This doesn’t excuse development teams either. Development teams should break down any backlogs they get into small tasks as well.
4. Not setting done criteria
What is the first thing a waiter asks you when you order a steak?
That’s right, they ask you how you would like it done.
If the chef doesn’t know what the done criteria is for your steak, the chef will have to decide what he or she thinks the done criteria is.
You may get back a well-done steak, or a rare steak, or something in between, depending on the personal preferences of the person cooking your meat.
This isn’t a good way to cook steaks and it isn’t a good way to cook software either.
In many software projects I often encounter lots of steaks being cooked but no one defining when they are done. The backlogs most often end up being “done” when time runs out.
It is very important to have explicit unambiguous done criteria for any backlogs being worked on by an Agile development team.
This means that the product owner should be defining some level of acceptance testing. It doesn’t matter if the tests are manual tests or fully automated tests (BATs), what matters is that some criteria is defined by which the team can evaluate whether they have achieved their goal or not.
Lacking this criteria is like a parent responding with “enough” when their child asks “how many more peas do I have to eat?”
Not having done criteria is demotivating and results in the finger pointing game of why the correct thing was not built.
I have found that if you tell someone exactly what is expected of them and what criteria they will be judged by, you get much better results than just throwing them in the ring, slapping them on the back, and saying “do a good job.”
5. Not letting the team be a team
Teams are strange organisms, especially Agile teams.
There are many dynamics that affect how teams act and interact. There are many ways to foster a healthy team and many ways to create unhealthy teams.
A healthy motivated team has synergy, a unhealthy gaunt team can be less effective than all of its members are on their own.
The key to having a healthy motivated team, is letting them be mostly autonomous. Regardless of your political persuasion, you would probably agree that when a country invades another country and sets up an unelected government to oversee the people, there are going to be problems.
The same thing happens in Agile development.
I’m not saying teams shouldn’t have accountability. But if you want to run a software project in an Agile way, you have to let teams self organize and self manage to some degree.
When big brother is always watching and interfering, team dynamics become very different. Naturally, teams usually develop their own cadence, leadership and roles (this is called norms.) However, when outside forces directly interfere with how the team does things, and they don’t have the power to decide their own fate, team members begin to realize that they need to look out for themselves.
Additional agile development resources
I’ve found it to be pretty difficult to find good resources on these topics, but here are a few books I’ve found useful that address some of the issues I described above.
- Succeeding with Agile: Software Development Using Scrum - This book is focused on Scrum but it applies to many different kinds of Agile projects as well. Mike Cohn and I usually agree on most Agile subjects.
- Agile Retrospectives - I’ve found this book to be useful for getting good idea for team retrospectives that can really help to break down some barriers and help teams learn to resolve issues themselves.
- Scrumban and Kanban and Scrum - Both of these books have excellent information about combining Scrum and Kanban and have great solutions for dealing with some of the problems I described above.
What do you think? Have I missed any of the major Agile project killers? Any useful tips for dealing with them?
Let me know in the comments below.
I always hear the advice that we should tackle hard problems first.
It seems like pretty legitimate advice, and many of the reasons for saying so make sense, at least at a surface level.
The reasoning usually revolves around the idea that by tackling the hard problems first you can eliminate your biggest risk early and everything will be smooth sailing from there.
When you really think about the reasoning for solving hard problems first though, most of it is not actually reasoning, but based of off one emotion… fear.
We tend to think it is best to solve hard problems first, because we are thinking about eliminating our fear, not because we are thinking about what approach has the highest chance of success or is the most optimal.
I call this FDD, or Fear Driven Development.
And when I think about it that way, I find myself hard pressed to find a real good reason for tackling hard problems first besides being able to abort early. Which, in some cases might be good, but I’d rather focus on success.
Here are 7 reasons why it might be a good idea to tackle the hard problems last instead of first.
1. Solving easy problems first gives you momentum
When a large ball starts rolling down a hill, it picks up speed rapidly and that large ball can bust through many barriers that it couldn’t before, simply because of one thing it has after rolling down a hill that it didn’t have before—momentum.
On the converse, trying to push a heavy ball up a hill is pretty hard. And if there are barriers on the way to the top of the hill, not only do you have to fight gravity, but you have to be able to push extra hard to get through those barriers.
Life is hard enough, why make it harder?
I recently received an email from a developer that was concerned that his team wasn’t gelling and that they didn’t have the expertise in the technology needed to solve the complicated problem ahead of them.
They were going to start the project by trying to integrate this fairly complex technology and he was afraid that it would cause them a large delay before they would be able to get version 1 out.
Start with your simple problems; get working software out there as soon as possible. Not only will the team gel much more easily as they are having success and making real progress, but that momentum will help them when it is time to solve the more difficult problem.
Even if they have to throw the first version away, when they get to the hard problem, the momentum alone will make them much more likely to reach success in the end.
I could give 100 examples of how solving easy problems to gain momentum can benefit you, but you probably already know intrinsically that this is true.
Long story short, get a running start before taking a big leap.
2. Avoid solving the wrong problem
The big problem with solving the hard problems first is that the hard problems usually require a large amount of context in order to fully understand them.
It is very hard to get the right context for a hard problem when we take it out of its natural order of progression and artificially cut it to the front of the line.
You’d probably like to start a college class by taking the final exam first, so you don’t have to worry about it, but the problem with doing so is that you’ll lack the context and information to understand the questions and to know the answers.
When we try to tackle problems out of order to avoid leaving the hard problems to the end, we end up losing all of the learning and context that would help us to solve the hard problems at the end and we are much much more likely to end up solving the wrong problem, which is a complete waste of time.
3. Someone else may solve the problem for you
Sometimes procrastination is a good thing.
Sometimes, when you purposely push off solving a hard problem till the end, you end up finding that someone else already solved your problem.
I spent a few minutes trying to troubleshoot it, but nothing I was trying was working, so I had to make a decision.
I had 3 choices:
- Keep trying to solve this hard problem before moving on.
- Go on and do other videos and send off a support request to see if they could handle it.
- Make a new project and re-edit all the clips.
Choices 1 and 3 involved tackling a hard problem right then and there.
Choice 2 was to tackle easy problems and see if support could solve my hard problem for me, and if not, I would solve it at the end.
I ended up choosing option 2 and it paid off. It turned out Camtasia support was able to solve my problem. By the time I needed the project to complete my course, they had solved my hard problem for me and I didn’t waste any time upfront trying to tackle it myself.
Now it could have worked out differently; I might have had to solve the problem myself at the end, but instead of assuming I would have to and wasting perhaps a day or 2, trying to solve the problem myself, I pushed it off and kept working on easy problems and I gave someone else a chance to solve my hard problem for me.
It doesn’t happen all the time, but many times if we push off the hard problems we face, we find that by the time we get to them, someone else has already solved the problem for us.
4. Your own subconscious mind may solve the problem
When I said someone else might solve the problem for you, that someone else might actually by you—at least your subconscious mind.
Have you ever had the experience of thinking about a problem and not being able to figure it out, but then you wake up the next morning and suddenly have the answer?
It seems that our subconscious mind is more powerful than we are aware of.
In many cases, if we know of the hard problem we need to solve and have thought about it a little bit, our subconscious mind will start working on the solution, even though we are not aware.
Obviously this isn’t going to work all the time, and your subconscious mind isn’t going to write a bunch of code for you, but in many cases there is at least some benefit to throwing the problem off to our internal “worker thread.”
5. You are more committed to solving the hard problem when you risk everything you do so far
One benefit to saving the hard problem for last is that you have some extra motivation in the form of loss aversion.
We can use this knowledge to our advantage by doing the easy work first and letting our loss aversion help motivate us to solve the harder problems, because we don’t want to lose all the work we put into a project so far.
By starting with easy problem, we put some “skin in the game.”
If we try to solve the hard problems first, we have nothing to lose, so we are much more likely to give up.
6. Hard problems are often easy problems bundled together
And it turns out that many hard problems (not all) are decomposable into many small easy problems.
If you strive to never solve hard problems and to always push them off, you may actually find out that you never have to solve hard problems.
Many times we can chip away at hard problems, by taking bits of them off a little at a time and solving those easier problems. Eventually, you may find that you’ve reached the tootsie roll center of your hard problem lollipop and it is filled with chocolate candy!
Now, some problems aren’t very easily decomposable, but a good majority of problems are. Once you develop the skills to chip off bits of hard problems into smaller easy problems, the world looks like a much more conquerable place.
So, saving hard problems for last and breaking off little pieces of them as you go, can be a good strategy to help you to wear down your opponent before you have to face him.
7. Some hard problems are never truly solved
One of the big problems with tackling the hard problems first is that they tend to fill up as much time as you’ll give them.
If I give you an easy problem, like write a function to reverse a string, there isn’t much to think about. You can solve it a number of different ways and there isn’t a huge difference in the efficiency of the different methods of solving it. It is pretty unlikely you’ll spend weeks revamping your solution and thinking that it’s not quite right.
But, if I give you a problem like, create an in-memory database, not only is it a hard problem, but it has a huge number of possible solutions and can be optimized from now until the end of time. You could spend 2 weeks working on that task or 2 years.
The problem is that many hard problems don’t have a good scope to them when they are solved in isolation.
If you design an engine for a car that isn’t built yet, you won’t know when you are done.
But if you design an engine for a car and you know how much it weighs and know what kind of transmission it will use and what kind of fuel efficiency it needs to have, you can have a much clearer understanding of when your engine design is done.
If you save the hard problems for last, the scope of those hard problems will be much more defined, which will keep you from wasting valuable time over solving a problem or, like I mentioned earlier, solving the wrong problem altogether.
There is a huge difference between a principle and a best practice.
Best practices are subjective and depend largely on context, while principles are eternal and universal.
After writing The More I Know The Less I Know, I received a few emails talking about how there are absolute best practices that should always be followed in software development.
I had already intended to write about principles, but that confusion made it clear to me that there should be a distinction made between best practices and principles. We don’t want to throw the baby out with the bath water.
Looking at some examples of best practices
First let’s take a look at some software development best practices, then we’ll contrast them to principles to better get an idea of the difference.
One of the most common best practices today in software development is the idea of unit testing. I’ve written about how I have my doubts about blindly following this best practice in the past, but whether or not we should follow it, is not what I am concerned with today.
Unit testing is extremely contextual. What I mean by this, is that almost anyone would agree that there are a certain set of circumstances that makes unit testing have value.
If you work in an environment where the execution of unit tests takes a really long time, or you are developing your software in a waterfall approach where you have a big upfront design and detailed requirements, unit testing starts to lose value rapidly.
But rather than get trapped into the argument of when unit testing loses its value, it is better to address when it has the highest value—we are much more likely to agree there.
Unit testing has the highest value when we are working in agile environments where changes are being introduced into a software system rapidly and refactoring is taking place. It also greatly increases in utility when you are able to execute and write the tests quickly, because that feedback loop makes it much easier to write the tests in a step by step approach, especially when doing TDD.
There are plenty of other best practices that have fallen out of favor, like heavily commenting code and documenting requirements with UML diagrams, but context also greatly played a part in the value of these practices.
When most developers wrote very short variable and method names, comments were really important. Before Agile processes became prevalent, getting detailed requirements upfront was critical.
But, most best practices are good!
Yes, you are right, most best practices do apply pretty broadly and are generally helpful in a large number of different contexts.
For example, it is considered a best practice to use a source control system and it doesn’t seem like there are many situations where this wouldn’t be the case.
So doesn’t that make it a concrete rule or a principle?
No, it is still too specific to be generally applied in all cases and the act of putting your code in source control does nothing to improve the quality of your software or software product.
If you were to blindly follow any best practice and not apply that best practice in a way that brings out the underlying principle, you would be very unlikely to actually receive any benefit.
You see, most best practices are actually derived from universally applicable principles that never change. That is why most best practices are good.
The problem is applying the best practice itself in no way assures the benefit of its underlying principle.
To put it plainly, there is something greater at work that makes it a good idea to check your code into a source control system. It is entirely possible to follow the action, but completely miss the spirit of the action.
More and more today, I see software development teams that are:
- Writing unit tests
- Using continuous integration systems
- Using source control
- Having Scrum meetings
- Pair programming
- Using IoC containers
Yet they are getting little to no benefit from it. Just a bunch more pain and hoops to jump through. The reason is simple…
It’s not the best practice that is effective, it is the principle behind the best practice
Principles are everywhere. They apply in all aspects of our life. You cannot go through the day without being affected by the results of 100s of different principles that have a constant influence on your life, just like the law of gravity does.
Gravity is actually a great way to understand principles. As far as we know, it is a universal force that is always in effect. It is impossible to escape the law of gravity, wherever you go in the universe it affects you.
Principles are like laws of nature except bigger. Principles are more like the laws of reality. Even though you may not be able to describe them fully or understand how they work, they always work.
Take for instance, the law of the harvest. Most people are familiar with this particular principle. It basically goes like this.
You reap what you sow.
How universal is this truth? How can anyone avoid it? How many times have you found yourself subject to this inescapable law about how reality works?
Many software development best practices are actually based on this principle. Think about best practices that have you make efforts to improve the quality of software early on in the process.
TDD or test driven development, is such a best practice. The basis of TDD is to introduce quality into the software development process as early as possible, so that the finished product is better.
If you apply the practice of TDD without understanding this principle, you are just following the motions and you won’t actually gain the benefit of the practice.
If you can’t understand at some level that the point of doing TDD is to sow some good seeds in your software that you will harvest later on, you won’t be writing the right kind of tests.
There is nothing magical about writing tests before writing code, but there is something valuable in purposely investing in upfront quality with the end goal of getting a big yield on that investment in the right season.
By the way, that is why I like Bob Martin’s book Agile Principles, Patterns and Practices in C#; it discusses many principles of software development that are timeless. Books like this one and the book I have mentioned probably 10 times in this blog, How to Win Friends and Influence People, are full of principles.
Also, check it out, you just learned what Agile really is. With principles in mind, now read the Agile manifesto. It was never designed to be a detailed process and set of best practices for developing software, it was always meant to be a recognition of a set of principles that guide software development.
So, just remember the next time you are arguing with someone over a best practice, or consider applying one to a project you are working on, if you don’t understand the underlying principle, no amount of ceremony and procedure will have the smallest amount of benefit.
With the coming and passing of the predicated date of the Mayan apocalypse, I got to thinking about what kind of final advice I would leave the development world in the event of my untimely departure.
I suppose this is quite a strange thing to ponder given the circumstances, since if the world did indeed end there wouldn’t really be anyone to pass this advice onto anyway. Also, given that I am writing this blog post now, it would already be too late.
Regardless, it got me thinking about the little things I have learned in my development career that might be helpful to others.
So without further adieu, if I could only leave one blog post to the future generation of developers, here is what it would be.
1. Break things into small tasks
This might seem like a strange thing to put at number one.
I mean really, of all the advice that you can give, breaking things into small tasks is number 1?
Yes, it is that important!
Breaking things into small tasks is not just a skill that will make you a better developer, but it is a critical thing that will determine whether or not you enjoy what you are doing.
I have found so many times that when I have become unhappy with my work or felt distraught by the overwhelming task ahead of me, it is mostly because I have failed to break down a chunk of work into small tasks.
Just this last week I re-realized how much a difference just breaking things up into small enough tasks can make on the overall morale of an entire team and my individual feel of satisfaction with the work I am doing.
Time and time I come back to this very real realization; that breaking things up into small tasks doesn’t necessarily make me a better developer, but it makes me a much happier developer. (Although it is quite easy to make the argument that it makes me better at my job as well.)
2. Help others
This one may also come as a surprise, but I’ve found that helping others is one of the best ways to help yourself.
Not that we should have selfish ambitions when helping others, but I want to point out something that many developers don’t realize when they are first starting out. Most developers never realize how much they actually benefit from helping other developers. I have been in plenty of great environments where every developer was willing to help anyone else who needed their help, but I have also encountered many environments where that was not the case.
Sometimes we can be trapped into thinking that helping others will reduce our own value, because we will be giving out our valuable knowledge to others for free.
This way of thinking could not be further from the truth. In contrast, I have found that helping others results in a much deeper understanding of things yourself. I’ve talked about how teaching something is really the best way to learn a subject, but it goes much deeper than that. When someone asks questions about something you take for granted, you have to answer a question you don’t normally ask yourself when you know how–why.
Helping others forces you to answer the question of why you are doing something a certain way and brings to the surface of your mind a much fuller understanding of a problem.
In addition to gaining deeper understandings of what you already know, helping others often will expose you rapidly to a large number of problems. I cannot express how valuable the increased exposure to problems is in accelerating your own learning.
3. Speak up tactfully
Perhaps you may not feel like what you have to say is important or that you don’t want to cause conflict or maybe you are just trying to be agreeable, but whatever your reason is, “stop it!”
You need to express your voice, because doing so is going to actually make you part of the team rather than being a passenger on the bus.
I often see that many developers either remain quiet or speak their mind with no temperance in what they say.
It is very important to say what you think, but it is just as important to do so in a tactful way. I realize this is much easier said that done, but mastering this skill will help you immensely in your career. I have found that very rarely will someone be offended by your ideas, even if they contrast their own, if you present your ideas in a tactful way and you are willing to change directions when you realize you are wrong.
It is amazing how many things many of us do over and over again each and every day and we never take the time to automate them. The more brainless tasks that we do which could be automated, the less time we have to do things that actually improve our skills and really matter.
I’ve found that I often fall into the trap of thinking that I don’t have enough time to automate tasks that I am doing, so I keep doing them over and over again.
I still have about a dozen things I can think of right now that need to be automated.
As software developers we should really take the time to automate these things, because software development is automation.
5. Understand first
Another major trap that we can fall into is trying to jump into things without fully understanding the problem.
I don’t know how many times I have tried to fix a bug without actually reproducing it first. Many times I prematurely jump into trying to solve a problem before I fully understand it, because I feel like I don’t have the time to spend learning about the problem.
This is always a wrong choice!
I’ve found time and time again that the moment you fully understand a problem you solve it. (There are some exceptions of course.)
6. Have a side project
Not only will it likely provide you a second source of income, but it will improve your skills and give you an important outlet to learn and explore without the outside pressure of time or politics.
It can be pretty difficult to have a full time job and still try and do something on the side. I know that after working for 8 hours the last thing you probably want to do is come home and work some more, but there is a very large value in doing something for yourself that adds up over time.
Most of us spend our day jobs working for someone else and making them money. Most of our day jobs involve doing work that only benefits us for a single paycheck. What I mean by this is that the work you do at your job doesn’t continue to pay you dividends over time. So if you work your regular job for 20 years at the end of 20 years, the only thing you will have is the money you earned from that work.
Now think about what happens if you devote 1 or 2 hours a day to doing some work for yourself every day as well. 1 or 2 hours a day might not seem like much, but over just 2 years that is 500 to 1000 hours of work on a side project. Think about what you could accomplish in 500 to 1000 hours.
7. Read books
I’ve talked about this idea many times, because it is so important. You should always be reading some kind of book that will improve your skills and knowledge. I do videos for Pluralsight, and videos and blog posts are great, but books are still vitally important.
Books require much more thought and are processed much more than any other medium.
Think about what goes into the process of writing a book. Books are outlined, then written, edited and revised and technically proofed. Books also tend to cover subjects much more deeply than a blog post or a video series.
So, I am not saying to avoid other mediums, but I am saying that quality books are an important part of improving your abilities as a software developer.
8. Do the right thing
Bob Martin had an great post on saying “no.” The basic idea is that you are a professional and because you are a professional sometimes you’ll have to basically disobey what your client or employer asks for and instead insist on doing the right thing.
I recognize there is often a great cost associated with this line of thinking– you might lose your job– but if you ever want to consider yourself a professional, you have to be able to make these kinds of tough choices.
In my experience, I have never been fired or even detrimentally affected by taking this stance.
I’ve always tried to take this position in a very respectful way and that has probably been a large reason of why I have had success with it. I have found that most employers and clients will respect you and value you more even if they are a bit ticked off initially. This goes back to the idea of speaking up and doing so in a tactful way. When you do this correctly and you insist on doing the right thing it is called integrity and even if someone disagrees with your position, most people will respect you for it.
9. Be kind
Seriously, is there any benefit to you or anyone else by being mean?
I know it isn’t always easy to be kind, but you should really strive to be, because it does no good to treat people poorly.
Alvin the Chipmunk once said that you’ll meet the same little people on the way down as you did on the way up.
I can’t make an effective argument of how being kind to others will benefit you, and even if I could I wouldn’t try, because that is really not the point. The point is we are all human beings and we are all deeply flawed, but every single one of us has value and that value can be appreciated.
So, just be nice to everyone you meet along your way and you’ll have a much better time of it. Call it Karma if you will.
I used to be very confident in my abilities as a software developer.
I used to be able to walk up to a group of software developers and tell them exactly what they were doing wrong and exactly what was the “right” way to do things.
I used to be sure of this myself.
It wasn’t even that long ago. Heck, when I look at the blog posts I wrote 3 years ago I have to slap myself upside my head in realization of just how stupid I was.
Not only was my writing bad, but some of my thoughts seem so immature and uneducated that it feels like a completely different person wrote them.
And I wrote those posts back when I knew it all.
The more I learn, the less I know
Lately I’ve been running into situations more and more often where I don’t have a good answer for problems.
I’ve found myself much more often giving 3 pieces of advice attached with pros and cons rather than giving a single absolute—like I would have done perhaps 3 years ago.
I’ve been finding as I have been learning more and more (the past 3 years have been an accelerated pace of growth for me,) that I am becoming less and less sure of what I know and more and more convinced that there is no such thing as a set of best practices.
I’ve even spent some time postulating on whether or not commonly held beliefs of best practices would be thrown completely out the window given a significant enough motivation to succeed.
My point is that the more doors I open, the more aware I become of the multitude of doors that exist.
It is not just the realization of what I don’t know, but also the realization of weakness of the foundation I am already standing on.
Taking it out of the meta-physical
Let’s drop down out of the philosophical discussion for a bit and talk about a real example.
Perhaps the biggest quandary I struggle with is whether or not to unit test or practice TDD and its variants.
The 3 years ago know-it-all version of me would tell you emphatically “yes, it is a best practice and you should definitely do it all the time.”
The more pragmatic version of me today says, in a much more uncertain tone, “perhaps.”
I don’t want to delve into the topic in this post since I am sure I could write volumes on my ponderings in this area, but I’ve come to a conclusion that it makes sense to write unit tests for code that has few or no dependencies and that it does not make sense to do so for other code.
From that I’ve also derived that I should strive to write code that separates algorithms from coordinators.
I still even feel today that my advice is not wholly sound. I am convinced it is a better approach than 100% TDD and units tests, or no TDD and unit tests, but I am not convinced there isn’t a deeper understanding and truth that supersedes my current thoughts on the matter.
As you can imagine this is quite frustrating and unsettling.
Silver bullets and best practices
What I am coming to realize more and more is that there are no silver bullets and more surprisingly there are not even any such things as best practices.
Now I’ve heard the adage of there being no silver bullets so many times that it makes me physically sick when I hear someone say it, because it is so cliché.
But, I’ve had a hard time swallowing the no best practices pill.
I feel like when I abandon this ship then I am sailing on my life raft in the middle of a vast ocean with no sails and no sense of what direction to go.
A corner-stone of my development career has been in the learning, applying and teaching of best practices. If these things don’t exist, have I just been pedaling snake oil and drinking it myself?
Best practices are simply concrete applications of abstract principles in software development that we cannot directly grasp or see clearly enough to absolutely identify.
Breaking this down a bit, what I am saying is that best practices are not the things themselves to seek, but through the pursuit of best practices we can arrive at a better understanding of the principles that actually are unchanging and absolute.
Best practices are optimal strategies for dealing with the problems of software development based on a particular context. That context is primarily defined by:
- Language and technology choice
- Meta-game (what other software developers and perceived best practices are generally in place and how software development is viewed and understood at a given time.)
- Individual skill and growth (what keeps me straight might slow you down; depends on where you are in your journey.)
There is a gentle balance between process and pragmatism.
When you decide to make your cuts without the cutting guide, it can make you go faster, but only if you know exactly what you are doing.
Where I am now
Every time I open my mouth I feel like I am spewing a bunch of bull crap.
I don’t trust half of what I say, because I know so much of it is wrong.
Yet I have perhaps 10 times more knowledge and quite a bit more experience in regards to software development than I did just 3 years ago.
So what gives?
Overall, I think I am giving better advice based on more practical experience and knowledge, it is just that I am far more aware of my own short-comings and how stupid I am even today.
I have the curse and blessing of knowing that only half of what I am saying has any merit and the other half is utter crap.
Much of this stems from the realization that there are no absolute right ways to do things and best answers for many of the problems of software development.
I used to be under the impression that someone out there had the answer to the question of what is the right way to develop software.
I used to think that I was picking up bit of knowledge, clues, that were unraveling the mystery of software development. That someday I would have all the pieces of understanding and tell others exactly how they should be developing software.
What I found instead was that not only does nobody know the “right” way to develop software, but that it is perhaps an unknowable truth.
The best we can do is try to learn from obvious mistakes we have made before, start with a process that has had some level of success, and modify what we do based on our observations.
We can’t even accurately measure anything about software development and to think we can is just foolishness.
From story points, to velocity, to lines of code per defect and so on and so forth, all of those things are not only impossible to accurately measure, but they don’t really tell us if we are doing better or not.
So, what is my point?
My point is simple.
I have learned that not only do I not have all the answers, but I never will.
What I have learned is always subject for debate and is very rarely absolute, so I should have strong convictions, but hold onto them loosely.
And most importantly, don’t be deceived into thinking there is a right way to develop software that can be known. You can improve the way you develop software and your software development skills, but it will never be based on an absolute set of rules that come down from some magical process or technology.
Let me ask you a question.
How would you develop your next software project if I told you that if you “succeeded” you would be given $1 million dollars, but if you failed you would get nothing?
Success of course is a very fuzzy term, but let’s assume that success means:
- You built a working functional product
- Your customers are happy with it, content enough to call it done
- You delivered it on time
- You delivered it within budget
- It can be fairly easily modified and maintained with its current design for the next 5 years.
This is an all or nothing proposition. Everything is on the line. If you were to succeed, you would be rewarded handsomely, but if you fail, your effort would be entirely wasted.
Oh, also, you can’t work more than 8 hours a day 5 days a week—no winning by heroics alone.
Why ask this question?
Simple, because after thinking about “how to build software” for the last 10 or so years, (in my beginning years, I didn’t really ponder this question) I have decided the answer to this question is the way that software should be built.
I don’t mean that developers should be given an all or nothing proposition. What I mean is that we should be building and designing software as if I proposed the above situation.
This might sound a bit crazy, but think about it for a bit.
Most developers today are paid a salary. Some salaries have bonuses attached for performance, but rarely are we put in a situation where making mistakes or not being as efficient as possible results in dire consequences. The worst that can happen in most cases is that we would get paid for the work we did and then be looking for a new job at the end of the project.
Losing all the time we put into the project and losing out on an opportunity to make a million dollars would be some consequence.
We’d think about software development quite differently
It’s really hard for me to answer the question of whether or not practices like TDD are worth it. I mean sure, I tell my manager it is. I emphasize quality over speed. I say we need to do this or that, but honestly I don’t really know.
I’ve never been put to the test.
No one has ever said to me, “succeed or die.”
No one has ever put the consequences of my choices directly in my own hands.
Never have I actually had to weigh the weight of extra time spent refactoring code, writing unit tests or using an ORM instead of writing SQL directly in my source code.
I choose technologies because they seem good to me or it seems like they’ll help me get things done faster, but I am never really staking much on that choice.
Sure, I may be staking my reputation, or I might lose my job, but chances are I’ll just blame some failure on some other reason or worse yet, I won’t even recognize my failure, because it will be disguised by “good enough.”
“Man, John, you are a big fraudulent a-hole! I can’t believe you go around touting to know stuff, leading people down paths that you yourself don’t even know are correct. Burning up other people’s money in your test lab of software development.”
Maybe it’s true. Maybe I am a big ignorant and arrogant jerk, but somehow I feel that I am not alone.
I’m not saying we don’t or shouldn’t have strong convictions.
I’m not saying that we don’t have good reason to believe the development methodologies and practices we suggest and follow are correct.
What I am saying is…
Our beliefs about software development are mostly theoretical
8 years ago I would have told you that you needed to create UML diagrams before you write any code.
5 years ago I would have probably emphasized the importance of always having an XML schema. I would have told you data without schema is meaningless.
I don’t hold either of those views today.
Perhaps you’ve changed your views in the past few years as well?
The point is that even though we may be down in the trenches doing software development and to some degree finding out what works and what doesn’t, we are basing the results on OUR motivations not the motivations of our customers or our employers.
So, we may eventually get good at making things easier for ourselves, but unless your feet are held to the fire, you’ll never really optimize for building the best software as efficiently as possible.
If you think you already are focusing on building the best software as efficiently as possible and I am just full of crap and you don’t operate that way, I would strongly suggest doing a thorough self-check and also please tell me the secret of the most efficient way to build software, because I definitely want to know.
What about startups or open source projects?
You might be tempted to think that working for yourself or not for money at all would change things, but I’m really not sure that is the case.
I’ve worked on plenty of my own software projects where my fate was directly in my hands.
I’ve built applications like PaceMaker where my financial success with the endeavor was directly tied my own efforts and my efforts alone.
But, do I think I built PaceMaker as efficiently as possible?
No, definitely not!
The problem with PaceMaker and other projects where there is a big potential upside, is that the upside potential is not guaranteed.
I can assure you that if someone said to me that I have 6 months to build PaceMaker and if I succeeded (according to the rules I stated above) that I would be given $1 million dollars, but if I failed I would be given nothing, I would have probably gone about the whole process an entirely different way.
I’m not saying I would have built it with perfect efficiency, but my motivations would have entirely matched up with the footprint of building software as perfectly as possible.
What I mean by this is that I would be trying as hard as I could to do what I absolutely believed was mostly likely to benefit me, and by doing so I would be trying as hard as possible to completely optimize the process of building that software.
What can we do about this?
Now I can’t tell you how I would have built PaceMaker had I been given the million dollar challenge.
There is no way that I could objectively put myself into that situation and give you an answer.
Would I have bothered with unit tests? Would I have have created backlogs and written UATs to make sure they pass? Would I have used an IoC container?
The plain and honest answer is, I don’t know. All I know is that I would have done things differently and on any software project you’ve worked on, you probably would also.
There are a few things we can gather from thinking about this hypothetical proposition.
- Don’t think you know the answer, but at the same time have convictions about what you currently believe. Just hold on to those convictions loosely.
- Even though you can’t realistically be completely objective and build software as if someone was going to pay you 1 million dollars (just like you can’t tickle yourself) you can still try to think as much as possible with this mindset. It is a worthwhile exercise to imagine as if the decisions you are making and the convictions you are holding onto are based on idealism or reality. We could also all examine and at least understand our own motivations.
- Recognize others are in the exact same boat you are. Don’t believe everything an expert says or even believe there are experts. Experience and wisdom count for something, but no one has anywhere close to all the answers.
I’d love to see this experiment actually played out. I’d really be curious to see what the outcome was, especially in a team situation.
Perhaps someday, someone will actually conduct this experiment and share the results.
What about you? How do you think being given the million dollar challenge would change the way you develop software?