I’ve seen and built many software building and deployment solutions over my career, and I have come to find that most software deployment can be boiled down to a simple process.

I’m not trying to give you a solution for your software deployment automation, nor am I trying to perfectly model your exact process.

What I am trying to do in this post, is to help you to simplify your process.

If you can identify the parts of your deployment process that fit into the simple steps I am going to outline below, it should be much easier for you to automate your deployment process.

The process

Even though software build processes, infrastructure and components are unique, I have found that most software deployment processes can be simplified into the following steps.

1. Build software without configuration
2. Create environment specific configuration.
3. Create a set of database changes.
4. Bundle software, configuration and database changes.
5. Apply new software
6. Apply new configuration
7. Apply new database changes
8. Start it back up

You might read through these steps and think “well duh.”

You might be tempted to say “my process is more complicated than that.”

I’m not going to argue with you.  You are right, your process is probably more complicated than that.  But, does it need to be?

Can you simplify your process to fit into these steps?

Sure, the implementation of these steps is likely to be fairly complex and vary for each type of software, but if you can distill the process into these steps, you can much more easily automate that process.

Where people go wrong

The big key to my simple version of deployment is

Build software without configuration

You MUST do this!  Departing from this step causes all kinds of pain and complexity.  Please don’t try to build your software and the configuration for an environment at the same time.  These things must be pulled out from the get g or you will have the pain of trying to tease them apart later – or you will have to create separate builds for each environment.

It is also critical that the same bits that were built by your build server are what is deployed to each environment!

I will say that this isn’t the easiest problem to solve.  You may need to have a separate build process that builds up the configuration for an environment.

Separating the two will also force you down the path of building a process to apply that configuration to an environment.

But, if you are willing to accept that this is just a must and bite through this pain, you’ll come out on the other side clean (even though you had to crawl through tunnels of crap.)

The whole story

Now that I’ve hopefully convinced you to separate your configuration from the building of your software, let’s go over the big picture of a deployment using the simple process outlined above.

It all starts out when you build your software.  Perhaps you have a continuous integration build server setup that is automatically building software on each check-in; perhaps you are manually kicking off a script.

Once you have built your software, you have some bits that you should be able to apply to any environment.  Nothing that you built here should be machine or environment specific in any way.

Now, you kick off another process, or perhaps one was kicked off simultaneously by your continuous integration server. This builds up the configuration for the environment you are going to deploy to.

A similar process is kicked off—also could be simultaneous, for generating a list of database changes that need to be applied to the target environment.

Now that you have your bits, configuration and database changes, you are ready to deploy.

If you are smart, you’ve even built these ahead of time and they are just waiting for when you need them.

Next, gather up the artifacts and move them to the deployment target where you actually apply them.

First, unpack your bits and put the new bits into place.  (You may or may not need to take your application fully offline to do this.)

Then apply the new configuration on top of your newly installed bits for that environment.

Finally, apply database changes for that environment.

Now you should be completely deployed and can start up your application.

But how do I do it?

Perhaps you agree with me that the actual process should be what I have outlined and described, but now you are at the point of implementing a solution.

How do you actually automate this stuff?

Good question.  If you figure out a simple answer, let me know.

This is the point where you might be writing custom tools and scripts to get all this working.  The key is to take it one step at a time.

There are at least two tools out there that I know of that help you do this.  I can’t speak for either of these tools, since I haven’t used them myself, but I have heard good things about them.

• Octopus
• Dropkick

One other thing to consider is how you are going to get the right stuff to the right server.  You will want to think about things like:

• Promoting build products
• Preloading promoted products to servers to make deployment faster
• Getting through firewalls by having the software or some other process PULL the upgrade to your target, rather than you PUSHING it there.
• Rollback, or some kind of mitigation strategy if things go wrong. (My recommendation here is not to get fancy.  I have NEVER seen a successful rollback, only a database restore followed by a manual code restore.  If you mess up bad, just count on restoring the machine and the database.)

Well perhaps not, but it might be the most important piece of programming advice you hear today.

“Switching gears is grinding gears.”

I’ve been doing this programming thing for quite a while now, and I have come to realize that the biggest thing that saps my productivity is switching gears.

What do I mean by switching gears?

Many different contexts

Switching gears can apply to a variety of different contexts, but it is basically whenever you are working on some task or technology and have to stop what you are doing and either do something else or change the type of thing you are doing.

This is really a bit different than multi-tasking.  Multi-tasking is really trying to do more than one thing at once.  It usually involves a large amount of rapid context switches, but for this post I am more concerned about the general idea of breaking rhythm.

I think it is easier to speak about this subject if we talk about some specific applications of the idea and why they are harmful.

Switching problems

In the Agile world of software development today, we often are required to switch from problem domain to problem domain as many of our iterations contain mixes of different features on different parts of the system and bugs to be fixed.

Have you ever been in the situation where you were just beginning to understand how some area of the code base worked, or how to work with a particular customer on a particular problem space and then had to switch to something else and start all over again?

Even when you switch back to the original context you were working with, if enough time goes by, it becomes new again.  You have to relearn much of what you had learned the last time and just when you begin to hit your groove, the cycle begins again.

This form of switching gears is at the very least frustrating, and at the most a complete demotivator and productivity sapper.

Unfortunately for most developers, this problem is out of your immediate control.  But, take note product owners and project managers, there is a reason why iterations should have a goal.

If you are in the precarious position of being a developer pinned against the ropes, try to make some noise and let your management and product owners know that your team will be much more efficient when you aren’t switching gears all the time.

In my experience, the results of replacing this constant context switching with the synergy of a common goal and related features in a segment of time has an astounding effect on productivity that is hard for any serious businessperson to ignore.  So speak up!

Switching technology

This is the primary reason why I have a distaste for JavaScript.  It is why even in this abundance of web development, I would still rather program a client application.

Switching technology is painful!

This is one of those things that is unavoidable in todays development environment.  If you are a web developer, you are going to be working with HTML, JavaScript, probably some sort of server side language, and most likely some form of SQL.

It really isn’t worth trying to fight this because you are going to be going against the technology grain to do so.  Although, I do suppose this may be one of the reasons for the recent popularity of Node.js.

What we can try to do is to minimize the context switch as much as possible.  We do this by sticking with a particular way of doing things and not chasing after each new technology of JavaScript framework that comes out each week.

I’m not saying to not learn new things.  It is very important to always be learning.

What I am saying, is to try to find some kind of rhythm with the technology stack you are working with and try not to switch that up.

Technology specific training can really help here as well.  I for one, need to learn JQuery better.  The problem is that when I am working on a web based feature, I am not forced to learn JQuery because I am not in that context long enough.

So what do I do instead?

I waste time Googling for answers.  I know that I have a short coming here and I need to just bite the bullet and spend some dedicated time to really thoroughly learn JQuery, because by Googling for little pieces of it at a time, I am not really making much headway and the context switch ends up stealing what I do learn from my memory.

One more aspect of this is the idea of focused teams.  Many software development groups do not like to specialize their developers onto one focus area.  I agree whole-heartily with the idea of non-specialization. 

But!  There is huge benefit to be gained by keeping the same group of developers working on a specific set of technology or part of the code base for a set period of time.  I’ll talk about this a bit more, towards the end of this post, but the basic idea is that it takes time for people to find their groove.

I think it is optimal to break any large team up into smaller technology area focused teams that regularly rotate every 3 months or so.  The idea here is that you give those teams enough time to get good at what they are doing and actually commit what they have learned to memory, but you rotate them often enough that they don’t end up becoming specialists which are unable to see the whole picture.

Switching teams

This one typically isn’t an issue, but it can be depending on the environment that you are working in.

Teams need enough time to go through that forming, storming, and norming set of phases.  If you are constantly interrupting this process by switching around team members, you are never going to get the team to act as an entity of its own.

Teams are generally more efficient than individuals, because they benefit from synergy, when 1 + 1 = more than 2.

But just like a big truck taking some time to get up speed, a team takes time to get going.  Also like a big truck, a team can gain momentum that individual developers seldom can achieve.

A smaller aspect of this is pair programming.  I suppose that many have been successful switching up programming pairs once a day or so, but I suppose that more teams have been successful switching up pairs at longer intervals.

For me, this interval varies.  Sometimes, I feel like I need to stay paired with a teammate for longer than 2 weeks, which is our regular interval, sometimes 2 weeks is fine.  It depends on what you are working on and how much momentum you have.

They key here is to make sure that you are giving teams enough time to take up their own flag and stake their territory.  Teams can take some time to find their common goal.  Self-direction can really help with this.

The groove

Ever tried to turn a flat head screw with a screwdriver, but it doesn’t quite turn, because you can’t seem to get the head of the screwdriver into the groove of the screw?

You twist that screwdriver around a bit until finally it slips into the groove.  Then when you turn the screwdriver the screw turns with it.

As humans, we tend to be like screwdrivers, we have to find our groove.  We all have this adjustment period where we are fumbling through things. 

It is very important to make sure that we aren’t switching gears so often that we are not actually turning any screws once we are in the groove.

Regardless of what kind of context switching you are doing—whether it is problem domain, technology or teams—it is important to make sure that you are not spending 90% of your time finding the groove and only 10% of your time “in the groove.”

Depending on what you are doing and what the context switch is, the proper amount of time before switching contexts is going to vary, but I think it is very important be aware of this phenomenon and plan around it.  If you don’t you are going to end up spinning your wheels, which is unfulfilling to say the least.

One of your friends calls you up and says:

“Hey, would you mind helping me move on Saturday, I am getting 4 or 5 people together and we are going to move my stuff to my new house?”

You of course reply:

“Sure I can help, what time do you want me to be there?”

“We are starting at 10:00 AM sharp!”

You might have guessed what is going to happen next.  You show up at your friends house expecting that everything is nicely packed in boxes and that you are just going to move some boxes and furniture into a U-Haul, but instead what you see is this:

Seriously?

He couldn’t have cleaned up the house and at least put the stuff in boxes before having 5 people show up at his house to move stuff?

So what should have been a 2 hour jobs turns into a 2 day ordeal as you and 4 other friends sit around waiting for moving friend to pack up his stuff so that you can put it in the truck.

All 5 of you can’t actually help with the packing of stuff and the throwing out of garbage because there is only one person who knows what is trash and what things need to go in what boxes.

If your friend would have cleaned up the trash and packed up everything in boxes before he had the 5 of you come over and help, things would have gone much much faster!

Your backlog list is your apartment

The same kind of thing happens when you call up 5 of your developer and QA
”buddies” and ask them to get some work done for a sprint.

If the team is trying to sort out what needs to be done, what is trash, and what things need to go in what boxes, they are going to be much slower at actually getting the work done.

You see, your list of un-groomed backlogs is not unlike a hobo.  Often times it could use a good wash and a proper shave.

When I am talking about backlog grooming here, I am not even talking about having a planning meeting, where you plan what you will do next.

Angela Druckman has a good description of a Grooming Session.

The idea is that periodically you groom the top x% of your backlogs so that they are in nice clean boxes ready to be worked on when the team picks them up.

The important thing here is that this is a team activity, everyone should be involved with backlog grooming.  The business really owns the backlogs in most cases though, so they should be directing the team as to which things go in which boxes so to speak.

What about just in time backlog grooming?

Why work on a backlog until the very last responsible minute?

Good question.

For the answer I point you back to the moving scenario I started this post with.

The reason why it is so inefficient for your friend to have everyone come over and try to move him before he has packed up everything and gotten rid of the trash is that you end up having a large amount of idle time waiting on one person.

Think about when happens when a team of 8 developers starts working on backlogs for an iteration at the same time.

If all of the backlogs are not “groomed”, but instead are a scattered mess of mismatched sizes and parts, each developer is going to have to talk to perhaps the same business person about the backlog that developer is working on.

What happens when this business person is out of the office?

What happens when this business person is trying to field questions from 8 developers?

What happens when there is a turn around time to get back to the developers on questions about the backlog?

Now contrast this with the scenario where all the backlogs that are going to be worked on in an iteration are nicely packaged in neat little boxes.

The backlogs may not contain all the technical details of what is going to be implemented, but they are broken down to small bite size pieces that are fairly well understood by the team.

In this scenario the team is going to be able to pick up the backlogs and start going to work.  Sure they will still have to ask the business questions about the backlog, but the kind of questions being asked change fundamentally.

Contrast this kind of question:

So what exactly is this backlog about?  What kind of things do we want to build to report on this data?

To this kind of question:

As we discussed previously and I see in the backlog, we are creating a custom report for this customer.  I also see that we had defined what this column A should do.  Should I be using calculation X or calculation Y to compute this column?

The first kind of question is better answered in a bigger meeting with all the right people.  To answer the first kind of question might require some research.  The first kind of question would hold up the completion of a backlog because it is not easily answered.  It requires thought and perhaps asking other people, talking to the customer, etc.

The 2nd kind of question is specific.  This is something that can be figured out usually by a single person.  Most importantly it does not hold up the work.  Developers can continue implementing parts of a solution and put in the algorithm for computing some value later.

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

In an Agile environment this question has even more relevance as the focus on teamwork is greatly emphasized.

Teams have a common objective

The most important component of a team is shooting for the same target.  The smaller and more defined the target the more tightly bound the team.

Consider the famous TV team “The A Team.”  Each episode of this television series the team is tasked with a common goal of some sort.  This pursuit of a common goal is one of the main components that binds them together as a team.

Sports teams provide an even better example.  In all team sports the entire team has the objective of winning.  More specifically though, there is always some other sub-objective that the team is involved in.  The more “team-like” the sport, the more the entire team is involved in that objective.

I have found that it is exactly the same in software development.  It is important to try to find small goals that focus the entire team.  When differing members of the team have different objectives, you may be operating on a team, but you might not be operating as a team.

This difference is critical because teamwork often produces synergy.  Synergy is when 1 + 1 = more than 2.  It is when skills are combined together to produce a greater result than the sum.

If you want to increase the feeling of “team” over individual, reduce the breadth of work going on at any given time and focus everyone on the same small goals.

In Agile this looks like having more team members working on a single backlog or story and reducing the number of stories that are put into progress at any given time.  Try putting a strict WIP (Work in Progress) limit on your team to force the team to work together and have a common objective.

Teams fates are bound together

Teams fail or succeed together.  I’m not much of a follower of professional sports, but I know from TV dramas produced about them that when different team members fates are not intertwined, teamwork breaks down and all hell breaks loose.

This seems obvious, but it is worth taking the time to consider whether your team members fates are truly bound together or not.

Want to find out quickly?  Here is a mental test you can do.  Pretend that one team member completely fails to do the work that he or she is responsible for.  Does the entire team fail at that point?  Do some team members get accolades and others get scolding?

If you honestly ask that question, you might find that indeed your team members fates are not bound together.

It is absolutely critical for a team to equally reward members at the failure or achievement of a particular goal! 

Call out the over achievers and you will destroy teamwork to create developer heroes who step outside of the team’s workload to do special tasks and take on herculean efforts in order to gain recognition.

Call out the under achievers and you will also destroy the team, creating an environment conducive to backstabbing and snitching.  One in which no one wants to be at the bottom of the totem pole, so everyone steps on everyone else’s heads and fingers to climb over them.

Don’t get me wrong here.  I am not saying to never evaluate the individual and promote or reward them.  I am just saying evaluate an individual in regards to their contribution to the team achieving the team’s goal.

Focusing on the team’s success or failures encourages team members to help each other, share knowledge and work together.

If I know that I am being evaluated based on what I get done versus what the team accomplishes, I am much less likely to spend my time helping other team members complete their work.  This isn’t to say that I don’t like helping others and collaborating, it is just common sense to prioritize what you are being evaluated on first.

Bottom line is, if you want to make a successful team, make sure that successes and failures are shared.  Sure, sometimes the whole team is going to be doing pushups for one team member’s mistakes, but it will force the team to deal with that weakness.

Teams deal with their own problems

Nothing demotivates a team more than bringing external influences into a team to control their fate.

Teams should control their own destiny.  It is much better to give a team a problem and ask the team to deal with it than to task a particular team member with a special project or responsibility.

Teamwork is all about sharing the burden.  This is closely tied to the idea of fates being bound together.

Imagine this scenario.  You have a team that has a common objective.  They have their fate bound together.  But, you come in and give one team member a special responsibility or assignment.

How does this team member view the world?

Differently than the rest of the team.  Now they have to competing goals.  The team’s current objective and their special objective.  Most people in this situation tend to ditch the team in favor of taking care of the responsibility that is squarely on their shoulders.

A better way to handle this situation is to give that “special task” to the team itself.  Let the team figure out how to get it done.  And let the completion of that objective fall squarely on the team and not on any individual.

If this is a little too scary, you might consider having a team lead.  This is a bit contrary to most Agile preachers, but in reality most teams have captains and leaders.  Take the extra step and let the team decide who the lead will be.

This brings us to another way team should deal with their own problems…

Teams should be responsible for both bringing new members into the team and kicking them off the team.  Interpret this as you will.  But teams have to deal with their own problems.  If they need more resources, they need a way to get those resources and if someone isn’t pulling their weight, they need to either deal with it or vote that team member off the island.

Applying this to Agile

I covered a pretty large amount of ground in this post, so let’s wrangle it all together and see how to effectively apply this to an Agile development environment.

Here are a few of my tips to improve your Agile team:

• Most important! Set the team on a common objective by only allowing 1 or 2 backlogs or stories to be worked on at once, regardless of the size of the team. (Imagine what would happen if a sports team had different team members focusing on different objectives.  All chaos would ensue.)  It is alright to have individual members focusing on different tasks, but those tasks should belong to a common objective.  Is this painful?  Hell yes!  Does it feel inefficient?  You bet, but the team will find a way to work together to be efficient.
• If your team is too big to have a common objective, break them up into smaller teams.  It is better to have smaller focused teams than big teams with split objectives.
• Don’t split the responsibility based on roles.  If a team shares an objective, then that team should see it all the way through.  You may have developer, QA people, and business people all on the same team, so it is difficult to break the mindset of “I do this, then this other person is responsible for the next step and so on.”  Instead the mindset should be “as a team we need to get this feature defined, developed and tested.”
• Never reward individual heroics and never call out individual failures.  This is hard, because it goes against one of my personal principles of always calling out both commendation and condemnation publically, but in a team setting it has to be the team that is called out, not the individual.  Trust the team to deal with the individuals.
• On the flipside, reward the entire team for success, call out the entire team for failure.
• Always give a task to the team, preferably in the form of a backlog or story.  Never give a task to an individual.
• Let them team decide how to handle any given responsibility.  Teams tend to find fair ways to assign and distribute work.
• Never hire unless the team approves.
• Never fire unless the team approves.
• Don’t make team members evaluate each other.  This will cause them to compete against each other and to think about merits as individual accomplishments versus team accomplishments.
• As much as possible let the team run itself.  If a team is not performing, you might need to appoint someone as a leader or facilitator of that team and give them the responsibility of improving the team’s performance, but in most cases teams figure out how to operate best when left to themselves.

I spent the last couple of months testing out running different VMs for each function of application development I was working on, and I am pretty convinced that technology has not yet reached the point where this makes sense.

I’m not saying you can’t do it.  I’m just saying it isn’t worth the cost.

My initial assumptions or theory

I first thought that this would be a good idea for several reasons:

• I might actually get a speed boost by having dedicated virtual machines for each function of application development, by not having the other cluttering programs installed on the machines that didn’t need them.
• I would be able to only use what I needed at the time, thus reducing the overhead I would incur.
• I would be able to keep each virtual machine clean to its purpose, so it would never get cluttered up and need to be repaved.
• I would be able to transport my machine to any computer just by transferring the VM.
• If I hosed a machine I could restore to a known good state from a backup.

Most of these were decent assumptions.  To give you an idea of what I was really aiming for, you have to understand my VM setup.

1. Web Development Machine
2. Mobile Development Machine
3. SQL Server Machine
4. Other / Side Project Development Machine
5. Test Dangerous Stuff Machine

The basic idea was isolate in order to make things clean and orderly and increase performance on single tasks by only using what I need.

It wasn’t a total disaster

But it wasn’t exactly what I had planned.

Things worked out pretty well, and I could have kept operating in that mode, but there were enough irritations that I decided to bite the bullet this weekend and axe the VMs for normal development work.

The biggest factor that made me end this experiment was performance.  I had my VMs running on my SSD drive, but I still felt that the performance was suffering quite a bit.

Compile times and general Visual Studio user experience just were not what I thought was tolerable for every day use.  Considering how often I compile in a day, even a small bit of a performance hit is magnified.

It was definitely nice to have development activities separated out to the point that I could just fire up certain VMs to do certain kinds of work, but it also would be a bit of a pain when I got to my machine and found the wrong VM was running or that I needed to apply windows updates to 5 machines.

Having SQL server on a separate box seemed like one of the best ideas but when I really think about it, SQL server running on your machine only really consumes memory when it is not actively being used and I have 16 gigs of RAM.  Also losing that auto complete you get for connecting to a local instance of SQL server was a fairly large price to pay.

Another big issue turned out to be tools and tooling.  Turns out many of the tools I use aren’t specific to a certain development task.  Things like text editors and R# needed to be installed on each VM.  Often I would find that the tool I wanted wasn’t installed on the VM that I was using and it became a headache keeping everything up to date and installed in the right place.

One issue I didn’t expect was the use of monitors.  Using VMWare I could expand a VM onto multiple monitors but I found that it was irritating when the VM was covering up a window on the main computer and I needed to switch back and forth between multiple machines.  In my configuration, I have 6 monitors hooked up to my PC, so for most people this probably wouldn’t be as big of an issue.

I had some big dreams of using Unity mode to allow me to use applications from different VMs and make it all feel like it was on the main PC, but that technology just isn’t quite what it needs to be to make it worth the cost of efficiency.  Right now it seems Unity mode is rather slow, error prone, and hard to use.

For me it comes down to this…

I want my workstation to be as fast as possible when I am doing disk or processor intensive tasks.  PCs are not at the point yet where we have so much CPU and disk speeds are so fast that everything is still virtually instantaneous in a VM.  If we get to that point, I won’t feel like I am making a sacrifice in a VM.

I want to be able to seamlessly switch tasks and monitors.  The reason I have 6 monitors on my computer is because I like being able to drag any window anywhere and keep right on going.  Running VMs put a bit of a stutter step into my window flinging.

I don’t want to maintain 5 different operating systems.  You don’t really notice the effort when you are just maintaining one, but when you are trying to keep 5 operating system up to date with patches, software updates and everything else, it become a pain.

Sure, I can’t have that nice clean, my computer just stepped out of a shower feeling as I open up my VM targeted exactly for the task I am about to do.  Yes, I have to put all kinds of junk into my registry and feel “icky” about it.  SQL server is always running in the background chomping up a gig or so of my RAM.

But!  I am running about as fast as I can and I have the agility I need to be more efficient.

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

It doesn’t have anything to do with my like or dislike for Scrum itself.  Plenty of teams that aren’t following Scrum have Scrum meetings.  Plenty of teams that don’t even really follow any kind of Agile process have Scrum meetings.

The reason why I am not a big fan is because most of the time they are very ineffective.

The idea behind a Scrum meeting is simple

The idea is that you have a daily stand up meeting where you basically determine how the work is progressing and if divine intervention is required to keep things progressing.

The idea is to provide a transparency where anyone can see what exactly the team is up to.

The basic format of a scrum meeting is pretty simple.  Each person on the team says three things:

1. What did I do since our last meeting
2. What am I doing until our next meeting
3. What is impeding me

On the surface these seem like sensible and valuable things for each team member to talk about, but…

So often these three things end up being BSed

No offense to anyone I’ve ever worked with.  Heck, I know I have BSed them myself.  It’s just too easy to do it.

When I say BSed here, I don’t necessarily mean blatantly lying about what you did and are going to do.  (Although that happens plenty as well.)

What I am talking about is saying what you did or what you intend to do without really having a purpose of why you are saying it.

How many times has this happened to you?  You walk into your Scrum meeting and panic hits!

Oh crapola! It’s coming around to me!  What am I going to say?

I didn’t really get much done yesterday.  I was fooling with the build, then I kept getting interrupted.  Then I was answering a bunch of emails and I ended up helping Joe with that task he was working on.  Then we started talking about how we’d like to start using this framework.

I don’t even know what I am really planning on doing today.  I’m just going to keep working on the backlog I am working on.  I don’t want to say that though, I need to say something more important.

So what do you do?  You kinda make up something important sounding, and you mix in some of what you did do and try and make it relate to things happening in the sprint.  You prattle on about stuff that no one really cares about, because you need to say something.  You’ll look like an idiot if you don’t.

This kind of thinking is contagious

Pretty soon you end up with the whole team just prattling on about things that aren’t really important to anyone else or just saying something like:

I continued to work on backlog X, I’ll continue to work on backlog X today.

Waste of time, waste of breath.  A scrum report is only valuable if it is providing usable information about the progress of an iteration and helping to bring impediments that can be resolved, to light.

I really don’t mean to sound harsh here but I want to be truthful.  We don’t benefit anything if we can’t examine what we are doing with a impartial lens and tear it apart if needed.

My solution

I am a firm believer that pointing out problems without offering solutions is just whining.  It has no value.  I wish I could say I myself was never guilty of it, but the truth is, I do my fair share of it.

With that said, I am offering a solution to this problem…

Commitment!

By happenstance, I just finished reading Clean Coder by Bob Martin.  (I’ll post a review on that book when I get a chance, but excellent read.  It hurts to read it, because it will call you to a higher level of accountability, but read it.)

Anyway, that book has a great chapter on commitment, which coincides nicely with a technique I came up with awhile ago to help make Scrum meetings much more valuable and harder to BS.

The basic idea is pretty simple.  I replace the 3 topics in a Scrum report with my sub-classed version:

1. What did I commit to doing yesterday and did I or did I not meet that commitment.  If not, why not.
2. What will I commit to getting done today.
3. What is impeding me that can be improved by bringing it up in this meeting.

Commitment can be scary

But, it is something that carries with it accountability.  See the problem with your normal Scrum meeting is that it is far too easy to BS.  There is no accountability, so you can basically make up whatever you want.

I’m sorry if this offends some people, but I am not placing myself above BSing a Scrum report.  I would venture to guess that a majority of Scrum reports given on this planet are heavily weighted to the BS side of the scale.  Once again, not necessarily saying people are lying.  I am just saying that your Scrum reports contain the value of feces from a male bovine.  Just saying…

What I try to do with my solution is add some accountability by focusing on commitment.

The idea is that it makes you think about what you are actually reasonably going to get done in the day.  It forces you to think about priorities and schedule.  It requires you to break down your work and think ahead a bit so that you can come up with something that you can commit to.

It also tends to make sure that what gets worked on is what should be getting worked on.  You don’t get to say what you worked on yesterday, you only get to talk about what you committed to that you either did or did not get done.  This prevents team members from talking about all the other non iteration related stuff they did.

It really makes you think before you jump off down some rabbit trail if you really want to say in Scrum the next morning that you didn’t do anything you committed to.

It really makes you think first before implying you are going to get something done.  Instead of saying “I’m going to try and finish up backlog X today,” you are more likely to break down some component of backlog X and say “I will commit to getting done task A in backlog X.”

The difference here is critical, because in it lies the value.  The value is that someone listening to your report can actually get an idea of the true progress of the team and of the backlogs being worked on by the team.  If every day team member just report they are working on backlog X and backlog Y, we aren’t really learning anything about if backlog X and backlog Y are really progressing.  On the other hand, if team members are required to make a commitment about what they are going to get done on backlog X and backlog Y, we start to get a real picture of how that work is progressing.

Keeping it relevant

The other thing this Scrum meeting modification does is to keep everything talked about relevant.  By talking about what was committed to and what is going to be committed to, it doesn’t leave much room for talking about other unrelated issues that are best left to another meeting.

A Scrum meeting is effective because it is a short status meeting.  When other stuff bleeds in, people start tuning out and pretty soon we are all standing around talking to the mirror.

I make a point that no one should be talking about impediments that cannot be reasonably be expected to be helped by bringing them up in the Scrum meeting.  What is the point of talking about something that can’t be fixed?

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

I basically summed the problem up as being one of software development on Agile projects being a really long set of continuous sprints on an already laid out path.

I have been thinking quite a bit about some possible solutions to this problem, and in this post I hope to provide some.

Racing the good race

One of the analogies of Agile projects is that of a race.  In Scrum we call the iteration a sprint. 

I think this analogy points clearly to the solution.

When I used to be on the track team in high school we would do sprints.  It was called interval training and it sucked.

Well, at first it didn’t.  At first you would do some of your runs, like a series of 100m dashes and feel great.  But, by the end of the session, if you ran it as hard as you could, your lungs would be burning and it wasn’t all that uncommon to puke up your guts.

People just aren’t meant to sprint over and over again without a break.  When we were pushed too hard on the track we would end up slowing down the entire time on the next training session in order to pace ourselves.

You have to find the right combination of sprints vs. rests.

Enter the cool down sprint

My most promising idea is: after a certain number of sprints the team slows down and basically takes a breather.

Not to say the team just sits around and doesn’t do anything.  Far from it.  Instead the team does what they think needs to be done and is important.

Kind of a self-reflection / sharpening the saw / building tools and making ourselves more efficient sprint.

I would guess it would be beneficial to do something like this every 5 or 6 sprints.

The idea here would be you just trust the team to work, but no one directs the team as to what that work is.

I think it is a good way to take some of the pressure off and reenergize the team to start the next set of intervals.

This next idea is stolen

I’m not sure exactly from where, but I heard of a team before that gave each member of the team some amount of “dollars” they could spend each iteration.

Product owners and managers were given “dollars” as well.  The idea was that you could spend your money on items that you wanted to put in the sprint.  The priorities were set based on where the dollars fell.

In order for something like this to work, you need to give far more dollars to product owner, support people and other people on the business side of the house, but by giving a certain amount of dollars to developers, you allow them to have some amount of say in what is worked on.

This kind of solution solves the problem by allowing developers to have more control over their destiny rather than feeling the path is laid out for them.  It gives them the ability to work on something they see valuable or even just fun without pressure or guilt because everyone understands the concept of spending their own money how they see fit.

Developer free time

This idea is stolen as well.  Actually it is stolen from Larry Silverman who is my current employer at TrackAbout.com.

This idea is pretty simple; here is a quote from our dev wiki describing it:

You may be familiar with Google’s policy that each employee is granted 20% free time to pursue their own interests. Of the projects that result, some of the best become live Google products. Google has nearly infinite resources, and with thousands of developers trying so many things, every so often one of them is going to be a winner and create a major new revenue stream. Obviously, TrackAbout does not have infinite resources. But we admire the spirit of Google’s free time policy and would like to emulate it to the degree we can.

The following Developer Free Time policy is intended to promote creative thinking and innovation by providing developers an opportunity to explore new ideas and technologies. In the spirit of Continuous Improvement, you may choose to take this time to explore an idea or improve your own skill set. Of course, we hope that some of work might prove beneficial to TrackAbout.

Most importantly, the policy is intended to allow you, the developer, to stretch your proverbial legs and have some fun!

The Details

• The program is voluntary, not mandatory. You don’t have to participate, but it’s encouraged.
• Each developer is granted 2 days per quarter to pursue their interests/projects.
• No approval is required for the project, but feel free to use Larry or other devs as a sounding board for ideas to pursue.
• Developers can take the 2 days at any time during the quarter (when inspiration strikes!), provided it doesn’t place at risk a release under development.
• Developers can team up on projects.
• After completion, the developer (or team) will submit a brief write-up to the Dev Team summarizing the work they did and what was accomplished, explored or learned.
• If the dev (or team) wants to present their work to the dev team or to the greater company, it’s encouraged but not required.

It’s about nurturing passion

After thinking about my last post for some time and what possible solutions to the problem I described, I realized it is all about nurturing passion.

Agile and Kanban like thinking tend to get us really focused on producing and producing.

Don’t get me wrong, I love to know that my team is operating efficiently and producing lots of value for the customer, but sometimes we need to really make efforts to balance that production with what really makes the developer engine run.

Sometimes we need to take a pit stop or do an oil change so we can keep that machine running smoothly and if we don’t, just like any machine, it will break down.

I realized that most people in our profession don’t get here because they love a paycheck, but because they really love writing code and exploring technology.

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

I have written in the past about some of the shortcomings of specific agile methodologies.  I’ve even suggested a new one myself.

The truth of the matter is that agile is pretty vague.

The point of this post is not to attack Agile, but rather to honestly reflect on one of the shortcomings that I have come to realize seems to be prevalent in all of the various flavors of Agile methodologies.

Driven instead of steered

I spent a good deal of time trying to think about what seemed to be missing from software development ever since I started really embarking on that Agile journey.

Ever since I’ve been wading around in Agile-land, from Scrum to XP to Kanban, there seems to have always been something missing.  Something that was there around the beginning of my career, but now seems to be somewhat lost.

I’ve noticed it in others as well.  I have noticed more and more software developers becoming more passionate about their side work than their main job, even when they are working on some pretty interesting stuff.

I don’t think it is a coincidence that so many open source projects have sprung up along with the rise in Agile development methodologies.

So what am I getting at here?  What do I believe is… missing?

I think it comes down to the difference between being driven and being steered.

Most Agile projects, regardless of which specific methodology they are adhering to, focus on the idea of having prioritized backlogs that are worked on in a specific period of time and measured.

The good thing about this kind of flow is that the customer or the business is setting the priority on the work being done.  The bad thing about this kind of flow is that the developer is being driven down a specific path of software development.

What I mean by this is that software developers in this kind of system do not have much control over their own priorities and what they work on in their day.

Backlog work is measured, non-backlog work is not measured and even frowned upon for the most part.

A certain amount of freedom is given up in exchange for efficiency.

My business mind says this is good.  My developer passion feels like it’s been stifled just a bit.

Now don’t get me wrong here.  I am not exactly complaining.  And I am not trying to derail Agile development or say we should stop doing it. 

What I am trying to do is point out something I think many people ignore in the name of progress.

I think sometimes the whole Agile mindset has a tendency to kill the parts of developers that got them interested in software development in the first place.  The idea of exploration and creativity and feeling they have the freedom to explore areas of code and technologies and projects that interest them the most.

This is why I believe so many developers have taken up side projects which do not even offer up a paycheck.

Think about it… Open source developers must be getting something they are missing in their 9 to 5 jobs or why would they be so passionate about it and be willing to expend so many hours for no monetary compensation?

Agile puts a gun to your head

Did I grab your attention with that one?

I mean it part in jest of course, but there is certainly some truth to it as well.

When I am working on an Agile project, I know that time is ticking away on the backlog I am working on and that it is going to need to be done in a relatively short timeframe.

I certainly have the tendency to look at some area of code that I would perhaps like to delve deeper into and refactor, but instead pass it by in the interest of completing the work at hand.  This can be argued as a good or bad thing, but I’m not interested in debating it at this point.

What I am interested in saying is that there is certainly an amount of pressure that is applied just by having work broken down and measured.  It is a constant pressure that doesn’t let up week after week. 

If you complete all the items in a sprint, you’ll have new ones the next sprint.

If you complete a backlog, you’ll need to grab the next one on the pile.

Before Agile came strong onto the scene, many software projects were still doing iterative development of sorts, but not really calling it Agile.

You couldn’t really avoid it, since most work is maintenance work, and it is pretty hard to not iteratively do maintenance work.

The key difference is that we were a little less efficient, but a little more free.

We tended to be steered into areas of work, but given the freedom to work on things as we saw fit.  Sure there were deadlines and certain goals that needed to be accomplished in certain timeframes, but day to day work was more about what do I feel like getting done today rather than what must I get done today.

I don’t want to paint this picture by indicating that we should get to our desks in the morning and work on whatever we want to work on each day, (I think that would be the other extreme of what I am suggesting,) but I do want to point out that there is a distinct difference in being driven and being steered and that Agile methodologies tend to drive developers down a specific path versus steering them in a general direction.

Why this problem is important

Now I could be going out on a limb here with this post.  Maybe I am the only one who thinks this problem exists.

Maybe many of the newer, younger developers don’t remember the time when you came to your desk and each day was a brand new path to explore instead of having the path laid out for you ahead of time.

But, I suspect that deep down many developers, like Neo in the Matrix, feel that there is just something that doesn’t seem right and that this might be it.

If that is the case, I think solving this problem is very important, not just to software developers, but to the companies they work for.

If we can solve this problem I think we can achieve benefits like these:

• Higher retention rates. I have been on many Agile projects in the last decade and the retention rates have been pretty bad.  I don’t have any data to back this up, but I suspect they run much higher than non-Agile projects.
• Improved architectures.  One related problem I have noticed on Agile projects is that architectures tend to get messy, because no one ever has time to focus on the architecture.  There are many things that developers would do that would improve architecture of projects that just would never make it onto a backlog.
• Improved efficiency through synergy.  I strongly believe there is a strong synergy component between the work that a person wants to work on and how much they accomplish.  Isn’t it amazing how fast skunk works projects end up being built?  Again I don’t have statistics to back this up, but in my experience skunk works code gets written 5-10 times faster than regular project code.  I think it has to do with this synergistic relationship between passion and work output.
• Increased adoption of Agile.  Some “old timer” run software development shops really resist the idea of doing Agile development.  Probably a large amount of this resistance comes from not wanting to give up their freedom.  Agile as it is now tends to trade freedom for dependability.
• Decreased half-ass Agile adoptions.  So many software development shops claim to be practicing Scrum or another Agile methodology, but they are not really doing it.  I suspect some of the reason for this is related to the constant pressure they know really following an Agile methodology might impose on all members of their organization.

I think I have pretty clearly highlighted the problem in this post, and in my next post, I hope to come up with some possible solutions to address this problem.

What do you think?  Am I totally off-base here?  Does what I say ring strangely familiar?

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out ElegantCode.comwhere I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

This is definitely one of the more challenging tasks you will face, and perhaps the most critical.

All of your efforts will be in vain if you can’t get to the point of creating new BATs for new backlogs that are finished, because if you don’t you’ll still be doing manual testing, and you’ll be playing catch-up all the time.

Start off small

The goal here is to eventually get to the point where every single backlog requires BATs to be written and passing in order for that backlog to be called “done.”

If you try to drop this lofty goal onto a team in one iteration, you are likely to get beaten and slapped with much vigor.

Instead, start with a simple goal of requiring that at least one backlog have BATs for all of the functionality of that backlog.  Start with this small goal, but enforce it with rigor.

By starting out small like this, you will get the team accustomed to the idea of building automated tests at the same time as building the functionality and you will give them a chance to have a small success without sacrificing much velocity.

The goal at this point should be to get people excited about writing BATs along with the backlog.

It’s actually quite fun

I have found that developers tend to really enjoy writing automated tests.  Make sure you let everyone who wants to share in the fun, not just QA and not just developers.

It is really very exciting to see the “magic” of an automated test clicking through buttons on your web page and doing all kinds of things that seemed so tedious before.

I have found that this is so much fun, that you might have the problem of developers not wanting to work on the real backlog, but rather to write the automated tests for it.

This is good, you really want to foster this attitude, it will greatly help when you finally…

Drop the bomb!

What bomb?  The bomb that says on it:

All backlogs will require BATs as acceptance criteria.  It’s not done unless it has automated tests that prove its functionality.

You should be getting to this point within 3 to 4 iterations on average.  Don’t go too fast to get here, but don’t go too slowly either.

I know this seems like an impossible task, but it really isn’t.  I’ve been here before and had much success with it on several different teams.  The key is making sure everyone is comfortable with writing automated tests and understands the value.

Whatever you do though, once you draw this line in the sand, Do not back down! I mean it.

You might start getting death threats.  People might directly laugh in your face all the while spitting skittle colored loogies in your coffee, daring you to “taste the rainbow,” but you have to hold your ground.

It will be worth it in the long run, because you will have built an awesome safety net for proving the functionality of your software over time.

It’s not all rainbows and butterflies

You are going to sometimes run into scenarios where automation will be too high of a price for a particular backlog.

This will especially happen in cases where you have some kind of process that spans multiple domains or software systems.

When you run into those situations, give it a solid effort to at least automate as much as possible, but don’t die on the hill and kill the whole automation project.  We have to be pragmatic or risk having our credibility called into question.

Don’t expect everyone to embrace writing BATs overnight, or to even be good at it.  When you first introduce this to the team, you are going to have a learning curve and your automation framework is still going to have some pretty big holes in it.

But, that is OK.  Over time you will build out that framework and produce some fierce battled scarred veterans able to automate web pages while scowling at you from afar, just don’t lose hope and keep beating that drum!

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple. Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week. Also, you can follow me on twitter here.

Spend a majority of your effort, all the time you would have spent writing unit tests, instead writing what I will call blackbox automated tests or BATs.

In this post, I am going to outline what I think it takes for a team to go from BAT-less to BAT-man (or BAT-woman) without going batty.

I want to take a practical approach to getting from 0 blackbox automated tests to building a sustainable process of developing BATs as a integral component of the acceptance criteria for calling a backlog item “done.”

Let me paint a picture for you

I want to paint the picture for you of the end goal that we are seeking to achieve, from start to finish, of a single backlog.

1. A backlog is selected for work.
2. Developers, QA and product owner work together to clearly define the acceptance criteria which will enable that backlog to be signed off or called “done.”
3. QA and developers work together to transform the acceptance criteria into a high level description of BATs which will be created to verify each aspect of the acceptance criteria.
4. Developer begins coding with this high level set of BATs in mind (BATs drive the development), QA begins developing BATs.
5. As developer completes enough functionality to pass a BAT and not break existing BATs that functionality is checked into trunk.
6. Each check-in causes the entire battery of BATs to be kicked off split across multiple VMs which run 100s of hours worth of automated tests in minutes.
7. Once all BATs for a backlog are passing, that backlog is done and those BATs become part of the battery of tests which are run with each continuous integration build.

We should keep this simple process in mind and strive to reach it, knowing that when we are able to reach this process we will be able to have a HUGE amount of confidence in the functionality and non-regression of our system.

Put aside for a second your notion that unit testing is the correct thing to do.  Put aside the idea that blackbox automated testing is too hard and too fragile, and imagine the world of software development that flows like the picture I painted above.

Now listen up, because this part is important…

Before you put forth the effort to write one more unit test, before you dip your test double pen in the ink well of “mock,” make sure you are first taking efforts to develop a process to create BATs.

The value proposition

We really need to take the effort to understand the value proposition being presented here.

I’m going to use some real fake numbers to try and really convey this important point that I think is likely to be overlooked.

Consider first the amount of time that you spend doing two things:

1. Creating infrastructure to allow your code to be unit tested in isolation.  (Dependency injection, interfaces, etc.)
2. Time spent writing unit tests.

Now, before we go any further, I just want to reiterate.  I am not advocating completely abolishing the writing of unit tests.  See my original conclusions post, or my post on unit testing without mocks for a better understanding of what I am advocating in regards to unit testing.

So back to my point.  Think about all the time it takes to do this.  To properly isolate and unit test code most developers would probably estimate that for every hour worth of work writing the code there is about another hour to two hours worth of unit testing and unit test prep time.

There is a reason why unit tests take longer to write than the code they test; it takes MORE lines of code to test a line of code in isolation.

BATs developed using a good automation framework are just the opposite.  While unit test code might have a ratio of 4 lines of unit test code to 1 line of production code or a 4:1 ratio, BAT code has a completely opposite ratio.  It is very likely that 1 line of BAT code can test 20 lines or more of production code, a 1:20 ratio.  (Where do I get these numbers?  Looking at some of my real production code being tested with unit tests and BATs.)

Even if unit tests and BATs were equally effective in preventing the regression of your software system, and equally effective at providing an absolute assurance of meeting the acceptance criteria (which I would argue that BATs are much more effective at both), you can see easily that you are going to get a much higher return on your investment by investing your time in BATs vs. investing your time in unit tests.

I don’t make these statements lightly or in a theoretical tone.  I have real world experience with successfully implementing automation frameworks for writing BATs that reinforce these conclusions.

How to get there

If I have done my job, you are now at least convinced that getting to the point of having BATs for all backlogs is a good goal, but if you are like most people I talk to, you are skeptical of the costs and feasibility of doing it.

What you need is a good guide.  A step by step guide on how to do it.

I am going to conclude the Back to Basics posts and segue into a new series with the goal of providing a step by step guide to getting to fully automated blackbox testing.

Let’s take a look at the steps that I will cover in the next series of posts.

1. Hiring an automation lead – it’s important to either find someone who’s done this before, or a developer resource to this role.
2. Deciding on the underlying browser driver or browser automation tool – assuming web apps here, there are several to choose from, WatiX, Selenium and more.
3. Designing an automation framework – building a framework tailored specifically to your application under test will create an effective domain specific language for testing your application.
4. Creating your first smoke tests – building some smoke tests will help you prove and build out your framework and provide you the highest value tests first.
5. Adding smoke tests to your build – with smoke tests being run as part of the build, you can immediately start seeing value.
6. Adding BATs to your acceptance criteria – we need to start out slowly here, but eventually make all backlogs require BATs for each acceptance criterion.
7. Scaling out – as you start to get a large amount of BATs you’ll need to figure out a way to virtualize more servers and parallelize the test runs to be able to run all the BATs in a short amount of time.
8. Building a true DSL – once you get this far, it may be time to start thinking about creating a domain specific language that even business analysts can write tests with.

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.