I spent a good time last night troubleshooting a “works on my machine” problem.
It takes pain to learn something; this pain perhaps was good. It reminded me of a concept that is really important in your software development infrastructure.
I have three golden rules of development environments and deployment:
- I should be able to run a software build locally that is the exact same build that will be run on the continuous integration server.
- Only bits that were built by the continuous integration server are deployed and the same bits are deployed to each environment.
- Differences in configuration in environments only exist in one place and on that machine.
You don’t have to follow these rules, but if you don’t, most likely you will experience some pain at some point.
If you are building a new deployment system, or you are revamping an old one, keeping these 3 rules in mind can save you a large amount of headache down the road.
I think it is worth talking about each rule and why I think it is important.
Rule 1: Local Build = Build Server Build
If you want your continuous integration environment to be successful, it needs to appropriately report failures and successes.
If your build server reports failures that are false, no one will believe the build server and you will be troubleshooting problems that are build configuration related instead of actual software problems. Troubleshooting these kinds of problems provides absolutely no business value. It is just a time sink.
If you report false successes when you deploy the code to another environment, you will discover the issue, will be wasting time deploying broken code, and you will have a long feedback loop for fixing it.
As a developer, I should be able to run the exact same command the build server will run when I check in my code. I would even recommend setting up a local version of the continuous integration server your company is using.
By being able to be confident that a software build will not fail on the build server or during deployment if it doesn’t fail when running it locally, you will prevent yourself from ever troubleshooting a false build failure. (The deployment still could fail, and the application could still exhibit different behavior on different environments, but at least you will know that you are building the exact same bits using the exact same process.)
Rule 2: Software Build Server Bits = Only Deployed Bits
Build it once and deploy those bits everywhere. Why?
Because it is a waste of time to build what should be the exact same bits more than once.
Because the only way to be sure the exact same code gets deployed to each environment (dev, test, staging, production, etc.), is to make sure that the exact same bits are deployed in each environment.
The exact same SVN number is not good enough, because an application is more than source code. You can build the exact same source code on two different machines and get a totally different application. No, I’m not loony. This is a true statement. Different libraries on a machine could produce different binaries. A different compiler could produce different binaries.
Don’t take a risk. If you want to be sure that code you tested in QA will work in production exactly the same way, make sure it is the exact same code.
This means you can’t package your configuration with the deployment package. Yes, I know you always have done it that way. Yes, I know it is painful to figure out another way, but the time it will save you by never having to question the bits of a deployment ever again will be worth it.
Rule 3: Environment Configuration Resides in Environment
Obeying this rule will save you a huge amount of grief.
Think about it.
If the only thing different in each environment is in one place, in one file in that environment, how easy will it be to tell what is different?
I know there are a lot of fancy schemes for adding configuration to the deployment package based on what environment the deployment is going to. I have written at least 3 of those systems myself.
But, they always fail somewhere down the line and you spend hours tracing through them to figure out what went wrong and ask yourself “how the heck did this work again?”
By making the configuration for an environment live in the environment and in one place, you take the responsibility of managing the configuration away from the software build process and put it in one known place.
There is an interesting constraint in release management that is pretty often ignored.
I think it is worth talking about because not too many people on Agile projects really realize the implications of this simple constraint.
You can either release based on a number of features or a date in time, but not both.
What do I mean by this?
Simple. Let’s say that you are doing a software release in an Agile project. (In any project really, this constraint happens to be one of the reasons waterfall projects often fail… death-march anyone?) How do you decide when to release? What is your release cadence so to speak?
Think about this for a second. If you answered “on this date, when these feature are done,” you are wrong. You can’t choose both.
I am saying that you can either say, “I will release when X number of features are complete,” or you can say, “I will release every 2 weeks,” but you can’t say both.
A running metaphor
Let us translate this problem to a different paradigm for a second to see if it becomes more clear. Many people right off will understand this concept, but others will be confused.
Think about running.
Have you ever done a running work-out or did some running training? Ran a race perhaps? Biking and similar sports apply here.
Think about the training programs or workouts you have done.
- You might have a workout that is “run for 20 minutes.”
- You might have a workout that is “run for 3.1 miles.”
Have you ever had a running program that instructed you to run for 3 miles and do it in exactly 25 minutes? Pretty doubtful.
I’m not saying it can’t be done. But, it would be pretty difficult to run for exactly 25 minutes and travel exactly 3 miles. Most likely you would miss one of those marks.
The same with software development. Features or time, but not both. Both is kind of silly.
But we do it successfully right now
No you don’t. It is called padding.
Back to the running metaphor. So there is a way to achieve both time and distance. If you really think about it hard, you’ll figure it out. If someone was going to pay you a huge amount of money to cross a finish line at a particular distance within 5 seconds of a set time, how would you do it?
I’ll give you a second to think about it.
Well, the strategy I would employ would be to run really fast as hard as I could and come right up to the finish line, but not cross it. Then I would sit and wait for the remaining time to tick down and cross the finish line, right on the money.
In software development, we call this “padding.” Padding is a bad word. Padding will take a productive development team and turn them into lazy sloths. Once you start padding it is extremely hard to stop. It becomes the norm.
That is why I say that if you think you are releasing software on a set date with set features, you are wrong. What you are doing is padding. Stop it! Stop wasting your resources and choose one. Features or time?