I worked on an interesting problem this week that might have looked like I was running around in circles if you just looked at my SVN commits.
The problem, and the eventual solution, reminded me of an important part of software development—of building anything really.
Sometimes you must tear it down!
No really, sometimes you build a structure only to tear it down the very next day.
It’s not a mistake. It is intentional and productive and if you are not doing it, you very well might be making a real mistake.
That is highly illogical
Imagine for a moment that you are tasked with the job of repairing the outside walls of a 2 story building.
There are of course many ways you could go about doing something like this.
Use a ladder and just reach the part of the wall the ladder allows you to. Then move that ladder, repeating the process as needed to complete the repair to the entire wall.
Try to lower yourself down from different windows to reach as much of the wall as possible.
Tear down the entire building and rebuild the building and walls.
I am sure there are plenty of other methods besides what I listed here.
Yet, a very simple approach would be to build a scaffolding.
A scaffolding is basically a temporary construction used just to help repair or initially build a building which is built for the very purpose of eventually being torn down.
Software is different, we don’t contend with physics!
You are absolutely right!
We contend with a much more powerful force…
Conceptually anything you can create in software could be created without any kind of software scaffolding. Unfortunately though, the complexities of the logic of a system and our abilities as humans to only contain so much of it in our brains impose a very real limitation on our ability to even see what the final structure of a metaphysical thing like software should be.
So what am I saying here then?
I’m just saying that it sometimes helps to remember that you can temporarily change some code or design in a way that you know will not be permanent to get you to a place in the codebase where your head is above the clouds and you can look down and see the problem a little better.
Just like there are physical ways to repair a wall on a 2 story building that don’t involve wasting any materials by building something that will be torn down, there are ways to do build software without ever building scaffoldings, but in either case it is not the most efficient use of time or effort.
Don’t be afraid to take a blind hop of faith
Source control is awesome!
Source control lets us make changes to a code base, track those changes and revert them if needed.
Working on a code base without source control is like crawling into a small crevice in an underground cave that has just enough room to fit your shoulders—you are pretty sure you can go forward, but you don’t know if you can crawl back out.
But, with source control, it is like walking around a wide open cave charting your trail on a map as you go. You can get back to where you want to any time.
So as long as you have source control on your side, and especially if you are using a distributed source control where you can make local commits all day long, you don’t have to be afraid to step out on a software ledge and see where things go.
Oftentimes I will encounter some problem in code that I know needs to be fixed and I know what is wrong with it, but I just don’t know how to fix it.
When I encounter a problem like this, I will often have an idea of at least one step I could take that should take me in the direction I believe the code needs to go.
A real example
To give you a real example, I was working recently on some code that involved creating a lot for a product. (Think about the lot number you might see on a box of Aspirin that identifies what batch of ingredients it came from and what machine produced it when.)
Up to the point in which I had been working with my teammate on refactoring this code, there had only been only one way to produce the lots.
We were adding some code that would add an additional way to create the lots and the labels for those lots. We also knew that there would be more additional ways in the future that would need to be added.
Well, we knew we wanted to have a generic way of specifying how lots should be created and labeled, but we didn’t know what the code that would do that would look like.
We could have rewritten the entire lot handling code and made it generic, but it was quite complex code and that would be equivalent to tearing down a building to fix a wall.
It was apparent there were some parts of the existing code that were specific to the way that the current lots were being produced, so we knew those parts of the code had to be changed.
So what did we do?
We started with what we knew needed to be changed and what we thought would move us in the direction we wanted to go, and we built some structures to allow us to refactor those parts of the code, knowing that we would probably end up deleting those structures.
The scaffolds that we created allowed us to have a midway point in the long journey across the Pacific ocean of complexity in our refactor trip.
The point here is that we had to take a bit of hop blindly in a direction we thought things needed to go and part of that hop involved creating some scaffolding type structures to get that code to a place where it still worked and we could examine it again for the next refinement.
The final refinements ended up deleting those scaffolds and replacing them with a more elegant solution, but it is doubtful we would have been able to see the path to the elegant solution without building them in the first place.
Show me the code!?
You may wonder why I am talking about code in such an abstract way instead of just showing you a nice little “scaffold pattern” or a real code example.
I’m not going to show the code, because it isn’t relevant to my point and it is so situational that it would detract from what I am trying to say here.
The point is not what our problem or solution was, but how we got there.
There isn’t a “scaffolding” pattern that you can just apply to your code, rather it is a tool that you can use and shouldn’t be afraid to use in order to move your code forward to a better design. (To its happy place.)
In my previous post, I talked about the idea of having a simple branching strategy and why I prefer one where everyone works off the same branch.
In this post I will show you how to create what I believe is the most simple and effective branching strategy.
Take a look at this diagram of a sample project’s code lines:
Walking through it
The idea here is very simple. Let’s walk through a development cycle together:
- Development starts. Everyone works off of trunk. Code is frequently checked into trunk, many developers checking in code 3-4 times a day, as they complete small quality sections of development.
- The continuous build server is continuously building and checking the quality of the code every single time code is checked in. Any integration problems are immediately fixed.
- Enough features are done to create a release. Trunk is tagged for release and a release 1 branch is created representing the currently release production code.
- Developers continue to work on trunk not being interrupted by the release.
- A customer finds a high priority issue in Release 1.
- A Rel 1 Hot Fix branch is created, branched off of Release 1 to fix the high priority issue. It turns out that a good fix will take some time. Team decides the best course of action is to apply a temporary fix for now.
- Rel 1 Hot Fix is done and merged back into Release 1 branch. Release 1 is re-deployed to production.
- In the meantime another emergency problem shows up that must be fixed before the next release. Rel 1 Hot Fix 2 branch is created.
- The bug fix for Rel 1 Hot Fix 2 is a good fix which we want in all future releases. Rel 1 Hot Fix 2 branch is merged back to Release 1 branch, and merged back to trunk. Release 1 is redeployed.
- In the meantime work has been going on on trunk, team is ready for Release 2.
- Release 2 branch is created…
Breaking it down
I gave a pretty detailed walk-through for a very simple set of actual steps. But, I hope you can see how simple this process really is.
The basic idea here is that we are trying to decouple releases from development as much as possible. The team is always going to keep chugging along, building new features and enhancing the code base. When we decide we have enough features for a release, we simply branch off of trunk and create the release branch.
We can even do some testing on the release branch before we go to production if we need to without impacting future development.
The release branch code-lines never come back to trunk. They don’t need to, they only exist so that we can have the exact production code and make modifications to it as hot-fixes if we need to.
We branch hot-fixes off of the release branch so that we can work on them independently, because not all hot-fixes go back to the main code-line. We can make a hot-fix just for the current release, or we can merge it back to trunk to make it a permanent fix.
That is all there is to it. This kind of branching strategy almost completely eliminates merges. The only merge you ever do is small merges for hot-fixes.
Your branching strategy does not have to be complicated. A simple strategy like this can fit almost any software development shop.
Frequently disputed points
Almost immediately when I introduce this simple system someone says:
What about half-completed features? I don’t want to release half-completed features. Using this strategy with everyone working off trunk, you will always have half-completed features.
So what? How many times does a half-completed feature cause a potential problem in the system? If the code is quality and incrementally developed, it should not impact the rest of the system. If you are adding a new feature, usually the last thing you do is actually hook-up the UI to it. It won’t hurt anything to have its back-end code released without any way to get to it.
Continuous integration, (especially running automated functional tests), trains you to always keep the system releasable with every commit of new code. It really isn’t hard to do this, you just have to think about it a little bit.
If worse comes to worst and you have a half-finished feature that makes the code releasable, you can always pull out that code on the release branch. (Although I would highly recommend that you try and find a way to build the feature incrementally instead.)
If you know you’re going to do something that will disrupt everything, like redesigning the UI, or drastically changing the architecture, then go ahead and create a separate branch for that work. That should be a rare event though.
I need to be able to develop the features in isolation. If everyone is working off of trunk, I can’t tell if what I did broke something or if it is someone else’s code. I am impacted by someone else breaking things.
Good, that is some pain you should feel. It hurts a lot less when you’re continuously integrating vs. working on something for a week, merging your feature and finding that everything is broken.
It is like eating a meal. All the food is going to end up in the same place anyway. Don’t worry about mixing your mashed potatoes with your applesauce.
If something someone else is doing is going to break your stuff, better to fail fast, then to fail later. Let’s integrate as soon as possible and fix the issue rather than waiting until we both think we are done.
Besides that, it is good to learn to always check in clean code. When you break other people and they shoot you with Nerf guns and make you wear a chicken head, you are taught to test your code locally before you check it in.
How to be successful
How can you be successful at this simple strategy?
- Make sure you have a continuous integration server up and running and doing everything it should be doing.
- When you work on code, find ways to break it up into small incremental steps of development which never break the system. Hook up the UI last.
- Always think that every time you check in code, it should be code you are comfortable to release.
- Check in code at least once a day, preferably as soon as you make any incremental progress.
- Test, test, test. Test locally, unit test, test driven development, automated functional tests. Have ways to be confident the system never moves backward in functionality.
- So important I’ll say it twice. Automated functional tests. If you don’t know how to do this, read this.
- Release frequently instead of hot-fixing. If you never hot-fix you will never have to merge. If you never have to merge, you will live a longer, less-stressed life.
- Don’t go back and clean up code later. Write it right the first time. Check it in right the first time.
Hopefully that helps you to simplify your branching process. Feel free to email me or post here if you have any questions, or are skeptical that this could work.
Source control management has always been one of those sticky topics which always causes many questions. Many veteran programmers are baffled by the in-and-outs of branching and merging. And for good reason; it is a difficult topic.
I’ve been around in many different organizations. I’ve been the person who was told what the SCM strategy was, and I have been the person who designed it. I’ve seen it done just about every way possible, and after all that, do you know what I think? (I’ll give you a hint, it’s the name of this blog)
Keep it simple. Working directly off the trunk is by far the best approach in my opinion.
It almost sounds like heresy when I actually type it onto my screen, but if you’ll bear with me for a moment, not only will I show you why I think this is essential for an Agile process, but I’ll show you how to make it work.
Continuous integration is the root
As a software development community as a whole, I think most of us can agree that CI is good and provides real value.
What is Continuous Integration?
Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily – leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible.
So the idea here is to integrate as soon as possible, the faster the better. If something is going to break, we want it to break sooner rather than later.
The best way for this to happen is that whenever I check in my code it is instantly integrated with everyone else’s code and we build and test it.
This is a change from the old way of doing things, in which I am working on my code in isolation for some period of time and periodically I show up with a bunch of code and merge it all together.
The only practical way to have continuous integration is to have everyone working off the same branch. It doesn’t have to be trunk, but it has to be the same branch. Sure, there are some ways to have people work off different branches and automatically merge them when they commit as a trial integration, but that is a large amount of effort, and it can’t handle conflicts.
One of the “big dogs” in Agile is collaboration. Collaboration is pretty important. Working on separate branches tends to go against this idea. I know you can still collaborate when you have separate branches, but it is more likely that you won’t.
In general I have found that software developers in particular are the kinds of people who will go off and do their own thing if given the chance. It is not a bad thing, but it means that we need to set up developers to collaborate. If your process isn’t explicitly fostering collaboration, it probably won’t happen (at least not as much as you would like it to.)
If we are all working on the same branch we have to communicate. We have to work together to accomplish a goal. I can’t just work in isolation, because what I am doing affects you, and what you’re doing affects me. We have to talk about it.
I’m more likely to watch your code come in and say “hey, where is your unit test, bub?” You’re more likely to watch my code come in and say, “is there a reason you’re not using superWidgetX to solve that problem?”
If I check in some junk and break the build, everyone is impacted. This might seem like a bad thing, but it fosters a team mentality instead of an individual mentality. Think about boot camp (or at least what you have seen on TV.) What happens when one guy screws up? EVERYONE does push-ups. Why? To build a team, a team mentality.
I have found in Agile processes like Scrum, the best quality work is done when developers are paired up working on a backlog item. This isn’t impossible to do with individual developer branches or feature branches, but it is much harder and less likely to happen.
Keeping it simple
One of the central themes of my blog… of my life… is keeping it simple. Simple is good, it makes everything easier.
I am notorious for walking into an organization and ripping out all the ridiculous process that provides no value and putting in very simple and lean process in its place. It is in my nature.
If you have ever spent some time in what I like to call “merge hell”, you’ll understand what I am about to talk about.
I have been working with source control for a very long time. I have been the “merge master,” merging huge numbers of branches together on large projects. I have worked on developer branches, feature branches, release branches and everything in-between, and I still can’t get it right!
Don’t get me wrong. I know the theory behind it, I know how to do it. But merging can be so complicated at times that the chance of making a mistake is very high.
You really have to ask yourself a question here: “Is all the overhead you are incurring from doing your complicated branching and merging strategy resulting in a real value that does not exist over a more simple strategy?”
My general rule of thumb is, never branch unless you have to.
A simple Agile branching strategy
In my next post, I will show you what I think is the most simple and effective branching strategy. A strategy I have effectively used in the past and have developed over time. I’ll sum it up briefly here.
- Everyone works off of trunk.
- Branch when you release code.
- Branch off a release when you need to create a bug fix for already released code.
- Branch for prototypes.
That’s it, stay tuned for next time when I talk about how to make this simple branching strategy practically work in your Agile environment.
Merging is source control Kung-Fu.
I’ve seen many people get taken to the mat when trying to merge code. Today, I’m going to give you a simple technique that can help save you the embarrassment of your favorite source control program kicking you right in the head.
Bring the plate to the food
Often as a kid the table would be set and dinner would be ready. I would try and take some food from the kitchen over to my plate on the table. (Grab the hamburger and carry it over to the plate.)
My dad would often tell me, “Bring the plate to the food.”
Which would mean that I would have to take the plate from the table. Bring it to the kitchen. Put the food on the plate. Bring the plate back to the table. Oh, what a hassle.
Less food ended up on the floor that way. Now it seems obvious to me. But, back then it didn’t.
So it is with merging
It is exactly the same way with merging.
Wise-man once say:
If you want to merge code to a location, you must first merge code from the location
Anytime you are about to merge code to some branch, always merge code from that branch first.
Let’s say that you created a branch off of your trunk. You started working in that branch and you are done with whatever you are doing there. You are ready to merge code back up to trunk.
- First merge trunk to your branch
- Resolve any conflicts
- Test on your branch
- Then merge your branch (that has the trunk changes already) into trunk
Why can’t I just merge to the destination, why merge in first?
It may seem like a bunch of overhead, but if you’ve ever merged to trunk or a release branch and broke it for everyone, then scrambled to try and fix it, you’ll probably see the benefit in making sure that all merges to release branches or trunk are trivial.
A trivial merge is a merge that can be automatically done by your source control. It doesn’t require human interaction.
If you merge in, and then merge out, the merge out will always be a trivial code merge. So in reality you’re not really adding any overhead at all. You are just handling the possibly difficult code merge on your branch as opposed to the trunk or release branch.
Another important reason is that you want to be able to test your changes with the other changes that have happened in the system since you branched off. Most of the time other changes will be happening at the same time you are making changes.
The only way to know what the interactions will be is to test them. The best place to test them is on your local branch so that you don’t interfere with everyone else.
My dev cave
Without further adieu, here are the pictures of my dev cave I set up for my new job.