It is time to pick the browser automation tool you are going to use for driving your automation efforts.

Before we can build an automation framework, we need a basic set of tools to build it on.  We want our automation framework to be tailored to our application under test specifically.  Our automation framework will act like a Domain Specific Language (DSL) for testing our application, but we need to build that automation framework on top of something.

What is a browser automation tool?

It is essentially what we will use to drive the web browser.

We need to pick a tool or framework that will easily allow us to interact with the web browser so we don’t have to write that code ourselves.

We will need to be able to click buttons, enter text, select drop downs, and check values on web pages.

Some of the things we will want to consider when choosing a browser automation tool include:

• Language we can use with the driver
• Browsers it supports
• How we interact with the browser (do we use XPath, JQuery, or some other mechanism)
• If we can access and manipulate everything we need to be able to in the browser
• Speed of execution
• Ability to execute in parallel
• Support and future development

Let’s take a quick look at each of these considerations.

What language can we use?

This is an important consideration because you will want to be able to utilize your other programming resources to help with building an automation framework and to eventually create automation tests.

If you pick a browser driver that supports a language your regular programmers don’t like, or don’t know, you may not end up with their support, and you are definitely going to want their support.

What browsers does it support?

This is a tricky topic to discuss.  It might seem that a browser automation tool would need to support all of the browsers your application supports, but that is not really true.

You really have to think about what your goal is with your BATs.  I could certainly devote a whole blog post to this topic, but I will try and summarize my main point here.

BATs should be designed to test functionality of the application, not to test compatibility.  (At least most of your BATs should be, although you might have some that are specifically for compatibility.)  With that goal in mind, a browser driver supporting each and every browser your application supports is not really necessary.

You do need to consider that the browser automation tool should support at least one of your supported browsers, preferably the main one.

How do we interact with the browser?

There are several different APIs which are used to communicate with the web browser that are employed by various different browser drivers.

Some tools have APIs that are very low level and allow you to very easily manipulate objects in the web browser while others are at a much higher level and rely more on your understanding of the brower’s DOM.

For example, one API might expose elements on a web page by their actual element name and let you interact with them directly.

browser.Button(Find.ByName(“submitButton”)).Click();

vs

instance.click(“submitButton”);

In the first example, the API is aware of element types like buttons.  In the second example, the API is more generic and will send a “click” to an element with the ID value you specify.

You’ll have to decide how important the ease of use is to your project at this level.  There are some tradeoffs to consider here.

The lower level the API is, the easier it will be to use, but the more dependent you will be on that API and the more language specific that API will be.

The higher level the API is, the harder it will be to use, but it will free you from depending so much on the API, because you will be interacting more directly with the browser’s DOM.

I prefer lower level APIs because I find that it is much easier to write the framework code on top of these.

You will also want to consider here the skill levels of the programmers who will be creating the framework.  If your API is lower level, native language skills are more important.  If it is higher level, DOM and HTML, and perhaps Javascript skills, are more important.

Can we access what we need?

If your application makes use of Javascript pop-up dialogs and your browser automation tool doesn’t give you a way to interact with them, you don’t want to find this out when you have already invested significantly into your automation framework.

Different automation frameworks support different features of browser; not all of them will allow you to interact with every single browser feature that exists.  As new features are introduced, you could also get stuck with a tool that is no longer being maintained and doesn’t add support for the new features.

You will want to consider things like how much your application uses AJAX or JQuery.  You will want to select a browser automation tool which would make it easy to interact with AJAX calls if your application relies on them heavily.

Speed of execution

Not all browser automation tools are the same in execution speed either.

This may not be important to you, depending on how you address the issue of concurrency, but you should at least consider that if you have a large number of automated tests, a small difference in execution speed can result in a large difference in total time to run the tests.

The faster the automated tests can be run, the more valuable they are.  I won’t get into the details here but there are many reason why this is true.

Also influencing the speed in which you can run your tests will be if you automation tool choice requires you to put pauses into the test to wait for the browser instead of responding to events that occur in the browser.

Ability to execute in parallel

This consideration will depend greatly on the volume of and speed in which your tests can execute.

At some time (sooner than you think) you will likely get to the point where you can no longer run all of your automated tests in 24 hours.  Perhaps even before this point, you will want to consider running your BATs in parallel to reduce the total time to execute the tests.

Some automation tools have built-in support and others will require you to build your own way to do this.  It is good to at least have an idea of how you are going to achieve parallel execution when considering what browser automation tool to use.

Support and future development

We are in a pretty rapid pace of browser development.  Many things in the web browser world are changing much more rapidly than ever before.  Tools to automate the browser must also change or you could end up in a very bad place.

If your application is going to take advantage of the latest browsers and browser features, you should make sure the automation tool you choose is in active development.

There is nothing worse than investing into an open-source project and then having that project die, resulting in you having to rip it out to replace it with another library.

You can always design your automation framework to be abstracted away as much as possible from the underlying browser driver, but that will be extra work, so consider this point carefully.

Name names sir!

Nope, I’m not going to do it.

I’m not going to tell you to use Watin or Watij or Selenium or WebAii or any other browser driver.  I don’t want to put the focus on the tool, since the real focus will be building the automation framework on top of the tool you use to drive the browser.

I would suggest that you try writing a few simple tests in each of the major browser drivers so that you can get a good feel of what the API is like and how it will work with your application.

I will say that I have used most of the major choices out there and there really is no clear winner in my mind.  It really is going to depend on what your environment is like and what kind of application you are testing.

I would also recommend picking a browser driver and sticking to it.  I’ve tried in the past to abstract away the browser driver from the automation framework and while it is possible, it can become quite messy and add quite a bit of overhead to your project.

My only other hints would be to not put too much emphasis on supporting multiple browsers or on using recording tools.  Neither of these things will benefit you much in the long run because you will find that you will not want to try and run all of your BATs on each browser. Recording tools will not be nearly as effective as writing your own custom framework (which I will talk about in my next post.)

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

I’m going to go against the grain here today though and tell you that it is not always required.

Many times code that should be refactored doesn’t get refactored due to the myth that you must always have unit tests in place before refactoring. 

In many cases the same code stays unimproved over many revisions because the effort of creating the unit tests needed to refactor it is too high.

I think this is a shame because it is not always necessary to have unit tests in place before refactoring.

Forgoing the safety net

If you go to the circus, you will notice that some acts always have a safety net below because the stunt is so dangerous that there is always a chance of failure. 

You’ll also notice that some acts don’t have a safety net, because even though there is risk of danger, it is extremely small, because of the training of the performers.

Today I’m going to talk about some of the instances where you don’t necessarily need to have a safety net in place before doing the refactor.

Automatic refactorings

This is an easy one that should be fairly obvious.  If you use a modern IDE like Visual Studio, Eclipse, or IntelliJ, you will no doubt have seen what I call “right-click refactor” options.

Any of these automatic refactors are pretty much safe to do anytime without any worry of changing functionality.  These kinds of automated refactors simply apply an algorithm to the code to produce the desired result and in almost all cases do not change functionality.

These refactoring tools you can trust because there is not a chance for human error.

Any time you have the option of using an automatic refactoring, do it!  It just makes sense, even if you have unit tests.  I am always surprised when I pair up with someone and they are manually doing refactorings like “extract method” or “rename.”

Most of the time everything you want to do to some code can be found in one of the automatic refactoring menus.

Small step refactors

While not as safe as automatic refactors, if you have a refactor that is a very small step, there is a much higher chance your brain can understand it and prevent any side effects.

A good example of this would be my post on refactoring the removal of conditions.

The general idea is that if you can make very simple small steps that are so trivial that there is almost no chance of mistake, then you can end up making a big refactor as the net effect of those little changes.

This one is a judgment call.  It is up to you to decide if what you are doing is a small step or not. 

I do find that if I want to do a refactor that isn’t a small step refactor, I can usually break it down into a series of small steps that I can feel pretty confident in.  (Most of the time these will be automated refactors anyway.)

Turning methods into classes

I hate huge classes.  Many times everyone is afraid to take stuff out of a huge class, because it is likely to break and it would take years to write unit tests for that class.

One simple step, which greatly improves the architecture and lets you eventually create unit tests, is to take a big ol’ chunk of that class, move it to a new class, and keep all the logic in there exactly how it is.

It’s not always totally clean, you might have to pass in some dependencies to the new method or new class constructor, but if you can do it, it can be an easy and safe refactor that will allow you to write unit tests for the new class.

Obviously this one is slightly more dangerous than the other two I have mentioned before, but it also is one that has a huge “bang for your buck.”

Unit tests, or test code themselves

Another obvious one.  Unless you are going to write meta-unit tests, you are going to have to live a little dangerously on this one.  You really have no choice.

I think everyone will agree that refactoring unit tests is important though.   So, how come no one is afraid to refactor unit tests?

I only include this example to make the point that you shouldn’t be so scared to refactor code without having unit tests.  You probably do it pretty frequently with your unit tests.

I’m not advocating recklessness here

I know some of you are freaking out right now.

Be assured, my message is not to haphazardly refactor code without unit tests.  My message is simply to use temperance when considering a refactor.

Don’t forgo a refactor just because you are following a hard and fast rule that you need unit tests first.

Instead, I am suggesting that some refactorings are so trivial and safe that if it comes between the choice of leaving the code as it is because unit testing will take too long, or refactoring the code without the safety net, don’t be a… umm… pu… wimp.  Use your brain!

Things that will bite you hard

There are a few things to watch out for, even with the automatic refactorings.  Even those can fail and cause all kinds of problems for you.

Most of these issues won’t exist in your code base unless you are doing some crazy funky stuff.

• If you’re using dynamic in C#, or some kind of PInvoke, unsafe (pointer manipulation) or COM interop, all bets are off on things like rename.
• Reflection.  Watch out for this one.  This can really kick you in the gonads.  If you are using reflection, changing a method name or a type could cause a failure that is only going to be seen at runtime.
• Code generation.  Watch out for this one also.  If generated code is depending on a particular implementation of some functionality in your system, refactoring tools won’t have any idea.
• External published interfaces.  This goes without saying, but it is so important that I will mention it here.  Watch out for other people using your published APIs.  Whether you have unit tests or not, refactoring published APIs can cause you a whole bunch of nightmares.

This list isn’t to scare you off from refactoring, but if you know any of the things in this list are in your code base, check before you do the refactor.  Make sure that the code you are refactoring won’t be affected by these kinds of things.

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

This time I want to talk about choosing between what I call vertical difficulty and horizontal difficulty.

Horizontal difficulty

Horizontal difficulty is difficulty that is associated with just doing the work as the current structure or tooling exists at that moment.

Consider the problem of moving a washer and dryer.  If you have no tools and you just have to lift it, there is some horizontal difficulty involved.

In programming terms horizontal difficulty might look like writing a complicated SQL statement with multiple conditional joins because the data is all over the place.  Or writing a web page without using a framework, because your application doesn’t have one.

Vertical difficulty

This is the difficulty associated with mainly building tools or frameworks.  It is the kind of difficulty that exists in simplifying a problem by going a layer up to “meta” solve the problem.

If you are familiar with Calculus in mathematics, Calculus is an example of what I would call vertical difficulty.  Many mathematical problems are solved through the use of Calculus by taking the level up one higher and solving the problem there.

To keep with the same example of moving a washer and dryer, the vertical difficulty would be building a cart or dolly to move the washer and dryer.  An important point here, which I will make again, is that in many cases the amount of raw effort required to build a dolly or cart, or even to figure out a way to procure one, will be equivalent to the effort required to move the washer and dryer.

In terms of code, vertical difficulty might be creating an error handling framework, creating a custom control for your web page, using views to simplify SQL data access, or even to repartition and move data to make a better model.

Where horizontal difficulty represents brute force, vertical difficulty represents mental fatigue.

What about that sawhorse?

If you are familiar with woodworking or construction, you will have no doubt seen a sawhorse.  A sawhorse is platform that can be used to hold something so you can cut it.

Sawhorses are usually constructed on the jobsite before any other work begins.

Why?

Well, have you ever tried to hold a piece of wood and cut it straight?  How about searching for different objects in your garage that you can prop the wood on so that you can get it high enough above the ground that you can put a saw through it?

Experienced craftsman build the sawhorse first.  They don’t start cutting pieces of wood and then build the sawhorse.  An experienced craftsman knows that by building the sawhorse first, he will save time by not wasting time on each cut.  His cuts will be more accurate and he might just be able to bring that sawhorse to his next job.

Every time you sit down to solve a programming problem, you should think about whether or not you should be building a sawhorse first.

Are you saying always build the sawhorse?

No, not at all.  If you are going to cut one piece of wood, do not build a sawhorse.  If you are going to cut two pieces of wood, don’t do it either.  I won’t tell you how many pieces of wood that it will take to pay off, but I will tell you 3 things.

1. It doesn’t take many cuts for a sawhorse to pay back the time it takes to build it.
2. The more sawhorses you build, the faster you get at building them.
3. You are always wrong about how many cuts you are going to make.  When you estimate 3 it might end up being 20.

Vertical vs horizontal difficulty

It is very important to weigh out the pros and cons of each before making a decision which way to go.  I am of course going to try and lean you towards choosing vertical difficulty over horizontal most of the time, but ultimately it is up to you.

Let’s look quickly at some pros and cons for each (very generalized.)

Horizontal

Pros:

• Can follow a well ridden path.  Usually there is an example of how to solve the problem already.  (Someone has done it before.)
• Less thinking, you just follow the approach and go, after some amount of hours you will be done.  (Consider copy and pasting each cell of an html table to a spreadsheet, vs writing a program to parse it.)
• Less risk, you are very likely to get to your destination with minimal problems.

Cons:

• Boring.  This is not really going to challenge that programmer blog reading brain of yours.
• You or someone else will be probably doing the same thing again.  Solving the problem once, only helps to beat down the weeds in the trail, but it doesn’t make it shorter.
• You might be building on top of a bad foundation.  By adding one-offs as individual solutions to the problem, the general case can become more hidden.  (If you want to solve the problem better later, you make it harder each time you solve it the horizontal way.)

Vertical

Pros:

• Simplified working space.  Once you solve a problem a vertical way, you end up building an abstraction that makes the problem seem easier at the lower level.  (Think about connectors on your motherboard vs individually connecting each wire.)
• Reuse.  Many times when you solve a problem the vertical way, you can reuse that solution to solve future problems in almost no time at all.  (Build connector couplings for wires and next time you can just snap them together.)
• Bigger picture understanding of the system.  When you take the time to go up a level and solve a problem, you can see the bigger picture better and can understand the system as a whole better.  This will lead to better solutions and fewer mistakes later.
• You are developing a skill that is multi-purpose and can be applied more widely than a very specific skill which might be developed in a horizontal solution.  (Thinking about working at McDonald’s vs running several McDonald’s.)
• Clean.  Usually you will end up with less code.  Less code means less bugs.  Changes happen in one place instead of 50.

Cons:

• It can be hard mentally.  It can require a higher level of skill.  Not everyone who can solve the problem horizontally can solve it vertically.
• Higher risk.  If you mess up along the path of the horizontal solution, you can probably go back a few steps and fix it.  If you mess up along the vertical solution, you might have to scrap it and start over.  (Building a house vs building a microchip.)

Okay, that’s it.

Wait what?  Did you say I forgot the biggest con of Vertical difficulty?

No, I didn’t.  I left it out on purpose.

Vertical difficulty does not always mean it takes more time.  Sometimes it is actually faster to do the vertical difficulty path even when “cutting one piece of wood.”

I have seen Perl programmers and gurus parse through text or whip up a meta-solution that can solve a problem faster than I could have done it manually once.  And they have a script around to do it again.

I have seen VI wizards edit the heck out of a text file much faster than I could point and click to do the same thing.

Scripting languages and editors like VI are designed for solving vertical problems.  When you are using VI and issuing commands to edit text, you are solving a vertical problem.  You are operating at a high level to edit a text file.

Many times you will find that the vertical solution is not only faster the first time you implement it, but it also makes the solution almost instant the next time around.

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

If you never had the fortune of being introduced to VI, it is basically a bare bones text editor designed to be used without a mouse and focused more on manipulating text than creating it.  VI is on just about every platform you can think of and will be for the foreseeable future.

It is not my point today to bash VI.  VI is a great text editing tool that can make you a wiz at slinging lines and words around your files.  VI is the evolution of text editing because when you are using VI you are actually programming your text.

The problem with IDEs

Basically there are two problems with IDEs.  One I think is a valid complaint, the other appeals to engineers wanton desire to be simple, pure and take things apart.

The Bloat

The first problem with modern IDEs is the bloat.  The IDEs are big beasts that take up lots of memory, are fairly slow to load and tend to have so many bells and whistles that they feel messy and clunky.

As I write this part of me thinks “what’s the big deal?”  I’ve got a pretty fast machine, it can run VS2010 pretty well.  But, there is something that doesn’t sit right for me and I am sure for other developers.  The IDE should feel clean, but it doesn’t.

I’m not sure if I can completely identify why IDEs have suddenly gone sour in our mouths.  Perhaps part of the bloat problem is that the IDE has kind of become a swiss army knife instead of a focused tool.

Strangely enough, I think part of the problem might be tools like Resharper, that are helping us a little too much.  The tool is so good that sometimes you wonder what life would be like without all those little hints and the IDE doing so many things for you.  Perhaps sometimes you feel like you are “cheating.”

The Imagined

Then there are the imagined problems with IDEs.  The ones that don’t really have any justification, but some of the “cool kids” like to talk about on their blogs.

If I can summarize what I gleaned from the argument here, I would say it basically is… IDEs that give me auto-complete, intellisense, and visual designers rot my brain.  To really program I should be able to write code without the help of an IDE.

I couldn’t agree more with that statement. 

As a matter of fact, for that reason I don’t use electric toothbrushes, because it is not really brushing my teeth. 

I also abhor graphing calculators, it’s not really calculus unless you are cranking it out by hand. 

Email? Psha, everyone knows the pure way to communicate is by registered mail typed from a typewriter.

Oh, and don’t get me started on those GPS things.  You are not really navigating if you aren’t using a map and a compass, seriously.

Sorry for all the sarcasm, but I hope you get my point.

What is the solution then?

Is it to abandon the IDE and jump over to VI and notepad to edit our files so we don’t “rot our brain?”

I know that is the popular stance among the best and brightest right now, but sometimes the best and brightest are wrong.  Sometimes they are so bestest and so brightest that they can navigate with a map and a compass better than you or I can with a GPS.

I think the solution is to bring more VI-ness to IDEs.  The good thing about jumping to VI is that you can sling text around like nobody’s business.  The bad thing about jumping to VI is that you are forgoing some of the most useful productivity tools in dealing with APIs and libraries.

Why can’t we take the good VI-ness and put it into Visual Studio?  Looks like someone already has (ViEmu.)

The other part of the problem is the bloat.  Honestly, I think Eclipse deals with this fairly well, by making everything modular.  Unfortunately, some of the modules look plain ugly and don’t integrate well into Eclipse, but with Visual Studio you have to pull out a swiss army knife with 50 gadgets on it when you are just trying to eat some beans with a fork.

The answer is modularization and perhaps some competition for Visual Studio and some of the other IDEs that are a bit bloaty.  Perhaps we need an IDE that is built up from a VI or Emacs heritage?

I know for sure the solution is not to throw the baby out with the bathwater.  IDEs have made some truly amazing advancements that help bring the level of abstraction of software development to a much higher level. 

Features like intellisense have made it easier then ever to hit the ground running with a new API like never before.

Automatic refactoring tools built into IDEs, and with add-ons like Resharper, have made refactoring code so much easier and so much more accessible.

Ctrl+Click goto definition and backwards / forwards navigation to jump to parts of the code greatly increase productivity.

I don’t need to go into all of the features of modern IDEs to make the point that there is value there and a large amount of it.

So before you abandon the IDE, consider strongly why exactly the IDE has failed us, and consider whether jumping to VI is really the best solution to the problem.  If you don’t know what problem you are trying to solve by jumping to VI, you might just be following the “cool kids” and drinking their “cool aid.”

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

In this post I will show you how to create what I believe is the most simple and effective branching strategy.

Take a look at this diagram of a sample project’s code lines:

Walking through it

The idea here is very simple.  Let’s walk through a development cycle together:

1. Development starts.  Everyone works off of trunk.  Code is frequently checked into trunk, many developers checking in code 3-4 times a day, as they complete small quality sections of development.
2. The continuous build server is continuously building and checking the quality of the code every single time code is checked in.  Any integration problems are immediately fixed.
3. Enough features are done to create a release.  Trunk is tagged for release and a release 1 branch is created representing the currently release production code.
4. Developers continue to work on trunk not being interrupted by the release.
5. A customer finds a high priority issue in Release 1.
6. A Rel 1 Hot Fix branch is created, branched off of Release 1 to fix the high priority issue.  It turns out that a good fix will take some time.  Team decides the best course of action is to apply a temporary fix for now.
7. Rel 1 Hot Fix is done and merged back into Release 1 branch.  Release 1 is re-deployed to production.
8. In the meantime another emergency problem shows up that must be fixed before the next release.  Rel 1 Hot Fix 2 branch is created.
9. The bug fix for Rel 1 Hot Fix 2 is a good fix which we want in all future releases.  Rel 1 Hot Fix 2 branch is merged back to Release 1 branch, and merged back to trunk.  Release 1 is redeployed.
10. In the meantime work has been going on on trunk, team is ready for Release 2.
11. Release 2 branch is created…

Breaking it down

I gave a pretty detailed walk-through for a very simple set of actual steps.  But, I hope you can see how simple this process really is.

The basic idea here is that we are trying to decouple releases from development as much as possible.  The team is always going to keep chugging along, building new features and enhancing the code base.  When we decide we have enough features for a release, we simply branch off of trunk and create the release branch.

We can even do some testing on the release branch before we go to production if we need to without impacting future development.

The release branch code-lines never come back to trunk.  They don’t need to, they only exist so that we can have the exact production code and make modifications to it as hot-fixes if we need to.

We branch hot-fixes off of the release branch so that we can work on them independently, because not all hot-fixes go back to the main code-line.  We can make a hot-fix just for the current release, or we can merge it back to trunk to make it a permanent fix.

That is all there is to it.  This kind of branching strategy almost completely eliminates merges.  The only merge you ever do is small merges for hot-fixes.

Your branching strategy does not have to be complicated.  A simple strategy like this can fit almost any software development shop.

Frequently disputed points

Almost immediately when I introduce this simple system someone says:

What about half-completed features?  I don’t want to release half-completed features.  Using this strategy with everyone working off trunk, you will always have half-completed features.

So what?  How many times does a half-completed feature cause a potential problem in the system?  If the code is quality and incrementally developed, it should not impact the rest of the system.  If you are adding a new feature, usually the last thing you do is actually hook-up the UI to it.  It won’t hurt anything to have its back-end code released without any way to get to it.

Continuous integration, (especially running automated functional tests), trains you to always keep the system releasable, with every commit of new code.  It really isn’t hard to do this, you just have to think about it a little bit.

If worse comes to worst, and you have a half-finished feature that makes the code releasable, you can always pull out that code on the release branch.  (Although I would highly recommend that you try and find a way to build the feature incrementally instead.)

If you know you’re going to do something that will disrupt everything, like redesigning the UI, or drastically changing the architecture, then go ahead and create a separate branch for that work.  That should be a rare event though.

I need to be able to develop the features in isolation.  If everyone is working off of trunk, I can’t tell if what I did broke something or if it is someone else’s code.  I am impacted by someone else breaking things.

Good, that is some pain you should feel.  It hurts a lot less when you’re continuously integrating vs. working on something for a week, merging your feature and finding that everything is broken.

It is like eating a meal.  All the food is going to end up in the same place anyway.  Don’t worry about mixing your mashed potatoes with your applesauce.

If something someone else is doing is going to break your stuff, better to fail fast, then to fail later.  Let’s integrate as soon as possible and fix the issue rather than waiting until we both think we are done.

Besides that, it is good to learn to always check in clean code.  When you break other people and they shoot you with Nerf guns and make you wear a chicken head, you are taught to test your code locally before you check it in.

How to be successful

How can you be successful at this simple strategy?

• Make sure you have a continuous integration server up and running and doing everything it should be doing.
• When you work on code, find ways to break it up into small incremental steps of development which never break the system.  Hook up the UI last.
• Always think that every time you check in code, it should be code you are comfortable to release.
• Check in code at least once a day, preferably as soon as you make any incremental progress.
• Test, test, test.  Test locally, unit test, test driven development, automated functional tests.  Have ways to be confident the system never moves backward in functionality.
• So important I’ll say it twice.  Automated functional tests.  If you don’t know how to do this, read this.
• Release frequently instead of hot-fixing.  If you never hot-fix you will never have to merge.  If you never have to merge, you will live a longer, less-stressed life.
• Don’t go back and clean up code later.  Write it right the first time.  Check it in right the first time.

Hopefully that helps you to simplify your branching process.  Feel free to email me or post here if you have any questions, or are skeptical that this could work.

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

Dev machines woes

I been having lots of fun trying to build and setup my new dev machine for my new job.  Ended up working over the weekend on it, but it has been a pretty good learning experience.

I have learned many things from the experience, including:

• Power supplies can make beeping noises.
• Just because your computer beeps, doesn’t mean it’s not working, check to see if there is video.
• You must pull the processor securing lever all the way up before panicking, calling everyone you know and screaming “MY PROCESSOR IS STUCK IN THE MOTHERBOARD, OH GOD HELP ME NOW!”
• Installing Windows in SATA mode, then switching to ACHI mode will probably require a reinstall.
• Installing Windows on a hd connected to a motherboard and processor, and switching to another motherboard and processor will probably require a reinstall.
• IIS is not installed by default.
  • When IIS is installed, ASP.NET support is not installed by default in the IIS install.  (I always install IIS to serve up static content in 2010… yeah)
• Drives raided together need to have their partitions recreated before a Windows 7 install will recognize the drive at all.
• 4 monitor stand clamps don’t work to well on glass desktops for supporting four 24” monitors.
• You can never have too many monitors.  It’s just not possible.

Making lemonade

Out of all the bad things that seemed to go wrong, I did learn a large amount of stuff, so overall I think it was worth it.

One of the really cool things I started learning is Powershell.  After setting up my dev environment for about the 4th time, I decided I would try to start to build something to benefit the rest of the team and new developers.

I started writing a Powershell script that would set up all the little tricky things that need to be done in order to get our development environment ready.  I am eating my own dog food, from my previous post.

One value I hold pretty highly is that if I have to do something manually more than 3 times, I need to figure out a way to automate it.  Powershell makes many automation tasks very possible.

I’ll include some pictures of my dev cave on my next post!

In doing so, I have once again felt the pain of getting a development environment configured.  I forgot how painful it can be.  This is in no way a reflection of TrackAbout, the truth is most development environments are a pain to get setup.  Unless you’re actively trying to build a painless development environment, it is going to probably be the opposite.

I’ve seen a large number of development environments and I’ve built my share of them.  From all this, I have a pretty good idea of what I consider ideal, what we should strive for.

The basic outline

1. Install non-scriptable tools or start with a fresh image.  (Basically getting IDE and SQL Server installed locally)
2. Get branch from version control.
3. Build database
4. Build code
5. Local deploy

The idea here is that I should be able to either get an image that has my base tools installed, or install them myself, then pull down one source control location and everything else that happens from there is the result of  build scripts or some other automated process.

I know, it is easier said than done.  Let’s break it down step by step and look at some of the possible solutions.

Install tools

If you are in an organization where everyone will have the same hardware, it is much easier to create an image of a developer machine with say Visual Studio and SQL Server installed.

Another possible solution is to create a dev VM that is maintained and updated regularly, so that it has all the required tools and you have a uniform structure.  I have tried this approach, and I find that the biggest problem is that many times you want to run native to get the performance improvements.  As hardware capabilities increase though, I am seeing this as a more viable route.

Finally, if you can’t get either of those situations, it is ideal to put all the tools that must be installed on a network share or some other easily accessible place.

Ideally, you want to keep the number of required tools down to an absolute minimum.  In most .NET environments this should be Visual Studio and SQL Server.  The other kinds of tools can be handled via dlls (usually).

Get branch from version control

Ideally, you should be able to point a person to one source control location, and that should get everything necessary for them to build and deploy the entire system locally.

If different applications your organization is developing have different branches, then you might need to check out one location per project, but even that can be automated to some degree with a “get latest” script or symbolic links.

Build the database

This one is kind of hard.  It requires quite a bit of forethought on how to get this working.  The idea here is that I should be able to build the entire database from a set of scripts.

The challenge is getting together a process which allows for the construction of the database from scratch and to populate tables that are required for the application, and be able to apply patches to existing database.  I won’t go into how to do that here.

Build the code

There is quite a bit lumped into here.  From a developer perspective I should just be able to run one build command that is the same build that will be run on the continuous integration server and everything that I need should get built for me.

From behind the scenes, this is a difficult step.

• You have to make sure everything works from relative paths or environment variables.
• You have to have your scripts check to see if things are installed and install them if not (registry keys, etc).
• You have to have all the libraries in a place that the build can find on the client machine.

The key to success here is to eliminate as much as possible and locate in one place, as much as possible, all configuration differences.

Local deploy

It should be very easy to do a local deployment of the application.  For .NET developers this usually isn’t a challenge, but in the Java world it can take some thinking on how to do this properly.

At anytime someone should be able to deploy locally to their machine.  Ideally, anyone should be able to take a build from the build server and deploy it with a single command.

It is all about the mindset

Basically, you have to think about zero configuration development environment from the beginning if you really want to be successful at it.  It is much harder to add it on later.

You do have to weigh the effort involved carefully though.  Most developers only set up their configuration once or twice. If you are going to have a growing team where you are constantly adding new developers, you should probably put considerable effort into getting as close to zero configuration as possible.  On the other hand, if you have a small team and don’t have new developers very often, it might not be worth the extra effort.  You have to find the balance.

In all honesty, my experience at my new job has been pretty good in contrast to some of the development environment setups I have seen.  There is a huge amount of consistency in configuration locations, which is good.

I’m looking forward to figuring out how to make it easier for the next guy though, once I understand everything better myself.

One of the first things I have been doing to get ready to start this new job is setting up my development workstation at home.  There are really quite a few considerations to think about when setting up your home office.

Hardware

I was debating between going all out with the Intel 6 core chip or getting the nicely priced AMD 6 core.  I ended up choosing the AMD chip because the Intel chip was 3 times the cost, and the chipsets on the motherboards for the AMD chip are a little more stable since they have been around longer.

The processor doesn’t really matter that much anymore since processors are so fast nowadays.  What really matters though is the hard disk.  On this case I opted for a super fast SSD drive.  256GB should be plenty of space, with an additional hard drive just for back ups.  The speed improvement when using a really good SSD drive is amazing.  It is the single best upgrade you can do in a developer machine.

I also went with a large amount of RAM because I know that if you need to run a virtual machine, RAM is going to be a big deal there.  16GB should meet any need just fine.

Finally, I am trying out a quad-monitor setup for the first time.  I have been using dual 24” monitors for a long time now, but I have always thought two more would be even better.  It is really important to be able to quickly see multiple things going on at once without having to switch between applications.

I’m planning on setting up the monitors like so:

1. IDE
2. Reference: web pages, API, etc
3. Communications: twitter, pidgin, email
4. Secondary IDE for debug, or SQL server

Desktop or laptop

Most developers are getting laptops with docking stations these days and while I see the appeal, I like a good old tower and a cheap laptop instead.

• With a desktop you can get beefier hardware for less $$$.
• You can pick the hardware yourself (which is a big deal for me since I do a large amount of research on each component).
• You can have more monitors natively (just buy a 4 port video card).
• More upgrade options.

I still have a laptop, but it is a cheap light one.  The advantage here is that I can just remote desktop into my powerful machine and get all the benefits of both worlds.  If I am really ambitious I can even remote in with my iPad or phone.

So, while I can definitely understand the appeal to many for having a laptop that they can just disconnect and carry with them, I still prefer the desktop.

Virtualize?

I almost did it this time.  I keep going back and forth on this one.  I really want to have my development machine be a VM so that I can just load it up and go, but after thinking about it more, I am not sure it is worth the cost.

I kept thinking about why I want to have a virtual machine for my development machine.  Really the answer comes down to me liking to have things in a nice separate little box.  Sure, I can drop my dev virtual machine on an external drive and take it with me, but I can achieve the same by remoting into it instead.

The thing that made me finally decide on no VM is the idea of optimizing for the rule rather than the exception.  The truth of the matter is when I am on my PC, I am going to be spending 90% of my time doing development work, and doing it on that one machine.

When I look at it that way, I can’t see a good reason to take the performance hit of virtualization for the 10% case.  I can achieve most of the “neatness” of virtualization by cloning my disk at a good configuration.

I’ll probably still have some sandbox VMs for testing out “crazy stuff”, but I think I am going to go native so I can really utilize the max out of my new hardware.

I’ll just have to treat my Windows install like I treat my source code.  “Leave it better than when I started.”

Anything I’m missing here or not seeing?

Someone recently asked what exactly this means.

It means that the database data that is in production never goes into dev, integration, qual, or whatever other environments you may have set up at your workplace.  I think more important than the “what” is the “why”, and while I’m at it, I’ll tell you why I won’t give you, as a developer, access to production data at all.

Why production data stay in production

Let’s start with a simple reason: security.  Production data in most cases is going to contain at least some amount of sensitive information.  If your company leaks sensitive information about customers or clients, you can almost guarantee that you will have a lawsuit on your hands.

It is much better to keep production data as safe as possible.  The best way to do that is to limit access to it as much as possible.  If you move production data to different environments, you will be greatly increasing the amount of places that data lives and correspondingly the number of people who have access to it.

Aside from just the security concerns, is the problem of reliance on exact data to recreate a problem.  Debugging an issue which is seen in production should not require the actual production data. If it does, there is a problem with the tooling you have created to support your application.  You might consider creating some development tools that will allow you to simulate any production issue in a different environment without having to use real production data.

I know it is much easer to just use production data, but it is a bad habit that masks other problems with your tooling and support.  It also has a large cost to the infrastructure and support of an application if you require frequent data migration between environments.

Why developers stay out of production

The reasoning is very similar.  We could talk again about security.  We could talk again about the problems with relying on production data to debug a problem in your application.

This really comes down to a discipline and constraint issue.  By enforcing this constraint and practicing this discipline, you gain quite a few things out of necessity:

• Logging must be improved to understand what happened in production
• You are forced to consider creating a user click tracking mechanism
• Test data generation is forced to be considered
• Testing becomes more complete, because your team is forced to recreate issues independently rather than using production data

It might not be immediately apparent to you how these things are connected, but if you enforce this kind of a policy, you will quickly find that you will need better ways of understanding what the user did.

It may seem counter-productive to make a developer’s job harder by preventing them from accessing production, but in the long run, being forced to create tooling and better logging to understand how the user is using the system, and having more complete testing will save time.

Some tips on making this reality

Yes, I know, easier said than done.  Let me help you get it done with these tips:

1. Build enough logging into the application to make it easy to understand what the system is doing and the flow through the system.
2. Build a detailed level of logging that can be turned on for different parts of a live running production system.
3. Build or use tools that take data from your web server logs and application logs and translate that into a digest of how the user clicked through the system.  (Doing this well will allow you to recreate most scenarios that may have happened in production.)
4. Create migration tools that allow you to migrate and cleanse a particular piece of data.  The goal here is not to take the whole database and dump it to another environment, but to take specific sets of data, cleanse them (get rid of all personally identifiable information), and put them somewhere else to examine and debug.
5. Have ways to generate volumes of clean test data to put into non-production environments.  This may look like a test script that runs though the application and clicks through many scenarios to generate test data, or a bunch of SQL scripts that populate test data tables.

Here is the thing about a dead horse.  Well, you just can’t really know it’s dead.  I mean, sure its eyes are closed.  Sure, it’s lying there in a bloody mess on the floor, but is it really dead?  The only sensible thing to do is beat it just one more time, just to make sure.  Let’s kill that horse good!

There is an underlying problem here… I call it

THE SCRUM BUBBLE

Remember the .com era.  Ah, wasn’t that great.

How about that whole stock market always rises decade…

Oh, and don’t forget the housing boom.  The housing boom was great!

But then… What happened?

Oh yes, we got nuked back to the stone age. 

The same thing is happening with Scrum.  Certification proliferation is fueling it.  What happens when everyone in the software development world is a Certified Scrum Master, and we are still building shitty software?  What then?

BOOM!

It implodes, just like a bubble.  That bubble will burst and when that happens it is going to take Agile right along with it.

A wise man once said

You can shear a sheep many times, but skin him only once.

When you arm people with useless certifications and send them into organizations to go and conquer the beast of software development, you are skinning that sheep.  All the consultants that are milking so much money out of Scrum will eventually find that tit dry.

Scrum is good

It is.  It is really a great tool for building software.  It makes sense, it is practical and if you apply it correctly you will get a good result.  The problem is applying it correctly.  The problem is it isn’t addressing the other things that need to change to really be able to do Agile development.  (Which is why I suggested a new process Kanbanand that does address those things.)

Training and consultants are good also.  Getting good hands on training from someone who really knows what they are doing can accelerate you on your learning curve like nothing else.  Consultants who have already been through a process and can look at things with a fresh eye can be invaluable in making positive changes in an organization.

The problem is Scrum is not the end-all-be-all to software development.  It is not the silver bullet that kills death march zombies.  It is a tool.  I really like Henrik Kniberg’s approach to Scrum and Kanban, where he talks about each being a tool to help you develop software.  In his book, (buy it this book is really good), he asks the question, which is better a fork or a knife?  A pretty silly question to ask, which illustrates the point well.

When we hype up Scrum to the point of rock star stardom, and then use it as a tool to extract large amounts of money out of the software development eco systems quickly, we are creating a bubble.  Just like any bubble, we should expect that to burst if we keep inflating it.

There are a lot of good people in Scrum

There are plenty of good people that have a sincere passion for the craft that are really investing into Scrum because they know it is a good way to deliver value to customers, and it is a simple framework that can be easily taught and implemented.  There are plenty of people like David Starr and Tobias Mayer that are honestly trying to do good in the world of software development and using Scrum to do it.  I honestly believe that.

But, there are plenty of other people that are riding on the success of Scrum and building careers out of being a certificate factory.  We have to take away the keys to the kingdom from those people if we want to avoid destroying the good that others are doing in the name of Scrum.