The worst that could happen (part 5) – The (not-so) Portable HDD

Joe and the (not-so) portable Hard Drive

My first day on the job, I was told to start fixing all of these broken programs. This was no surprise. During the interview, I was warned about how bad things were and I was ready to dig-in.

For starters, there was no source code for any of these broken programs. Yes, that is right. No code. NONE! I took the job anyway because the programs had been written in .NET 1.1 and I knew about a decompiler called “reflector”. I planned on decompiling all of these programs and systematically knocking them out.

(Translation for non-techies: “no code” = “can’t program”. Programs are written in code and compiled into a program. A decompiler turns a program back into code.)

Now that I was an employee, it was okay to ask the question: “What happened to all of the source-code anyway? You said it got lost, but how did it get lost?”

The House that Joe Made

It started when they hired Joe, right out of college. He was wicked-smart. He cranked-out programs like nobody’s business. He was always coming up with fresh new ideas too. He worked well with the sys admin (who was self-taught). It seemed like every day for the users was a new day, full of surprises and innovations.

After Joe had been there for a year, there seemed to be a lot of broken programs. It seemed to be slowly getting worse as time went-on. Joe didn’t seem to be fixing things, because he was always busy innovating and producing new programs.

After a year and a half, the bugs were getting out of hand. Pressure had been rising. The IT department added another developer to help Joe. Instead of teaching the new girl to share the existing work, he assigned a few projects for her to take-over. It was much easier than trying to share the code and avoid stepping on each other, constantly. He had his stuff and she had hers.

Joe’s boss thought it would be easier to share the code if the two programmers used a SCM (Source Code Management system). Joe read that SVN was the popular choice, but the new girl wanted to use SourceSafe, because it seemed to integrate better with Visual Studio. Joe decided to use both. They would each try different SCM systems and see who liked theirs best. It seemed like a good idea.

Taking This Joe on the Road

At the two-year mark, Joe announced that his wife got a job in Chicago and so he would be leaving in two weeks. He would spend his last two weeks making documentation that showed how to maintain all of these systems.

Joe didn’t have a job lined-up in Chicago yet, so he offered to work remotely, as a consultant. He would log his hours and bill for them. It seemed like a reasonable plan until the boss could find a replacement for Joe.

To make things easier, Joe took his computer with him. Nobody else was using it anyhow.

Things started out pretty well. Joe got some stuff done and showed some decent progress. After a few weeks, Joe didn’t seem to be getting as much done. Little by little, things slowed down. The boss had a hunch what was going on and had been actively searching for a consultant who could fill-in for a while.

Finally, the boss found a local consultant to take the baton from Joe. Joe was asked to bring back his PC. Joe wasn’t headed that direction any time soon. So he had the PC shipped back to the office.

The PC arrived within a few days and the consultant was ready to dig-in. However, when the consultant hooked up the machine and tried to boot it. Nothing happend. There were a few lights, some beeps and a message saying something about a hard drive not being found. He opened the machine and yes, the hard drive was right there. Yet, the machine insisted that it could not find the hard drive.

They gave Joe a call and he confirmed that everything was working fine when he boxed it up. It was all there. There was no reason to see an error. “Good luck”.

The Backup Plan

It was at that moment that the boss realized that there were no backups of Joe’s machine. Just then the boss remembered the SCM that Joe had been evaluating.

They took a look at SVN’s web interface and found Joe’s source code. Unfortunately, it turns out that he had only evaluated it for two weeks and quit using it. Nothing had been checked-in within the past eight months. Plus, it looked like he had only checked-in the two projects that he was working-on that week. The other programs (maybe 40 or 50) were not checked-in. They must all be on the hard drive of his machine. The dead hard drive.

A quick phone call to Joe confirmed that this was true. So what could be done about it? Joe didn’t have any ideas. He never made a backup because, he never imagined that he would need one.

Essentially, all was lost. Including all hope. (Only, until the team learned about decompilers).

Lessons learned: 1) Always make a backup of all code. The storage space is cheap. 2) Use a SCM all of the time. All code goes into the SCM. ALL OF IT! 3) The boss needs to enforce each of these rules and follow-up to make sure they are being followed, without exception. 4) Never send a machine on the road. Make a copy of the HDD and send that instead. 5) If you spend all of your time working on new stuff, you will never have time to fix the bugs. 6) Young new programmers have lots of energy and creativity, but they might not have very many “lessons-learned” to draw upon. So when they say “what’s the worst that could happen?”, they really mean it. They just can’t imagine things going this badly.

About Tim Golisch

I'm a geek. I do geeky things.
This entry was posted in IT Horror Stories, Lessons Learned. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s