Switching from dev-process to maint-process

I was going to name this article “Maturity model for maintenance programming”, but every time I read that title, I cringed. Maintenance programming is not much fun. It sounds a bit like “prison guard” or “mortician”, or some other unpleasant profession.

It is a tough job, but someone has to do it. More than that, the job needs to be done well. Imagine the results from a prison guard or mortician who is a slacker. You know the results would be unpleasant and people would be pretty unhappy with you. It is serious work and therefore, it needs to be done by someone with “guts”. If you have the guts to do it, then let me offer you my thanks in advance.

When you take-over a project from someone else (another person, or another team), there is a bunch of work to do. You will quickly find how the project went. It probably went like this: At first, your predecessors meant well. They started with some great plans and expectations. Everybody was optimistic and striving for perfection. As the project progressed, the roadblocks got bigger and the team eventually became weary. The closer the project got to the end of the timeline, the more all-of-that-talk about “the right way” tended to fall apart as the project got squeezed by a timeline, a budget and the remaining feature-set (and bugs). Something had to give. People made compromises and got things done. It was good-enough. Now that the project has been handed over to you, your job will be to find all of the sub-par stuff and create a scope-of-work to fix it.

The good news is that, once it is handed to you, the focus on timeline seems to go-away and the focus turns to “reliability”. Which really means [fix the bugs immediately]. Of course, what they really want is a bug-free system, but the term is misleading, because your predecessors will claim that they delivered a bug-free system and there is not much to do.

Removing “the suck”

There are several layers of improvement/maturity that you can take the project through. Each tier makes the system easier to maintain. However, each one also takes increasing discipline, planning, skill and effort to get the system to progress to the next level.

Here are the levels of maturity:

  1. Released – The project runs and has cleared testing. Hurray! Ship it!
    At this point, there are no expectations about maintainability. Up-to-now, the goal has (most likely) been to complete the project by a certain date, with a list of features-completed and passing some reasonable level of testing. If the authors were not required to maintain the code, there would have been no incentives for making the system maintainable.
  2. Customer Assessment – There are way-more customers than there were testers. The customers (end-users) are going to be more creative than your testers. They are a million monkeys sitting at keyboards and they will give you some Shakespeare eventually. And by “Shakespeare”, I mean “bugs that you never would have dreamt-of”. When those bugs arrive they will be waves crashing on your embankment and your level of resourcefulness will be the key to resolving the problems quickly.
  3. Stabilizing – The bug waves typically come in a pattern: 2-2-2-6. 2 days till the first wave, then 2 weeks later, then 2 months later, and it mostly subsides in 6 months. You will need to deploy your fixes between these waves of complaints. As you stabilize the app, you would expect the intensity of the complaints would gradually reduce. However, that doesn’t seem to actually happen. The complaints just become less frequent, after a while.
  4. Supportability – After 2 months, you will start to think about how much time (too much) you are spending on monitoring your app, or wading through code or data, to find problems. At this point, you will start to think of utilities or features in your program, which would make it easier to support. These productivity tools will give a fairly nice ROI. They will make your support-efforts move quicker and more-decisively.
  5. Performance – After you have several waves of bug fixes and updates, eventually some joker is going to complain that your site is slow. It wasn’t this slow in test (because you didn’t have any data). At this point, you need to do a performance study and optimize the app and database. Expect another of these in 18 months.
  6. Feature enhancement – After the bugs are (mostly) resolved, people will start chattering about how some of the features are not as good as they’d hoped, or it would be better if… So, there will be enhancements. This means new development. Which means you will have to switch gears back to requirement gathering and scope containment. How soon can the users get this new stuff? No, that is too long. Get it done sooner!

Not every project goes this smoothly or follows this timeline. This is the path a project is supposed to take. Of course there are some variances, based on project size, team size, budget, etc. However, if your project does not climb these levels, it is usually because your leaders are not aware of the natural progression.

It helps to know what to expect, so you can measure yourself against reasonable expectations. Also, if someone has unreasonable expectations, it is nice to have a nice/effective way of suggesting something more reasonable.

So plan your lifecycles, set expectations and stay l33t.

Posted in Lessons Learned, Methodology | Tagged , , | Leave a comment

How to get people to follow a process

Years ago, I worked on a team who was making an online trivia game. The concept was a very brilliant solution to a common problem. The customer was a huge company who processed credit applications for car purchases. Their financial services mgt had come up with a new initiative to boost sales. They were having problems convincing “the old-dogs” to try it. They were like “look, I don’t need this gimmick. I’ve been doing this for [x] years. I already know what I’m doing. Just leave me alone so I can do my job”. Management was frustrated with the uncooperative attitudes of some of these people, who were unaware that they were getting rusty.

The solution was a trivia game. It was designed to confirm that (at-least) each person had absorbed the training and the new concepts. Each week, new questions were posted and people gained points from correct answers. It tapped-into the competitive nature of people and motivated the staff to learn the materials, just so they could win the game. It was quite irresistible and it was a big success.

I wish this strategy could be applied to every problem, which involves stubborn people. Unfortunately, it is probably not strong enough. For instance, imagine if you were to come up with a new software development process, or documentation regimen for your staff. Ugh. Documentation. I’d rather eat liver, or brussel sprouts. How could you possibly get people to get on-board that wagon? I don’t think a competitive game amongst co-workers would be quite enough.

The solution lies in the end-game. Same as the trivia game, previously mentioned, the end-game is to win at whatever you are doing. The reason people hate documentation is because a developer will spend a bunch of time on it and nobody will ever read it or get any ROI from it. At least, that is the perception. Likewise, what is it about this new-fangled software dev process that makes it so superior to our old one? I have to break all of my habits (arguably good ones) to form new habits. I won’t drink the cool-aid unless I’m certain it will be much better-tasting.

As a leader, these are your points-of-focus:

  1. Help the team see and understand the problems that you are trying to solve
  2. Help them understand why the problems are bad
  3. Keep statistics for the side-effects of those problems
  4. Point-out that the current process gets you a “B-“, but you really want an “A”.
  5. Explain how the new process is designed to address the main problem(s) that are holding-you-back

One little caveat to go along with this: You need to be able to deliver on these five points. If you don’t, then you honestly deserve to get resistance from your folks. They will believe that either you a) cannot back it because this info is not legitimate, or b) you aren’t willing to put-in the effort to back-it. Basically, you don’t really believe in it, but you expect them to care more than you.

At the end of the day, people really want to take pride in their work and don’t want to do senseless stuff. If they can see the carrot, and understand that carrots are good, they will reach for it.

If they truly understand things, they may even contribute additional ideas and become invested in solving the root of the problem. They may even feel inspired to solve other problems. How cool would that be?

Posted in IT Psychology, Professionalism, Team | Tagged , , , , , | Leave a comment

Refactoring code that is too big

“If your code is so big that you need Regions to collapse areas of it, then your code is probably too big and you should consider refactoring it”.

I’m sure everyone has encountered a programming task, where someone wrote a huge page or library or function, etc. and you had to maintain it. This kind-of feels like you’ve been dropped in the ocean at night. You can’t see the shore and it is hard to determine which way to swim. Even choosing to swim west, might be pointless if there is a current going east.

Fundamentally, the concept of refactoring large files into smaller files seems logical. So, when I see a really large file or function, my first inclination is to dive-in and refactor it.

I recall one particular time, when I took a large class/file and started refactoring it. I cut large chunks of code from the file and pasted it into new files and connected them back together by making calls between them. As I did this, I found that some of the code was stateful and relied on local (private) variables and properties. So I had to pass those into the functions that I just moved. As I did this, I realized that it was all becoming tightly coupled.

Once I was done, I evaluated what I had just done: refactoring this class/file took a bunch of time, I changed a lot of code (which added risk) and the finished work was tightly-coupled (not ideal). In the end, the refactored code seemed to work fine. So I guess it was maybe worth the risk. I still wasn’t sure how much value I had actually provided. It could be argued that I added or subtracted value.

So essentially, what I found is that refactoring was not so bad in functional (modular or non-OO programming). You just cut the code in half or in tenths, and you are done. However, in OO, you can’t just split up the code in “an object”. Some of the fundamental nature of OO programming is encapsulation. Encapsulation basically means that your object needs to contain all of its own code and state (variables). Splitting it up is a clear violation of OO principals, unless you are pulling the them into another class, which is independent of the other classes (loosely bound).

Sometimes an object wraps something really big. Sometimes, you can’t just refactor it and make it smaller. And sometimes, when you refactor it anyway, you end up with utility libraries and static classes and functions, that merely spread the mess around and it is no-longer very true to OO design.

Another down-side to this style of refactoring is when you refactor without consolidating any code. In that case, your code seems smaller because you have seventy files that are ten lines long, but still have seven thousand lines of code. When you step through the code, you start to get lost in the seventy files that your logic traverses. Good luck keeping track of that. I don’t really believe that it is an improvement of any kind.

The most certain way that you can tell that you have over-factored a function/class is when you have subs/functions where you are passing dozens of parameters into it, or if you have to use ByRef variables (you change the state of a param, rather than returning it as a result from a function). This is terrible and is evidence that your code and logic is fragmented and will be more-difficult to maintain now.

Most developers who encounter this kind of problem pick one answer: “This should be rewritten”. However, all truly experienced programmers will tell you that a rewrite is very unlikely to produce a better outcome. It will just burn through a bunch of time & money.

My point is this: some blocks of code are just big. Before you spend months working on the perfect refactor, think about the ROI that you will get. Just because “the experts are saying it”, doesn’t mean it actually makes sense or is practical in most circumstances. Please get a grip and show some judicious discernment. Then discipline yourself to pick the low-hanging fruit (if there is any). Get that done and then step away from it, and call it a day.

You don’t have to solve every problem. Pick your battles. Know when to be elite and when to be efficient.

Posted in Lessons Learned, Optimization, Professionalism, Programming | Tagged | Leave a comment

How your processes save your backside

When you have a well-defined process, it can be tedious at times. Sometimes you even want to throw unpleasant terms on it, like “bureaucracy”.

Bureaucracy sounds nasty. I’m trying to think of someone who could make it sound palatable. Maybe Mr. Rogers or Martha Stewart or Samuel L Jackson could make it sound nice, or at least cool. Well, stand-back, because I’m about to try to add my name to that list. Let’s see if I can sell you on a little bureaucracy.

Have you ever asked someone at work, to do something? Maybe your colleague even said they already completed your request, “yep, it’s done. You’re welcome”, and then it turned out that it actually WAS NOT done. So you asked them what happened and suddenly that person has some kind of memory loss. “I didn’t say that. I don’t recall agreeing to that. You never asked me to do that”. You just got hung-out-to-dry. Maybe you are even a little mad about it. I know I would be.

I must confess, I’ve even done this to myself a few times. Please believe me: I was not trying to deceive or trick myself. I just got rushed to do a bunch of important things and I overlooked one or two important steps and didn’t realize it until much later. I was pretty mad at myself and maybe even vowed to never let it happen again.

I think we can agree, these are two types of instances where “following-a-plan” could have helped. Here is how things would have gone differently.

Scenario 1:
Me: Hello fellow-co-worker, could you please accomplish this very time-sensitive task.
Co-worker: Absolutely. I am glad to participate.
Me: (later, via email). Here is the plan. It shows what I am doing and what you are doing. It also has durations, and an ETA (for your tasks and mine). Could you please confirm that this is correct?
* If the co-worker doesn’t respond or can’t commit, I retry, but CC a manager or two, just to ensure that I am okay to assign work to that person (and maybe I am even applying a little muscle).
Co-worker: Yes
Me: (later, following up) Hey, I just wanted to touch-base. I’m sure you have it handled. I just wanted to see if you need anything from me, or if anything is in-your-way.
Co-worker: (who nearly forgot) Yes, I am about to start on it right now.

The take-away from this is:
• Confirm the commitment from each team member
• Ensure everyone is aware of the timeline, tasks, work distribution
• Follow-up, with plenty of time before the deadline. So, if the person forgot, there is still time to complete the task on-time.
• Keep following-up (very politely), until each task is complete
• CC a manager if you need a little more muscle, but don’t over-do it. Keep things friendly.

Scenario 2:
1. Make a plan for myself
2. Determine what I am doing each day
3. Make it into a checklist
4. If I discover a new task, add it to the list and the timeline
5. If the plan shifts, then shift the plan, so I can tell if my plan is still feasible

The take-away from this is:
• Work a plan, so you don’t forget stuff
• When plans change, it is easier to make realistic decisions about timelines, if everything is in-front-of-you at-once

If your plans always work perfectly, this might seem like a waste-of-time. However, if you ever want to scale beyond a team of one (yourself). You need to have a written plan, so your teammates can prepare and coordinate. Also, some people are just hopeless optimists (or way-over-booked) and have trouble being realistic about timelines and ability to meet deadlines. A written plan is just rigid (and honest) enough to bring the needed amount of realism to the project timeline.

Even if you scoff at this type of process, I would say, give it a try once or twice. See if it made anything better. It is a trade-off and it is a new skill. It might even take you one step closer to elitism. Welcome.

Posted in IT Psychology, Lessons Learned, Professionalism, Team | Tagged , , | Leave a comment

My most epic project

The biggest project of my career was a Y2k project for a state agency. The project was a huge challenge by itself. To make things more interesting, it had a fixed timeline (Y2k) and a customer who didn’t typically do well with fixed timelines. That combination is a near guarantee to make things challenging at times. It was a pretty big opportunity for me and I was optimistic about it, in spite of the obvious challenges.

On day 1, we started by formulating our strategy. We needed a set of ground-rules that would guide our decisions, to ensure success. This was the crux of our strategy:

  1. Contain the scope (and be vigilant about it)
  2. Build a solid proof-of-concept (program/prototype)
  3. Focus on architectural models that would prove each of the elements of the final product (eliminating unknowns and accelerating development)
  4. Train the staff
  5. Cooperate
  6. Commit to improvement (within reason)

We figured that if we were committed to those guidelines, then our development would accelerate (meaning: the rate of completing work would steadily increase), the timeline would not be able to outrun us, and the chance for surprises would be minimized. The timeline would be predictable. We could do it, on-time.

We started-off strong with all six of these. Of course, progress always seems to start-off a little slow during the first few weeks of any project. It is a real gut-check until you pick up steam. I was only 30 years old, and I got my first 5 gray hairs during those 3 months.

During that time, my dev lead expressed concern about making the timeline. The task was gigantic and he couldn’t imagine how we could accomplish so much in such a short time. I explained how it would all work (a few times) but he still had concerns. He had never been part of anything like this (and technically, neither had I). He wasn’t as optimistic as me. In fact, you could say he even sounded somewhat pessimistic. I figured, he probably just needed some solid reassurance from me.

After trying a few times, to reassure him, I realized that there wasn’t anything that can be said to reassure a young developer, about a project as big as ours. So, I changed my pitch: “If you honestly don’t think that we can complete this, then we really only have two options: 1) Find new jobs right away. Or 2) Work as hard as we can and do our best, and at the end of it all, if we come up short, we can rest-assured that nobody else could have done a better job than us and been more successful”. (of course, I was being a little sarcastic with the first option).

He acknowledged that there was really only one reasonable choice: do our best, and leave nothing on the table. So he committed to our endeavor. I did point-out that I was personally determined to do anything necessary (within the boundaries of good ethics) to get the project done on time. We had some great support and resources at our disposal. We were even told that “getting a B” would still pass, but we were going to settle for nothing-less-than an A.

We pressed-on with diligence, constantly checking our quality, progress and vigilantly containing scope. One time, our scope-containment effort even resulted in a rather heated disagreement with a serious player. I was a small dog in a fight, but I treated it like the careers of my team (18 of us) depended on this decision. We stood our ground and worked our resources pretty hard. I used up all of my “get out of jail free” cards that day, but prevailed. The scope was contained and we sprinted towards our due date.

In the end, we finished 2 weeks early (Sep 16, 1999) and the program exceeded the expectations of everyone, for performance and stability. Everyone was astounded. We rolled-out the project to the entire state of Michigan, over the next few months. 14000 state workers picked it up without a hitch. Y2k was a non-event for my team.

Of course, I cannot take sole credit for the win. I had a great PM, excellent dev managers, I had one or two certified geniuses on the team and management was behind us 100% (even covered my backside from that dog fight). We couldn’t have done it without all of them. Each accomplished some legendary feats during the project.

I still feel that the winning ingredient in all of it was the solid foundation (6 guidelines) and the commitment from everyone. At times, we got scared and wavered, but that fear also drove us to accomplish things that (I’ll even admit) were pretty colossal. We were driven and that made all of the difference.

Since that project, I have not faced very many challenges that can compare to that one. I am fortunate to have seen how a great accomplishment can be achieved. I was there and I know what goes into it.

If you ever get offered a chance to accomplish something epic, do a gut-check, be realistic about your own resources. If you go all-in, be ready to back it, all the way. It might be the most legendary win of your life, but only if you apply the effort needed to get that win.

Posted in Career, Lessons Learned, Methodology | Tagged , , , , , , | Leave a comment

Event code: 3008. A configuration error has occurred

This week, we were deploying some new files to our project (running on IIS). After adding the files, the app didn’t run, and we got the following error in the IIS logs:

Event code: 3008
Event message: A configuration error has occurred.

… [snip]

Exception information:
Exception type: ConfigurationErrorsException
Exception message: Could not load file or assembly 'ProjectName_old' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
at System.Web.Configuration.CompilationSection.LoadAssemblyHelper(String assemblyName, Boolean starDirective)
at System.Web.Configuration.CompilationSection.LoadAllAssembliesFromAppDomainBinDirectory()
at System.Web.Configuration.CompilationSection.LoadAssembly(AssemblyInfo ai)

The error looked like maybe there was some kind of config error. Except we didn’t make any changes to the config. We just deployed a new version of a DLL in the /bin folder.

Something about it sounded familiar. I’ve seen this misleading error a dozen times before.

The problem was caused by a simple mistake on the part of the server admin. He renamed old dll before he put the new one on the server. Unfortunately, the way IIS works, it will read all DLL files in the /bin folder and pull them into memory. Unfortunately, this will cause some serious conflicts if you have two copies of the same DLL, but with different names. IIS won’t know how to handle the conflict.

It would be great if the error message said something accurate like “DLL Conflict: two DLLs contain the same classes/namespaces and are conflicting with each other. The files are named Copy1.dll and Copy2.dll” but nooooooo. Instead it gives this dandy error message.

So, hopefully you know better now. If you ever deploy content to an IIS server, please just zip the whole folder and store that backup somewhere (not in the wwwroot, etc). Then delete the contents of the folder, and then put the new content there. It is the safest way to avoid conflicts, while preserving your options to restore everything, in the event of a problem.

Posted in Errors, IT Horror Stories, Lessons Learned | Tagged | Leave a comment

Underachieving, overachieving

When I was in high-school, I had an interesting discussion with a guidance counselor. I was taking some pretty hard classes in school and I was getting so-so grades (B or C, not A). His advice was to take easier classes. I told him that I really liked to be challenged and I don’t think I would get better grades if I had easier classes. I confessed that the problem was that I was just lazy.

He persisted that my grades were going to be used as more of an indication of my intellect, and the challenge/difficulty-level of my classes would be overlooked. Basically, if I wanted to go to college, I needed good grades, not a good education. He told me that my grades made me look like an under-achiever and that was really bad.

After high school, I went into the US Army for a while and did okay. Some things, I did great but others, not so much. A few people gave me great advice about self-improvement, but I had trouble embracing it. I must confess that I was still a little lazy and I moved a little faster when a sergeant or officer was um, “motivating” me.

In college, things turned around. My junior year at U-Mich, my classes got pretty hard and the dean of engineering told me I had to bring up my grades or they would kick me out. I couldn’t believe it. What a wake-up call. I cancelled a lot of my leisure activities and replaced that time with studying. I even started studying things that I didn’t have to study. I started learning some really cool stuff and wanted to apply it.

After graduating, I kept applying this idea: I looked for things to do or learn, which went beyond what was asked of me. At work, I started looking for new ideas for learning and growth. I started looking for the things that needed to get done, but weren’t, or places where I could apply some newly-gained knowledge, to make something work-better or simplify something. It was pretty fun and satisfying.

Unfortunately, I got a little carried away and started paying too much attention to these other great/fun/rewarding things and not enough to the work that was assigned to me. My boss had to have a talk with me and set me straight.

It is funny about how these things come back around. I have also seen this among some of my co-workers. Some work is just so much fun to do, and some of it is boring and tedious. At work, you might begin to do the fun stuff and let the boring stuff slip a little. Maybe too much.

If you are thinking “this doesn’t happen to me”, then answer this question: What have you done for paperwork this month? Written any docs lately? How about your TFS check-in comments? I’m not trying to name names, because I’m certain we all do this sometimes.

My point is this: at work, you still get a grade. It just isn’t on a report card. It is good to be an overachiever, but don’t lose sight of how it affects your grade. Do your homework, study for the test, be thorough. Go for that “A”.

Posted in Career, Education, Lessons Learned, Professionalism | Tagged | Leave a comment