Refactoring code that is too big

“If your code is so big that you need Regions to collapse areas of it, then your code is probably too big and you should consider refactoring it”.

I’m sure everyone has encountered a programming task, where someone wrote a huge page or library or function, etc. and you had to maintain it. This kind-of feels like you’ve been dropped in the ocean at night. You can’t see the shore and it is hard to determine which way to swim. Even choosing to swim west, might be pointless if there is a current going east.

Fundamentally, the concept of refactoring large files into smaller files seems logical. So, when I see a really large file or function, my first inclination is to dive-in and refactor it.

I recall one particular time, when I took a large class/file and started refactoring it. I cut large chunks of code from the file and pasted it into new files and connected them back together by making calls between them. As I did this, I found that some of the code was stateful and relied on local (private) variables and properties. So I had to pass those into the functions that I just moved. As I did this, I realized that it was all becoming tightly coupled.

Once I was done, I evaluated what I had just done: refactoring this class/file took a bunch of time, I changed a lot of code (which added risk) and the finished work was tightly-coupled (not ideal). In the end, the refactored code seemed to work fine. So I guess it was maybe worth the risk. I still wasn’t sure how much value I had actually provided. It could be argued that I added or subtracted value.

So essentially, what I found is that refactoring was not so bad in functional (modular or non-OO programming). You just cut the code in half or in tenths, and you are done. However, in OO, you can’t just split up the code in “an object”. Some of the fundamental nature of OO programming is encapsulation. Encapsulation basically means that your object needs to contain all of its own code and state (variables). Splitting it up is a clear violation of OO principals, unless you are pulling the them into another class, which is independent of the other classes (loosely bound).

Sometimes an object wraps something really big. Sometimes, you can’t just refactor it and make it smaller. And sometimes, when you refactor it anyway, you end up with utility libraries and static classes and functions, that merely spread the mess around and it is no-longer very true to OO design.

Another down-side to this style of refactoring is when you refactor without consolidating any code. In that case, your code seems smaller because you have seventy files that are ten lines long, but still have seven thousand lines of code. When you step through the code, you start to get lost in the seventy files that your logic traverses. Good luck keeping track of that. I don’t really believe that it is an improvement of any kind.

The most certain way that you can tell that you have over-factored a function/class is when you have subs/functions where you are passing dozens of parameters into it, or if you have to use ByRef variables (you change the state of a param, rather than returning it as a result from a function). This is terrible and is evidence that your code and logic is fragmented and will be more-difficult to maintain now.

Most developers who encounter this kind of problem pick one answer: “This should be rewritten”. However, all truly experienced programmers will tell you that a rewrite is very unlikely to produce a better outcome. It will just burn through a bunch of time & money.

My point is this: some blocks of code are just big. Before you spend months working on the perfect refactor, think about the ROI that you will get. Just because “the experts are saying it”, doesn’t mean it actually makes sense or is practical in most circumstances. Please get a grip and show some judicious discernment. Then discipline yourself to pick the low-hanging fruit (if there is any). Get that done and then step away from it, and call it a day.

You don’t have to solve every problem. Pick your battles. Know when to be elite and when to be efficient.

Posted in Lessons Learned, Optimization, Professionalism, Programming | Tagged | Leave a comment

How your processes save your backside

When you have a well-defined process, it can be tedious at times. Sometimes you even want to throw unpleasant terms on it, like “bureaucracy”.

Bureaucracy sounds nasty. I’m trying to think of someone who could make it sound palatable. Maybe Mr. Rogers or Martha Stewart or Samuel L Jackson could make it sound nice, or at least cool. Well, stand-back, because I’m about to try to add my name to that list. Let’s see if I can sell you on a little bureaucracy.

Have you ever asked someone at work, to do something? Maybe your colleague even said they already completed your request, “yep, it’s done. You’re welcome”, and then it turned out that it actually WAS NOT done. So you asked them what happened and suddenly that person has some kind of memory loss. “I didn’t say that. I don’t recall agreeing to that. You never asked me to do that”. You just got hung-out-to-dry. Maybe you are even a little mad about it. I know I would be.

I must confess, I’ve even done this to myself a few times. Please believe me: I was not trying to deceive or trick myself. I just got rushed to do a bunch of important things and I overlooked one or two important steps and didn’t realize it until much later. I was pretty mad at myself and maybe even vowed to never let it happen again.

I think we can agree, these are two types of instances where “following-a-plan” could have helped. Here is how things would have gone differently.

Scenario 1:
Me: Hello fellow-co-worker, could you please accomplish this very time-sensitive task.
Co-worker: Absolutely. I am glad to participate.
Me: (later, via email). Here is the plan. It shows what I am doing and what you are doing. It also has durations, and an ETA (for your tasks and mine). Could you please confirm that this is correct?
* If the co-worker doesn’t respond or can’t commit, I retry, but CC a manager or two, just to ensure that I am okay to assign work to that person (and maybe I am even applying a little muscle).
Co-worker: Yes
Me: (later, following up) Hey, I just wanted to touch-base. I’m sure you have it handled. I just wanted to see if you need anything from me, or if anything is in-your-way.
Co-worker: (who nearly forgot) Yes, I am about to start on it right now.

The take-away from this is:
• Confirm the commitment from each team member
• Ensure everyone is aware of the timeline, tasks, work distribution
• Follow-up, with plenty of time before the deadline. So, if the person forgot, there is still time to complete the task on-time.
• Keep following-up (very politely), until each task is complete
• CC a manager if you need a little more muscle, but don’t over-do it. Keep things friendly.

Scenario 2:
1. Make a plan for myself
2. Determine what I am doing each day
3. Make it into a checklist
4. If I discover a new task, add it to the list and the timeline
5. If the plan shifts, then shift the plan, so I can tell if my plan is still feasible

The take-away from this is:
• Work a plan, so you don’t forget stuff
• When plans change, it is easier to make realistic decisions about timelines, if everything is in-front-of-you at-once

If your plans always work perfectly, this might seem like a waste-of-time. However, if you ever want to scale beyond a team of one (yourself). You need to have a written plan, so your teammates can prepare and coordinate. Also, some people are just hopeless optimists (or way-over-booked) and have trouble being realistic about timelines and ability to meet deadlines. A written plan is just rigid (and honest) enough to bring the needed amount of realism to the project timeline.

Even if you scoff at this type of process, I would say, give it a try once or twice. See if it made anything better. It is a trade-off and it is a new skill. It might even take you one step closer to elitism. Welcome.

Posted in IT Psychology, Lessons Learned, Professionalism, Team | Tagged , , | Leave a comment

My most epic project

The biggest project of my career was a Y2k project for a state agency. The project was a huge challenge by itself. To make things more interesting, it had a fixed timeline (Y2k) and a customer who didn’t typically do well with fixed timelines. That combination is a near guarantee to make things challenging at times. It was a pretty big opportunity for me and I was optimistic about it, in spite of the obvious challenges.

On day 1, we started by formulating our strategy. We needed a set of ground-rules that would guide our decisions, to ensure success. This was the crux of our strategy:

  1. Contain the scope (and be vigilant about it)
  2. Build a solid proof-of-concept (program/prototype)
  3. Focus on architectural models that would prove each of the elements of the final product (eliminating unknowns and accelerating development)
  4. Train the staff
  5. Cooperate
  6. Commit to improvement (within reason)

We figured that if we were committed to those guidelines, then our development would accelerate (meaning: the rate of completing work would steadily increase), the timeline would not be able to outrun us, and the chance for surprises would be minimized. The timeline would be predictable. We could do it, on-time.

We started-off strong with all six of these. Of course, progress always seems to start-off a little slow during the first few weeks of any project. It is a real gut-check until you pick up steam. I was only 30 years old, and I got my first 5 gray hairs during those 3 months.

During that time, my dev lead expressed concern about making the timeline. The task was gigantic and he couldn’t imagine how we could accomplish so much in such a short time. I explained how it would all work (a few times) but he still had concerns. He had never been part of anything like this (and technically, neither had I). He wasn’t as optimistic as me. In fact, you could say he even sounded somewhat pessimistic. I figured, he probably just needed some solid reassurance from me.

After trying a few times, to reassure him, I realized that there wasn’t anything that can be said to reassure a young developer, about a project as big as ours. So, I changed my pitch: “If you honestly don’t think that we can complete this, then we really only have two options: 1) Find new jobs right away. Or 2) Work as hard as we can and do our best, and at the end of it all, if we come up short, we can rest-assured that nobody else could have done a better job than us and been more successful”. (of course, I was being a little sarcastic with the first option).

He acknowledged that there was really only one reasonable choice: do our best, and leave nothing on the table. So he committed to our endeavor. I did point-out that I was personally determined to do anything necessary (within the boundaries of good ethics) to get the project done on time. We had some great support and resources at our disposal. We were even told that “getting a B” would still pass, but we were going to settle for nothing-less-than an A.

We pressed-on with diligence, constantly checking our quality, progress and vigilantly containing scope. One time, our scope-containment effort even resulted in a rather heated disagreement with a serious player. I was a small dog in a fight, but I treated it like the careers of my team (18 of us) depended on this decision. We stood our ground and worked our resources pretty hard. I used up all of my “get out of jail free” cards that day, but prevailed. The scope was contained and we sprinted towards our due date.

In the end, we finished 2 weeks early (Sep 16, 1999) and the program exceeded the expectations of everyone, for performance and stability. Everyone was astounded. We rolled-out the project to the entire state of Michigan, over the next few months. 14000 state workers picked it up without a hitch. Y2k was a non-event for my team.

Of course, I cannot take sole credit for the win. I had a great PM, excellent dev managers, I had one or two certified geniuses on the team and management was behind us 100% (even covered my backside from that dog fight). We couldn’t have done it without all of them. Each accomplished some legendary feats during the project.

I still feel that the winning ingredient in all of it was the solid foundation (6 guidelines) and the commitment from everyone. At times, we got scared and wavered, but that fear also drove us to accomplish things that (I’ll even admit) were pretty colossal. We were driven and that made all of the difference.

Since that project, I have not faced very many challenges that can compare to that one. I am fortunate to have seen how a great accomplishment can be achieved. I was there and I know what goes into it.

If you ever get offered a chance to accomplish something epic, do a gut-check, be realistic about your own resources. If you go all-in, be ready to back it, all the way. It might be the most legendary win of your life, but only if you apply the effort needed to get that win.

Posted in Career, Lessons Learned, Methodology | Tagged , , , , , , | Leave a comment

Event code: 3008. A configuration error has occurred

This week, we were deploying some new files to our project (running on IIS). After adding the files, the app didn’t run, and we got the following error in the IIS logs:

Event code: 3008
Event message: A configuration error has occurred.

… [snip]

Exception information:
Exception type: ConfigurationErrorsException
Exception message: Could not load file or assembly 'ProjectName_old' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
at System.Web.Configuration.CompilationSection.LoadAssemblyHelper(String assemblyName, Boolean starDirective)
at System.Web.Configuration.CompilationSection.LoadAllAssembliesFromAppDomainBinDirectory()
at System.Web.Configuration.CompilationSection.LoadAssembly(AssemblyInfo ai)

The error looked like maybe there was some kind of config error. Except we didn’t make any changes to the config. We just deployed a new version of a DLL in the /bin folder.

Something about it sounded familiar. I’ve seen this misleading error a dozen times before.

The problem was caused by a simple mistake on the part of the server admin. He renamed old dll before he put the new one on the server. Unfortunately, the way IIS works, it will read all DLL files in the /bin folder and pull them into memory. Unfortunately, this will cause some serious conflicts if you have two copies of the same DLL, but with different names. IIS won’t know how to handle the conflict.

It would be great if the error message said something accurate like “DLL Conflict: two DLLs contain the same classes/namespaces and are conflicting with each other. The files are named Copy1.dll and Copy2.dll” but nooooooo. Instead it gives this dandy error message.

So, hopefully you know better now. If you ever deploy content to an IIS server, please just zip the whole folder and store that backup somewhere (not in the wwwroot, etc). Then delete the contents of the folder, and then put the new content there. It is the safest way to avoid conflicts, while preserving your options to restore everything, in the event of a problem.

Posted in Errors, Lessons Learned, IT Horror Stories | Tagged | Leave a comment

Underachieving, overachieving

When I was in high-school, I had an interesting discussion with a guidance counselor. I was taking some pretty hard classes in school and I was getting so-so grades (B or C, not A). His advice was to take easier classes. I told him that I really liked to be challenged and I don’t think I would get better grades if I had easier classes. I confessed that the problem was that I was just lazy.

He persisted that my grades were going to be used as more of an indication of my intellect, and the challenge/difficulty-level of my classes would be overlooked. Basically, if I wanted to go to college, I needed good grades, not a good education. He told me that my grades made me look like an under-achiever and that was really bad.

After high school, I went into the US Army for a while and did okay. Some things, I did great but others, not so much. A few people gave me great advice about self-improvement, but I had trouble embracing it. I must confess that I was still a little lazy and I moved a little faster when a sergeant or officer was um, “motivating” me.

In college, things turned around. My junior year at U-Mich, my classes got pretty hard and the dean of engineering told me I had to bring up my grades or they would kick me out. I couldn’t believe it. What a wake-up call. I cancelled a lot of my leisure activities and replaced that time with studying. I even started studying things that I didn’t have to study. I started learning some really cool stuff and wanted to apply it.

After graduating, I kept applying this idea: I looked for things to do or learn, which went beyond what was asked of me. At work, I started looking for new ideas for learning and growth. I started looking for the things that needed to get done, but weren’t, or places where I could apply some newly-gained knowledge, to make something work-better or simplify something. It was pretty fun and satisfying.

Unfortunately, I got a little carried away and started paying too much attention to these other great/fun/rewarding things and not enough to the work that was assigned to me. My boss had to have a talk with me and set me straight.

It is funny about how these things come back around. I have also seen this among some of my co-workers. Some work is just so much fun to do, and some of it is boring and tedious. At work, you might begin to do the fun stuff and let the boring stuff slip a little. Maybe too much.

If you are thinking “this doesn’t happen to me”, then answer this question: What have you done for paperwork this month? Written any docs lately? How about your TFS check-in comments? I’m not trying to name names, because I’m certain we all do this sometimes.

My point is this: at work, you still get a grade. It just isn’t on a report card. It is good to be an overachiever, but don’t lose sight of how it affects your grade. Do your homework, study for the test, be thorough. Go for that “A”.

Posted in Career, Education, Lessons Learned, Professionalism | Tagged | Leave a comment

EMV transactions in the iSC480

My boss’s boss was informed that the credit card laws were changing and therefore, we needed to deploy new credit card readers and change our program to interface with the new readers.

I had already gotten started with development and was interfacing with the iSC480 very well (see previous article for more details).  My one remaining task was to detect when a user was trying to “swipe” an EMV card.  The EMV card must be inserted and not swiped.

The documentation said several times, “If the swiped card is an EMV card then the cardholder will be prompted to insert the card” (Dev Guide, sec 3_6_18, variable 413).  Well, that never happened.  So I had to write that code myself.

It was pretty easy.  I just had to determine if a card is an EMV card.  Track 2 contains a service code (positions 21-24).  If position 21 contains a “2” or “6”, then the card is an EMV card and must be inserted.  I detected it and changed the screen message and I was back on track again.  No biggie.

When I inserted a EMV card, nothing happened.  The vendor said that the device should start automatically running through screens.  I never saw any of that.  Finally, my team had a phone conference with a tech guy from the vendor.  He pointed out a few VERY important pieces that were necessary to process EMV transactions:

1. EMV must be turned on in the device.  By default, it is NOT turned on.  So, you will never get anywhere until you know this critical step.  To turn on EMV in the device, you need to send a “M60” command to change setting “19” to a “1” instead of the default of “0”.  The support guy was surprised that I hadn’t read that tidbit because it was “easy to find” somewhere in the 826-page-long PDF document (Dev Guide).  I eventually found it under the section titled “Support for Voice Referral for EMV”.  I guess they figured the whole “voice referral” part would not dissuade me.  They were mistaken.

2. Once the user inserts the card, a 09 command is received (via the API). At this point, you (the programmer) needs to send the following:
– M14 (use transaction type 01 = Sale)
– M13 (Set amount, without the decimal point)
– The device will respond with a 33.02 message
– M04 (set the transaction to “B” for credit or “A” for debit, etc)
– M13 (set the amount again, for some reason)

Then, the device will take-off and do all of the EMV magic.  The user will have to do some actions on the screen.  These actions will vary, based on the settings inside of the EMV chip in the card.

3. When the user completes all of the required input, the device will send a 33.03 message asking you (the programmer/program) to do a credit auth.  You need to respond with a M22.04 message.

4. The device will ask the user to eject the card.  The device might ask the user to sign.

5. When the user is done signing, the device will not give any indication that the user is done signing.  That is of course, unless you already knew that you had to set variable “0009” to “1” and you have already done it, before you started any of this.   If you already took care of this, the device will send a “20.0” message to indicate that the signature is done. Otherwise, you could always set up a polling loop (timer thread) to check if a signature message has been recently received.

6. Done.

So, as long as you know all of these little tricks, the EMV stuff is pretty easy to deal-with in the iSC480.  Otherwise, the EMV stuff may not seem very intuitive and it will have you scratching your head for a while.
I was disappointed that, Goog/Bing didn’t have any info about it. I guess people aren’t supposed to talk about the SDK. I have been intentionally vague in this article, to avoid heat. So, if this article didn’t make any sense, it is probably because you don’t already have the SDK.

Final note: Snaps to the tech support at the vendor. I always hate calling for tech support. It feels the same as stopping at a gas station and asking for directions. *ugh*. However, in this case I wish I hadn’t burned two weeks before I called. The support folks at the vendor, got me back-on-track quickly. The phone call and solution took like, 5 minutes.

Hopefully, you found this guide on the web and it saved you a lot of headaches.

Posted in Programming | Tagged | Leave a comment

Training your manager

Disclaimer: This is not about my current workplace. This is about all workplaces.

Whether you are a team lead, software architect, PM or a junior developer, you count on your team for stuff. Maybe if you are a one-man-team, you do this to a lesser-degree, but you still would have to count on yourself to fill those roles.

Each of the roles on a team, are there because they have a function in the process of writing software. Everybody has a job to do. If every job is done well, the whole thing works like a charm.

Believe it or not, management also has a job to do. I’ve met people who honestly thought the job of management was simply to watch you work and badger you. I’ve known others who thought the job of management was to set up snares to catch you when you made a mistake and then make you pay for it. I’m inclined to think that those people just didn’t have a realistic understanding of things. Maybe they were pessimistic or obtuse, or maybe just downright paranoid.

The really good managers are always there to help and you can see the value they contribute. For example, a good manager will do the following:

  • Ask if you need anything to do your job. If you need something, they acquire it for you or help you acquire it. (within reason)
  • Attend meetings for you and handle issues, so you can focus on your work.
  • They keep you informed, so you understand the big picture and see how you are contributing to the success of your organization.
  • They plan ahead. Short term plans are very detailed. Long term plans are more vague, but become more detailed as they approach.
  • They work with you to determine solutions and estimates, and technologies.
  • They do more asking than telling.
  • They encourage you to think about how you can grow your organization and yourself.
  • They get you the recognition you deserve.

Of course, not all managers are good managers. The career goal of a manager is to make other management believe he is doing a good job. That is how you get advancement. However, this is easily confused with the higher goal, of making management believe his people are doing a good job. Note the difference “I am doing a good job (regardless of my people)” vs “My people are doing a good job (therefore, I must be doing a good job)”.

Not all managers are born with the innate ability to recognize this higher goal, but that is no reason to distress. If you find yourself in the hands of a manager who is not taking care of you, you just need to work on that person, like you work on any of your other skills. Managers can be trained, they just need to see the carrot.

If your manager has some room-for-improvement, try asking him do to the stuff that you need him to do. Help him understand how this helps you and moreover, helps him.

  • If he is not giving you recognition, try showing him how this is done. Write a weekly document that highlights a few things
    • Accomplishments for the week
    • Plans for next week
    • Roadblocks or resources that you need
  • If you deliver something really valuable (like a weekly document highlighting accomplishments) and the manager says you don’t need to do this. Tell him that you like to do this and you know of other managers who ask for this kind of thing. Therefore, you feel obliged to do this yourself. Then mention how it must make his job easier, because his manager must need this kind of info from him. He can just copy it out of your status report and send it upward.
  • Do your own monthly evaluations and invite your manager to participate. After a few of them, the manager will see the pattern and join-in.
  • Find out what other kinds of paperwork a manager should be doing. Do it for him and give it to him. He will see that it is not that hard (since you were able to do it). Also, modifying an existing document is much easier than starting from scratch.
  • Make the time to talk about progress. Don’t just talk about the current problems, talk about small solutions to the current problems. Build momentum and a “can do” attitude. Turn it into a habit.

If you’ve never done this for a manager, you might be thinking that I must be kidding. It sounds like I’m suggesting that you do your manager’s work and he will get all of the credit for it. Well, you are half right.

This kind of paperwork does of course benefit you too. Visibility is rarely a bad thing for you (unless you have something to hide). Plus, some day, you might take a vacation day/week. That is a great time to pass the baton.

Training a manager might be something you’ve never tried, but like anything, you can learn to do it well, and reap the benefits.

Posted in Career, IT Psychology, Lessons Learned, Professionalism, Team | Tagged , , | Leave a comment