Magnitudes of technical debt

The term “technical debt” really covers a lot of areas of concern. It is not simply relegated to “messy programming”. That is part of it, but there is more to it.

Like with many unpleasant things, the impact of technical debt, can be measured in magnitudes. Some kinds of technical debt are easy to deal-with and some are monumental. When a developer says “let’s resolve the technical debt around here”, be careful about writing a blank check.

Like with all development efforts, it is a good idea to “contain your scope”. You will generally maximize ROI and build momentum when you start with the low-hanging fruit and avoid items of high impact.

Low impact tech-debt (quick ROI) examples:

  • Compiler build warnings – These are usually easy to resolve. Make a habit of resolving these as they appear, or work to eliminate a dozen of these a day/week.
  • Coding conventions violations – When your code is “conventional”, it is easier to maintain. Pick a coding convention and enforce it. Then there won’t be “my code” and “your code”. It will all look identical.
  • Configs – Hard coded settings in a project can be evil. You don’t want to modify your program every time you move it to a different server or file path. That is bananas, and very risky. Make sure you have those settings in config files. Make sure the configs are clean.

Med impact tech-debt examples:

  • Obsolescence, falling behind – Don’t let your program turn into “the next COBOL”. Even if it is actually written COBOL, you can still take periodic steps to keep it updated and modern.
  • Awkward programming styles – Ninja programmers can come up with some impressive stuff, but you don’t want your code written in a manner where only a genius can maintain it. If you haven’t generously applied “the kiss principal”, then you are carrying some tech debt with a heavy burden, and it will cost you.
  • Lack of error handling – If a user complains about your buggy program and you have to ask what they did when an error occurs, then you don’t have adequate error handling. Your error handlers are metal detectors to show you the needles in the haystacks. You don’t ever want to live without them. Be generous with error handlers and logging.
  • Inadequate tracing – See error handlers, and imagine that something went bad, but didn’t result in an error that could be caught. Without a trail of breadcrumbs, it will take a frustrating amount of time to recreate the conditions. Eventually, you will wish you had tracing to tell what was happening.
  • Things like: global variables (a needle that seems to move around in your haystack)
  • DLL hell, GAC hell, SP hell, all of the other hells that come from reuse/sharing of code. – Brochureware always sells the idea of code reuse, like it is so great and never turns your life into a nightmare. If you’ve been around the block, you know better. Caveat emptor.

High impact tech-debt examples:

  • Architecture – If your architecture paints you into a corner, you might be in the worst kind of checkmate. Starting-over, might be your only hope.
  • Bad designs – There are all kinds of ways to design a system, which makes them impossible to maintain or close-enough to it. Untying some knots, can turn into epic tales.
  • Security – Thanks to the internet, security is a big enough topic that it a career path for many people. Dabbling in security is like being your own lawyer. It is not a good place to learn from your mistakes.
  • Platform (OS, data, language) – I’ve always laughed at IT people who warned you about a technology like SQL Server or .NET because “you don’t want to be locked into a single platform”. It is a ridiculous claim and the people who fall for it eventually realize that people don’t switch platforms for any good reason. If you ever do switch platforms, you will never do it twice. Locking into a platform is a way to get stability. Y’know, like a foundation. You don’t just switch it periodically.

I’m not saying you can never tackle the topics on the High list. Sometimes you just need to do those. However, don’t do it on a whim. Rome wasn’t built in a day. Don’t start with Mt Everest.

Most important of all: The time to be the most careful with these things is BEFORE you start programming and not afterwards. Like a foundation, you want to start with a solid one, in a reliable place. Make sure the big items are done correctly by someone who knows their stuff (and not just some silver-tongued sales person or giddy fan-boy).

In the words of Bertrand Russell, “Fools and fanatics are always so certain of themselves, but wiser people so full of doubts.” Measure yourself by those words.

Posted in Architecture, Lessons Learned, Review | Tagged , , | Leave a comment

LLT – M53: unknown product code (add it in ‘LLT.prm’ file)

Once again with the LLT. I just got a new PC and had to reinstall all of my software and re-learn the tricks for connecting LLT, etc.  Btw, the help file for LLT has better directions than the rest of the docs. I wasted an hour trying to figure out how to get the device into LLT mode.  Hint: read the LLT docs, section 6.  Btw. the “up” key is the minus -.

When I tried to connect LLT to my device, I was getting this connection error:

“M53: unknown product code (add it in ‘LLT.prm’ file)”

The LLT.prm file is in the same folder as your LLT.exe.  It has a list of all of the possible devices, but it stops at M52.  The iSC480 is supposed to be M53.  For some reason, in the past 2 years, nobody has managed to update this file.  *ehem*.  So we have to do it ourselves.  No biggie, I guess.

Use notepad.exe to edit the LLT.prm file.  Put “M53: iSC480” on the last line (before the line that says “M??:all products”).  Save, and re-launch LLT.  Since it probably took you five minutes to make the change, your iSC480 has probably restarted, so you need to do that trick to get back into LLT mode. 🙂

Btw, the entry for M53 is optional.  The LLT will connect to your device and work etc. even with this error.  It is really just a warning.

Posted in Errors, Lessons Learned | Tagged , | Leave a comment

LLT_LOAD – RAS Error 633

Today, I needed to upgrade the RBA for my device.  I started by using LLT to set up my connection and make sure everything worked.  I worked-out the configs and was able to get a file list in LLT.  So I was certain that everything was good to go. Configs were right.

When I ran LLT_LOAD.bat (for my device), I was getting LLT “Run-time error ’13’: Type mismatch”, then the LLT_LOAD screen gave me a red error message “!!!!ErrorCode: #### DOWNLOAD FAILED – PPP CONNECTION PROBLEM!!!!”.

The message didn’t really give any useful information, so I read the bat file, to get an idea what was going on.  It referred to a log file, which I found (relative to LLT_LOAD.bat) in ..\log\ResBatch.txt.  In the log file it said the problem was caused by RAS Error 633.  Whatever that means!  Searching Goog and Bing didn’t really help.  I put a few debug/trace statements into the bat file, to see what was going on and tried a few other things.  Eventually I guessed at the problem.

Things that were not wrong:  Device was in LLT mode, device was connected, I know I had my LLT and connection configured properly. You don’t need to run LLT_LOAD as admin.

Solution: it was a bonehead mistake on my part, but easy to make.  The prompt says

“ENTER COM PORT # or [ENTER] for COM1 :”

Since the prompt referred to the COM port using the format “COM1”, I figured it would need my answer (COM3) in the format “COM3”.  Nope.  It just needs the digit “3”.  Once I gave it just the numeric part of the COM port, everything ran like a champ.

I’m now on the new RBA.  Hope I helped you solve things too. Good luck!

Follow-up: the “LLT User Guide.pdf” does talk about RAS 633 in sec 10.3, but it just says to check if your cable is connected and that the USB plug/played.

Posted in Errors, Lessons Learned | Tagged , | Leave a comment

Why management wants more technical debt

*** Disclaimer: this is not about my current project. It is about every project. ***

I think it is safe to say that every program contains some technical debt.  This is because perfection is (somewhat) impractical.  If you actually try to achieve it, you have probably invested more time into the program than is necessary.  It is more practical to achieve “good enough” or “really good”.  Even “excellent” is an achievable goal for some programs, but “perfection”, is more than just lofty.

I’ve already talked about the costs of retaining technical debt and the consequences of accumulating too much of it. So why allow any of it in the first place?

One Man’s Treasure…

Any manager who is not deeply familiar with software development, will not be aware of the value of clean code. When you talk with management about technical debt and ask for time/budget to resolve it, you may often encounter resistance or disbelief. “How did we get technical-debt, in the first place?”. “Do you mean to tell me that the programmers are writing sloppy code?” “The program actually works now, and we are living with it, so maybe this is not really a problem. So, just leave it as-is”.

[Ignoring technical debt] could be equated to: manufacturing a new car and not painting it. Yes, the car runs and there is no rust today. Next week the rust will be minimal, but in two years, it will be falling apart and be un-drivable. It is a very difficult message to convey, and even harder (sometimes) to get management to acknowledge it.

Many teams know that resolving tech-debt, is a feature.  It may sound strange, because your users don’t see it with their eyes. However, like with any feature in a program, you can place it on your priority list, and apply more or less effort to it. You can also postpone it or completely omit it.  It all depends on how much value you choose to place on that feature.

If you choose to place a high-priority on [resolving tech-debt], you will perform reviews and enforce standards for your project:

  • Developer self-reviews and refactoring
  • Peer reviews
  • Formal code reviews
  • Architecture and system reviews
  • Quarterly and Annual upgrades to technology
  • An overall attitude of constant improvement

Of course, don’t forget the most important part of the review process: the resolution.  All of the reviewing in the world, won’t make a bit of difference if you cannot commit-to resolving the problems that are found during these reviews.  It is like seeing a house fire, and saying “someone should put-out that fire” and then walking away, without doing anything to help.

Another Man’s Trash

If you are a developer, you probably read that list of reviews and thought “I’m going to print that list and glue it to the wall in front of every manager.”
If you are a manager, you read that list and thought, “wow, that is a list of increasingly lofty goals.”
If you were a VP, you are still probably thinking “we can live without all that stuff. Cut it from scope.”

Bottom line: in general, many managers believe they add value by squeezing more efficiency out of their people and their department. Some of them cannot resist the temptation to squeeze a timeline too much. Initially, it may seem great that the work got done so quickly. Therefore, upper management may or may-not ask the right questions.

The wrong question is “how did you do it?” Because the answer will probably be “The programmers were slackers. I applied some pressure, and they stopped wasting a bunch of time on unproductive crap”. The manager is really just gaming-the-system, because the corners which were cut are usually the ones which produce a higher-quality product. The more quality you cut, the quicker things seem to get “done”. It doesn’t take much skill to do, and it usually results in some nice short-term rewards. Therefore, the incentive is very tempting.

Cutting-out “tech debt resolution” is a dangerous practice and the victims are entire organizations who are impressed by superficial results without an understanding of the problems lurking beneath and accumulating. By the time the project reaches technical bankruptcy, the ones who were responsible have usually moved-on.

In my next post, I will talk about ways to analyze your work, to assess how much debt you are carrying and the level of effort to resolve the problem.

Posted in Lessons Learned, Methodology | Tagged , , | Leave a comment

Technical debt and interest

In case you are not familiar with the term “technical debt”, it mostly means “messy programming”.  Unfortunately, it is more serious than just ordinary messiness.  It usually happens because a programmer was in a hurry and couldn’t think of a better way to make the program at-that-moment.  Later-on, other programmers will see the code and think “why was this written like that?  It would be much better if…”

Like with financial debt, there is interest. The “interest payment” for technical debt is “paid” by all of the developers who come in contact with the debt-laden code.  Everyone takes a little longer to comprehend the code, or apply a fix without breaking everything.  It slows everyone down, and this is the “interest payment”.  It may not seem like much, but it all adds up.  If the debt is big enough or continues to accumulate, eventually you could spend more time paying interest on your debt than doing actual work.

Have you ever wondered “why does it take these developers so long to complete (seemingly) simple or ordinary tasks?”  This is a pretty good indicator of the amount of technical debt in a program.  Imagine something simple that you might do, like making a PB&J sandwich, except the bread isn’t sliced and you are out of PB and J, and those are sold at different stores.  That is technical debt.  The task would take seconds if everything was arranged for your convenience and speed.

Payment plans

It sounds logical that sending a hundred dollars every month to a creditor, should pay down your debt eventually. However, that is largely dependent on your amount of debt and interest rate. If 9/10 of your payment goes to interest, then your progress is going to be low.  A double payment is going to have a much larger impact, but you might not be able to afford it.

With technical debt, once a developer has waded through a file and negotiated his way through it, he will have a better understanding of the debt itself, because he has paid the interest.  Now he can do his work (the reason he was here in the first place).  There is something more he could do right now.  Since [knowing the problem] is half the battle, he sees the technical debt (paid the interest). He could take a few minutes, or even hours to resolve the tech debt.  Once the debt is paid, the team can enjoy the lasting benefit.

Finding a balance

Of course there are two problems:
1) YMMV –  If the developer suggests that you just rewrite the whole thing, then you are less likely to get a good ROI.  You really need to have a team policy about “how good are we aiming towards”.  Don’t say “perfection” because that means something different to everyone.
2) Knowing when is a good time to fix it – Of course, it seems like a good idea to fix it as-soon-as-you-see-it, but that is not compatible with most timelines.

I’ve had good success with granting developers a budget of 2 hours per week, of tech-debt resolution time.  The time can be applied at the discretion of the developer, but still needs to be documented and tracked.  We don’t want developers getting carried away, after all.  Any tech debt that takes more than 2 hours, goes into TFS first (or other work-tracking / requirements pool) and then it gets more evaluation: consider approaches/alternatives. Management can evaluate it and prioritize it.  Just be careful to not burn up 4 hours to evaluate an issue that can be fixed in 2 hours.  Show some good judgment.

Paying down debt it achievable and the ROI can be substantial.  If you want your developers to get more work done quicker, then plan on making a few “double payments” and get some of that technical debt out of their way.

Posted in IT Psychology, Methodology | Tagged , , | Leave a comment

Technical Bankruptcy

*** Disclaimer: This is not about my current project, it is about every project everywhere ***

Technical-debt is a natural byproduct of software development.  If you don’t recall seeing technical debt in every sw-dev project, it is because most developers prefer to resolve technical debt immediately as it appears.  However, if you have a project with an aggressive timeline, you will see technical debt and you will probably accumulate some of it. This is a natural byproduct of aggressive timelines.  The more aggressive the timeline, the worse the byproduct.

The reason technical debt accumulates is fairly obvious.  It takes time to resolve/remove technical debt.  When developers are pressed-for-time, they have less time to review and refactor their own code. They have to decide between “clean it now” or “put-it-off till later”.  When the program works and it passes testing, only the developer will know the difference.

Too Much Debt

If you’ve never heard of “Technical bankruptcy”, write-it-down.  It is a thing.  You know you have reached the point of Technical Bankruptcy when you realize that you are overcome by an insurmountable amount of technical debt.  You no-longer have the time and resources to resolve the tech debt because you are slowly suffocating under a creeping blanket of support calls/tickets.

If you are not sure if you have reached technical bankruptcy or not, take this quick quiz: Is anyone on your team saying “We would be better-off if we just started over from scratch. Let’s throw it all away and start over”.  When you believe it is better to “start over”, you have officially reached the point of technical bankruptcy.

We’ve all seen it before.  It is pretty bleak.  It is hard to imagine how you got to this place. “How did things get this bad?!”  It is even harder to acknowledge the real cause of this, because everyone ignored the problem and in-some-way allowed, caused or contributed to it.

Turning things around

To avoid technical bankruptcy, you need to actually pay-down your tech debt.  Unfortunately the very-nature of technical debt is that it slows you down and prevents you from developing software effectively.  If you are spending all of your time, dealing with complaints, you have very few options.  If you haven’t already done so, you absolutely must cut-back on other projects and assign a higher priority to the process of paying-down-debt. If I’m making it sound easy, let me correct that notion.  This is going to require a significant effort.

On the other hand, if you think you will do better by starting over, think again.  Before you try it, you really need to take a deep look at yourself and your processes, so you don’t arrive at the same point, over-and-0ver again.

It is a long road back and it begins with things like “living within your means”, being realistic about your ability to produce and maintain software.  Committing to improving your processes and paying attention to quality. This last concept is the most significant one.

It is a lot of work and a lot of commitment, but you can do it. Keep your head above water.  Live within your technical means.  Don’t accumulate technical debt.  Stick with these rules and some day, you won’t have to spend all of your resources, paying interest. Remember: Accumulating debt and never paying it down, will lead to one place: bankruptcy.  Don’t go there.

In my next series of posts, I will be discussing technical debt, why your manager loves it, and some things you can do to monitor and manage it.  So check back next week.

Posted in Lessons Learned, Review | Tagged , , , | Leave a comment

Current Mission Statement

When I read the book “7 habits of highly-effective people”, I remember my reaction to the suggestion about having a “personal mission statement”.  It struck me as being a little silly.  Am I really going to do that with all aspects of my life?  Do I really need that sort of thing? Where would I even start?

Well, at work my attitude is much more focused.  At home I can wing-it, whenever I want, because I am still figuring some things out and sometimes it is more fun to live with fewer boundaries.  Honestly, I spend a lot of my time at home pondering problems from work instead of working-out problems at home. So I’m more inclined to have a plan for work.  Regardless, I apply much more structure to my work goals, regimen and mission statement.

I just thought it would be interesting to some of you to see my current mission statement.

  1. Provide the best value that I can to my employer and customer
  2. Determine what-it-is that the customer wants
  3. Determine what is out-of-scope and counter-productive, regardless of its value
  4. Never allow my ego to get in the way of the customer’s needs
  5. Watch my zones
  6. Determine communication routes and routines
  7. Plan ahead as much as possible, but adjust the plan, as needed
  8. Always have several backup plans
  9. Be transparent
  10. Identify the (increasing) points that the bar can/should be raised-to
  11. Determine the path to accomplish this (increasing improvements)
  12. Be patient and try not to agitate my colleagues, while pursuing these goal
  13. Always do my research before opening my mouth
  14. Prepare proof-of-concepts before discussing ideas for fixes
  15. If something seems strange, find out why it is that way, before proposing to correct it

There are a few more things that I do, but most of my decisions are founded on these items, and I do my best to weigh my choices against this list.

Every month, I review my mission statement and make sure my plans and work are in alignment with it.

Posted in Professionalism | Tagged , , | Leave a comment