Triage: picking your battles vs being lazy

A few years back, I joined a team that had a lot of problems. I mean A LOT of problems. To give you an idea: the team wasn’t even fixing the problems. They retreated to the point of documenting “the order in which to reboot servers, when things crashed”. Their stuff actually broke that often, and they felt it was the best solution they could offer. I was brought onto the team to pull them out of the well that they had fallen into.

For the first week, I observed how support was handling things. Each morning I would report to my boss about the top issues that I faced yesterday, and which ones I fixed. After a few weeks, I managed to fix a pretty popular complaint and my morning report was pretty upbeat, “Yesterday, I deployed a fix which should resolve the top two bugs in prod”. The head of support asked if that meant all of the bugs were fixed in that system. I told her that there were still a few bugs in there, but for now, I was avoiding their gaze. Everyone in the meeting got quiet and seemed perplexed about my statement. It took me a second to remember, there is a homonym for “gaze”. I continued, “You know, like vampires! If you look them in the eyes, then they got you, and you have to deal with them”. Half of the people understood and I guess the other half thought I was avoiding gay vampire bugs or something. So I reworded my response, “It is good enough for now”.

The head of support was not satisfied by this and said something implying that I was being lazy or doing a half-job or something. I felt like it was out-of-line, because I just squelched the top two issues for the project and she was disappointed that I didn’t build Rome in a day. It hit a nerve, but whenever someone questions my integrity, I always give it some consideration. I know I have an ego and sometimes I need to confirm that my ego is not driving my decisions. Nope. This time, it was a business decision.

Too much work
When I get some work to do, I get in there and just knock it out. When I get too much work to do, I start by prioritizing what needs to be done first and which can wait. However, when I am avalanched by an unmanageable amount of work, I know that I need to keep my head together and make some important strategic choices. My great mentor, Robert Baldyga taught me this crucial lesson:
When everything catches on fire and people are screaming at you to fix it NOW, you need to do the following:

    1. Don’t succumb to the chaos or panic. People are counting on you to keep a clear mind and make good decisions. Bad decisions will waste your precious time or make things worse. So make sure you know the difference between a bad decision and a good one. Take a second to make sure, before you act.
    2. Before you make any changes, make sure you can undo them.
    3. Before you start making changes, understand the problem.
    4. If you don’t have enough information to understand the problem, then your first priority must be to gather the information to understand the problem.

Of course, there are plenty of things you can do before a problem occurs, to make it easier to isolate the cause. However, if you are fixing someone else’s problem, then you are on your own. It is really hard to put a timeline or ETA on the fix. You simply have to do your best to get things headed in the right direction and chip-away at the problem until it is all-done.

My boss (at this place) understood what I was doing and was satisfied with my strategy and progress. Eventually, I was able to resolve all of the problems and take the pressure off of the support team. Of course, it messed up all that documentation about “what order to reboot the servers”, because you didn’t need to reboot stuff anymore. That was a whole different battle to pick, and another story for another day.

Posted in Professionalism | Tagged | Leave a comment

Knowing what you don’t know

A few weeks ago, I had the joy of experiencing a new twist on an old problem: gathering requirements.  I have a small project which has missed a deadline or two.  Naturally, I’ve heard question a few times: “when will it be done?”  I am a naturally optimistic person. Therefore, I am always inclined to say something like, “this Friday, for sure”, but that is what I have said for the past two weeks, and it didn’t happen.  So, based on previous experience, is it realistic to give the same answer again?  How many times do I need to be wrong before I ask the question-behind-the-question: “why don’t I know when it will be done?”

If you are on a project and you don’t know when it will be done, then there clearly is a problem. If you lack experience, that is one thing, but if you have a few decades doing this sort of thing, and you usually meet your deadlines, then you need to take a closer look at the deadlines that you miss.  Missed deadlines are usually the result of one of these:

  • Scope creep – Arbitrary amounts of time get added into your schedule, thereby making it unpredictable
  • Unrealistic expectations – If you are done and it is good, but can’t get approval, that is a management challenge
  • Aggressive timeline – If somebody needs it yesterday, then that probably isn’t going to happen
  • Inaccurate estimation – Estimation is a science, if you pick arbitrary dates, with nothing to substantiate them, then you are just gambling and hoping you get lucky
  • Lack of requirements – If you don’t know what “done” means, then how can you expect to put a number on it?

Of these five, they all tend to be connected to the last one.  If you don’t have clear requirements and expectations, then the whole process tends to get really “free and loose”.  The only way to firm it up, is to clarify/improve/re-gather your requirements.  The bad news is: that takes time and skill.  If your project is already behind, people don’t like to hear that the solution is to gather requirements (weeks after it was supposed to be done, instead of weeks before starting).

The problem that I ran into, had a lot more to do with us not-knowing that we needed more requirements.  It sounds funny to say that “we didn’t know what we didn’t know”. Like duh.  I guess it is more a matter that we thought we knew the requirements, but it has become more-clear, that we do not know the requirements.

How do you not know?

Here is the scenario:   A year ago, we changed technology vendors for some tech devices.  We already had users, with experience using the old devices, so everybody already knew what the new devices were supposed to do (obviously, the same as the old devices, but better).  We observed how everyone used the old devices, and even tried it a few times for ourselves.  We gathered a bunch of requirements and bingo.  Scope, plan, timeline; it was all predictable and went like clock-work.

This time, we are moving to the next generation of technology (like moving from a wall-phone to smart phone, or PC to tablet, or DVD to Netflix, you get the picture).  So, our users don’t know what will be different and neither do we.  I wrote the new software to do the old stuff, but it doesn’t match-up.  It is a square peg for a round hole.  So now what?

Now that you know what you didn’t know

It hurts to say it, but we need to bite the bullet and face reality

  • Re-gather requirements
  • Acknowledge that it is unlikely that we will gather all requirements this time either
  • Make a new plan to take-into-account that we need some users to help figure out what they will be doing from now-on
  • Split the timeline up, into multiple iterations of build, observe, repeat
  • Set expectations accordingly

Sometimes you just can’t know everything.  When you run into a situation like this, you need to make big changes to your process and accommodate the process of learning and gathering requirements as you acquire them.  It is sometimes called “being on the bleeding edge”.  It is hectic, but it can be managed if you acknowledge your situation and change your processes to meet the needs of the project.


Posted in Lessons Learned, Requirements | Tagged , , , , | Leave a comment

Knowing something you don’t

Now that I’m all grown up, I can actually laugh when someone says (in a condescending manner) that they know something that you don’t. Sometimes they may even say it with a tone of disbelief: “You don’t know that!!? I thought everyone knew that!!!!”. It used to make me feel a little defensive. Now it is just funny.

My kids are starting to go off to college now, to study engineering. They sometimes express to me some fears that this stuff is too hard, or there is just too much to know. They are worried that they won’t pick it all up in four years, and they will get to their first job and the CEO of the company will berate them, because they don’t actually know everything yet, “Whaaaatttttt!!!??? You don’t know the rotational velocities and masses of all of the moons of Jupiter?!!!! Outrageous! What rock did you crawl out from underneath?!”. I suppose it is funny to imagine this conversation with british accents and shakesperian exaggerated interjections. Hahaha. Sorry, it is funny because I’ve had dreams like this.

Anyway, my advice to my kids is this: being an engineer doesn’t mean that you know everything. It just means that you are willing and able to square-off-with difficult challenges, and be resourceful. You don’t give-up easily, and if you don’t know an answer, you have a good idea where to find it, or who to ask. It sounds too easy right? Especially if you are still in college, facing exams. Well, the exams in college are a little more intense than your average workday as an engineer. However, once you pass those exams and get to a job, you will have proven that you are ready to face new challenges.

My main point is that you can’t know it all. If you do know it, that is great. If you don’t, then the next best thing is a willingness to learn or the resourcefulness to find the answer. So I don’t mind if you know something that I don’t. If I want to know it, eventually, I will.

Posted in Education, IT Psychology, Uncategorized | Tagged , , , | Leave a comment

Troubleshooting, and 3rd parties

Working on a project, with another vendor, is not always the walk-through-the-park that one might expect.  I’ve done this a few times in my career. Like with anything, there is some good and some bad.

When things are going good, everything is cool.  Hopefully, things start-out that way, so you can build a positive foundation and establish good communication and rapport.  Eventually, something is going to go wrong.  After all, this is IT you know.  Stuff breaks.  We solve it.  Repeat.

Depending on how involved, or on-the-ball your vendor contact is, at some point, you are going to experience some kind of problem where you won’t be able to tell if the problem is yours, or the vendors.  There are 4 paths to take, but choose wisely, because this quickly turns into a “choose your own adventure” type story.

When you can’t tell if the problem is on your-side or theirs, do you:

  • Tell the boss that the problem is theirs.   Wait for one of these responses:
    • They do the research and determine the truth
      • If the problem is theirs, they fix it and you are all set
      • If the problem is yours, they tell you how to fix it and save you a big bunch of time
    • They feel lazy or optimistic or whatever, and don’t do any research, and claim that they did and that the problem is not on their end, but don’t give you any hints about how to solve it
      • You believe them. Continue to the next option.
      • You don’t believe them (you still want them to do the research or something). Go back to the start. Repeat.
  • You do the research to determine the root cause
    • The problem is yours and you fix it
    • The problem is theirs and you can now prove it
      • They believe you. You work together and solve the problem.
      • They don’t believe you. You show them how you came to that conclusion and convince them.
      • You decide to be jerk about it and embarrass them.  You burn a bridge
      • They decide to be jerks about it and you must make them eat it now

So, from this list, you can see that some of these result in a solution.  On the other hand, some of them take you down a dead-end road, which makes it harder to fix things next time.  Some of them don’t really seem to lead anywhere.  Eventually, you need to choose one of the paths that results in an outcome (unless you choose to give-up and leave things broken or abandon your project or something).

When you look at it like this, it becomes pretty obvious that the only ones that lead to success are the ones that require someone to do some work.  I once heard a saying about what to do if you want something done right.  Have you heard that saying?

And of course, don’t forget that there will probably be a next-time.  When it happens, people will expect the same thing that happened last time:

  • You did your homework and proved that you know your stuff. Next time, you will have much more credibility. If you say the problem is on the vendor’s side, then the vendor starts looking, because you have a reputation for being right and not wasting people’s time.
  • Last time you showed that you were a lazy putz who expects others to do your work.  You waste people’s time like it is no big deal. The vendor may or may-not work harder than you, depending on incentives.  Your boss will probably catch-on.
  • You are the kind of dirtbag who blames others, even before you know whose fault it is.  The vendor will focus on covering his backside and gathering evidence against you, rather than focusing on fixing stuff.

Bottom line: Be a classy person and do the homework. The respect that you earn from others will pay-off. Also, focus on fixing things and preventing the next problem or detecting it earlier.  Stick with this plan and everyone’s jobs will become easier and you will earn a few hero badges.

Posted in Professionalism | Tagged , , , , , | Leave a comment

Embracing pain

Back in the 80’s there was an exercise slogan “no pain, no gain”. The original concept was simply to work-out with an intensity and duration so you began to feel discomfort. If you weren’t experiencing some discomfort during your work-out you probably weren’t pushing yourself enough to result in growth and progress. The marketing campaign was eventually abandoned, because too many people were misinterpreting the message, and were simply injuring themselves during a workout, and often had to abandon their workouts altogether.

In the software development and IT (generally), there are different kinds of pain too. Some of them are bad and you want to avoid them: blowing a deadline, crashing a prod server, security breech …those are bad.

In contrast, there are other types of pain, which show that you are pushing the envelope and growing.

  • Blowing a checkpoint early in a project, and then making adjustments/corrections to get things back on-track
  • Crashing a test or pre-prod server instead of a prod server
  • Struggling to get a prototype to work correctly
  • Abandoning a bad design
  • Refactoring part of a project
  • Finding a “show stopper” bug in QA or Pre-Prod

Each of these are the right kinds of mistakes to make. Some of these show that your test/QA/pre-prod environments are providing a good value, or that you are taking some risks, or cutting losses in a way that will protect the majority of your project.

Even if you do experience the bad kind of pain, there are still benefits to gain: They say you can learn more from your mistakes than from your successes. Of course, if you dodge the responsibility or outright deny that there is/was a problem, then you will probably learn nothing. But if you address the problem, discover the root-causes and take steps to prevent similar problems the future, then you have probably gained some real ROI from something bad.

It comes down to attitude. Don’t avoid the discomfort that goes-along-with exerting yourself. Push yourself a little harder. Encourage that growth. Earn your muscles.

Posted in Lessons Learned | Tagged , , , , | Leave a comment

Why communication is hard, and what to do about it

I know IT folks sometimes seem under-socialized and (to varying degrees) awkward. However, they don’t have the market-cornered on communication problems. Gosh, just turn on any TV sitcom and you will probably see a people who are having conflicts because they are miscommunicating and misunderstanding each-other. I even recall feuds between family members because “she said this to her, which was totally out-of-line”, but later they apologized because it was all just a big misunderstanding. Right? You’ve never seen that before, have you? LOL. Maybe every-day, right?

IT folks are very much like everyone else, on the inside. Any person’s primary motivation for communicating is to benefit themselves. Even if they are doing it to help others, the primary motivation is to advance their own initiative or current task. Teamwork is good too, but I like my team better than any other team. You get the pattern. It is human nature.

So when things go wrong with communications, the problem usually stems from two things.

  1. People tend to think about themselves. When you are communicating with someone else, that person is trying to think about what they need or how you are going to help them. If your side of the conversation is strictly geared towards helping yourself, the other person might have trouble relating-to what you are trying to communicate. Sometimes this can become quite baffling for folks. Like if I say I want a burger and your response is “I want a taco”, I might think, “I don’t want a taco. Why is he talking about tacos? WTH”
  2. People tend to expect from others, what they expect from themselves. On one occasion, I recall dealing with a person who was convinced that I was trying to subvert him in some way. Which is completely against my nature. I just don’t operate like that, and I was astounded that the other person would even suspect that I could be capable of such guile. It took me a while to remember this point: the other person probably had a subversive nature and was simply expecting me to act the same. It was astounding (to that person) to think that I could be genuinely offering to help without some spectacular agenda for myself. After that exchange I was very careful about how I interacted with that person. VERY careful.

In the end, it comes down to empathy. If you spend your time thinking about yourself, then you will have trouble motivating others. Once you have empathy, and understand what motivates others, you can begin to work on agendas that will benefit both of you.

Of course, to have empathy means you need to understand others. It only works if you know what makes other people tick. So, you need to start by getting to know others. You probably need to ask questions and listen to the answers. Think about what the other person said, and why. Eventually, you may need to read a little more about what makes others tick. Mostly, because not-everyone is wired the same and therefore is not motivated by the same things.

Most people really do want to be understood and are glad to work with you. If not, you will notice when a person’s actions and words don’t line up. That is a discussion for another time.

Posted in IT Psychology, Professionalism, Team | Tagged , , | Leave a comment

Why bother with a plan

I have met plenty of cowboys in my day. I’ve written about this topic several times. One thing that cowboys hate, is writing-up a plan. It makes me think of the times when I MAKE my kids do homework. Their effort to avoid doing homework, usually is greater than the homework itself.

It probably isn’t a surprise that I’ve also worked with people who were not cowboys, but still refused to write-down their plans. Maybe you are you familiar with the saying “failing to plan is planning to fail”. It is a catchy saying, but how accurate could it be? After all, if it was true, a cowboy would always fail. Right? However, cowboys don’t seem to constantly fail. Perhaps this is iron-clad evidence which contradicts this catchy slogan.

To plan, or not to plan

So-many people do-not believe in plans because they have operated without a plan and have experienced varying degrees of success. Therefore, to them, a plan is just some BS busy-work that has low ROI. I’ve heard the same about Algebra homework. (deep sigh, and eye-roll).

Well, good news/bad news: This negative rhetoric may be accurate under some circumstances, but I cannot name very-many times where [having a plan] was actually a complete waste of time. So, let’s start with some conditions where [having a plan] will probably end-up being a total waste of time.

  • Very small tasks
  • Busy-work
  • Tasks where creativity is the fundamental goal
  • Tasks which only involve one, maybe two people
  • Tasks where the outcome is unclear and it is impractical to plan “research” or “discovery”
  • Tasks which require a cowboy (emergency outage in production, or a panicked fix)

In contrast, I guess you could say, nearly every other condition is a time when you want to have a plan. For example:

  • Large projects (or even medium projects), which require coordination and timing
  • Projects which are being watched by important people
  • Projects with a significant budget and expected ROI
  • Projects which should be predictable
  • Tasks which require surgical precision and zero margin for error

For work like this, there really is no excuse for not having a plan. If you’ve done something before, you know what you are doing, you can predict how things will go, then make a plan to show that you really do know what you are doing. As you hit your marks, it will increase confidence in your process and skills. Clearly you must know what you are doing if this is so predictable, and you predicted it so well.

Of course, there will always be some surprises on any project, so document your expectations about those too. If you encounter more surprises than you anticipated, then adjust your plan and adjust your work to bring the project back to the plan. It might sound inconceivable to predict surprises, but with some experience, you will come to expect surprises and you will prepare to 1) recognize them early, and 2) react to them and handle them so they don’t derail your plan or timeline.

Bottom line, if you know what you are doing AND you are not a cowboy, then cowboy-up: write-down your sweet plan and sport it like the magnificent elitist that you and I know you are.

Posted in Lessons Learned, Professionalism | Tagged , , , | Leave a comment