A few years back, I joined a team that had a lot of problems. I mean A LOT of problems. To give you an idea: the team wasn’t even fixing the problems. They retreated to the point of documenting “the order in which to reboot servers, when things crashed”. Their stuff actually broke that often, and they felt it was the best solution they could offer. I was brought onto the team to pull them out of the well that they had fallen into.
For the first week, I observed how support was handling things. Each morning I would report to my boss about the top issues that I faced yesterday, and which ones I fixed. After a few weeks, I managed to fix a pretty popular complaint and my morning report was pretty upbeat, “Yesterday, I deployed a fix which should resolve the top two bugs in prod”. The head of support asked if that meant all of the bugs were fixed in that system. I told her that there were still a few bugs in there, but for now, I was avoiding their gaze. Everyone in the meeting got quiet and seemed perplexed about my statement. It took me a second to remember, there is a homonym for “gaze”. I continued, “You know, like vampires! If you look them in the eyes, then they got you, and you have to deal with them”. Half of the people understood and I guess the other half thought I was avoiding gay vampire bugs or something. So I reworded my response, “It is good enough for now”.
The head of support was not satisfied by this and said something implying that I was being lazy or doing a half-job or something. I felt like it was out-of-line, because I just squelched the top two issues for the project and she was disappointed that I didn’t build Rome in a day. It hit a nerve, but whenever someone questions my integrity, I always give it some consideration. I know I have an ego and sometimes I need to confirm that my ego is not driving my decisions. Nope. This time, it was a business decision.
Too much work
When I get some work to do, I get in there and just knock it out. When I get too much work to do, I start by prioritizing what needs to be done first and which can wait. However, when I am avalanched by an unmanageable amount of work, I know that I need to keep my head together and make some important strategic choices. My great mentor, Robert Baldyga taught me this crucial lesson:
When everything catches on fire and people are screaming at you to fix it NOW, you need to do the following:
1. Don’t succumb to the chaos or panic. People are counting on you to keep a clear mind and make good decisions. Bad decisions will waste your precious time or make things worse. So make sure you know the difference between a bad decision and a good one. Take a second to make sure, before you act.
2. Before you make any changes, make sure you can undo them.
3. Before you start making changes, understand the problem.
4. If you don’t have enough information to understand the problem, then your first priority must be to gather the information to understand the problem.
Of course, there are plenty of things you can do before a problem occurs, to make it easier to isolate the cause. However, if you are fixing someone else’s problem, then you are on your own. It is really hard to put a timeline or ETA on the fix. You simply have to do your best to get things headed in the right direction and chip-away at the problem until it is all-done.
My boss (at this place) understood what I was doing and was satisfied with my strategy and progress. Eventually, I was able to resolve all of the problems and take the pressure off of the support team. Of course, it messed up all that documentation about “what order to reboot the servers”, because you didn’t need to reboot stuff anymore. That was a whole different battle to pick, and another story for another day.