The worst that could happen (part 4) – The President’s Firewall

Working for a growing company in small metropolis has its ups and downs. If your company has the right people, the company can grow pretty rapidly and you can pick up some nice customers, simply based on your reputation and quick success. This sort of thing becomes self-perpetuating for a while. A growing company will also tend to attract higher talent. This is great for both parties. The only problem occurs when old-guard conflicts with new-guard.

I had just hired into this kind of growing, local consulting company. A few weeks later, they landed an account with one of the biggest companies in town. It was very exciting because I got to jump-in and face the challenges and opportunities that came with this big new account.

The first project was a slam-dunk. Clearly it was a slow-pitch designed to test our skills. The 2nd project was more involved: fix a project that was struggling. That project took more creativity, but went really well. The customer must have recognized it because, before that project wrapped up, we got three more projects, at once.

The first two were interesting, and right in-line with our skills. We would handle them, no problem. The third was gigantic: redesign the web site for this huge company AND HOST IT.

I couldn’t believe that they were serious. You could fit our entire consulting company into the server room of the customer. They had more people in their HR department than our entire company. To think that we could host their web site better than them, was mind-blowing.

I recommended (privately) to my boss that we should step-away-from the hosting part of that project, but my boss didn’t want to turn away such an opportunity. He was going to put our two best resources on that project: two hippie-looking web design guys who were still in college and showed up to work in their pajamas sometimes. Their design skills were top-notch. However, something about them, gave me the impression that, maybe, they lacked discipline occasionally. Call it a hunch.

The reason that I was so scared about hosting the web site was because I had been in the “server room” before. In fact, I was in there every day. Because the “server room” actually doubled as a break room. In the corner, there were two server racks (with no locks or anything), blinking-away. Occasionally, if you ran the coffee maker, and the microwave at the same time, it would pop a circuit and it would take-down all of the servers. Somebody usually noticed in a few minutes, and flipped the circuit breaker on again. The customers rarely-ever noticed the outages.

I was relieved to hear that the “server room/break room”, was not our ONLY home for servers. We also had a server co-located at a local ISP’s site. This would be the home for the web site of the huge customer.

The web-site redesign went very well. Those two hippies in their pajamas knew their stuff and cranked-out a really great looking new site. Our back-end programming was solid too. Pretty soon, we were ready to demo it to the customer and pick a go-live date.

Going Live Without Your Pants
The go-live went really well. The site came up and the web traffic seemed pretty steady. I was ready to eat my words. We were actually doing a good job at hosting the web site for this huge company.

After the dust settled on the initial launch, the customer ran a security scan of the server and asked us to “firm-up” some of the firewall rules. Nobody had ever messed with the firewall before. We called the hosting company and they said that they did not manage the firewall. It was owned by us. We just didn’t know which one of us.

After some asking-around, we found out that the firewall was managed by the president of our company. Apparently, he was the only person who could be trusted with such a serious responsibility. I found it hard to believe that nobody else could be trusted. I’m not suggesting that he should hand it over to the hippies, but there were plenty of other knowledgeable and responsible people who worked there. *ehem*. *cough* *cough*

I still had never met the president of the company, but I had seen him. I guess his background was as a server admin. Supposedly, he was pretty accomplished, back-in-the-day. Now, his day mostly consisted of sales calls and paperwork. We got the president’s phone number and left him a message. We were pretty clear on the importance of the matter. We figured he would call us back pretty quickly.

By the next day, we had not heard from him. The customer called us to ask why the security changes were not complete yet. What could I say? I did a little tap-dance for the customer and then I left another voicemail for the president. No response. We ask the VP if he could get in touch with the president or get login permissions or something. The VP was glad to help and said he would handle it.

The customer kept calling (every two hours) asking for an ETA. After the third or fourth call like this, we started to feel a little stupid. We had tell them that there was no ETA. Plus, now the VP had left the office and was not returning our calls either.

We started thinking outside of the box. Perhaps there were other alternatives: Could we close ports on IIS? Could we install another firewall on the web server directly? Could we buy a Cisco PIX and take it to the ISP ourselves?

Finally, the VP and president showed up and said, “Let’s get that firewall configured”. Just like that. The president whippped out a laptop and started clicking away. We tried to look over his shoulder to see what he was doing, but he turned his laptop as if it was some kind of secret. Okey-dokey. Just as long as it gets done.

After clicking around for a few minutes, he said “There you go”. He dramatically pushed the enter key and leaned back in his chair. “In a minute or two, it should come up and we will be all set”.

I was like “Did you just say that it would be UP IN A MINUTE OR TWO?!”. He gave me a blank stare. I darted to my computer and checked. No! The #%$& site was DOWN! I ran a ping to see how long until it came up. There was nothing else that I could do now.

A minute passed. Then two. I walked over to get a coffee, because staring at ping wasn’t making it go faster, and I could feel my blood pressure rising a little.

I came back in a few minutes and nada. The site still wasn’t up. It had been five minutes.

I found the president, so I could see if he could take another look. He was on the phone and gave me the “wait one” finger gesture. After politely waiting two or three minutes, I tried to pantomime that the server was still down. He rotated in his chair so I wouldn’t distract him from his phone call.

I stepped-out to find the VP and explain the gravity of the situation. As I took one step outside of the president’s office, I was immediately intercepted by the receptionist. She shoved a phone in my hand, “It is the customer. They are really upset, because they noticed that their web site is down.” I explained to the customer that we were all working on it already and I would call them back in a few minutes. They asked to wait as I worked, instead of waiting for a call-back. I hadn’t even finished my sentence and the receptionist said there were two more calls (on hold) from the customer, “Line 2 and Line 3”. Gosh, those guys were pro-active.

I got the VP and asked him to interrupt the president. “What is it?!” responded the president in an abrupt manner, as he held his hand over his phone. The VP quickly explained the problem. The president told the person on the other end, that he would call them back. “Why didn’t you tell me it was so important?”

He opened his laptop again and started clicking. “Hmm. Hmmmm. Well, I can’t get in. I must have made a mistake with one of the firewall rules.” I felt my stomach ache, at the thought of it. “No problem” the president declared. “I will just drive over there and take care of it.”

“How long will that take?” I asked. “About twenty minutes to drive there. Then it should be up pretty quickly”, responded the president.

“What should I tell the customer?” I asked. “Tell them anything you want to. I’m on my way.” he responded.

So, I spent the next fifteen minutes stalling for the customer until they got exasperated and asked me to call when it was done. I didn’t hear from the president. However, 90 minutes after the site went down, one of the hippies came into my office and said “by the way, the site is up again”.

I wish I could tell you that we all learned a lesson from this fiasco and things changed. However, they did not. This scenario played-out SIX more times over the next six weeks. The customer somehow endured it, but I could not. I found a new job.

Lessons learned. Oh my gosh. Where do I begin? 1) Hire trustworthy people so your president doesn’t need to be the only person that you can trust to configure the firewall. 2) NEVER EVER mess with the firewall during business hours. 3) Don’t make a firewall change (or other server change) that can lock you out of it, unless you are sitting (physically) in front of it and can back-out the change. 4) If your gut tells you that you are getting-in over your head, you are probably right. “Upping your game” is advisable, but might not be good enough. 5) If you are not in charge, then you should be careful about telling customers that you are responsible and can handle things that are clearly out of your control. 6) If your president is messing up the company, then you might want to get out of there before you get blamed and fired.


About Tim Golisch

I'm a geek. I do geeky things.
This entry was posted in IT Horror Stories, Lessons Learned. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s