Your Tax Dollars at Work, Episode #17

A little story about the workings of State IT

Posted by 03/15/2007

I lost connection to the web server this week at work. What follows is a boring story about the 3 full days it took to get up and running again, which serves to illustrate some underlying deficiencies in the IT department where I work - and which, I contend, is typical of most IT departments.


Several people in my department notice that we can no longer upload stuff to the web server. Since we upload something pretty much everyday, usually multiple times a day, this is kind of disruptive to the work flow. So it is time to contact the server administrator. The one and only person that has any knowledge whatsoever of the single server that houses the internet and intranet for 30+ separate web sites, running web sites for 2 large state agencies with upwards of 50,000+ employees and 100,000+ web pages. This is a problem in and of itself. One box. One administrator. One point of failure.

It does not surprise me when he doesn't answer the urgent email I send. That is common. In the past 5 years he has been notorious for the "zero response strategy" as I like to call it. This is the customer service strategy employed by the entire IT department. You send an email and then it goes into a vacuum. A black hole. There is no response whatsoever. No "I'm working on it", No "I'm busy I'll get to it when I get a chance", just - zilch. This is brilliant because it avoids confrontation. And also keeps the workload low.

Next I try sending roughly the same email except with a CC to his boss. This never works either because it assumes the boss cares - whereas the boss is employing the same 'zero response strategy'.

Next is the phone call. The phone will never, ever, ever be answered by someone in IT. But you can leave a message. Which will also disappear into a void of nothingness. Pager? Same thing.

Then finally I walk over to the actual building to try and find someone and physically force them to respond. This approach has varying degrees of success - if you can find the person. Because people are sometimes friendly when you corner them. In fact it's weird. They can be so friendly you think everything will be different from now on. "I finally found someone helpful over at IT. Our troubles are over!" That is an illusion though, don't be fooled.

So I look up the name of the one and only web server administrator in the Global Address Book and go to the 3rd floor - section A. There is no one there by that name. There is no cube name tag with that name. Also, no one in the area has ever heard the name, and they have no idea where he could possibly be. "This is the mainframe section". Ask me if I'm surprised by this.

The answer is no.

Looks like today is a wash.


Around 9:00 a.m. I get an email that says:

"Please contact the help desk in the future. What is the error message you are getting. The FTP server seems to be running fine".

from the aforementioned server administrator. So I figure, "great he's working on it now". What is the error message? Well when I try to upload files via FTP the server returns a message "home directory inaccessible". I figure it's a permissions problem. All indications from the Google-borg are to that effect. It's just a matter of making sure I still have read/write permissions to some directory somewhere. So it will be fixed soon. Right?

Hours go by and I do not hear from anybody. I guess that means I have to create a help ticket after all. When he said to make a help ticket "in the future" he actually meant the future of today - not the next time a problem is encountered. So I create the help ticket. Maybe the formality is necessary. This is the usual conversation once communication is established:

"I know you are but what am I?"

"I know you are but what am I?"

Which I remember from Pee-wee's Big Adventure. In other words, it must be your fault. I have never once worked with IT where they did not immediately assume it was my fault. It must be part of the training. The "I know you are but what am I? strategy".

Mind you, I am portraying this all rather calmly - which is because I myself am calm, but everyone around me is starting to freak. They are livid. It's going up the command chain, to my boss's boss's boss's boss. Who is contacting the powers that be in IT. Because he knows her personally and maybe he can call in a favor. Everyone is trying to get a ball rolling that weighs 1000 pounds.

We have stuff to put online. It's backing up. Our customers are getting angry. My managers can't believe how inept and unresponsive the server administrator is. They have forgotten that it's always like this.

Our customers associate us with the web - so if the web is not working they are not going to understand the subtleties of the difference between those that put content on a web server vs. those that administer a web server. It's hard to explain that the "web people" actually have no control whatsoever over the web server:

"How come my stuff isn't up yet? I've scheduled meetings around it. We have training scheduled over this new material. There is new policy effective today. What is up there now is wrong. Can you at least delete that. What is the problem?"

You get the idea. Lawsuits have been threatened before - although not this week. All we can say is "Sorry, 'technical difficulties'".

So angry emails are flying. The head of IT asks for the help ticket# so she can "expedite" things. The day ends. No one has done anything. No one has contacted me except for that one email.


Hours go by in the morning. More emails are flying - people keep asking me if anything has happened. I have to run around in panic mode. In the meantime the help ticket is marked as "low priority" and has not been assigned to anybody. And it has been routed to the wrong department - who have no idea what to do with it - so they are just ignoring it. Of course, we don't know this yet. We are just sitting there waiting to hear from someone. In the ideal world someone calls us at this point.

No one is calling though, so somebody takes the initiative to follow up with the Help Desk. Then the byzantine organizational structure makes itself known. This is the sad fact that, even in a large organization with thousands of pages of rules, policies and procedures to cover every scenario that could possibly occur ever, it all boils down to somebody having the secret knowledge of who to call. And this is invariably someone that has been there for a long time, and someone who is persistent. "I'll call this person, because they may know who will know someone that will know who to call..."

So now we know the ticket is low priority and has not been assigned to anyone - and was routed to the wrong department. More angry emails fly. Hours go by. Nothing happens. Lunch happens, and if you work for the state you know that nothing can happen from about 11:00 - 1:00 because it is likely that somebody is at lunch.

So the hours of 11 to 1 pass. Everyone comes back from lunch. Still nothing. Then, all of the sudden at 2:00 I get a call from someone I've never heard of. He says he is trying to fix the problem because no one can find the "person who usually takes care of the servers...". So he fixes part of the problem. Then he tries the other part and says "Oh. I just crashed the server. I'll call you back". I never hear from this person again. I think he may have been killed.

At 3:00 I get an email from the original guy I was trying to find in the first place. He says my group needs to be deleted, but just to help out he has a new partition solution that he thinks will be helpful. Unfortunately, this solution is not helpful at all - in fact it's just more work for me. So I respond to him "Can you please just restore things the way they were. If you have any new ideas about how to structure things that's fine, but I need some kind of warning time to adjust. 1 week is fine. But immediately isn't. Also, are you aware that there is another guy working on this?". I hit send. About 30 minutes pass without event.

Then the phone rings. It's the original server-guy. The one whose whereabouts are unknown. He is speaking to me from inside a cave:

"Try now. Is it working?"


"Okay... (click, click). (12 seconds of awkward silence). Try again. Is it working now?"

"No, I don't think so"

"Hang on a second ... (click, click). (12 seconds of awkward silence - with breathing). I have another idea. Okay it's propagating... I'll call you back in 5 minutes"

10 minutes pass - I test it. And it seems to be working, amazingly enough. It's 4:00 or so by now. Instead of 5 minutes he calls back in 30 minutes:

"Is it working?"

"Yeah I think so"

"So what was the problem?" I ask. Was it some Daylight Saving time patch or something (I was wondering)?

"Sorry. I was doing a little 'house-keeping' this weekend and deleted your accounts by mistake".

So it took 3 full days to do 5 minutes worth of work - to erase a totally bone-headed mistake. Welcome to state government IT.

Last Thing

Finally, at 5:00 at the end of the day an email is circulated originating from a higher up at IT. "I have not received a response to my request for assistance. I have escalated this another level. I should have a status report in the morning". Well, no one told you did they? - it's all over now. But thanks.


Total: 0.33 Python: 0.06 DB: 0.27 Queries: 19