Today the server of an important client went down. Was not reachable. Not that interesting as, with the enormous amounts of servers we are managing these days, that happens more often.
What was nice about it is the process surrounding this issue. The server was the mail relay server. So the client could no longer send mail from his website to the outside world, as this mail goes via the relay server.
Unfortunately the admin was, for a brief moment not reachable to use remote powerboot to fix it, so I uttered the, in hindsight, dangerous words;
this is a mail relay, why don’t we just set another relay for the moment?
Apparently the 10 minutes (even less…) required to change the relay in Postfix, were translated into the company (that hired me) workflow to:
1. ask John to create an issue in Jira
2. assign the issue to the person in charge of the admin work load distribution, Frank
3. Frank will check the roster of the admins and assign one; Paul
4. Paul will estimate the time needed to change the relay (a whopping 8 hours!) and assigns the issue back to Frank
5. Frank fixes the estimate by Paul by setting the estimate to estimate*PI and assigns back to John
6. John assigns the issue to sales
7. Sales talks to the client and asks for estimate*PI*120%*$100 ~ $3000
8. After approval the issue is assigned back to John
9. John assigns to Frank
10. Frank assigns to someone other than Paul (as that is how it goes)
During step 2 the admin came back and rebooted the server.
Unfortunately the above is not strange; any person working at a bigger company knows that this is actually not too bad compared to even bigger companies. In fact; Frank or Paul in steps 2,3,4 should have sent the issue back to John for lack of a clear RFP. And the estimate is actually quite low, considering someone has to write an RFP, FO and TO for this problem. And no, I am not kidding, as I see it happen every day.
Enterprises are weird entities that spend massive amounts of $ on nothing at all. Wondering if it is possible to run one without all that crud… It is a disease and I am waiting for 10 years already for IT to kick in and make at least the almost enterprises or middle sized companies a lot leaner than this. Hasn’t happen though. People still buy ERP systems for millions that are unusable, they still hire ‘enterprise Java’ guys for $200/hour and consultants for $2500/day to tell them how to work Websphere and so on. A sick, sick world.
(the names have been changed to protect the innocent)