Our out-sourced operations team broke the portal of a large customer (to give a sense of perspective: their revenue in 2006 was a bit over 5 billion Euro).

It was “only” an outage of about twenty minutes.

Unfortunately the customer’s top management was holding a presentation of the portal at the same time.

Luckily I wasn’t involved in any of this, and the customer did approve the change for this time and date, but I’m sure I’ll hear about this again. They already canceled all changes for the rest of the week, and I look like a bit of a fool for following the company line and telling our customer how competent the operations guys are.

Update, the day after: Yes, I was right. Hans and I have been doing nothing but damage control all day today.