The setting

I worked as the lead for mobile apps at Vodafone NZ. We were working on a major new release of our app which had approximately 1.4 million customers across iOS and Android. The new features were being launched on a specific date supported by TV and radio advertising, billboards, bus signage, etc.

In the week prior to launch the iOS tech lead refused to share the digital certificates for the app, taking the only copies of them with him on holiday to Brazil. His parting statement was “If you don’t know how to create new certificates you shouldn’t have this job”.

The mistake

My team and I discussed the problem, and I decided to create new certificates as there were no apparent drawbacks to doing so.

From there everything went smoothly. Development was completed, the new app passed our quality assurance testing and Apple’s App Review. On launch day we released it to our customers.

The realisation

At this point we discovered that the keychain, where a customers login name and password are securely stored on their phone, is tied to the digital certificate of the app.

Which we had changed.

I had inadvertently wiped the saved login name and password of 850 thousand customers.

The result

A cross-functional response team was assembled to handle the incident - we quickly diagnosed the issue, and I immediately owned it. The incident team collectively decided the only real option was to keep going as is.

How this affected Vodafone was dramatic, but far more interesting was how it was handled. During the day various key people explained in person the impact on their teams:

• The call center manager explained how the call volumes were 6 times normal, and they’d had to put extra staff on for the next week, including pulling some back from leave.

• The devops lead explained how the load on the authentication servers had gone the highest they’d ever seen, and how that was adversely affecting other systems.

• The marketing manager explained how this was all negatively affecting the reception of their new campaign and newly launched features.

Everyone was very polite and matter of fact. No one yelled. Instead the consequences of my actions to the entire company were made crystal clear to me.

By the end of the day I was pretty devastated. My manager came over and with some amusement asked how my day was. I offered to resign, and he responded with:

“We just spent a lot of money and time teaching you to not do this again, let’s not waste that.”

Vodafone didn’t do everything right, but the blameless way they handle incidents is a gold standard everyone should aspire to.

The big picture

Everyone makes mistakes.

How you handle them, how your company handles them, tells you a lot about both.

Perspective helps. Next time you make a mistake just remind yourself - at least it’s not as bad as Dan pissing off 850 thousand people in one day.

The key to handling mistakes like a champ

1. People make mistakes. Know that no-one is focussing on it happening, but on how you respond.

2. Mistakes can be embarrassing, but owning them shows integrity and creates trust. Which is the only thing people really remember.

3. Mistakes are lessons learned the hard way. Don’t waste them.

Follow up note: I’ve had some pointedly charmless people inform me that my managers response was very close to an apocryphal story and thus my story must be fake. I contacted the manager and he told me he’d heard that story and liked the idea and approach so he re-used it here. He also said the internet is full of "ignorable fuckwads". Good advice.