When I ran Metaswitch Operations, we spent a lot of time figuring out how to avoid mistakes.
My team was shipping out a lot of boxes full of hardware, with each shipment subtly different – and if we shipped the wrong cable or entered the wrong mailing address or installed the wrong software version… it would be costly and embarrassing to fix these mistakes.
A $900M mistake
However, none of our mistakes were quite as costly or embarrassing as this:
Last August, Citigroup Inc. wired $900 million to some hedge funds by accident. Then it sent a note to the hedge funds saying, oops, sorry about that, please send us the money back. Some did. Others preferred to keep the money. Citi sued them. Yesterday Citi lost, and [the hedge funds] got to keep the money.
Matt Levine, Bloomberg Opinion
Normally if a bank sends you money by accident, you have to give it back, but in this case the story is a bit more complicated. The hedge funds had loaned money to one of Citi’s clients, and Citi wired exactly the right amount of money to pay off the loan (instead of just making their interest payment, as they intended), so the court said they could keep it.
Which is all funny to read about, but I’m sharing this because we can learn a lot from why they made the mistake.
Poka-yoke
Back in Metaswitch Operations, we learned all about Lean Manufacturing, including the idea of poka-yoke, which is the Japanese term for “mistake proofing”.
The general idea is that people will sometimes make mistakes, so you need to design processes, products and tools so that it becomes easy to do the right thing, and difficult to make a mistake.
For example, you can’t run the microwave with the door open, or insert a plug the wrong way into a socket. In my team, this meant a lot of color coding, software tools to automatically warn us of errors, and so on.
Back at Citigroup, the software they used to transfer the money was… weak in this area.
- In this particular example (for complex reasons), they needed to transfer the principal (the full amount of the loan) to a dummy internal account as part of making the interest payment in their software. Not a great design. Strike One.
- In order for this to work, they had to enter a dummy account number and check the PRINCIPAL box on the screen below (which they did), but they also needed to check the boxes for the FRONT and FUND components. Three people approved the transfer and none of them knew this weird rule. Strike Two.
- When they hit submit, they got a warning pop-up (good) which was confusing and didn’t really explain the problem (bad). So the user ignored it (bad). Strike Three.
And that’s how you lose $900M.
Error-proof processes
Personally I don’t have $900M to lose, but I care about my team’s work and I want to avoid mistakes as much as possible. I imagine you feel the same.
So if you’re responsible for a repeatable process, here are some quick lessons to remember.
- Write clear instructions – which could be detailed instructions for a beginner or just a checklist for an experienced team member.
- Configure your tools to simplify the process as much as possible – for example by using templates when provisioning in MetaView Web.
- Configure (and review) warnings – keep a clean alarm panel in your switch so new alarms are easy to spot, and make use of statistical thresholds to spot issues with call quality.
If your processes are inefficient or you’re having quality issues, drop me a line and let’s see if we can help you out.
