In the very first Sherlock Holmes short story, A Scandal in Bohemia, the great detective creates a small fire (and plenty of smoke) in order to reveal a secret. He’s trying to find a scandalous photograph, and in the panic of the apparent fire, his adversary runs to retrieve the photograph – thereby giving up the hiding place.
We can use the same tactic in our networks… and no, I am not encouraging pyromaniacal tendencies in the central office. Really, no! But we can create unusually stressful situations in order to learn more about the resilience of our network.
Let’s consider some examples.
#1: High call volumes
You could use a bulk test tool such as SIPP or Hammer to push a large volume of calls through your network (during a maintenance window) and discover where the bottlenecks lie. Maybe there’s a licensing limit on your SBC, or maybe the CPU on your router maxed out… it’s easy to speculate, but the truth can be smoked out through testing under stress.
#2: Overflow routes
What happens if your primary toll trunk goes down? Or if your connection to the local tandem goes down? Can you smoke out the answer by disabling some trunks or even (if you want to be dramatic) pulling some cables?
We can certainly theorize about what happens by reviewing your routing logic, but will your backup route accept the calls? It’s hard to know until you test with fire.
#3: Resilience to failure
Do you have a fully redundant network, that is resilient to the failure of any one piece of equipment? If so, have you proved it?
When I used to write code (a long time ago) we lived by the maxim “Not tested? Not working.” There are plenty of overconfident developers who think they’ve written perfect code that will work first time… guilty as charged! However, we proved (over and over) that until the code had been rigorously tested, we shouldn’t assume our logic was correct.
In the same way, just because you have a nice diagram that shows you can handle a router failure, or a power failure, or a server failure, that doesn’t mean it’s true. Not tested? Not working.
Fire shows the truth
If you really want a strong, resilient network that provides service even when something goes wrong, you need to develop and execute a test plan that puts your network to the test, and smokes out the problems. Just as with Sherlock Holmes, the planned fire can reveal truths that would otherwise have remained hidden.
