Do you have any plans for Labor Day weekend?
Back in 2011 my plans were rudely interrupted. At the time I was managing the Metaswitch support team responsible for new customer installations, commissioning and migrations. This was a fun job, but mostly fairly predictable – after all we were working on multi-month projects which followed a fairly standard sequence of events.
This time, however, was different.
On August 26 New York Governor Andrew Cuomo declared a state of emergency due to Hurricane Irene which caused widespread flooding on August 27-29.
Luckily for me, I lived in Cleveland, Ohio at the time – so wasn’t directly affected – but the hurricane nevertheless disrupted my plans for the holiday weekend.
A small town in central New York state was hit hard by the flooding, and the local independent telco ended up with 42″ of standing water in their central office. It turned out that their existing DCO switch wasn’t waterproof, and so their entire customer base lost phone service.
And so it came to pass, that on August 31, 2011 they ordered a brand new Metaswitch VP2510 and I got a phone call asking how quickly we could get it into service.
Now a typical project took anywhere from four to nine months, but I got the impression they were looking for an expedited process in this case.
The big question: how quickly could we move?
The importance of preparation
As we approach Labor Day 2019 many of you are making business plans for 2020 – identifying key priorities for your network, allocating budgets, and so forth.
You’re doing this ahead of time – rather than waiting until January 1 – because taking time to plan and prepare is critical to making good decisions and meeting your goals.
In the same way, good preparation was critical back in 2011. You see, August 31 is the last day of Metaswitch’s financial year, and so it turned out that the Metaswitch Operations team had pre-packed a VP2510 chassis which they were able to ship out that very same day – shipped overnight to arrive on the morning of Thursday September 1.
Status after 24 hours: equipment on-site.
As you look forward to 2020, what can you do to prepare well? In the rest of this article I’ll highlight six different ways of looking at your voice network, which should help to figure out which projects should get priority for the new year.
#1: Redundancy
There are many different aspects of redundancy – from geo-redundant Perimetas, to distributed media gateways, to simply having spare cards on the shelf.
The amount of redundancy appropriate for your network will vary. It certainly wouldn’t be appropriate for a small ILEC with a few hundred lines to build a coast-to-coast geographically redundant voice network – but you do want to understand the impact of different events, and figure out a response plan for each type of scenario.
In 2011 there was no redundant switch, but there was a redundant location. The local phone company had rack space, power and connectivity available in another building a couple of miles from the CO – on higher ground.
Metaswitch sent out a team of physical installers who got the equipment racked and powered at this second location by the end of September 1.
Status after 48 hours: Equipment installed.
Back in the office, I was having an internal debate about what to do. On the one hand, one of the privileges of being in management is that you get to choose how work is assigned.
In my days as a lowly support engineer, I was never assigned the accounts in Guam or Hawaii or the Caribbean – and no I’m not bitter. Much.
Compared to these sunny trips, working overtime on a holiday weekend in a natural disaster area didn’t sound so appealing. Exactly the kind of task it would be good to delegate to someone else.
Unfortunately, through a mix of conscience and a misguided hero complex, I decided to handle this myself.
So on Friday September 2, I found myself driving my rental car towards this small town, only to encounter a new problem: a literal road block with some kind of security personnel standing guard.
#2: Security
Is your network secure? What does that even mean?
- Could someone guess a password on one of your SIP lines and steal service?
- Could someone gain access to a phone’s web interface and automatically forward all calls to North Korea?
- Could someone hack one of your admin interfaces?
- Are you vulnerable to a distributed denial of service (DDOS) attack?
- How long would it take you to notice if one of these things happened?
Service providers across the country are getting hurt by toll-fraud – and the cost can often be in the tens of thousands of dollars. If you haven’t already implemented security best practices to protect your network and your subscribers this really needs to be on your list for 2020.
It turned out that when FEMA is active in a disaster area they put up road blocks – perhaps to prevent gawkers and/or looters from visiting a town devastated by a flood. Thankfully they bought my story about helping to restore phone service, and I was allowed to continue.
The next few hours I was able to meet my contact Mike, find the Metaswitch, get IP addresses configured and so forth. All good.
Then it came time to start entering the switch configuration:
Me: “So can I get a list of point-codes, TCICs, and so on to build out the SS7 routing and ISUP trunk groups?”
Mike: “Well all that data is in the DCO… which doesn’t work any more. Because the CO got flooded. Remember?”
Me (feeling dumb): “Oh.”
Mike: “We do have some paper records of our trunk orders, would that help?”
Me: “Yes, that sounds great. Where are they?”
Mike: “We keep them in a filing cabinet…”
Me: “Great!”
Mike: “… which was also in the CO… so they’re… wet.”
Me: <frowns>
After some searching we were able to find these paper records, and sure enough they were rather damp. Feeling like art restorers / archaeologists we very carefully peeled apart these sheets of sodden paper and amazingly were able to find all the trunking information we needed.
Status after 72 hours: SS7 links active and PSTN trunks in service
But we were in store for another shock, because some bad weather was on the way – a thunderstorm – and as if flooding wasn’t bad enough, the next disaster was just around the corner: a lightning strike!
This left me feeling pretty shaken. [Was that the best link to the next section ever?]
#3: STIR/SHAKEN-ready
The FCC is pushing all major carriers to implement STIR/SHAKEN by the end of 2019, and although there may be less pressure on smaller independent telcos there are nevertheless real commercial pressures on smaller carriers to keep up with these standards or else their subscribers will be relegated to second class citizens of the PSTN.
You have a few different options to get your calls signed – including a hosted service from Metaswitch & TNS, and a hosted offering from TransNexus – but you’re still likely to have some challenges with your TDM-based trunking.
In any case, if you don’t already have a plan, you may want to figure out your approach early in 2020.
The lightning hit another part of the telco’s network – so I was left all alone while Mike went off to deal with the problem.
Having sorted out PSTN connectivity (and made a few simple SIP test calls) I figured I should take a look at the subscriber side, and that’s when we discovered a new problem. All the original subscriber lines were using TR-08 T1 interfaces, which (for those who don’t remember) is an older form of POTS line technology that the Metaswitch did not support.
So in the midst of everything else, we needed to convert all their subscribers to GR-303. (Remember this was 2011.)
#4: Network Evolution
While you’re unlikely to need to upgrade your subscribers to use exciting new inventions like GR-303, the technology used in the voice network is continually evolving, and you should consider what plans are appropriate for your network.
Based on projects we’ve worked on recently, you might be considering the following:
- Replacing an old switch
- Refreshing the hardware on a switch
- Moving from TDM to VoIP for your subscriber connections.
- Moving from TDM to VoIP for your PSTN trunking
- Consolidating several switches into one – creating a single redundant network rather than a bunch of small independent switches.
These are all potentially big projects – where do they fit on your list of priorities, and which ones do you need to get started in 2020?
Back in 2011 we were through the worst of the drama. (Just as you’re through the worst of this article.) We were able to get some kind of subscriber data dump which I turned into a Metaswitch import file, and then the local staff began the laborious task of rewiring all the subscriber lines over to their GR-303 shelves to restore service one-by-one.
In the meantime, I was able to turn up their MF trunks to 911 and operator services and validate TCAP services (CNAM, toll-free lookup, etc).
Ultimately we got the first subscribers back in service 4 days after the new switch was ordered, and I was able to fly home on Labor Day evening… feeling tired, but with a good story to tell.
#5: Launching the right products for the future
After a natural disaster all the focus is on restoring service – getting things back to how they were – but if you’re going to survive and thrive in the roaring twenties (whoops, wrong century) you need to make sure you’re offering voice services that are still relevant and can make money in the modern age.
We spend a lot of time working with independent telcos to help launch and grow their Hosted PBX / UCaaS services for business customers, but depending on your market the greatest need may simply be for SIP trunking to PBXs, or if you’re a wireless service provider it may be time for add intelligence to your network to allow multi-persona users to support the mobile workforce using MaX by Metaswitch.
#6: Building the right team
Finally, any strategic planning exercise should include a review of the organization from a people perspective.
- What skills do you need on your team?
- Do you have the right people in the right roles?
- Do you have development plans in place – particularly for those on an upward trajectory?
- Who would be well placed to contribute to, or even take responsibility for your strategic projects?
Checklist
I know most of you will have been taking careful notes as you’ve been reading, but just in case you weren’t, here’s a checklist you can use as you perform your own planning exercise for 2020.
- Do you have appropriate redundancy built-in to your network?
- Is your network secure?
- Do you have a plan to implement STIR/SHAKEN?
- How does your network need to evolve – what migrations / upgrades / consolidations should you consider?
- Are there any new products that need to be launched or revamped?
- Do you have the right team in place to execute your plans?
If you’d like to chat about any of these areas, please drop us a line and I’d be happy to set up a free 30 minute consultation to answer any questions I can.
If it’s helpful, we also offer a strategic review service where we work with your team to answer these questions in detail and identify the specific projects that make most sense for your situation. Again, contact us if you’d like to discuss your situation. And let me know if I need to bring my waders.
