Distributed SBCs aren't just lightning insurance

In this article I’ll explore how you can use a geographically distributed network to actually improve call quality – not just as a marketing slide to show how resilient you are to the forces of nature.

If you offer VoIP service to customers across a large geographic area – e.g. nationwide in the US – then you’ve probably spent some time considering how to build a geographically redundant network. Having equipment in multiple geographies (e.g. Atlanta and Denver) gives your business subscribers a warm fuzzy feeling that neither earthquake nor hurricane, neither tornado nor ice-storm, neither flood nor lightning strike can prevent their phones from accessing your service.

Great. Good job marketing team.

No really. Resilience to a natural disaster is important, but in this article we’ll explore how you can use your geographically distributed network to actually enhance the customer experience – not simply protect against a freak event.

Let’s start by looking at a non-distributed network, and a typical long-distance call made by your subscriber. I’m going to be using high-tech diagrams created with a pen and paper.

I’ve obviously used an extreme example, but in this diagram we have the following situation.

A subscriber in New York places a long-distance call to Los Angeles.
The caller’s media routes first to your network in (say) Denver, and then from there you route the call over a SIP LD trunk to (say) Atlanta.
We don’t know exactly what happens after the call enters the LD carrier’s network, but at the very least it’s going to have to travel from Atlanta to LA.
Assuming the media path is the same in both directions, the audio must travel 5500M in each direction, or 11,000M round-trip between when the caller speaks and when an immediate response is heard.
That’s certainly an under-estimate because we have no idea how the traffic is routed after it enters the LD carrier’s network.

A brief dive into some math:
Based on Wikipedia’s article on latency each km of distance introduces at least 5 microseconds of latency – and that’s before we even consider how many different IP or VoIP routers the traffic must traverse between point A and point B. 5 microseconds / km can more easily be understood as 5 ms / 1000km, or 8 ms / 1000 miles.

However, using https://tools.keycdn.com/ping I did some real-life ping tests from different US locations to a fixed point, and while the results vary, in most cases the actual latency is at least 60% higher than this theoretical number. So for all my analysis below I’ll be assuming that the real-life latency due to distance on IP traffic is 13ms per thousand miles.

Back to our example

So in our example we have 11 thousand miles of round-trip distance, which gives us approximately 143ms of latency due to distance. Most experts say that callers start noticing latency once the round-trip time exceeds 250ms, so we’re still below that threshold, but unfortunately distance is not the only contributor to latency in the network.

We also need to consider the impact of the buffers used by VoIP devices to compensate for the jitter (variability in arrival time of RTP packets) found in IP networks. The size of these jitter buffers may vary but a typical buffer might introduce 40ms of delay. Unfortunately the one thing we don’t know is how many devices are introducing jitter buffers into the voice path. Certainly any time the media is converted from VoIP to TDM (or from VoIP to audio – i.e. at a phone handset) there will be a jitter buffer, but we don’t know how many times that happens in the “unknown” part of the network, and it’s not impossible even for VoIP-VoIP devices to add jitter buffers if they think it’s necessary to maintain audio quality.

The result: at the very least we have 143ms of latency due to distance plus 80ms of round-trip latency due to jitter buffers on the two end-user phones, which brings us to 223ms for the call.

Umm… but that’s still fine. What is your point exactly?

You’re right. 223ms is probably fine. However, it wouldn’t take much to push this call over the edge. What would happen if…

there was congestion somewhere in the IP network
the carrier routing from Atlanta to LA also bounces through a few other cities rather than going direct from point-to-point
the carrier routing from Atlanta to LA switches between TDM and VoIP a few times introducing more jitter buffers
the subscriber in LA has set up call forwarding to her cell phone, and she’s currently traveling in Minnesota
the called line is actually an auto-attendant, which then hands the call off to a call-center in Florida, or Phoenix, or India.

Any of these factors could push this particular call over the edge (perhaps significantly over the edge), and many of these factors are totally out of your control as a service provider – so it’s our job to minimize latency as much as possible in our network, thereby increasing the chances of a great user experience.

It’s our job to minimize latency as much as possible in our network, thereby increasing the chance of a great user experience.

A distributed network of SBCs

So if you accept the premise that minimizing latency is important to ensure high quality calls for our geographically dispersed subscribers, what’s the best way to do that?

I’d like to suggest that you geographically distribute your session border controllers (SBCs) around the country, and use these distributed SBCs to minimize latency. Here’s how it might look.

In this network we have distributed 4 SBCs around the United States, and with this model we can configure the end-user SIP phones to register with the SBC that is geographically closest to the phone. What’s more, since the entire call is voice-over-IP, we can configure the session border controllers to connect media directly from the phones to the carrier SIP trunk, without bouncing the media through your core network.

Based on the given example, this would reduce the total media path distance from 5500M each-way to about 2800M each-way (almost a 50% reduction). That gives us a round-trip distance of 5600M which equates to about 73ms of distance-based latency – a saving of 70ms compared to the original design – and that gives you a much greater chance of a high quality call, even if there are inefficiencies elsewhere in the call path.

But you faked it! Your LD carrier “just happens” to be in Atlanta

Okay, you got me. My example very conveniently happens to use an SBC (and a customer phone) that are fairly close to my long-distance SIP carrier – so of course the results look impressive when I get to pick the examples.

But wait! Isn’t this just part of a good design? What if you had a network of (say) 4 geographically distributed LD SIP trunks (either from different providers, or distributed trunks from the same provider), and then you intentionally positioned your SBCs so that they were close to the SIP carriers?

If you designed your network that way, and configured each SBC (and your core switch translations) to always prefer the SIP trunk that’s co-located with your SBC, then you effectively eliminate one leg of the media path.

So for example, in this diagram your mid-western subscriber is registered to your mid-western SBC, which happens to be close to your mid-western SIP carrier, who then routes the call (over their network) to Los Angeles.

Or in this example, your Texas-based caller is registered to his nearest SBC in Phoenix, which is designed to be close to another SIP trunk provider, who then routes the call to LA.

So in the worst case, perhaps your subscriber is registered to an SBC that’s 800 miles in the wrong direction, but by creating a network of 3-4 geographically distributed SBCs (and carrier SIP trunks) you have dramatically reduced the distance your media has to cover, thereby dramatically reducing the latency experienced by your subscribers.

Of course, you could take this further – if four SBCs is better than one, then surely ten SBCs is better still, right? And of course the answer is yes… but the benefits of each additional SBC become smaller and smaller (the reduction in distance for the farthest subscriber is less dramatic), and of course your network logic becomes more complex – a classic example of the law of diminishing returns.

In conclusion: don’t go crazy over this, but if you’re expanding a nationwide VoIP network, you should absolutely consider the benefits of distributing your first few SBCs geographically, providing better call quality to your subscribers, as well as the more obvious resilience against natural disasters.

About Andrew