I once saw a proverb that said something like “If you want to quickly, go alone, if you want to go far, go in a group”. That’s a nice sentiment, but in my experience of going hiking with friends, if you go in a group then you can only go as far as the slowest/weakest member – so I’m not convinced that this latter part is true. But the first part – I definitely agree with that.
In our house, it seems to take forever to get the whole family out of the door. The kids often take their sweet time to even begin to get ready, and then we have the inevitable trips back to pick up the forgotten sweater, water bottle, phone, glasses, yarn, or whatever it might be.
But most frustrating of all is when I’ve been waiting by the door ready to leave for several minutes, and I call back in frustration “are you guys ready yet?”, only to hear “I was waiting for you!” Somehow, despite all the practice, we sometimes end up with each member of the family waiting for all the others, not realizing that everyone else was already to go. Sigh
A few weeks ago, Dan (on our team) was telling us about a trouble ticket where the root cause was basically this same problem. But before we get into it, I need to explain a concept called media latching, which is a cool feature used by SBCs to traverse NAT.
As a reminder, Network Address Translation (NAT) is a function performed by firewalls to convert from a private IP address space inside a home or business (e.g. 192.168.x.y) to the public IPs of the public internet. So if you have a phone on your desk in a business, that phone will likely have a private IP (let’s say 192.168.5.100), which makes things interesting when it wants to make and receive calls through a Session Border Controller (SBC) with a public IP of (say) 5.6.7.8.
The firewall will prevent any inbound traffic from reaching the private IPs on the private network at all, and the NAT function on the firewall will convert the source of any outbound traffic so that rather than having a source of 192.168.5.100, the messages will come from the firewall’s public IP (let’s say 100.2.3.4). The firewall may have many devices behind it, but it handles this by using a unique port for each outbound request. So if the phone sends a SIP REGISTER message from port 5060 on the phone, the firewall may send this out from 100.2.3.4 on port 8342, and it then knows that any response back to port 8342 should be routed back to port 5060 of 192.168.5.100 (i.e. the phone).
For security reasons, firewalls don’t keep that “pinhole” open indefinitely – they are designed to allow a response to a request made by the local device, and usually the pinhole will close after 30-60 seconds so that no nefarious person can try to hack the devices on the private LAN.
This works great for outbound phone calls, but as you might imagine, most people also want to be able to receive inbound calls – and not only during the 30 second window after they last made an outbound call. So one of the core functions of an SBC is to keep that pinhole permanently open by modifying the expiration interval of the SIP REGISTER messages received from devices behind a NAT, to force these phones to re-register every 30 seconds (or whatever period is needed based on that particular firewall).
This works well for the SIP signaling. However, you may recall that the RTP media streams are not sent on the same port as the signaling. So when an inbound SIP INVITE is sent to the phone, the SBC will say (inside the SIP message), “Please send media to 5.6.7.8 using codec G.711 on port 32768”, and then the phone will respond saying “Please send me media to 192.168.5.100 on port 2345”.
As you may have guessed, this poses a problem because the SBC can’t reach 192.168.5.100 on port 2345.
However, the SBC is smart, and it knows that the signaling messages for this device are arriving from the firewall’s IP of 100.2.3.4 on port 8342, so it knows that the RTP media packets will need to be sent to 100.2.3.4 to get back to the phone through the firewall. But what port should they be sent to?
If you said 8342 then… you’d be wrong. That port is being used for the signaling traffic, so if you sent RTP to that port then those packets will indeed reach the phone on port 5060, but the phone will ignore them because it’s only expecting to receive SIP signaling packets on that port.
So how does the SBC know where to send the packets? Well… it doesn’t. There’s literally no way for it to know where to send them, because it has no way of knowing which port the firewall will pick when setting up the NAT pinhole for the RTP connection on port 2345.
So the SBC waits, patiently. There’s no way to know in advance which port will be used, but if we monitor the network, and wait for the RTP stream to begin FROM the phone to the SBC, then we should see a new stream of RTP packets arrive at the SBC from the 100.2.3.4 IP address on a previously unused port (let’s say 4567). When the SBC sees this new stream, it assumes that this new stream of media must be the media from the new call that was just established, and it latches together that stream of media with the connection it needs to make to send audio to the phone, and it sends inbound media back to 100.2.3.4 on port 4567. This is called media latching – and as you can see the SBC has to be really quite smart to successfully traverse these firewalls and communicate to devices on a private LAN.
Now that all of that is clear, let’s get back to Dan and his trouble ticket.
Dan was working with a client who had a SIP trunk connection to another carrier and they were reporting that there was no audio on any of the calls. Dan did some investigation, and he established that there was a Perimeta SBC on our side, with NAT traversal and media latching enabled, and the far-end had a Sonus SBC, which also had NAT traversal and media latching enabled.
As a result, both sides thought that the far-end was behind NAT, and so both ends were waiting for the other side to send media. But just like my family, if everyone is ready to go, but everyone is waiting for the other person to go first, then the consequences is that no-one goes anywhere. So neither side was sending media, and no-one could hear anything.
Once Dan figured this out he was able to change the configuration of the SBCs, and allow media to flow unimpeded in both directions. And he had a funny story to share with the rest of the team.
One of our core values at Award Consulting is “Be part of the team” which is partly about our role in supporting you, our clients, but it also means that we invest time every single day in meeting together as a group to share what we’ve learned, so that we can all grow our skills and learn from each other’s experiences – and therefore serve you better.
If you’d like to have our team in your back pocket, to better support your team, then please drop us a line and we’ll be happy to tell you more about our support retainers.
