Switch consolidation: a perfect union or the road to perdition?

It’s easy to see the attraction of switch consolidation.

Owning a class 5 switch is a big investment – there’s significant capital expenditure on hardware, federal funding is drying up, that same hardware needs to be refreshed when it reaches end-of-life (EOL), and on the operational side you need specialist expertise, you need to provide power and cooling to your equipment, you are paying for TDM trunking to any number of local and long distance carriers… the list goes on.

So why not band together with other local IOCs – with whom you may already share some infrastructure (e.g. a fiber ring) – and consolidate all your switches into a single piece of equipment? There’s often plenty of excess capacity on the systems, so you may not even need extra hardware – and then you’d have a single set of equipment to manage, a shared set of trunk groups, significant savings in both CapEx and OpEx.

Why not consolidate all your switches into a single piece of equipment?

This conversation is taking place in regional cooperatives and independent operating companies across the United States, and the merits of the argument are compelling.

From a financial point of view a service provider can save hundreds of thousands of dollars on switch hardware upgrades, while also reducing operating expenses (fees for SS7 links, for T1s/DS3s, power costs, equipment maintenance fees, etc). In the past a new switch could result in an increase in NECA settlements, which made the capital expense easier to bear, but that’s no longer the case, and on the operational side one service provider told me they expected to save 15% on switching costs, even after paying fees to the host switch provider.

Taking all of that together, it’s easy to see why retiring an old switch and consolidating that function into a neighboring service provider is an attractive path.

But is it really that simple?

By “Pictures of Money” on Flickr.

Yes and no. While the business case often makes sense, the implementation can be a significant challenge. So on this article I’ll be highlighting some of the technical and logistical challenges that come with a switch consolidation plan. These are not insurmountable problems, but executives considering this path (and the operations teams tasked with implementing the migration) would be wise to weigh all of these factors.

1 – Transport

I imagine this isn’t a surprise, but switch consolidation only makes sense if you have a rock-solid transport network (whether that’s IP or TDM) between the subscribers and the original central office and the new switch location. Personally I’d be aiming for all traffic to be running over IP if at all possible, rather than routing a bunch of different T1s or DS3s between sites for different purposes, but in either case, you need a reliable and redundant transport network. This is why service providers with a shared fiber ring are particularly well suited to taking this step, as they already have that reliable, high-bandwidth and redundant infrastructure in place.

2 – Subscriber Connectivity

Most service providers have subscribers and businesses connected to their class 5 switch using a variety of technologies, and each would need to be handled differently in a consolidation.

VoIP endpoints (e.g. SIP phones, SIP/MGCP CPE). This is the simplest case. In a migration you would need to update the IP address of the switch that these are connecting to (which could be on a Session Border Controller, rather than the individual device), but otherwise this is just an IP device connecting to a different server. It shouldn’t be an issue.
POTS lines through a VoIP concentrator (e.g. a Broadband Loop Carrier (BLC) from someone like Adtran / Calix). In this case the local IOC would retain responsibility for the copper to the customer premises, and the only change would be in the BLC – to update that device to use a new host switch. The connection from the BLC to the class 5 switch would use the IP network just as with a pure VoIP endpoint.
GR-303. Architecturally this is very similar to a BLC – in that you have analog lines going to the customer premises, but then GR-303 provides TDM connectivity (over multiple T1s) back to the class 5 switch. In order to migrate this service to a new switch you would either need to transport those T1s to the location of the new host switch, or else take this opportunity to move these lines to a BLC so that you can connect to the new switch over VoIP and avoid the additional transport costs.
SIP trunking to a PBX. See VoIP endpoints above.
ISDN PRIs. Although premise-based PBXs are increasingly using SIP for connectivity, many still expect a PRI connection for PSTN connectivity. While it would be possible to take this T1 and transport it all the way to the new host switch, I would instead recommend that carriers insert a SIP-PRI converter (such as the Adtran TA900) at the customer site so that calls to and from this business can be handled over the IP network.

3 – PSTN connectivity

One of the biggest implementation challenges in a switch consolidation project is the PSTN connectivity, consisting of a mix of ISUP, MF and SIP trunk groups that are used for long distance calls, local calls through a tandem, local EAS calls, 911 calls and operator calls. And of course, each service provider will have a unique set of translations (routing rules) to decide which calls are sent over which trunk group and with what parameters.

There are two main challenges in consolidating this connectivity into the new host switch:

First, we have the trunk groups themselves. Does the host switch have existing trunks to cover all these bases – to all the same carriers, all the same local EAS switches, all the same 911 PSAPs? If not, do we want to set up new trunk groups for the missing items, or can we live without those services? Is it possible to move the existing trunk groups to the new host switch without changing the configuration (through point-code proxy / point-code sharing)? If we take that path do we have the capability of transporting those T1s to the new switch location?
Second, we have the translations. Is it possible to use the existing switch translations for the consolidated subscribers, or do we need to expand them to cover some extra trunk groups – or do we even need a separate set of translations for calls from the consolidated subscribers?

And then there’s billing, but I’ll give that a whole separate section.

4 – Billing

Even if a service provider no longer has a switch they still own their customers, they are still managing the local network to the customer premises, and they still need to be able to invoice their subscribers for service. Some aspects of the bill are straightforward – flat monthly fees for service or for certain call features (e.g. voicemail, call forwarding, etc) – but if you charge your subscribers usage charges (for LD calls, for international calls, for premium rate calls) then you need to consider how the CDRs will be processed on the host switch. Most likely the host service provider will have an existing billing company and you’ll need to work with that billing company along with the host service provider to gain access to CDRs generated by your customers.

On the flip side, if you receive fees for completing inbound calls (termination fees) to your subscribers then you’ll also need to make sure that it’s still possible to assess those fees after the consolidation (both legally and practically). It may be necessary to maintain separate PSTN trunk groups in some instances to allow the carrier to allocate fees separately between the two local service providers.

Simple network prior to consolidation

5 – Migration

Even if everyone’s clear about the goal, migrating a network to consolidate all class 5 switching functionality into a single endpoint without disrupting service is a big under-taking. At a high level, the project will go through the following phases.

Outbound testing: in this phase inter-machine trunks (IMTs) are set up between the two switches (ideally SIP trunks) which are then used to test outbound calls to various locations through the host switch. It’s during this phase that we would validate if the host switch has all the necessary PSTN connectivity and appropriate translations to handle outbound calls from the new subscribers. We can also begin to test the billing side of things.

Add a SIP IMT to permit outbound call testing

Inbound testing and trunk migration: in this phase we update the LERG and routing at interconnected switches so that all inbound calls route through the host switch (and then over the SIP IMT to the original switch). This may involve migrating PSTN trunks to the host switch, or it may involve simply retiring the old PSTN trunks if existing trunks on the host switch can be used. Once this phase is complete the original switch has no direct connectivity to the PSTN.

Route all inbound calls via host switch (now acting as a tandem) and remove PSTN connectivity from EOL switch

Subscriber migration: The final phase is to migrate subscribers to the host switch. This can be done in small or large groups – based on the urgency of the migration and the logistics of the particular situation. I’d recommend service providers start with a handful of friendly subscribers, and only begin large scale moves once everyone is confident that calls and billing are working well on the host switch.

Migrate all subscribers to host switch. Migration complete.

There’s a lot of complexity hidden in that overview. How will LNP (or toll-free or CLASS) TCAP queries work during (and after) the migration? How will subscriber configuration data be imported into the host switch? How are we handling voicemail and auto-attendant services? Can we really be confident that calls will still work after an instantaneous LERG change? Can we instead use point-code proxy to migrate trunks one-by-one in a controlled (and reversible) manner? How are we going to actually update the IP address of the registrar on hundreds or thousands of SIP devices?

6 – Post-Consolidation Management

Once you’ve successfully migrated all services to the host switch day-to-day operations will be very different. How will subscribers be provisioned and managed? Who has permission to access the host switch configuration? Can users be restricted to administering only some parts of the configuration – perhaps only their own subscribers? How will customer trouble tickets be handled? Who has the ability to access troubleshooting tools and logs?

If the host switch is a Metaswitch then it’s possible to use MetaView Web with delegated management groups (DMGs) to handle the provisioning and subscriber management side of things, but the host service provider will need to manage all centralized switch functions (such as translations changes and PSTN trunking). For troubleshooting Service Assurance Server is a fantastic tool, but doesn’t yet (correct me if I’m wrong) allow diagnostics to be partitioned by the DMG of the subscriber.

As with everything else, there are ways to handle these issues, but the involved parties should make a plan up-front, rather than discovering the challenges at the end of the switch consolidation process.

Conclusion

Consolidating one or more switches makes a lot of sense. It saves money and allows for technical expertise to be centralized, while allowing each local service provider to focus on serving their customers.

However, service providers should not under-estimate the complexity of the migration process itself, and the adjustment required in operational processes after the consolidation is complete. Even in this lengthy article, I’ve barely scratched the surface for what is required to execute a successful switch consolidation – so executives and operational staff should work together to understand all that is involved in a consolidation project to ensure they have a good plan and have allocated sufficient time for the migration project.