When BGP Goes Rogue: Dissecting Verizon's September 30th Outage

On Monday, September 30, 2024, Verizon Wireless customers spent most of the day staring at "SOS-only" on their phones. Downdetector logged somewhere between 66,000 and 1.5 million user reports. Estimates vary widely, but it is clear this was not a minor hiccup. While Verizon's official statement blamed a vague "network issue," I dug into the routing data and found something interesting.

Having recently completed Georgia Tech's Computer Networks course (highly recommended for those who want to understand how internet routing actually works), I could not resist investigating this widespread connectivity failure and immediately dove straight into RIPE's RIS data.

What did the data show? Two competing BGP paths for Verizon Wireless (AS 6167) were flip-flopping throughout the outage. One was a clean North American route. The other? A questionable path that leaked private network identifiers and took a detour through Hong Kong. When your phone's data session depends on stable routing, this kind of chaos can easily break connectivity.

The Root Cause?

While the actual cause of the outage remains officially unknown, the following is my analysis of publicly available data covering the outage period on September 30, approximately between 9:00 a.m. ET and 5:30 p.m. ET. The data from RIPE's RIS Route Collector 00, collecting data through peer AS 131477, during this period, captured repeated oscillations between two significantly different paths from the collector to Verizon Wireless.

The short clean path (131477 → 60068 → 7922 → 701 → 22394 → 6167) remains within North America after leaving the collector; however, the second path proved quite troublesome. That path (131477 → 65511 → 140096 → 150684 → 3491 → 701 → 22394 → 6167) raised immediate red flags.

First and most importantly, the second path is routed through AS 65511, which is a special-use ASN reserved for private networks. These AS's should never appear on the public internet. For anyone familiar with basic networking and not BGP, this is equivalent to a 192.168.0.0/24 appearing on the public internet. Next, the path took an interesting twist; instead of routing from Europe directly to North America over one of the high-speed underwater cables connecting the continents, it first took a multi-stop detour through the ASNs of multiple Hong Kong-based infrastructure providers, Jinx Co. and PCCW Global.

Here is what the raw RIB data looked like during the oscillations:

U|A|1727713781.000000|singlefile|singlefile|||131477|103.102.5.1|174.207.160.0/19|103.102.5.1|131477 60068 7922 701 22394 6167|6167|7922:409 7922:3000 60068:201 60068:444 60068:2000 60068:2150 60068:7120||
U|A|1727713965.000000|singlefile|singlefile|||131477|103.102.5.1|174.207.160.0/19|103.102.5.1|131477 65511 140096 150684 3491 701 22394 6167|6167|3491:2000 3491:2004 3491:9002||
U|A|1727717597.000000|singlefile|singlefile|||7018|12.0.1.63|174.207.160.0/19|12.0.1.63|7018 701 22394 6167|6167|7018:5000 7018:37232||
U|A|1727717657.000000|singlefile|singlefile|||131477|103.102.5.1|174.207.160.0/19|103.102.5.1|131477 60068 7922 701 22394 6167|6167|7922:409 7922:3000 60068:201 60068:444 60068:2000 60068:2150 60068:7120||
U|A|1727717855.000000|singlefile|singlefile|||131477|103.102.5.1|174.207.160.0/19|103.102.5.1|131477 65511 140096 150684 3491 701 22394 6167|6167|3491:2000 3491:2004 3491:9002||

Observe the AS-path column (field 12), which flips between the clean North American route and the Hong Kong detour with the leaked private ASN 65511 for the duration of the outage.

Three Theories: Hijack, Leak, or Glitch?

For those unfamiliar with BGP, think of it as a highway system for the internet where every city trusts road signs posted by neighboring cities. A careless road crew or a malicious actor can redirect traffic miles off course before anyone notices and corrects it.

Theory 1: Private ASN Leak

AS 65511 is the smoking gun here. Private ASNs are intended to remain internal, similar to private IP addresses (10.0.0.0/8, 192.168.0.0/16). Seeing one in global routing tables typically indicates a misconfigured router. This type of misconfiguration would be akin to Verizon's 2019 Allegheny incident all over again, where a similar configuration error disrupted service for hours.

Theory 2: Malicious Hijacking

Someone could have crafted bogus route announcements to inject the alternate path. Every ASN in the path exists and maps to real organizations, but this becomes more concerning in a geopolitical context. We've seen BGP hijacking incidents originate from this region before. PCCW Global (AS 3491) is approximately 18% foreign-government owned, which raises eyebrows. Without deeper investigation, definitive evidence of malicious intent remains elusive.

Theory 3: Internal Reconvergence Bug

Verizon's own routing logic might have oscillated between paths due to flaky internal systems, disrupting the GRE/IPsec tunnels that mobile infrastructure depends on. Occam's razor would more likely suggest internal misconfiguration over sophisticated nation-state tampering.

Without Verizon publishing root cause analysis, I am left connecting dots.

Why Internet Routing Breaks Your Phone

Modern cellular networks are not like the old days. Every voice call and data session from LTE/5G cell sites rides IP backhaul networks. Here is where it gets interesting: when BGP routing changes disrupt the tunnels between cell towers and Mobile Switching Centers, the cellular equipment cannot properly authenticate your device properly, your phone falls back to emergency-only mode.

While SS7 once served as the signaling backbone of mobile communications, today's cellular networks rely heavily on TCP/IP and BGP to establish and maintain connectivity. Voice and data traffic from LTE and 5G networks traverse IP backhaul networks, often encapsulated using GRE or IPsec tunnels. BGP governs how this traffic is routed, while MPLS provides fast, deterministic paths inside carrier networks between switching centers.

This architecture means that mobile sessions depend on stable, policy-driven IP routing, rather than just legacy signaling protocols. When BGP behaves unpredictably, it not only confuses routers but also disrupts the tunnels and control plane operations that mobile connectivity depends on. The reliance on BGP reveals the hidden brittleness of our hyper-connected world: your phone's ability to make calls depends on global internet routing working correctly.

Four Lessons for Network Operators

Through all of this, there definitely are some lessons for network operators.

First, filter private ASNs at your borders. They belong inside your network, not in the global routing table. No exceptions.

Second, operators need to deploy RPKI ROA (Route Origin Authorizations) validation universally. ROV (route origin validation) alone wouldn't have stopped this incident because the prefix origin (AS 6167) remained valid. You need additional mitigations, such as private-ASN filtering, provider authorization, or strict route-policy controls. erizon had started deploying RPKI at the time, but it hadn't been implemented universally across all of their autonomous systems, which is another reason why this routing anomaly slipped through.

Third, monitor for routing anomalies proactively using tools like RIPE RIS or Cloudflare Radar to alert on unexpected geography changes and private ASN appearances before they cause outages.

Finally, operators should always publish incident post-mortems. Transparency is not a security risk; it is how the internet community learns and prevents repeat incidents. Silence helps no one.

This incident reminds us that the internet's routing system still operates primarily on trust. And trust, as we have seen, is not always enough.