Serious Times — Serious Leaks

At 17:13 UTC on March 31, 2020, the AS50048 (NEWREAL-AS) leaked, in total, 2658 IPv4 network prefixes to the Tier-2 transit provider Transtelecom. Those prefixes included Orange, Akamai, Rostelecom and more than 300 other companies’ networks.

This route leak, in particular, is significant because of its spread. There were several waves, containing separate subnetworks. In fact, the total amount of separate IP addresses (approximately 13.5 million) was even higher than in the largest wave! Due to the periodic nature of the session's capture (and keeping in mind that each wave contains mostly unique prefixes), we suppose that Transtelecom has a route rate limit set with each customer. This can be one of the reasons why this incident didn't transform into an epic disaster — just the simple rate limit.

Compared to the day-to-day BGP operations, such a route leak could constitute a critical threat to the functioning of computer networks all over the world, while those networks are already under stressful conditions by almost worldwide quarantine.

Right now, we can only guess why exactly this particular route leak occurred and spread so far. The owner of the AS50048 has made some mistakes in network configuration, although it's still unclear how NEWREAL-AS upstream AS20485 could accept and further propagate those prefixes. We know that the NEWREAL-AS connection with this provider has been established long ago; however, during this incident, we observed several session restarts due to congestion.

As we've already written, due to the — almost worldwide quarantine measures against the coronavirus pandemic, there is a forceful redistribution of vast amounts of traffic between users endpoint connections. Quite simply, most of the people have already moved away from offices and their corporate networks. They are working from home, which causes significant traffic redistribution, changes in the traffic profile, and the overload of transit and peering facilities, globally.

In such a situation, the majority of ISPs and operators would be working sleepless nights, holding things together with a piece of chewing gum and string, quickly fixing and re-architecting their networks and peering facilities. This inevitably leads to human errors and, given the current sad state of the BGP security, could cause and, as we can see, already caused disrupting consequences.

We strongly warn everyone that this would certainly not be the last serious incident during the coronavirus crisis.

Update: Diverted Netherlands to Brasil traceroute example

Before:
trace from Amsterdam to Brazil at 07:46 Mar 30, 2020
1 *
2 80.249.208.50    NL-AMSIX-PEERINGLAN1           Amsterdam   Netherlands       0.786
3 109.105.98.33    nl-sar.nordu.net               Amsterdam   Netherlands       1.416
4 109.105.97.125   uk-hex.nordu.net               Amsterdam   Netherlands       5.556
5 195.66.224.153   linx.lon.seabone.net           London      United Kingdom   88.265
6 5.178.43.214     TI Sparkle Seabone POP         Fortaleza   Brazil          257.636
7 5.178.43.174     TI Sparkle Seabone POP         Fortaleza   Brazil          253.064
8 177.104.118.174  T-NET WIRELESS E INFORMÁTICA   Itaitinga   Brazil          253.902

During:
trace from Amsterdam to Brazil at 17:15 Mar 31, 2020
1  *
2  80.249.210.177   NL-AMSIX-PEERINGLAN1            Amsterdam    Netherlands   12.617
3  188.43.5.114     kzn06rb.transtelecom.net        Kazan        Russia        71.279
4  188.43.5.113     Chuenkov-gw.transtelecom.net    Voskresensk  Russia         71.38
5  195.222.173.101  Russia Moscow, GOLDEN Telecom   Moscow       Russia        68.692
6  *
7  5.178.43.174     TI Sparkle Seabone POP          Fortaleza    Brazil       249.881
8  177.104.118.174  T-NET WIRELESS E INFORMÁTICA    Itaitinga    Brazil       246.094

"Active measurement helps to illustrate the traffic misdirection that takes place when a routing leak occurs. Traceroutes show the impact on latency and the geographic detour that simply can’t be understood from BGP alone. In this example, traffic from AMSIX to Brazil is inadvertantly misdirected through central Russia as a result of AS50048's leak."

Quote and traceroute example were provided by Doug Madory, Director of Internet Analysis, Oracle Internet Intelligence.