Mistake, Mistake, Blackhole

Three Mistakes in a Boat (To Say Nothing of the Outage)

Yesterday, on 12.11.2018 a BGP configuration mistake happened at Mainone Cable Company (AS37282), a Nigerian ISP. It mainly hit two content providers: Google (AS15169, AS36384, AS36492, AS43515) and Cloudflare (AS13335). Leaked routes were accepted by its direct upstream, China Telecom (AS4809), further advertised in Russia to TTK (AS20485) and finally learned by NTT (AS2914) in Europe. After reaching the Tier-1 providers level leaked prefixes propagated globally, redirecting traffic to unusual Europe-Russia-China-Nigeria route.

This incident affected 214 prefixes and lasted more than an hour, leading to an increased network latency and possibly resulting in DoS.

The victims may accuse both the leaker AS and transit ISPs, which propagated leaked prefixes globally. TTK and China Telecom, which have already got an attention due to Naval War College’s paper, prove to have improper ingress filters at their links. The NTT role is less obvious since Google and Cloudflare are included in TTK’ AS-SET.

This was not the first time when Mainone Cable Company activity was detected by Radar monitoring system. For months AS37282 was constantly leaking prefixes from own customer cone between its providers. Such a weird behavior might occur as a result of the next sequence of misconfigurations:

• Union of customer’s route objects is used to build egress filters at links with upstreams/peers;
• No use of BGP communities to check the source of the route.

As a result, if such ISP receives customer prefixes from a provider or peer, it redistributes those to other providers and peers. Network with such configuration can be compared to a time bomb: customers could not disconnect from such a network, even if they drop the BGP session; and if something odd is added in its AS-SET, it would result with an incident similar to what we saw yesterday.

Making BGP configuration more native and user-friendly, thus helping to avoid such mistakes, may be hard to achieve, but worth trying. Our protocol extension draftaddressing this particular problem should have significant progress in the near future. So stay tuned, check the IETF mailing lists and scroll through upcoming software updates list.