Last Friday, June 12th at around 4:30pm, Telekom Malaysia’s (TM) UniFi and Streamyx broadband internet services experienced a disruption that would last for around two hours until 6:30pm, at least on its official clock.
The behavior of the outage is such — all corner of Malaysia were affected, except curiously in some regions. To the rest of us, almost all websites not hosted locally were inaccessible, and there are reports that even The Star Online can’t be loaded. Even for folks who are on Time Fiber and Maxis Fiber (including us), experienced a severe slowdown, although websites were able load eventually for us, during that time frame.
TM did not immediately acknowledged this, and naturally its support lines have been flooded with help requests. This has led to the internet community to take it to the LowYat.NET forum to discuss the issue. Another discussion can be found at Hacker News. As one poster described it, this outage is a route leak where an ISP declares it is the best route to reach sites that contain, say facebook.com and google.com, effectively causing the ISP to DDoS itself or one of its downstream partners.
CloudFlare’s status page confirmed this issue at around 6pm, but did not disclose TM’s name, stating, “An ISP in Asia is leaking routes to a Tier 1 transit provider, this may affect reachability issues for some ISPs to the global internet.”
BGPmon, now part of OpenDNS, later corroborated that there were a “massive route leak” initiated by TM (AS4788), which caused “significant network problems” for the global routing system. Level3 Communications, formerly known as Global Crossing or GBLX (AS3549), was the primary internet service provider that’s affected. At 4:43pm Malaysian Time on June 12th, TM announced about 179,000 of prefixes to Level3, which in turn accepted these prefixes and propagated them to their customers.
TM has effectively inserted itself in between thousands of these prefixes and Level3, making itself responsible to deliver these packets to their intended destinations. This has resulted in a massive packet loss and internet slowdown in many parts of the world, particularly in the Asia Pacific region. The graph above shows the packet loss measured by OpenDNS between London and Hong Kong, over Level3, but the same loss patterns can be observed from other Level3 locations globally like Singapore, Hong Kong, and Sydney.
Because of this, at the same time, the round trip between these destinations also shot up, as shown by this OpenDNS graph:
The exact time the issue began can also be seen by the number of Border Gateway Protocol (BGP) messages processed by BGPmon — around 4:43pm MYT — as seen in the graph below. Around 6:40pm MYT is when improvements were observed, and 7:15pm MYT is when connections began to clear up.
Telekom Malaysia’s official statement for this issue on June 12th itself, has been updated a day later to mention the route leak issue:
Internet Services Disruption
Update on 13 June 2015Telekom Malaysia Berhad (TM) wishes to update on the service related issue detected yesterday, 12 June 2015 affecting a number of our internet services customers that caused a deterioration in connection performance.
We identified the root cause and our network team immediately took steps to optimize traffic flows, while we worked to restore connectivity to its expected level of performance. The services were restored at 6:30pm on the same day.
We would like to clarify that during a network reconfiguration exercise, we had unintentionally updated traffic routing information which caused congestion and packet loss to our international connectivity. This had affected the internet traffic flow for some of our customers and some international traffic routes.
We apologize for any inconvenience caused by the service disruption and would like to assure the customers that we are undertaking all the necessary measures to ensure customers continue to experience uninterrupted services.
Meanwhile, customers who have any inquiry or require further assistance can email us at help@tm.com.my or tweet to us via @TMConnects on Twitter.
Thank you.
The post Telekom Malaysia initiated a route leak causing internet disruption appeared first on TechAttack.my.