Commercial Network Services
Network Issues
You are here: Portal Home > Network Issues > Resolved

Test Status Last Upd Next Upd Upd Interval Jun Uptime May Uptime
Commercial Network Services (LA) Ping
UP
07:08 AM 07:13 AM 5 min 100.000% 100.000%
Commercial Network Services (NYC) Ping
UP
07:08 AM 07:13 AM 5 min 100.000% 100.000%
Commercial Network Services (UK) Ping
UP
07:08 AM 07:13 AM 5 min 100.000% 100.000%

Website monitoring by Hyperspin

Undersea cable fault near Singapore (Resolved)
Affecting Other - Traffic in/out of Malaysia and Singapore | Priority - Low

Subscribers connecting from Malaysia may experience higher latency to their hosted service due to a submarine cable fault.  If you are experiencing an issue, please contact CNS support.  We may be able to route service around the trouble for you.  The cable is expected to be repaired by March 31st.

We have sent an urgent peering request to Telekom Malaysia, to peer in Europe.  This should resolve the issue for all mutual subscribers.  We will update after we hear back from them.

Update 3/9:  Telekom Malaysia has responded positively to our peering request to peer in UK.  This will help both traffic to both our UK and US datacenters.  We will update this report again as soon as peering is established.

Update 3/14:  Telekom Malaysia has scheduled to turn up peering with CNS in UK on the night of 3/23-24.  Telekom Malaysia already peers with CNS in Los Angeles, however this is over the faulty cable.  Adding a peering point in the UK should substantially improve connectivity to all CNS datacenters.

Update 3/30:  We received information from Telekom Malaysia that the peering has been rescheduled for April 4-5.

No further updates have been received from Telekom Malaysia.  We are also not receiving any reports of trouble from affected subscribers.  As a result we are closing this issue but will re-open if necessary.

--

http://subtelforum.com/articles/update-from-telekom-malaysia-on-restoration-works-to-repair-submarine-cable-fault/


Press Release

Telekom Malaysia Berhad (TM) earlier announced that we have detected a fault on the Asia Submarine Cable Express (ASE) system off Singapore, which affected our Cahaya Malaysia submarine cable linking Malaysia to North Asia and the United States. ASE consists of 6 fiber pairs of which TM owns 2 fiber pairs, named as Cahaya Malaysia.

 

We would like to reiterate that these cable faults have caused international link outages which affected the browsing experience of not just Internet customers of TM, but also other Internet users in Malaysia and in the region as well. This is as most of the regional traffic would have to pass the affected submarine cable systems.

 

During this period, Internet users may experience some degree of service degradation such as slow browsing and high latency while accessing contents hosted in the United States (US), North Asia and Europe via the affected cables. However, we wish to note that our IPTV service, HyppTV is not affected by the outages as the service utilises TM’s domestic backhaul network.

 

Whilst we are working with our consortium members in other countries to restore Cahaya Malaysia, we are also proactively rerouting traffic to alternative routes to minimise impact to our customers. The rerouting options available in South East Asia are also hampered by minor faults and planned maintenance on several other sea submarine cables around the region, namely Asia Pacific Cable Network 2 (APCN 2), South East Asia – Japan Cable (SJC), South East Asia – Middle East – Western Europe 4 (SEA-ME-WE 4). We are actively managing this dynamic situation together with consortium partners and would like to assure customers that we are undertaking all necessary measures to ensure that customers continue to experience uninterrupted service.

We will continue to provide necessary updates on the progress of the restoration works via our official customer support accounts on Twitter @TMConnects as well as on Facebook at Everyone Connects, and will monitor feedback from our customers on the quality of service experienced.

We wish to thank our customers for their understanding and patience for this capacity reduction affecting operators in the whole region. Should you have any questions or require any assistance, you can reach us at @TMConnects? or via Everyone Connects? Facebook page?. We can also be contacted via email at help@tm.com.my. ?

Date - 03/03/2016 07:09
Last Updated - 05/27/2016 08:44

Peering to NYIIX offline (Resolved)
Affecting Other - Peering in NYC to NYIIX | Priority - Low

Our peering link to NYIIX is experiencing trouble and currently offline.  There should be minimal impact as we have Equinix NYC and DE-CIX NYC peering active in the region.  However, some subscribers may experience slightly higher latency until the circuit is repaired.  We are waiting on an ETA from the provider.

RESTORED


Date - 04/18/2016 22:15 - 04/19/2016 09:36
Last Updated - 04/19/2016 09:36

UK Network Maintenance (Resolved)
Affecting Other - UK Network Connectivity | Priority - Low

During our maintenance window opening Friday, April 8 (after all markets have closed), we will perform work to return an out of service router back to service.  During this time some peering links in Europe will reset, causing a brief interruption to connectivity for some end users while BGP routes reconverge.  This process takes less than five minutes.

Date - 04/08/2016 19:00 - 04/09/2016 17:03
Last Updated - 04/09/2016 07:30

Network Upgrades (Resolved)
Affecting Other - Many routers - all datacenters | Priority - Low

During this maintenance window, some services may experience periods of high latency as we work to upgrade routing and switching equipment.

All upgrades are complete except UK.  We are working to resolve a maintenance related issue.  As a result, LINX and Equinix Connect peering is currently offline for an extended period during this maintenance window.



COMPLETE

Date - 04/01/2016 19:00 - 04/03/2016 12:00
Last Updated - 04/03/2016 21:05

Any2 IX offline (Resolved)
Affecting Other - Any2 peering traffic in/out of Los Angeles | Priority - Medium

We have detected Any2 Los Angeles IX is offline.  Any2 IX is the largest Internet traffic exchange on the US west coast.  Traffic transiting thru Any2 will be affected until the associated networks heal or Any2 comes back online.

Update 1:57PM:  We have seen stability over the past 7 minutes.  We are continuing to monitor and will drop the IX if it bounces again until we hear back from the IX NOC.

Update 3:05PM:  We have observed stabilit for the past hour.  We will continue to monitor but for now are considering the issue closed.





Date - 03/08/2016 13:43
Last Updated - 03/08/2016 15:05

Platform Updates (Resolved)
Affecting Other - All virtualzied services | Priority - Low

Please be advised many VM's and services hosted on VM's will be hibernated for up to one hour while we complete important updates to our platform.  No reboots are anticipated.  This is a continuation of last weeks maintenance and so if your service was affected last week it is unlikely to be affected this week.


Date - 02/27/2016 00:00 - 02/29/2016 16:08
Last Updated - 02/25/2016 06:52

Platform Maintenance (Resolved)
Affecting Other - VM Hosting Platform | Priority - Low

Many VM's will be paused for up to one hour while we complete maintenance to our hosting platform.  No reboot is planned.  The VM will resume as soon as maintenance is complete.

Date - 02/20/2016 00:00 - 02/21/2016 00:00
Last Updated - 02/21/2016 13:58

Maintenance (Scheduled) (Resolved)
Affecting Other - Hosted VM's | Priority - Low

CNS will be completing maintenance over the weekend of November 13.  Many hosted services will be paused briefly so we can update our infrastructure.  New provisioning will be held until maintenance is complete.


 

Date - 11/13/2015 19:30 - 11/15/2015 07:51
Last Updated - 11/14/2015 12:15

Network Updates (Resolved)
Affecting Other - All hosted services | Priority - Medium

We are applying a security patch to all public facing routers.  There may be brief periods of no connectivity.  This procedure will be completed well before markets open.

Date - 11/01/2015 08:51 - 11/02/2015 11:15
Last Updated - 11/01/2015 08:52

Transit within Europe from UK DC (Resolved)
Affecting Other - edge reachable by transit in UK | Priority - High

An upstream transit provider in UK is experiencing techncial issues.  This has caused periods of intermittent connectivity for some traffic entering the CNS network via transit links.

Most all traffic in/out of our UK datacenter travels through peering links and is unaffected.

We have isolated the issue specifically to an upstream transit network and have configured our network to route around them by utilizing our transit networks reachable from NYC and Los Angeles.  This will increase latency for traffic that travels out these links while the issue is resolved.

We are working with the transit provider to setup alternate peering.  Depending on the issue within their network, this may be accomplished rather quickly.  We will continue to optimize the detour until we hear back from the transit provider.

We will update this report after more information is available.

UPDATE 8/21 10:04AM PST:  Alternate routes optimized.  Transit latency is close to normal.  We will continue to monitor.  Please contact CNS support if you are experiencing any issues.

UPDATE 8/24 11:33AM PST: We have applied previous re-routing configuration after detecting more transit issues in the UK.  We will post further updates as we have more information to report.

UPDATE 8/25 6:38AM PST: This issue is potentially resolved.  We will return European routing to normal after we observe one hour of stability on the transit providers network.    Until then, we will continue to route around in order to maintain stable connectivity.

UPDATE 8/25 9:56AM PST: We are still seeing dropout on the european transit links.  We have further optimized the detour around, and so latency should be much better -  but not great.  We will post further updates as we have more information to report.

UPDATE 8/26 2:53PM PST:  Upstream tranist issues have been declared resolved by the provider for one of two legs.  After having observed one hour of no packet loss, we have now brought the transit route back online.  Latency for transit routes has returned to normal.  We will continue to observe the routes closely and report any issues here.

UPDATE 8/27 9:40PM PST: We will bring up the second leg of the UK transit providers uplink during our regular maintenance window this weekend. 

RESOLVED



Date - 08/19/2015 17:34
Last Updated - 08/27/2015 21:42

« Prev Page   Next Page »