Commercial Network Services
Network Issues
You are here: Portal Home > Network Issues > Resolved

Test Status Last Upd Next Upd Upd Interval Nov Uptime Oct Uptime
Commercial Network Services (LA) Ping
MAINTENANCE
- - 5 min 100.000% 100.000%
Commercial Network Services (NYC) Ping
MAINTENANCE
- - 5 min 100.000% 100.000%
Commercial Network Services (UK) Ping
MAINTENANCE
- - 5 min 100.000% 100.000%

Website monitoring by Hyperspin

Power issue in NY (Resolved)
Affecting Server - VMM | Priority - Low

We are working to resolve a power distribution issue affecting a limited number of VMs in NY.  The issue is causing CPU modulation, which results in a slow server.

UPDATE:  Engineers are setting up temporary power to bring back full performance while damaged equipment is replaced.

INFO:  A power issue destroyed a PDU, network switch and multiple PSU's in about 5 servers.  The network in this rack switched to a redundant switch and the damaged switch has already been returned to normal service.  We have completed running new (temporary) power and replacing PSU's to restore normal quality of service while a proper repair is completed.

Please let CNS Support know if you are experiencing any issues.

As QoS has been restored, priority of this issue has been reduced.  We do not anticipate any further impact to services.

UPDATE:  Full redundancy restored.  All systems normal






Date - 09/04/2017 10:40
Last Updated - 09/06/2017 12:45

Peering to AMS-IX (Resolved)
Affecting Other - Peering to AMS-IX | Priority - Medium

Peering to AMS-IX has flapped.  This will cause brief connectivity issue for traffic that transits the connection while routes reconverge.  We are investigating the cause.  We will update this notice with more information as soon as a review is complete.

---

The original notice listed the IX as DE-CIX.  It is actually AMS-IX

---

We are observing continued instability in the circuit and so the port has been administratively shutdown until the issue is resolved.  Traffic will route around the IX.  No major impact is expected.

---

The problem circuit has been repared and the network has been normalized.

The circuit vendor has responded they are experiencing a major outage.  We will leave the IX offline until they call all clear.



Date - 05/15/2017 07:32
Last Updated - 05/17/2017 07:44

Amsterdam P2P down (Resolved)
Affecting Other - Peering links into Amsterdam | Priority - Medium

Service to Amsterdam has been restored.  The network has been normalized.
____________

A dedicated circuit connecting our network to an IX in Amsterdam has gone down.  Possible fiber cut.  We have raised a ticket with the vendor.


This may cause increased latency to parts of Europe and Africa, but it will not create an outage because there are multiple redundant routes to the affected networks.

We will update this notice as soon as service to Amsterdam IX is restored.




Date - 05/04/2017 08:55
Last Updated - 05/04/2017 21:35

FXCM routing thru China Mobile (Resolved)
Affecting Other - FXCM routing | Priority - Critical

We have received a report of high latency to FXCM.  Our investigation has revealed that China Mobile is advertising their prefix, causing this traffic to route through their network.

We are working to filter the advertisement.

UPDATE:  We have filtered out the hyjacked route.  All appears normal.

We will continue to monitor.



Date - 01/29/2017 18:53 - 01/29/2017 23:15
Last Updated - 01/29/2017 19:07

Subscriber Control Panel Upgrade (Resolved)
Affecting Other - Subscriber Control Panel | Priority - Low

Please be advised the CNS Subscriber Control Panel will be unavailable during scheduled maintenance so we can install software upgrades.  If you require assistance during this time, please call the helpdesk at any number below:

San Diego, CA: +1 (619) 225-7882

Los Angeles, CA: +1 (213) 769-1787

New York, NY: +1 (646) 930-7435

London, UK: +44 (2035) 191453

 

 

Date - 07/02/2016 00:30 - 07/04/2016 07:43
Last Updated - 06/30/2016 15:51

Circuit issue in NY region (Resolved)
Affecting Other - NY POP | Priority - High

Two circuits in the NY region have been cut somewhere outside our cage.  We are working with the datacenter to get them re-run as fast as possible.  In the mean time, increased latency may be experienced as we have routed around the trouble.

One of two have been replaced.  Currently waiting on second line run and we will then normalize the network.

Damaged fiber has been replaced.  We are now testing before normalizing the network.

All tests have passed with no packet loss.  We are proceeding to normalize the network.

Network has been normalized.  We are observing to confirm.

Some routes are still reconverging.  we are observing












Date - 06/13/2016 08:03 - 06/14/2016 07:57
Last Updated - 06/13/2016 12:05

Network Maintenance [extended] (Resolved)
Affecting Other - Network | Priority - Low

We are performing maintenance on our network.  During this time some routes may be intermittent.  We are working to complete the maintenance as fast as possible.

Hardware issue during maintenance created complications.  We are finishing it up.  Apologizes for the delay.


Date - 06/12/2016 00:18 - 06/12/2016 18:20
Last Updated - 06/12/2016 10:37

Packet Loss off network (Resolved)
Affecting Other - UK | Priority - High

We are investigating reports of packet loss.  It appears rooted off our network.  We have routed some ISP's out and are announcing UK prefixes out of NYC (in addition to UK) while we investigate. 

It does not appear to be affecting local UK links.

More to follow as it becomes available.

Connections stable.  We are still investigating the source

Network has been normalized.  Countermeasures have been deployed in response.

We will of course continue to monitor.






Date - 05/27/2016 07:28 - 05/27/2016 11:20
Last Updated - 05/29/2016 09:53

Undersea cable fault near Singapore (Resolved)
Affecting Other - Traffic in/out of Malaysia and Singapore | Priority - Low

Subscribers connecting from Malaysia may experience higher latency to their hosted service due to a submarine cable fault.  If you are experiencing an issue, please contact CNS support.  We may be able to route service around the trouble for you.  The cable is expected to be repaired by March 31st.

We have sent an urgent peering request to Telekom Malaysia, to peer in Europe.  This should resolve the issue for all mutual subscribers.  We will update after we hear back from them.

Update 3/9:  Telekom Malaysia has responded positively to our peering request to peer in UK.  This will help both traffic to both our UK and US datacenters.  We will update this report again as soon as peering is established.

Update 3/14:  Telekom Malaysia has scheduled to turn up peering with CNS in UK on the night of 3/23-24.  Telekom Malaysia already peers with CNS in Los Angeles, however this is over the faulty cable.  Adding a peering point in the UK should substantially improve connectivity to all CNS datacenters.

Update 3/30:  We received information from Telekom Malaysia that the peering has been rescheduled for April 4-5.

No further updates have been received from Telekom Malaysia.  We are also not receiving any reports of trouble from affected subscribers.  As a result we are closing this issue but will re-open if necessary.

--

http://subtelforum.com/articles/update-from-telekom-malaysia-on-restoration-works-to-repair-submarine-cable-fault/


Press Release

Telekom Malaysia Berhad (TM) earlier announced that we have detected a fault on the Asia Submarine Cable Express (ASE) system off Singapore, which affected our Cahaya Malaysia submarine cable linking Malaysia to North Asia and the United States. ASE consists of 6 fiber pairs of which TM owns 2 fiber pairs, named as Cahaya Malaysia.

 

We would like to reiterate that these cable faults have caused international link outages which affected the browsing experience of not just Internet customers of TM, but also other Internet users in Malaysia and in the region as well. This is as most of the regional traffic would have to pass the affected submarine cable systems.

 

During this period, Internet users may experience some degree of service degradation such as slow browsing and high latency while accessing contents hosted in the United States (US), North Asia and Europe via the affected cables. However, we wish to note that our IPTV service, HyppTV is not affected by the outages as the service utilises TM’s domestic backhaul network.

 

Whilst we are working with our consortium members in other countries to restore Cahaya Malaysia, we are also proactively rerouting traffic to alternative routes to minimise impact to our customers. The rerouting options available in South East Asia are also hampered by minor faults and planned maintenance on several other sea submarine cables around the region, namely Asia Pacific Cable Network 2 (APCN 2), South East Asia – Japan Cable (SJC), South East Asia – Middle East – Western Europe 4 (SEA-ME-WE 4). We are actively managing this dynamic situation together with consortium partners and would like to assure customers that we are undertaking all necessary measures to ensure that customers continue to experience uninterrupted service.

We will continue to provide necessary updates on the progress of the restoration works via our official customer support accounts on Twitter @TMConnects as well as on Facebook at Everyone Connects, and will monitor feedback from our customers on the quality of service experienced.

We wish to thank our customers for their understanding and patience for this capacity reduction affecting operators in the whole region. Should you have any questions or require any assistance, you can reach us at @TMConnects? or via Everyone Connects? Facebook page?. We can also be contacted via email at help@tm.com.my. ?

Date - 03/03/2016 07:09
Last Updated - 05/27/2016 08:44

Peering to NYIIX offline (Resolved)
Affecting Other - Peering in NYC to NYIIX | Priority - Low

Our peering link to NYIIX is experiencing trouble and currently offline.  There should be minimal impact as we have Equinix NYC and DE-CIX NYC peering active in the region.  However, some subscribers may experience slightly higher latency until the circuit is repaired.  We are waiting on an ETA from the provider.

RESTORED


Date - 04/18/2016 22:15 - 04/19/2016 09:36
Last Updated - 04/19/2016 09:36

« Prev Page   Next Page »