Commercial Network Services
Network Issues
You are here: Portal Home > Network Issues > Resolved

Test Status Last Upd Next Upd Upd Interval Jul Uptime Jun Uptime
Commercial Network Services (LA) Ping
UP
11:08 AM 11:13 AM 5 min 100.000% 100.000%
Commercial Network Services (NYC) Ping
UP
11:08 AM 11:13 AM 5 min 100.000% 100.000%
Commercial Network Services (UK) Ping
UP
11:08 AM 11:13 AM 5 min 100.000% 100.000%

Website monitoring by Hyperspin

LINX Packet Loss (Resolved)
Affecting Other - UK | Priority - Low

During this past week, we experienced  a period of high bandwidth and packet loss on IX interfaces in NY and UK and across our NY-UK backbone.  We believe this was caused by a routing loop with an unknown peer between Europe and America.  The issue went away after many peers on LINX in the UK were dropped.

We have started reconnecting peers one at a time in order to identify the loop.  At that time, we will address the loop either through an internal filter, working with the peer, or both.  (We need to identify it in order to determine how to best address it).

There can be expected a brief period of packet loss as we work to reproduce the issue.  We will work to minimize it as much as possible.

 
----

This problem has been mitigated. 

We believe this issue occured because of a routing loop with a peering network.   Most peers on LINX have been cut away until we can identify it after markets close.

---

We are experienicng major packet loss on traffic entering our network at LINX in London.  We are in the process of cutting the IX away until the issue is resolved.



Date - 06/13/2018 09:48 - 06/17/2018 11:50
Last Updated - 06/16/2018 08:07

NYIIX temporary disconnect (Resolved)
Affecting Other - NYIIX Connectivity | Priority - Low

Due to technical issues being experienced by the New York Internet Exchange (NYIIX) and their intention to interrupt services on the IX over the week to apply a fix, we will be temporarily shutting down all peering sessions on the exchange until after the work is complete.  Very little impact is expected because most peers have redundant links through DE-CIX NY or Equinix NY.

It is possible some networks will revert to transit service until after the work is complete and all peers are reconnected.



Date - 06/17/2018 12:19 - 06/22/2018 09:57
Last Updated - 06/15/2018 12:22

UK Connectivity (Resolved)
Affecting Other - UK | Priority - Low

A power anomaly in the datacenter has damaged PDUs powering key switching equipment.  Power has been re-routed and the damaged equipment will be replaced during our maintenance window.

 

At this time all services are operational.  Please contact our helpdesk if you are unable to connect or experience any issues.

This event appears to have damaged a router.  It has been removed from service.

Multiple power supplies were destroyed.  They have been already or will be replaced during our maintenance window.

All work to repair or replace damaged equipment is complete.

We believe a power supply blew and sent a surge back into the PDU, damaging connected equipment or causing it to malfunction.

------------

 

 Detail:

 

We are investigating reports of connectivity issues in UK.

We appear to have lost both power legs to key equipment.  We are  working with Equinix to investigate.

CONFIRMED blown PDU(s) have taken down key switches.  We are working to re-route power as quickly as possible.

Power to key equipment has been restored through temporary circuits while work is completed to replace damaged PDU's.

2:45PM PST At this time all services should be operational.  Please contact our helpdesk if you are not online.

Full replacement of power equipment will be conducted during our maintenance window after markets have closed to avoid any possible interruption to services.

Status changed to "Scheduled"

Updates will be provided as available.


Date - 06/05/2018 10:56 - 06/09/2018 18:19
Last Updated - 06/09/2018 18:19

Abusive traffic - NYC (Resolved)
Affecting Other - NYC Network | Priority - Critical

CNS has mitiageted abusive traffic entering our network.  Please contact CNS Support if you are experiencing a degraded network so we may investigate.

Date - 06/01/2018 08:25 - 06/03/2018 13:13
Last Updated - 06/01/2018 08:26

Maintenance Updates (Resolved)
Affecting Other - All Services | Priority - Low

All services are subject to a brief pause on Saturday so we can complete security audits and platform maintenance.

Date - 04/21/2018 00:00 - 05/13/2018 11:06
Last Updated - 04/20/2018 08:52

Equinix NY Peering Maintenance (Resolved)
Affecting Other - All peering inside Equinix NY/NJ | Priority - Low

Uplinks to peering routers inside Equinix will be out of service on Saturday for maintenance to new lower latency equipment.  During this period traffic will route around over alternate routes. Date - 04/28/2018 00:00 - 04/20/2018 08:50
Last Updated - 04/13/2018 16:49

NYC Connectivity (Resolved)
Affecting Other - Abnormal traffic | Priority - Critical

After experienicng a period of several hours of no abusive traffic, we are now closing out this incident.  The new filters mitigating this traffic will remain in place.

--

We are continuing to mitigate abusive traffic entering our network.  This traffic earlier caused some connectivity issues in our NY datacenter.  All services are reachable.  Please contact our helpdesk by telephone, chat or support ticket if you are experiencing issues.

We will continue to monitor and adjust as required.

Date - 04/10/2018 01:38
Last Updated - 04/10/2018 08:11

NYIIX Packet Loss (Resolved)
Affecting Other - NYIIX Packet Loss (IX) | Priority - Low

We have temporarily shut down all peering sessions on NYIIX due to packet loss on the exchange.   We do not expect much impact due to redundancy at DE-CIX in NY.

Date - 03/26/2018 13:13 - 04/02/2018 09:11
Last Updated - 03/26/2018 13:15

Peering at NYIIX Port Migration (Resolved)
Affecting Other - Network Peering at NYIIX | Priority - Low

Our IX port at NYIIX (peering exchange) will be shut down temporarily so that it can be moved to their new platform.  There may be a brief interruption to network connectivity during this period.  This will begin about 30 minutes after all markets have closed.


Date - 03/16/2018 14:30 - 04/02/2018 09:11
Last Updated - 03/16/2018 14:19

Credit Card Payments (Resolved)
Affecting Other - Credit Card Payments | Priority - Low

We are experiencing problems processing credit card payments.  We are working on a resolution and will have it operational again as soon as possible.

We apologize for emails you may have received of a declined charge.  We will attempt the charge again after the gateway is back in service.

RESOLVED
Date - 02/28/2018 12:42 - 02/28/2018 13:17
Last Updated - 02/28/2018 13:17

« Prev Page   Next Page »