Commercial Network Services
Network Issues
You are here: Portal Home > Network Issues > Resolved

Test Status Last Upd Next Upd Upd Interval Sep Uptime Aug Uptime
Commercial Network Services (LA) Ping
UP
10:18 AM 10:23 AM 5 min 100.000% 100.000%
Commercial Network Services (NYC) Ping
UP
10:18 AM 10:23 AM 5 min 100.000% 100.000%
Commercial Network Services (UK) Ping
UP
10:18 AM 10:23 AM 5 min 100.000% 100.000%

Website monitoring by Hyperspin

Some brokers OFFLINE (Resolved)
Affecting Other - Some NY Hosted Brokers | Priority - Critical

Some brokers in NY hosted off our network are down due to an interruption at their hosting provider.  This is a hard down.  They are unreachable from any network.   Your terminals will automatically connect when service is restored.

This page will be updated as we receive new information.


UPDATE:  Connectivity has been restored.  Please contact your broker if you are still unable to connect.



Date - 09/14/2018 10:16
Last Updated - 09/14/2018 11:41

New York (Resolved)
Affecting Other - New York | Priority - Low

A failure of the conneciton to three key IX's failed, causing a cascading failure and reducing all connectivity in/out of NY to a single transit circuit.  All service is now normalized and we will continue to analyze the failure and make appropriate adjustments where necessary.

--

Connectivity has been restored to these IX's.  We are cautiously moving some routes back as we work to investigate the cause.  At this time all services should be reachable.

 

We have lost connectivity to three IX's.  This initially created a connectivity issue for many routes.  We have re-routed these affected routes out another transit provider while the issue is being investigated.  At this time all services should be reachable.



Date - 08/01/2018 02:32
Last Updated - 08/01/2018 11:19

LINX Packet Loss (Resolved)
Affecting Other - UK | Priority - Low

During this past week, we experienced  a period of high bandwidth and packet loss on IX interfaces in NY and UK and across our NY-UK backbone.  We believe this was caused by a routing loop with an unknown peer between Europe and America.  The issue went away after many peers on LINX in the UK were dropped.

We have started reconnecting peers one at a time in order to identify the loop.  At that time, we will address the loop either through an internal filter, working with the peer, or both.  (We need to identify it in order to determine how to best address it).

There can be expected a brief period of packet loss as we work to reproduce the issue.  We will work to minimize it as much as possible.

 
----

This problem has been mitigated. 

We believe this issue occured because of a routing loop with a peering network.   Most peers on LINX have been cut away until we can identify it after markets close.

---

We are experienicng major packet loss on traffic entering our network at LINX in London.  We are in the process of cutting the IX away until the issue is resolved.



Date - 06/13/2018 09:48 - 06/17/2018 11:50
Last Updated - 06/16/2018 08:07

NYIIX temporary disconnect (Resolved)
Affecting Other - NYIIX Connectivity | Priority - Low

Due to technical issues being experienced by the New York Internet Exchange (NYIIX) and their intention to interrupt services on the IX over the week to apply a fix, we will be temporarily shutting down all peering sessions on the exchange until after the work is complete.  Very little impact is expected because most peers have redundant links through DE-CIX NY or Equinix NY.

It is possible some networks will revert to transit service until after the work is complete and all peers are reconnected.



Date - 06/17/2018 12:19 - 06/22/2018 09:57
Last Updated - 06/15/2018 12:22

UK Connectivity (Resolved)
Affecting Other - UK | Priority - Low

A power anomaly in the datacenter has damaged PDUs powering key switching equipment.  Power has been re-routed and the damaged equipment will be replaced during our maintenance window.

 

At this time all services are operational.  Please contact our helpdesk if you are unable to connect or experience any issues.

This event appears to have damaged a router.  It has been removed from service.

Multiple power supplies were destroyed.  They have been already or will be replaced during our maintenance window.

All work to repair or replace damaged equipment is complete.

We believe a power supply blew and sent a surge back into the PDU, damaging connected equipment or causing it to malfunction.

------------

 

 Detail:

 

We are investigating reports of connectivity issues in UK.

We appear to have lost both power legs to key equipment.  We are  working with Equinix to investigate.

CONFIRMED blown PDU(s) have taken down key switches.  We are working to re-route power as quickly as possible.

Power to key equipment has been restored through temporary circuits while work is completed to replace damaged PDU's.

2:45PM PST At this time all services should be operational.  Please contact our helpdesk if you are not online.

Full replacement of power equipment will be conducted during our maintenance window after markets have closed to avoid any possible interruption to services.

Status changed to "Scheduled"

Updates will be provided as available.


Date - 06/05/2018 10:56 - 06/09/2018 18:19
Last Updated - 06/09/2018 18:19

Abusive traffic - NYC (Resolved)
Affecting Other - NYC Network | Priority - Critical

CNS has mitiageted abusive traffic entering our network.  Please contact CNS Support if you are experiencing a degraded network so we may investigate.

Date - 06/01/2018 08:25 - 06/03/2018 13:13
Last Updated - 06/01/2018 08:26

Maintenance Updates (Resolved)
Affecting Other - All Services | Priority - Low

All services are subject to a brief pause on Saturday so we can complete security audits and platform maintenance.

Date - 04/21/2018 00:00 - 05/13/2018 11:06
Last Updated - 04/20/2018 08:52

Equinix NY Peering Maintenance (Resolved)
Affecting Other - All peering inside Equinix NY/NJ | Priority - Low

Uplinks to peering routers inside Equinix will be out of service on Saturday for maintenance to new lower latency equipment.  During this period traffic will route around over alternate routes. Date - 04/28/2018 00:00 - 04/20/2018 08:50
Last Updated - 04/13/2018 16:49

NYC Connectivity (Resolved)
Affecting Other - Abnormal traffic | Priority - Critical

After experienicng a period of several hours of no abusive traffic, we are now closing out this incident.  The new filters mitigating this traffic will remain in place.

--

We are continuing to mitigate abusive traffic entering our network.  This traffic earlier caused some connectivity issues in our NY datacenter.  All services are reachable.  Please contact our helpdesk by telephone, chat or support ticket if you are experiencing issues.

We will continue to monitor and adjust as required.

Date - 04/10/2018 01:38
Last Updated - 04/10/2018 08:11

NYIIX Packet Loss (Resolved)
Affecting Other - NYIIX Packet Loss (IX) | Priority - Low

We have temporarily shut down all peering sessions on NYIIX due to packet loss on the exchange.   We do not expect much impact due to redundancy at DE-CIX in NY.

Date - 03/26/2018 13:13 - 04/02/2018 09:11
Last Updated - 03/26/2018 13:15

« Prev Page   Next Page »