Commercial Network Services
Network Issues
You are here: Portal Home > Network Issues > Resolved

Test Status Last Upd Next Upd Upd Interval May Uptime Apr Uptime
Commercial Network Services (LA) Ping
UP
02:38 PM 02:43 PM 5 min 100.000% 100.000%
Commercial Network Services (NYC) Ping
UP
02:38 PM 02:43 PM 5 min 100.000% 99.937%
Commercial Network Services (UK) Ping
UP
02:38 PM 02:43 PM 5 min 100.000% 100.000%

Website monitoring by Hyperspin

Malicious packets NY (Resolved)
Affecting Other - NY IP services | Priority - Critical

We are working to mitigatge malicious packets directed at our NY datacenter.

We have mitigated the traffic and are continuing to review and respond.

Possible route hijack.  Minimal effect due to large number of peers.  We are continuing to review and respond.

UPDATE: This issue appears to be resolved.  We will continue to monitor and respond quickly to any abnormalities.


Date - 10/08/2018 02:45 - 10/08/2018 12:21
Last Updated - 10/08/2018 12:05

NY Network Maintenance (Resolved)
Affecting Other - All NY based services | Priority - Low

Our NY datacenter will go offline briefly so we can complete network maintenance.

Details:  We are experiencing trouble removing a problem router from the rack.  In order to minimize any possible damage, we will need to shut down and remove other routers to remove the problem router.  We anticipate this procedure will just take a few minutes.

Date - 09/30/2018 11:53 - 09/30/2018 15:22
Last Updated - 09/30/2018 12:11

Router failure (Resolved)
Affecting Other - NY Network | Priority - Low

A router in our NY datacenter has failed and services automatically rolled over to another router.  This caused a very brief disruption while some routes reconverged.

We are working to identify the problem with the router to repair it.

UPDATE:  Early analysis indicates malicious traffic has caused the router to fail because it was unable to keep up logging it.  We have taken steps to mitigate it while we continue to review.


Date - 09/27/2018 05:57 - 09/30/2018 08:02
Last Updated - 09/27/2018 06:13

Some brokers OFFLINE (Resolved)
Affecting Other - Some NY Hosted Brokers | Priority - Critical

Some brokers in NY hosted off our network are down due to an interruption at their hosting provider.  This is a hard down.  They are unreachable from any network.   Your terminals will automatically connect when service is restored.

This page will be updated as we receive new information.


UPDATE:  Connectivity has been restored.  Please contact your broker if you are still unable to connect.



Date - 09/14/2018 10:16
Last Updated - 09/14/2018 11:41

New York (Resolved)
Affecting Other - New York | Priority - Low

A failure of the conneciton to three key IX's failed, causing a cascading failure and reducing all connectivity in/out of NY to a single transit circuit.  All service is now normalized and we will continue to analyze the failure and make appropriate adjustments where necessary.

--

Connectivity has been restored to these IX's.  We are cautiously moving some routes back as we work to investigate the cause.  At this time all services should be reachable.

 

We have lost connectivity to three IX's.  This initially created a connectivity issue for many routes.  We have re-routed these affected routes out another transit provider while the issue is being investigated.  At this time all services should be reachable.



Date - 08/01/2018 02:32
Last Updated - 08/01/2018 11:19

LINX Packet Loss (Resolved)
Affecting Other - UK | Priority - Low

During this past week, we experienced  a period of high bandwidth and packet loss on IX interfaces in NY and UK and across our NY-UK backbone.  We believe this was caused by a routing loop with an unknown peer between Europe and America.  The issue went away after many peers on LINX in the UK were dropped.

We have started reconnecting peers one at a time in order to identify the loop.  At that time, we will address the loop either through an internal filter, working with the peer, or both.  (We need to identify it in order to determine how to best address it).

There can be expected a brief period of packet loss as we work to reproduce the issue.  We will work to minimize it as much as possible.

 
----

This problem has been mitigated. 

We believe this issue occured because of a routing loop with a peering network.   Most peers on LINX have been cut away until we can identify it after markets close.

---

We are experienicng major packet loss on traffic entering our network at LINX in London.  We are in the process of cutting the IX away until the issue is resolved.



Date - 06/13/2018 09:48 - 06/17/2018 11:50
Last Updated - 06/16/2018 08:07

NYIIX temporary disconnect (Resolved)
Affecting Other - NYIIX Connectivity | Priority - Low

Due to technical issues being experienced by the New York Internet Exchange (NYIIX) and their intention to interrupt services on the IX over the week to apply a fix, we will be temporarily shutting down all peering sessions on the exchange until after the work is complete.  Very little impact is expected because most peers have redundant links through DE-CIX NY or Equinix NY.

It is possible some networks will revert to transit service until after the work is complete and all peers are reconnected.



Date - 06/17/2018 12:19 - 06/22/2018 09:57
Last Updated - 06/15/2018 12:22

UK Connectivity (Resolved)
Affecting Other - UK | Priority - Low

A power anomaly in the datacenter has damaged PDUs powering key switching equipment.  Power has been re-routed and the damaged equipment will be replaced during our maintenance window.

 

At this time all services are operational.  Please contact our helpdesk if you are unable to connect or experience any issues.

This event appears to have damaged a router.  It has been removed from service.

Multiple power supplies were destroyed.  They have been already or will be replaced during our maintenance window.

All work to repair or replace damaged equipment is complete.

We believe a power supply blew and sent a surge back into the PDU, damaging connected equipment or causing it to malfunction.

------------

 

 Detail:

 

We are investigating reports of connectivity issues in UK.

We appear to have lost both power legs to key equipment.  We are  working with Equinix to investigate.

CONFIRMED blown PDU(s) have taken down key switches.  We are working to re-route power as quickly as possible.

Power to key equipment has been restored through temporary circuits while work is completed to replace damaged PDU's.

2:45PM PST At this time all services should be operational.  Please contact our helpdesk if you are not online.

Full replacement of power equipment will be conducted during our maintenance window after markets have closed to avoid any possible interruption to services.

Status changed to "Scheduled"

Updates will be provided as available.


Date - 06/05/2018 10:56 - 06/09/2018 18:19
Last Updated - 06/09/2018 18:19

Abusive traffic - NYC (Resolved)
Affecting Other - NYC Network | Priority - Critical

CNS has mitiageted abusive traffic entering our network.  Please contact CNS Support if you are experiencing a degraded network so we may investigate.

Date - 06/01/2018 08:25 - 06/03/2018 13:13
Last Updated - 06/01/2018 08:26

Maintenance Updates (Resolved)
Affecting Other - All Services | Priority - Low

All services are subject to a brief pause on Saturday so we can complete security audits and platform maintenance.

Date - 04/21/2018 00:00 - 05/13/2018 11:06
Last Updated - 04/20/2018 08:52

« Prev Page   Next Page »