Commercial Network Services
Network Issues
You are here: Portal Home > Network Issues > Resolved

Test Status Last Upd Next Upd Upd Interval Jun Uptime May Uptime
Commercial Network Services (LA) Ping
UP
07:08 AM 07:13 AM 5 min 100.000% 100.000%
Commercial Network Services (NYC) Ping
UP
07:08 AM 07:13 AM 5 min 100.000% 100.000%
Commercial Network Services (UK) Ping
UP
07:08 AM 07:13 AM 5 min 100.000% 100.000%

Website monitoring by Hyperspin

LINX Packet Loss (Resolved)
Affecting Other - UK | Priority - Low

During this past week, we experienced  a period of high bandwidth and packet loss on IX interfaces in NY and UK and across our NY-UK backbone.  We believe this was caused by a routing loop with an unknown peer between Europe and America.  The issue went away after many peers on LINX in the UK were dropped.

We have started reconnecting peers one at a time in order to identify the loop.  At that time, we will address the loop either through an internal filter, working with the peer, or both.  (We need to identify it in order to determine how to best address it).

There can be expected a brief period of packet loss as we work to reproduce the issue.  We will work to minimize it as much as possible.

 
----

This problem has been mitigated. 

We believe this issue occured because of a routing loop with a peering network.   Most peers on LINX have been cut away until we can identify it after markets close.

---

We are experienicng major packet loss on traffic entering our network at LINX in London.  We are in the process of cutting the IX away until the issue is resolved.



Date - 06/13/2018 09:48 - 06/17/2018 11:50
Last Updated - 06/16/2018 08:07

UK Connectivity (Resolved)
Affecting Other - UK | Priority - Low

A power anomaly in the datacenter has damaged PDUs powering key switching equipment.  Power has been re-routed and the damaged equipment will be replaced during our maintenance window.

 

At this time all services are operational.  Please contact our helpdesk if you are unable to connect or experience any issues.

This event appears to have damaged a router.  It has been removed from service.

Multiple power supplies were destroyed.  They have been already or will be replaced during our maintenance window.

All work to repair or replace damaged equipment is complete.

We believe a power supply blew and sent a surge back into the PDU, damaging connected equipment or causing it to malfunction.

------------

 

 Detail:

 

We are investigating reports of connectivity issues in UK.

We appear to have lost both power legs to key equipment.  We are  working with Equinix to investigate.

CONFIRMED blown PDU(s) have taken down key switches.  We are working to re-route power as quickly as possible.

Power to key equipment has been restored through temporary circuits while work is completed to replace damaged PDU's.

2:45PM PST At this time all services should be operational.  Please contact our helpdesk if you are not online.

Full replacement of power equipment will be conducted during our maintenance window after markets have closed to avoid any possible interruption to services.

Status changed to "Scheduled"

Updates will be provided as available.


Date - 06/05/2018 10:56 - 06/09/2018 18:19
Last Updated - 06/09/2018 18:19

Abusive traffic - NYC (Resolved)
Affecting Other - NYC Network | Priority - Critical

CNS has mitiageted abusive traffic entering our network.  Please contact CNS Support if you are experiencing a degraded network so we may investigate.

Date - 06/01/2018 08:25 - 06/03/2018 13:13
Last Updated - 06/01/2018 08:26

Maintenance Updates (Resolved)
Affecting Other - All Services | Priority - Low

All services are subject to a brief pause on Saturday so we can complete security audits and platform maintenance.

Date - 04/21/2018 00:00 - 05/13/2018 11:06
Last Updated - 04/20/2018 08:52

Equinix NY Peering Maintenance (Resolved)
Affecting Other - All peering inside Equinix NY/NJ | Priority - Low

Uplinks to peering routers inside Equinix will be out of service on Saturday for maintenance to new lower latency equipment.  During this period traffic will route around over alternate routes. Date - 04/28/2018 00:00 - 04/20/2018 08:50
Last Updated - 04/13/2018 16:49

NYC Connectivity (Resolved)
Affecting Other - Abnormal traffic | Priority - Critical

After experienicng a period of several hours of no abusive traffic, we are now closing out this incident.  The new filters mitigating this traffic will remain in place.

--

We are continuing to mitigate abusive traffic entering our network.  This traffic earlier caused some connectivity issues in our NY datacenter.  All services are reachable.  Please contact our helpdesk by telephone, chat or support ticket if you are experiencing issues.

We will continue to monitor and adjust as required.

Date - 04/10/2018 01:38
Last Updated - 04/10/2018 08:11

NYIIX Packet Loss (Resolved)
Affecting Other - NYIIX Packet Loss (IX) | Priority - Low

We have temporarily shut down all peering sessions on NYIIX due to packet loss on the exchange.   We do not expect much impact due to redundancy at DE-CIX in NY.

Date - 03/26/2018 13:13 - 04/02/2018 09:11
Last Updated - 03/26/2018 13:15

Peering at NYIIX Port Migration (Resolved)
Affecting Other - Network Peering at NYIIX | Priority - Low

Our IX port at NYIIX (peering exchange) will be shut down temporarily so that it can be moved to their new platform.  There may be a brief interruption to network connectivity during this period.  This will begin about 30 minutes after all markets have closed.


Date - 03/16/2018 14:30 - 04/02/2018 09:11
Last Updated - 03/16/2018 14:19

Credit Card Payments (Resolved)
Affecting Other - Credit Card Payments | Priority - Low

We are experiencing problems processing credit card payments.  We are working on a resolution and will have it operational again as soon as possible.

We apologize for emails you may have received of a declined charge.  We will attempt the charge again after the gateway is back in service.

RESOLVED
Date - 02/28/2018 12:42 - 02/28/2018 13:17
Last Updated - 02/28/2018 13:17

NYC Migration Saturday (Resolved)
Affecting Other - All NY Hosted Services | Priority - High

We will be moving equipment to a new and larger location within the NY datacenter over the next couple Saturdays.  During this time services may be unavailable as equipment is physically moved.  We will resume the services as soon as possible, as they come back online in the new location.
 
No reboots are expected, but services will be unavaiable during the actual move of the respective machine(s).

UPDATE:  Most all services in NY are back online.  We are working on a couple of remaining items to bring everything up as fast as possible.

All work is complete.  During this move, some backend equipment was also upgraded with new bonded 10Gb uplinks.  enjoy






Date - 12/09/2017 01:02 - 12/10/2017 12:04
Last Updated - 12/10/2017 12:04

« Prev Page   Next Page »