Today, at approximately 07:00 CEST, we experienced a loss of connectivity to several upstream providers and Internet Exchange Points (IXPs), including DE-CIX and AMS-IX. As a result, some customers may currently experience reduced available bandwidth and increased latency.

During our investigation, we learned that Equinix FR5 had informed its customers of an ongoing water leak within the data center. FR5 is a major interconnection and peering location in Frankfurt, and the incident is therefore affecting a significant number of network operators and service providers.

According to the information currently available, the water leak impacted the facility's cooling infrastructure, resulting in rising temperatures and subsequent equipment and connectivity failures.

Equinix on-site engineers have since contained the leak and are working to restore cooling capacity and bring temperatures back to normal operating levels. Due to water remaining on the data center floor, customer access to the facility is currently restricted for safety reasons.

Thanks to our redundant network design and connectivity through multiple independent carriers and locations, our network remains operational and redundant. However, with connectivity to several major peering points currently unavailable or degraded, customers may continue to experience reduced network performance, including higher latency and lower available bandwidth.

We are continuing to monitor the situation and will provide further updates as the recovery progresses.

On May 5th at approximately 23:15 CEST, our monitoring systems detected that multiple infrastructures became unreachable.

Our engineering team immediately initiated an investigation and quickly identified that the affected systems were experiencing issues with domain name resolution. Further analysis revealed a common factor across all alerts: all impacted services rely on the .de top-level domain (TLD).

Shortly thereafter, major DNS providers such as Cloudflare also reported DNSSEC-related issues affecting .de domains. The root cause appears to be a misconfiguration within the infrastructure of DENIC, the registry responsible for the .de TLD.

Public sources, including Heise Online, are reporting on this global incident as well:
https://www.heise.de/news/DNS-Probleme-de-Domains-nicht-erreichbar-11283192.html

Please note that this incident is external to our platform. Our services and infrastructure remain fully operational. This notification is provided for informational purposes only.

One of our upstream providers will perform operating system updates on routers and switches to which we are connected. This maintenance will also affect our DE-CIX and AMS-IX connectivity. During the maintenance window, traffic will be rerouted via alternative carriers; however, you may experience increased latency.

There is currently a network disruption in Germany. Our technicians are working hard on a solution.

A few minutes ago we observed a network flap at our datacenter location in Frankfurt. We're currently investigating and keep you posted.

This morning around 06:53am CET our network engineers could identify a full reset of all BGP sessions of one of our core network devices. The device itself was up and running and still forwarding L2 packets, however, this might had impact to the availability of your services. Services have been restored ~10 minutes later, we are now investigating the root cause of this issue.