Fires in data centers and server rooms are of particular interest to ORR Protection due to our specialization in protecting mission critical spaces. On March 10, 2021, OVHcloud (formerly OVH, a cloud computing company in France) had a major fire at their Strasbourg data center. When large enough, data center fires make the news. The coverage is often due to the data outage caused, rarely it's because flames are shooting out of the building.
The OVH facility is unique. It isn’t small, but it isn’t as massive as other cloud provider installations. It is a multi-story complex built in stages, and by reading the news articles, it seems there are four separate buildings of data centers—SBG1 through SBG4—with each building containing multiple data halls. Apparently, the fire was large enough that SBG2 was completely destroyed along with four halls of SBG1. The damage led OVH to permanently close both data centers. Repair work, both structurally and from an IT perspective, was required to get SBG3 and SBG 4 back up and running.
This was obviously a very large event. A data center fire of this magnitude has not been seen since the Shaw fire (Canada 2012), the Cowboyminers fire (Thailand 2014), or the Samsung SDS fire (South Korea 2014). Information on the incident is somewhat limited, but not completely unavailable and it all points to a set of specific situations within a complex building that caused such a massive fire. It took more than two weeks for data operations to restart in the facility—a significant downtime by any standard. Here is a video of the fire:
This is an image of the aftermath:
Source: https://regmedia.co.uk/2021/03/22/ovh_strasbourg_fire_shutterstock_editorial_only.jpg
Looking at this event purely from the perspective of good fire protection raises a lot of questions. Here are some of the bigger ones:
-
-
- How did the fire get so out of control?
- Was there not an active fire protection system to control the size of the fire?
- How did the fire spread in the way that it did?
- How long did it take for the fire department to respond?
-
Apparently, this is a unique data center installation. OVH has been very creative in trying to keep their overall cost of service down. They manufacture their own servers, use shipping containers to build their data centers, and have unique cooling schemes to reduce their energy consumption. It was actually some of these practices that seem to have contributed to the severity of the fire.
The shipping container floors are covered with plywood, allowing combustible materials to creep into the data center. Articles on OVH indicate that they use convection cooling and a vertical structure to their data center to efficiently cool their servers. That practice must have contributed to vertical fire spread in the building. SBG1, which has been mothballed since the fire, was a portion of the Strasbourg site built with shipping containers stacked on top of each other with site-built construction between the stacks.
No reports have shown that there was an automatic fire detection or fire suppression system installed at the site. Reports from the night of the fire indicate that workers detected the fire and had to abandon the building quickly, call for emergency services, and wait for the fire department.
Strasbourg is a city of around 270,000 people and has a multi-station governmental fire department (Sapeurs Pompiers - Service Départemental D’Incendie Et De Secours Du Bas-Rhin). The fire department was reported to have responded quickly to the fire. Looking at maps of the city, the fire site is approximately 12-15 minutes away from the closest station.
Video showed a defensive fire attack and limited access to all sides of the building. The data center is located on a strip of land surrounded on two sides by the Rhine river. An access road runs along the front of the building, but the rear is blocked by railroad tracks. This area appears to be used for switching railcars. The fire department had to deal with a rapidly spreading fire, and combustible finishes inside the data center and the apparent lack of rated fire partitions may have contributed to the fire spread.
So, what started the fire? All reports point to two recently repaired uninterruptible power supplies (UPS) units causing the fire. The firefighters entering the building used thermal imaging cameras and identified two UPS units which were extremely hot and, because of the repairs to those units, they appear to be to blame. Interestingly, it is ORR Protection’s experience when investigating fires for our customers that UPS units are frequently at fault for thermal events causing fire suppression system releases.
OVH Founder and CTO Octave Klaba commented in the wake of a 2017 power outage that took down the data centers that it was “the worst incident that can happen to us.” It seems he was wrong. To make matters worse, five days later while the recovery efforts were underway, batteries inside SBG1 began smoking and required a second fire department response. Fortunately, no flames broke out.
The best thing to do with this incident is to learn from it. There are lessons for fire code officials, engineers and architects, and IT professionals. The building construction methods, cooling system, electrical system, fire protection features, and IT recovery efforts all bear investigation so that another facility never has to bear the same burden as OVH.
As we have followed this story in the U.S., we appreciate the lessons in crisis communications. Octave Klaba has been very communicative and forthcoming in how the company is handling this issue. A lot of their customers and web services were affected by this fire, but OVH has also kept the public abreast of the effects of the fire and the company’s subsequent actions during and after the incident. We hope they never have to go through an event like this again.
Sources:
OVHcloud may have taken risks with fire suppression and wooden construction, claims news report, Datacenter Dynamics, Peter Judge, April 15, 2021, https://www.datacenterdynamics.com/en/news/ovhcloud-may-have-taken-risks-with-fire-suppression-and-wooden-construction-claims-news-report/
OVH has a hunch about what caused the fire that destroyed its data center, Techradar.pro, Mayank Sharma, March 2021, https://www.techradar.com/news/ovh-has-a-hunch-about-what-caused-the-fire-that-destroyed-its-data-center
OVH News: Strasbourg datacentre: latest information, OVH.com, April 23, 2021, https://www.ovh.com/world/news/press/cpl1787.fire-our-strasbourg-site
OVH writes off another data centre – SBG1 – and reveals new smoking battery incident, TheRegister.com, Simon Sharwood, 22 March 2021, https://www.theregister.com/2021/03/22/ovh_sbg1_written_off_restoration_update/
