OVH Strasbourg fire

[Selaya]

Apparently there had been a big fire at OVH's Strasbourg facility, their entire SBG2 datacenter has been totalled.
Hope y'all had/have backups.

 

[Bill_Bright]

Any service worth its salt will have mirrored sites just in case of such of an event.

A quick look with Google shows OVH has 17 data centers in France alone, and 32 globally. One center going down is little more than a hiccup.

 

[Solaris17]

Don't use OVH but blows my mind they had 4 DCs on the same premises.

Hope y'all had/have backups.

so do they:

If your production is in Strasbourg, we recommend that you activate your Business Recovery Plan."

 

[Bill_Bright]

blows my mind they had 4 DCs on the same premises.

I wondered about that too. I cannot imagine that would mean they are all co-located in the same big room - at least without some sort of significant fire-breaks between them. They should be on totally isolated power sources and cooling systems too. If that is not the case, I would be moving my data to another provider.

The report said of the 4 data centers at that location, one was destroyed and the other damaged. But it does not say if the 4 data centers are physically separated and isolated in some way, or just virtually. I would hope not virtually.

As far as activating their Business Recovery Plan, I have to assume that is to expedite getting back into production. If that one location is the only site for those businesses (meaning there is no mirrored site for their data) then again, if I was the business owner or CIO, I would be looking for a new cloud storage provider.

I don't know, but it just seems odd to me too that the place would turn into an inferno so quickly. It makes me think the building was not built initially as a data center. It seems to me the building itself should have been constructed with fire-proof and/or retardant materials, and there should have been a robust fire suppressant system in the server rooms. If the building was initially built as a big wooden warehouse that was then converted into these data centers, then there should have been a building built within that building, built to house the data centers.

If there were a bunch of offices in the building too, filled with flammable furniture and other combustible stuff, that office space should have been isolated (with fire blocks) from the server rooms.

For sure, fire suppression in server rooms can be a real challenge - especially since Halon was banned years ago. But if the building was built from the foundation up as a data center for "mission critical" systems, a robust plan and system should have built into it from the foundation up too. I'm just saying - something smells fishy here.

 

[TheLostSwede]

Rust developer warns EU players of massive data loss after fire

25 servers were lost in a fire at the datacenter

www.polygon.com

 

[Selaya]

The fire occurred at the OVH Datacenter in Strasbourg, Germany.

what year is this, 1890?
also, some people just learnt it the hard way that offsite backups are recommended for a reason :^]

 

[John Naylor]

I used to keep a backup in a Fire Safe but but was too much of a pain the ass to open it every night. Live in a 200 year old dairy barn wih office in the hayloft and keep a "Go Bag" next to night table with shoes, clothes, jacket, wallet flashlight and "that night's backup". Weekly backup is stored off site. Plan for Spring is to set up nightly cross backups to other office.

 

[Daniel15]

Luckily all staff were safe and accounted for.

I don't know, but it just seems odd to me too that the place would turn into an inferno so quickly. It makes me think the building was not built initially as a data center.

Have you seen photos of their data center? The crates they had servers in literally look like shipping containers. I doubt they had any serious fire suppression system, fire walls, etc inside those. OVH servers are very cheap, so I guess that's where they cut corners.

https://twitter.com/i/web/status/1369559995491172354

https://twitter.com/i/web/status/1369538028243456000

 

[Bill_Bright]

I am not going to

The crates they had servers in

Crates? That's pretty much a guess on your part. If you look at the picture in my link above, they clearly are buildings that are ablaze. Here's another image. You can see 3 stacked shipping containers to the right. The building is not made of shipping containers.

But even if they are shipping containers, those are steel lined inside and out and so "should" have been easier to contain to just a few.

So while there appears to be shipping containers around the buildings, there's no way of telling if the operational servers are in those containers - unless you read that some where. If so, got a link?

OVH servers are very cheap

Oh? How do you know that?

 

[Daniel15]

Oh? How do you know that?

It's the main reason people use OVH - Their prices are very hard to beat in many cases, particularly if you look at their budget brands like Kimsufi. They don't just have cheap virtual servers / VPSes, they also have dedicated servers for less than $10/month (https://www.kimsufi.com/us/en/servers.xml). Sure, they're low-end Atoms, 11-year-old AMD Opterons (Opteron 4122), or 12-year-old Xeons (W3520), but it's still physical hardware sitting in a data center. People don't go to OVH when they need 100% uptime on a high-quality network... They go to OVH when they want cheap servers that mostly work well, on a network that mostly works well (but is often very congested).

So while there appears to be shipping containers around the buildings, there's no way of telling if the operational servers are in those containers - unless you read that some where. If so, got a link?

Not sure about their newer data centers, but they did use shipping containers at one point: https://www.datacenterknowledge.com/archives/2012/06/13/ovh-deploys-container-cube-data-center

The new OVH data center in Strasbourg, France uses a stack of shipping containers to re-create the "Cube" effect the company created in its data center in Roubaix, France. (Photo: OVH)

The French firm OVH is one of the world's largest hosting companies, with more than 120,000 servers. It has also earned a reputation for its innovative data center designs. DCK readers may be familiar with OVH for the innovative Cube-shaped data center it opened last year in Roubaix, France, which houses servers in an exterior corridor built around an open center, allowing for easy airflow through the facility.

Earlier this year OVH opened a new data center in Strasbourg, France that recreates many elements of the cube design using stacks of shipping containers housing servers. The facility features 12 containers, which are stacked three-high in two rows. Outside air enters the facility through louvers in the exterior wall. The air travels through the racks of servers, and then exits the IT corridor via large fans behind the racks, which vent the air outside into the open center.

 

[Bill_Bright]

Kimsufi is reseller. And cheap prices does not imply cheap quality. Hosting is a very competitive business. Just because they have low monthly rates, that in no way implies they cut corners in their facilities in terms of practicality, safety, reliability, etc. Fancy buildings cost money - money that could be used better elsewhere.

There are other indications they might have cut corners. Like their failure to comment on their fire safety practices. That raises suspicions, but it is still too early to come to any conclusions.

As for the Cube effect using containers, I don't see a problem with that - if done right. In fact, it could be a wise, cost effective approach. Buildings out of shipping containers is nothing new.

I think we need to wait until the fire investigation is complete and we see the findings - if we ever do - before we pass judgement and jump to conclusions as to the cause and whether there was a failure in management where they cut corners in safety, or not.

 

[Daniel15]

Kimsufi is reseller.

Kimsufi is fully owned by OVH. See OVH's press release from when they originally launched the Kimsufi and SoYouStart brands:

Third Internet hosting provider in the world*, OVH is launching its most recent lines of dedicated servers targeting new markets to keep its growth going. Through 3 distinct brands, “OVH.com”, “So you start” and “Kimsufi”, the hosting provider aims to reach a wider range of users, from fully developed markets (North America, Europe) to emerging ones (South America, Asia, Africa). OVH is now covering every type of business applications of its users.

Kimsufi is still listed as one of OVH's registered trademarks (https://www.ovh.com/world/support/terms-and-conditions/) so I'm fairly sure it's still an OVH brand today, rather than a reseller.

And cheap prices does not imply cheap quality.

When I said their servers are cheap, I was just referring to the price. I guess my sentence was ambiguous.

As for the Cube effect using containers, I don't see a problem with that

Sure, I was just replying to your sentence "there's no way of telling if the operational servers are in those containers - unless you read that some where" with a link showing that they did indeed have servers in shipping containers at some point

Edit:
Someone posted a graph of their server's temperature on Twitter... It jumped from 31C to 95C pretty quickly. I'm surprised it was able to collect data for as long as it did.

https://twitter.com/i/web/status/1369740303142359040

 

[Bill_Bright]

Sure, I was just replying to your sentence "there's no way of telling if the operational servers are in those containers - unless you read that some where" with a link showing that they did indeed have servers in shipping containers at some point

Yep. And I appreciate the link. My bad for not referring to my friend, Bing Google first!

 

[Fleurious]

Have rented OVH servers of varying specs for years to host games on dedicated servers, usually through soyoustart or kimsufi brands and from their Montreal location so luckily not impacted by this.

 

[snp688]

Containers are SBG1 which is partially destroyed. There were servers inside, I had the access to one before the fire - source: ovh panel server location information. More details here - http://travaux.ovh.net/?do=details&id=49484

 

[GerKNG]

they don't have at least 3-4 decentral backup servers?
glad i never heard of them..

 

[Daniel15]

they don't have at least 3-4 decentral backup servers?

They're a budget unmanaged hosting provider where you're expected to take backups, handle OS upgrades, etc. yourself... I don't know of any budget hosting services that include fully redundant backups as part of their basic service. Apparently they did have a paid backup service, but some of the backups were in the same room, so the backups were destroyed along with the servers

Protip: Don't store your server backups in the same building as the servers.