@Freek said:
Didn't they have a fire as well a couple of years ago? I can't find the news article anymore.
If memory serves me, there was a fire on a building close-by. They talked about adding (or already having added) some external fire-protection plates to prevent such incidents from putting them in danger.
Can't find the article neither.
@DataRecovery said:
Is it me, or even despite the fire, this OVH SGB2 datacenter looks like ... just a bunch of containers in Strasbourg?
This image probably shows what SGB1, SGB2 and SGB3 are best.
No, OVH SGB2 is that tower that burned. SGB2 had the capacity to house ~30000 servers.
SGB1 is the bunch/two rows of containers in the foreground; some of those containers are damaged/burned as well hence the SGB1 is damaged. Those containers have the capacity to ~12000 servers.
The tower next to the right is SGB3. This tower also has the capacity to house ~30000 servers.
The bunch of containers (if they are still there) to the right/north of the SGB3 tower is SGB4. Like SGB1, those containers have the capacity of ~12000 servers.
@bikegremlin said:
If memory serves me, there was a fire on a building close-by. They talked about adding (or already having added) some external fire-protection plates to prevent such incidents from putting them in danger.
Can't find the article neither.
@DataRecovery said:
Is it me, or even despite the fire, this OVH SGB2 datacenter looks like ... just a bunch of containers in Strasbourg?
This image probably shows what SGB1, SGB2 and SGB3 are best.
No, OVH SGB2 is that tower that burned. SGB2 had the capacity to house ~30000 servers.
SGB1 is the bunch/two rows of containers in the foreground; some of those containers are damaged/burned as well hence the SGB1 is damaged. Those containers have the capacity to ~12000 servers.
The tower next to the right is SGB3. This tower also has the capacity to house ~30000 servers.
The bunch of containers (if they are still there) to the right/north of the SGB3 tower is SGB4. Like SGB1, those containers have the capacity of ~12000 servers.
looks like the separation (fire-protection?) between SBG2 and SBG3 might have done its job and withhold quite some damage.
obviously doesn't say anything about cabling for network and power. I wouldn't expect to see anything out of SBG getting up and running today. if they get lucky and are even allowed to enter the area, maybe they can setup some emergency cabling in the next few days.
my condolences to everyone affected, but please already set your expectations straight... this is not only severe but catastrophic. do not rely on any hopes that anything from SBG2 is recoverable. set up your service from backups (or scratch if you have none) in some other DCs. do not wait for nothing. ?♂️
@DataRecovery said:
Is it me, or even despite the fire, this OVH SGB2 datacenter looks like ... just a bunch of containers in Strasbourg?
This image probably shows what SGB1, SGB2 and SGB3 are best.
Mein Gott!
Had no idea on how the famous OVH looks in real life. I'm not in France, but we also have former industrial areas, which are being transformed into data centers.
But even here I neither seen, nor imagined anything like that.
@Falzo said: looks like the separation (fire-protection?) between SBG2 and SBG3 might have done its job and withhold quite some damage.
My thinking was more "damn those firefighter were on point in preventing spread"...those things are really close together
I doubt there is much, they could really do about that, based on how they can or cannot locate themselves. but sure they might have tried to cool down the connection parts to avoid spread. didn't work so well on the other side towards the SBG1 containers though...
this gives an additional hint about the structure and the whole thing not really being a cube at all:
Once a "jinx-prone" colleague was on his way to fix our electrics, I phoned a friend at the fire department, who also knows the guy:
"This is probably the first time you got a 'schedule' call! See, Lucky is on his way to fix our electric cables, so I advise you to get ready and slowly be on your way to our offices." It all went fine, we all just had a laugh, and Lucky is a top-class guy - but he is a jinx which has been our running joke over the years.
Oh Lord! I feel for OVH and their customers. Horrible incident. I am glad that nobody died or got hurt, as it seems!
But I must also admit that postings like "How do I activate the Disaster Recovery Plan in the OVH panel?" amuse me a bit. Do people really think that a Disaster Recovery Plan is something that your provider offers as an addon? This world...
Amitz, a very stable genius (it's true!) and Grand Rectumfier of the official LESLOS® (LES League of Shitposters).
Certified braindead since 1974 and still perfectly happy.
@WSCallum said: I believe it was used to trial out the locations
I think mostly also because it simply was cheap. didn't they also get awards for kind of inventing or pushing that way of building DC infrastructure?
Yep that is literally the intention - Cheap and quick to build which allowed them to trial out demand in SBG before making residence there permanently, demand was so high while SBG 2/3 were being built SBG4 was also made out of shipping containers - I saw somewhere though that they were meant to be decommissioning 1/4 and merging them into 3 essentially. Unsure on the awards side but wouldn't surprise me.
I noticed that structure on some of the images from the fire. I'm not a fire engineer (if thats even a thing) or anything but that made me think that the whole structure must act like a giant chimney when on fire, creating a lot of drag and plenty of oxygen to fuel the fire.
Comments
Monitoring system clearly not functioning either - http://status.ovh.com/vms/index_sbg2.html
Yikes
yabs daddy
I'm glad everyone at the scene is ok and that all the users have good backups.
Woah!!! thats massive
HN discussion: https://news.ycombinator.com/item?id=26407323
WoW, you just know someone has already submitted an SLA claim for a $2 service
https://inceptionhosting.com
Please do not use the PM system here for Inception Hosting support issues.
the building seems badly burnt, probably gonna take a while.
I bench YABS 24/7/365 unless it's a leap year.
That video doesn't show the scale of destroyed really well. This earlier image maybe does show the scale a bit better!
Never heard of OVH SGB or SGB2.
Must be new.
"The imitator dooms himself to hopeless mediocrity." — Ralph Waldo Emerson
Free NAT KVM | Free NAT LXC
Yikes .. a whole DC gone!
Didn't they have a fire as well a couple of years ago? I can't find the news article anymore.
LinuxFreek.com — Thank you for your attention to this matter
this is really not good. i've always wanted to try a vps in SBG but maybe not possible anymore.
I bench YABS 24/7/365 unless it's a leap year.
If memory serves me, there was a fire on a building close-by. They talked about adding (or already having added) some external fire-protection plates to prevent such incidents from putting them in danger.
Can't find the article neither.
🔧 BikeGremlin guides & resources
More pics:
https://www.dna.fr/faits-divers-justice/2021/03/10/strasbourg-important-incendie-dans-une-entreprise-situee-sur-un-site-seveso-au-port-du-rhin
Free NAT KVM | Free NAT LXC
Was anyone hurt?
Yes, I see they said "Everyone is safe," but that's not the same.
🔧 BikeGremlin guides & resources
Is it me, or even despite the fire, this OVH SGB2 datacenter looks like ... just a bunch of containers in Strasbourg?
☰ VDS & Storage — Budget AMD EPYC VDS (ref) from $4/m ／ Big HDD VPS (ref) from $2/TB/mo ／ EPYC/Gold/Platinum dedis (ref) w/ GPU support
Night photos look scary - the OVH cloud datacentre turned into a big black cloud...
☰ VDS & Storage — Budget AMD EPYC VDS (ref) from $4/m ／ Big HDD VPS (ref) from $2/TB/mo ／ EPYC/Gold/Platinum dedis (ref) w/ GPU support
Damn.. That escalated quickly.
Ympker's VPN LTD Comparison, Uptime.is, Ympker's GitHub.
This image probably shows what SGB1, SGB2 and SGB3 are best.
No, OVH SGB2 is that tower that burned. SGB2 had the capacity to house ~30000 servers.
SGB1 is the bunch/two rows of containers in the foreground; some of those containers are damaged/burned as well hence the SGB1 is damaged. Those containers have the capacity to ~12000 servers.
The tower next to the right is SGB3. This tower also has the capacity to house ~30000 servers.
The bunch of containers (if they are still there) to the right/north of the SGB3 tower is SGB4. Like SGB1, those containers have the capacity of ~12000 servers.
But weren't SGB1 and SGB4 going to be decommissioned and their servers moved to SGB3.
Oh and more pictures!
It's Strasbourg and not Strasgourb.
"The imitator dooms himself to hopeless mediocrity." — Ralph Waldo Emerson
Those new Intel CPUs running real hot
I'm not sure if I am thinking of a different incident than you are, but I remember there was a fire in a building next to one of Scaleway / Online.net's datacenters 2 years ago: https://blog.scaleway.com/saint-ouen-laumone-fire-incident-security-measures-allow-dc5-to-escape-the-disaster/ .
Or they got a lp0 on fire error.
Or someone ran a "halt and catch fire" instruction.
And let's not forget about all those hotspares...
looks like the separation (fire-protection?) between SBG2 and SBG3 might have done its job and withhold quite some damage.
obviously doesn't say anything about cabling for network and power. I wouldn't expect to see anything out of SBG getting up and running today. if they get lucky and are even allowed to enter the area, maybe they can setup some emergency cabling in the next few days.
my condolences to everyone affected, but please already set your expectations straight... this is not only severe but catastrophic. do not rely on any hopes that anything from SBG2 is recoverable. set up your service from backups (or scratch if you have none) in some other DCs. do not wait for nothing. ?♂️
Mein Gott!
Had no idea on how the famous OVH looks in real life. I'm not in France, but we also have former industrial areas, which are being transformed into data centers.
But even here I neither seen, nor imagined anything like that.
☰ VDS & Storage — Budget AMD EPYC VDS (ref) from $4/m ／ Big HDD VPS (ref) from $2/TB/mo ／ EPYC/Gold/Platinum dedis (ref) w/ GPU support
Fire from Kernel 5.11 effect ?
Someone pushed their ryzen to 6GHz
Now twitter, at least the responses to OVH Cloud CEO's tweet, are on fire.
People say they do not have any data recovery plan:
And this is (supposedly) "a leading platform for collecting national and international news", at least according to google translate.
Now, when the fire is over, some even decided the timing is perfect to activate the disaster recovery plan:
☰ VDS & Storage — Budget AMD EPYC VDS (ref) from $4/m ／ Big HDD VPS (ref) from $2/TB/mo ／ EPYC/Gold/Platinum dedis (ref) w/ GPU support
My thinking was more "damn those firefighter were on point in preventing spread"...those things are really close together
Hope every one is safe!! This will impact their business again!
PeekaBoo at https://handyhost.net
(Not affiliated with the Russian Handyhost Website in any way.)
I doubt there is much, they could really do about that, based on how they can or cannot locate themselves. but sure they might have tried to cool down the connection parts to avoid spread. didn't work so well on the other side towards the SBG1 containers though...
this gives an additional hint about the structure and the whole thing not really being a cube at all:
Once a "jinx-prone" colleague was on his way to fix our electrics, I phoned a friend at the fire department, who also knows the guy:
"This is probably the first time you got a 'schedule' call! See, Lucky is on his way to fix our electric cables, so I advise you to get ready and slowly be on your way to our offices."
It all went fine, we all just had a laugh, and Lucky is a top-class guy - but he is a jinx which has been our running joke over the years.
OVH maintenance schedule:
http://travaux.ovh.net/?do=details&id=49016
🔧 BikeGremlin guides & resources
The LET Effect - People posting their server/rack numbers in their tweets.
Have you tried turning it off and on again?This gives a new meaning to Pyrocumulous Cloud
blog archives
We accept Karma donations for the last flan. 🍮 affbrr
Here more pictures from the accident.
Oh Lord! I feel for OVH and their customers. Horrible incident. I am glad that nobody died or got hurt, as it seems!
But I must also admit that postings like "How do I activate the Disaster Recovery Plan in the OVH panel?" amuse me a bit. Do people really think that a Disaster Recovery Plan is something that your provider offers as an addon? This world...
Amitz, a very stable genius (it's true!) and Grand Rectumfier of the official LESLOS® (LES League of Shitposters).
Certified braindead since 1974 and still perfectly happy.
Probably looking to have their bandwidth doubled.
☰ VDS & Storage — Budget AMD EPYC VDS (ref) from $4/m ／ Big HDD VPS (ref) from $2/TB/mo ／ EPYC/Gold/Platinum dedis (ref) w/ GPU support
Just updating about the status:
Aren’t most datacenters one big room? Why did OVH go with a shipping container design?
Only SBG1 and SBG4 are shipping containers, 2/3 are actual buildings (although still very much have a container style to them)
I believe it was used to trial out the locations before investing in actual buildings at the time.
Clouveo - SSD/NVMe Cloud VPS & Hosting
Performance & AMD EPYC Cloud | DDoS Protected | Snapshots | Auto Backups
I think mostly also because it simply was cheap. didn't they also get awards for kind of inventing or pushing that way of building DC infrastructure?
Yep that is literally the intention - Cheap and quick to build which allowed them to trial out demand in SBG before making residence there permanently, demand was so high while SBG 2/3 were being built SBG4 was also made out of shipping containers - I saw somewhere though that they were meant to be decommissioning 1/4 and merging them into 3 essentially. Unsure on the awards side but wouldn't surprise me.
Clouveo - SSD/NVMe Cloud VPS & Hosting
Performance & AMD EPYC Cloud | DDoS Protected | Snapshots | Auto Backups
that was 2017 in regards to the power loss incident there and afair more related to the grid design and not neccessarily the containers itself ;-)
That would make more sense!
Clouveo - SSD/NVMe Cloud VPS & Hosting
Performance & AMD EPYC Cloud | DDoS Protected | Snapshots | Auto Backups
This is a nightmare for some...
Nexa Racks - Reliable Web Hosting Company
I noticed that structure on some of the images from the fire. I'm not a fire engineer (if thats even a thing) or anything but that made me think that the whole structure must act like a giant chimney when on fire, creating a lot of drag and plenty of oxygen to fuel the fire.