@corbpie said:
Aren’t most datacenters one big room? Why did OVH go with a shipping container design?
Probably following the same idea as Google 12 years ago:
Amitz, a very stable genius (it's true!) and Grand Rectumfier of the official LESLOS® (LES League of Shitposters).
Certified braindead since 1974 and still perfectly happy.
@Freek said:
Didn't they have a fire as well a couple of years ago? I can't find the news article anymore.
If memory serves me, there was a fire on a building close-by. They talked about adding (or already having added) some external fire-protection plates to prevent such incidents from putting them in danger.
Can't find the article neither.
That was online.net, their "firewall" did work for them though back then.
In some countries, this could've meant over a 100 deaths. It's good that nobody got hurt. Yes, we can mock people for not keeping backups, but still, the next few days are going to hurt like hell.
Unfortunate for all those involved. I am surprised it spread so far in the one building though. I would have expected them to have some pretty robust fire doors at least to ensure that it was a localised incident.
@ApeWeb said:
Unfortunate for all those involved. I am surprised it spread so far in the one building though. I would have expected them to have some pretty robust fire doors at least to ensure that it was a localised incident.
Looking at the building, it looks pretty ghetto to me. Not even sure whether they had a working fire suppression system.
♻ Amitz day is October 21.
♻ Join Nigh sect by adopting my avatar. Let us spread the joys of the end.
@yoursunny said:
This is why I have all the code, config, and content on Bitbucket.
If my server vanishes, I can redeploy my website in two hours.
Where do you backup Bitbucket to?
The website is in three git repositories. They are on my laptop and Bitbucket.
My server contains all the code including input files (Markdown), but not the git history.
VPS vanishes (5% probability): rsync from laptop
Laptop breaks (2% probability): clone from Bitbucket
Bitbucket deadpools (1% probability): push from laptop to GitLab
Laptop and Bitbucket lost at the same time (0.02% probability): scp from VPS, but I lose the git history
All three at the same time (0.001% probability): redirect to the Internet Archive, and start making new website
1) last year we had a problem with the fire ... but even the pompiers was surprised because the servers is not combustible so the fire was powered down by "inert gas" (the gas what we use for not let oxigen in the dc room in case a fire is detected) so pombiers was not do nothing and the fire was stopped.
2) looking to the picks what is shared in internet with the owh i see a bulding .... with xxxxxxx who know how many servers and not have a proper fusible for the electrical cable ? not one like 4-7-10 if is need etc.
3) technically is ALMOST imposible to get so agresive fire in all the bulding including a constant fire in all parts , almos same intensity ...
4) where is the fuking detection ?
5) been a bulding construed with containers ... what power of fire can this generate ?
6) the most important and maybe the comunity is help me (i mean people who is affected can share the servers model) , let supose in all bulding was only old servers with clients what is not so rentable anymore and i am pretty sure the insurance is one of the big one ... so can be 10000 factors for this fire (maybe is SF for someone but for me is not because i have see many in this life)
7 ) i reeally feel bad for them , to see part of your business burining i am sure is not a pleasure but i hope this will learn some lesson.
8) Many people this days is mining because of hight value of crypto so i sugest to all owners for this business to make doble check in the electrical system
9) All this was not to critic or defame OVH , was my point of view.
PHP-Friends to the rescue. They do temporarily offer free hosting to affected OVH customers to keep their services online. You can contact them via their email. Nice gesture
OVH support probably is drowning right now and if clients can be hosted elsewhere for a bit that'd be quite helpful, I imagine.
@deank said:
How do they know whether someone is an OVH customer to begin with?
Not sure about their procedure, but a nice gesture Ofc, like most nice things, some ppl will try to exploit it.
I can imaging they might ask for a link to an invoice with the affected service. In my OVH panel I can just get a HTML-link to an invoice, which is visible for all to see. That's hard to cheat with because OVH basically gives you all data you need to verify that a person had that service there.
@deank said:
How do they know whether someone is an OVH customer to begin with?
Not sure about their procedure, but a nice gesture Ofc, like most nice things, some ppl will try to exploit it.
I can imaging they might ask for a link to an invoice with the affected service. In my OVH panel I can just get a HTML-link to an invoice, which is visible for all to see. That's hard to cheat with because OVH basically gives you all data you need to verify that a person had that service there.
I was somehow expecting to see Hetzner sell quite a few a server on the server auction, given that this incident most probably has affected quite a lot of people. But still not too much movement in the availability count.
@Bochi said:
I was somehow expecting to see Hetzner sell quite a few a server on the server auction, given that this incident most probably has affected quite a lot of people. But still not too much movement in the availability count.
If you didn't have a backup, and new server doesn't really help you.
@Aaron said:
If you didn't have a backup, and new server doesn't really help you.
Well, as sad as it is for those affected: you are certainly right!
And as entertaining as it might be to watch the comments over on twitter on this, I have to keep in mind, that I am also not always tracking the backup activities of my servers... However, it's just personal stuff where loosing a few days of data wouldn't hurt to much. But I suspect we will see quite a few "businesses" going down over this.
I have a kinsufi server in rbx if that can help anyone transfer backups from there. Also have one at BHS if that matters. Sympathy to everyone affected.
@joepie91 said:
And apparently [they were using water sprinklers, not gas suppression
The water based fire suppression system was probably just installed to comply with building fire code and keep the insurance company happy, but doubt they ever planned to use it as it would kill the servers.
It seem like OVH penny pinched their data centre build a little too much wooded floor and inadequate fire suppression systems.
I don't know much about fire suppression system just wondering dose anyone know roughly the cost of a proper gas based system.
Netcraft estimates 3.6 million websites across 464,000 distinct domains were taken offline because of this fire.
Some of those will of course return when parts of SGB1, SGB3, SGB4 are put back online. However many of those will be changed into a dark cloud of smoke!
@rajprakash said: 3.6 million websites, each losing millions in revenue every minute of downtime (/sarcasm) .... trillions of revenue loss might crash global markets.
it's rather press-bs to blow up the numbers. 460k domains is not dramatic enough.
if by websites they'd really mean different independent websites and not just (sub-)pages that means on average each domain needs to host 7 different independent websites.
instead for sure some smartarse probably meant 3.6m simple pages, even if they belong to the very same website.
but obviously it sounds much more dramatic if you tell the world 3.6 million websites are down. ?♂️
I saw something in one of the French news articles linked further up that the firefighters had trouble at first because they couldn't shut off the electricity to the DC, fwiw.
Comments
agreed. and if the power gets cut, luck might have it the generators kick in and survive a while to have the vents/AC still blowing into it...
This is very bad
食之无味 弃之可惜 - Too arduous to relish, too wasteful to discard.
Remember what I always cite.
If you are a follower of the Nigh sect, you will keep backups.
♻ Amitz day is October 21.
♻ Join Nigh sect by adopting my avatar. Let us spread the joys of the end.
Sorry, I couldn't resist....
In case of OVH, double the Mbps/Gbps limit for their unmetered* Bandwidth.
Have you tried turning it off and on again?"Smoke on the water, fire in the sky."
What was the reason?
And why did it burn so well?
Probably following the same idea as Google 12 years ago:
Amitz, a very stable genius (it's true!) and Grand Rectumfier of the official LESLOS® (LES League of Shitposters).
Certified braindead since 1974 and still perfectly happy.
That was online.net, their "firewall" did work for them though back then.
Hey teamacc. You're a dick. (c) Jon Biloh, 2020.
its pretty bad for those without a backup...
my wife's email is down because of it.
hopefully recovers well for all.
In some countries, this could've meant over a 100 deaths. It's good that nobody got hurt. Yes, we can mock people for not keeping backups, but still, the next few days are going to hurt like hell.
Have you tried turning it off and on again?Being a datacenter means there isn't generally anyone there.
Which ironically is probably why the whole building was burnt down; no one was there to notice early.
♻ Amitz day is October 21.
♻ Join Nigh sect by adopting my avatar. Let us spread the joys of the end.
Guess how many users have their backups in the same datacentre?
Action and Reaction in history
Having "a backup" on the same location isn't a backup. Backup 101.
♻ Amitz day is October 21.
♻ Join Nigh sect by adopting my avatar. Let us spread the joys of the end.
According to Twitter:
This is why I have all the code, config, and content on Bitbucket.
If my server vanishes, I can redeploy my website in two hours.
Stay away from WordPress and make static websites!
We accept Karma donations for the last flan. 🍮 affbrr
Honestly, I don't expect the whole datacenter to go down. If this were to happen to me, I'd probably have to go back at least two weeks ago.
Action and Reaction in history
Guide please
Unfortunate for all those involved. I am surprised it spread so far in the one building though. I would have expected them to have some pretty robust fire doors at least to ensure that it was a localised incident.
Where do you backup Bitbucket to?
Looking at the building, it looks pretty ghetto to me. Not even sure whether they had a working fire suppression system.
♻ Amitz day is October 21.
♻ Join Nigh sect by adopting my avatar. Let us spread the joys of the end.
The website is in three git repositories. They are on my laptop and Bitbucket.
My server contains all the code including input files (Markdown), but not the git history.
https://yoursunny.com/t/2017/yoursunny-com-git/
https://hexo.io/
We accept Karma donations for the last flan. 🍮 affbrr
Git is kinda nice, you already have 3 different backups by default.
Free NAT KVM | Free NAT LXC
All the firewalls, oh, the irony.
"The imitator dooms himself to hopeless mediocrity." — Ralph Waldo Emerson
My 2 cents :
1) last year we had a problem with the fire ... but even the pompiers was surprised because the servers is not combustible so the fire was powered down by "inert gas" (the gas what we use for not let oxigen in the dc room in case a fire is detected) so pombiers was not do nothing and the fire was stopped.
2) looking to the picks what is shared in internet with the owh i see a bulding .... with xxxxxxx who know how many servers and not have a proper fusible for the electrical cable ? not one like 4-7-10 if is need etc.
3) technically is ALMOST imposible to get so agresive fire in all the bulding including a constant fire in all parts , almos same intensity ...
4) where is the fuking detection ?
5) been a bulding construed with containers ... what power of fire can this generate ?
6) the most important and maybe the comunity is help me (i mean people who is affected can share the servers model) , let supose in all bulding was only old servers with clients what is not so rentable anymore and i am pretty sure the insurance is one of the big one ... so can be 10000 factors for this fire (maybe is SF for someone but for me is not because i have see many in this life)
7 ) i reeally feel bad for them , to see part of your business burining i am sure is not a pleasure but i hope this will learn some lesson.
8) Many people this days is mining because of hight value of crypto so i sugest to all owners for this business to make doble check in the electrical system
9) All this was not to critic or defame OVH , was my point of view.
Cheers all.
PHP-Friends to the rescue. They do temporarily offer free hosting to affected OVH customers to keep their services online. You can contact them via their email. Nice gesture
OVH support probably is drowning right now and if clients can be hosted elsewhere for a bit that'd be quite helpful, I imagine.
Ympker's VPN LTD Comparison, Uptime.is, Ympker's GitHub.
How do they know whether someone is an OVH customer to begin with?
♻ Amitz day is October 21.
♻ Join Nigh sect by adopting my avatar. Let us spread the joys of the end.
Not sure about their procedure, but a nice gesture
Ofc, like most nice things, some ppl will try to exploit it.
Ympker's VPN LTD Comparison, Uptime.is, Ympker's GitHub.
Some? A lot will try.
As soon as "free" comes to play, dignity vanishes.
♻ Amitz day is October 21.
♻ Join Nigh sect by adopting my avatar. Let us spread the joys of the end.
I can imaging they might ask for a link to an invoice with the affected service. In my OVH panel I can just get a HTML-link to an invoice, which is visible for all to see. That's hard to cheat with because OVH basically gives you all data you need to verify that a person had that service there.
That's nice
Ympker's VPN LTD Comparison, Uptime.is, Ympker's GitHub.
I was somehow expecting to see Hetzner sell quite a few a server on the server auction, given that this incident most probably has affected quite a lot of people. But still not too much movement in the availability count.
If you didn't have a backup, and new server doesn't really help you.
They are old data centers that opened in 2012.
https://purplehost.com.br - Reliable, secure and affordable game hosting.
Well, as sad as it is for those affected: you are certainly right!
And as entertaining as it might be to watch the comments over on twitter on this, I have to keep in mind, that I am also not always tracking the backup activities of my servers... However, it's just personal stuff where loosing a few days of data wouldn't hurt to much. But I suspect we will see quite a few "businesses" going down over this.
Me too :'(
That’s SBG/SBG2, not SGB or SGB2 ?
"The imitator dooms himself to hopeless mediocrity." — Ralph Waldo Emerson
I have a kinsufi server in rbx if that can help anyone transfer backups from there. Also have one at BHS if that matters. Sympathy to everyone affected.
Is ExtraVM France server in SBG? I hope it's not.
Neither did OVH.
FRAKVM2 is down for 1 day: https://status.extravm.com/?m=5ef8c084b648c772cb2ff2f4
Ympker's VPN LTD Comparison, Uptime.is, Ympker's GitHub.
It appears that this was the construction of SBG2:
And apparently they were using water sprinklers, not gas suppression.
So yeah, fire waiting to happen.
The water based fire suppression system was probably just installed to comply with building fire code and keep the insurance company happy, but doubt they ever planned to use it as it would kill the servers.
It seem like OVH penny pinched their data centre build a little too much wooded floor and inadequate fire suppression systems.
I don't know much about fire suppression system just wondering dose anyone know roughly the cost of a proper gas based system.
Netcraft estimates 3.6 million websites across 464,000 distinct domains were taken offline because of this fire.
Some of those will of course return when parts of SGB1, SGB3, SGB4 are put back online. However many of those will be changed into a dark cloud of smoke!
Got data on how many spammers taken offline?
I am guessing 3.5 million out of 3.6.
♻ Amitz day is October 21.
♻ Join Nigh sect by adopting my avatar. Let us spread the joys of the end.
3.6 million websites, each losing millions in revenue every minute of downtime (/sarcasm) .... trillions of revenue loss might crash global markets.
it's rather press-bs to blow up the numbers. 460k domains is not dramatic enough.
if by websites they'd really mean different independent websites and not just (sub-)pages that means on average each domain needs to host 7 different independent websites.
instead for sure some smartarse probably meant 3.6m simple pages, even if they belong to the very same website.
but obviously it sounds much more dramatic if you tell the world 3.6 million websites are down. ?♂️
I saw something in one of the French news articles linked further up that the firefighters had trouble at first because they couldn't shut off the electricity to the DC, fwiw.