In the day 12.05.2021 we have find 1 big storage netup down with+ 3000 clients involucrated in this so we have try to recuperate the data but unsucces.
unfortunatly from 400 hdd of 12tb each we have recuperate like 50 until now. he big problem in this period is buy new hdd due of the shit coin chia because the price in new hardware is almost 3x more exensive than normal so i have dedicated all my time infind a solution to have new hdd as rapid i can to have the posibility to migrate the data from the affected hdd.
What i think to do and hope is restablish the old service until the clients can migrate the data and offer a same service with 1 year for free to all clients involucrated in this (To note from all our services the issue is only in some storage plans , the rest is up an running so the compensation will be only for the clients from this Rack wich was fucked.)
I have not connect here to ask merci because i dont need it , i have connect to update a little the problem as i have been driving 3 times the way oradea to bucharest in 1 week.
We will came with a final decision Monday as i need 1 day for me because i am almost depressed due of this .
To me cociu comes across as someone who genuinely wants best for his clients, yet he's too nice in a sense that he under prices his stuff, which in turn causes him to get into financial trouble. He then tries to 'solve' this with some workarounds like double credit promos or shifting stuff (on paper) towards his Perfume company, but in the end that's just buying time and not tackling the actual problem: his prices are not sustainable.
I have a different guess. Let assume everything he says is actually true. Lets do this quick and dirty,,,,
Any type of 2u dual E5 server. (as cheap as possible) - 2u
(The main criteria for the 2u server is having both 2x 10g sfp+ or better networking and the option for a SAS HBA.)
10x CSE-847 847JBOD-14 ... those- are 45 disks in 4u -> 10 units in the rack=> 40u
gives you a total disk capacity of 45x10 -> 450 3,5" disks + up to 12x 3,5" disks from the server in one 42u rack.
then he probably runs 2x 10g lagg or something to one of his other racks where the vps nodes sit. and the rest is network storage mounts.
here our single points of failure are:
Network
raid controller
the 2u server
edit: he also stated he uses 12TB disks. so those are potetially SMR ... so a lot of fun when attempting to rebuild a raid array or zfs pool.
I would assume he uses the 3.5" ones for HDDs and 2.5" ones as OS disks.
These are 2U.
Let's say he uses 46U racks (rare but let's assume)
There is also a 14 x 3,5 Slot Version of the Dl380 wich we use for our Storage Solutions, but that is also not enough for 400 drives in one rack.
But I also can't believe what He told has happen. (I am nearly sure He install the OS on the same array or use a USB drive / SD card for)
We got 3 Power outages since I startet in 2013, but there were never more that a Hand full drives dead after that happens.
@serverian said: Very optimistic guess thinking he'd achieve that kind of cluster setup running fine
When you are running that kind of bare budget set up, anything is possible I guess. History if nothing else shows us that budget hosts only need one thing to happen in order to bring them down. With HS there appears to be many of those one things ready and waiting to happen.
jsg said
TL;DR 2: "cociu only has less than 3000 IPs" was a side story anyway and one that is irrelevant for the topic of this thread.
OK, so lets get to the meat of it then. Do you truly believe a 'UPS failure' fried 400 drives? You come off pretty smart, there's simply no way you believe it. No one does.
Here's my take:
#1: He's been overselling his storage space via thin provisioning (qcow2, LVM thin provisioning). With the roll out of Chia, there's no compression/deduplication possible, and those users went wild and ran through their space.
When you run out of space on a thin provision LVM pool, you can't actually start the pool unless you have at least a tiny bit (even a couple GB) of space spare on the volgroup for the metadata. Without that, it won't start. Infact, for the longest time, the volgroup would destroy itself (eat its own metadata partition) when it ran out of space. If he's on an old enough kernel/lvm version, could've happened.
#2: - He owes his datacenter a lot of money as has been mentioned in the filings and the DC simply cut some of his power to slow down the debt from growing much more.
#3: - A crack pot one, and more so a joke than anything, he sold the drives.
#4: - The outright lie - a UPS fried it.
Lets dive into #4 a bit. A datacenter known as WebNX had a generator (that was indoors) catch fire, set off the sprinklers that were not only above the generators, but also above the servers. The sprinklers soaked the servers full on. Not just a sprinkle. I literally went to the datacenter to help a friend of mine recover his data and the drives all have massive water marks on them.
Even still, they have 1000's, maybe even 10's of thousands, of servers online and they didn't lose 400 drives in their mess. They lost some, but I'd be surprised if it was 100.
I've had nodes go offline because of a UPS on the fritz and it wasn't double cabled. Even still, not a single drive was physically damaged.
If there was such a catastrophic UPS failure in his datacenter we'd have heard news of someone else getting affected by it. There's not a single host that's even heard such a thing as a rumor, never mind actually experiencing it.
I think it's #1 or #2, maybe both.
He didn't have a UPS failure that fried 400 drives.
And remember. Any power strip that'll be allowed to be used in a datacenter has surge protection in them with built in breakers. Those things would've tripped and protected the gear.
Infact, don't APC/Tripp include insurance with every strip for this very event?
jsg said
TL;DR 2: "cociu only has less than 3000 IPs" was a side story anyway and one that is irrelevant for the topic of this thread.
OK, so lets get to the meat of it then. Do you truly believe a 'UPS failure' fried 400 drives? You come off pretty smart, there's simply no way you believe it. No one does.
Here's my take:
#1: He's been overselling his storage space via thin provisioning (qcow2, LVM thin provisioning). With the roll out of Chia, there's no compression/deduplication possible, and those users went wild and ran through their space.
When you run out of space on a thin provision LVM pool, you can't actually start the pool unless you have at least a tiny bit (even a couple GB) of space spare on the volgroup for the metadata. Without that, it won't start. Infact, for the longest time, the volgroup would destroy itself (eat its own metadata partition) when it ran out of space. If he's on an old enough kernel/lvm version, could've happened.
#2: - He owes his datacenter a lot of money as has been mentioned in the filings and the DC simply cut some of his power to slow down the debt from growing much more.
#3: - A crack pot one, and more so a joke than anything, he sold the drives.
#4: - The outright lie - a UPS fried it.
Lets dive into #4 a bit. A datacenter known as WebNX had a generator (that was indoors) catch fire, set off the sprinklers that were not only above the generators, but also above the servers. The sprinklers soaked the servers full on. Not just a sprinkle. I literally went to the datacenter to help a friend of mine recover his data and the drives all have massive water marks on them.
Even still, they have 1000's, maybe even 10's of thousands, of servers online and they didn't lose 400 drives in their mess. They lost some, but I'd be surprised if it was 100.
I've had nodes go offline because of a UPS on the fritz and it wasn't double cabled. Even still, not a single drive was physically damaged.
If there was such a catastrophic UPS failure in his datacenter we'd have heard news of someone else getting affected by it. There's not a single host that's even heard such a thing as a rumor, never mind actually experiencing it.
I think it's #1 or #2, maybe both.
He didn't have a UPS failure that fried 400 drives.
And remember. Any power strip that'll be allowed to be used in a datacenter has surge protection in them with built in breakers. Those things would've tripped and protected the gear.
Infact, don't APC/Tripp include insurance with every strip for this very event?
Francisco
I like JSG. But, the part I didn't understand was, his two comments.
Where he didn't really consider BGP tools and just assumed an answer.
Even if someone is using holding companies to hold on to their range, they still need to announce them. Marius announces everything under one ASN. So this makes 0 logical sense.
There is a lot of hearsay and people making assumptions or guesses as to what has happened, but to me, the two hard facts below speak the loudest.
Servers are down for 8 days
Company owner goes radio silent for 7 days
If you then consider previous downtimes, strange network configurations, unsustainable pricing, double credit promos and the potentially unconvincing explanation about this current downtime it starts to paint a pretty interesting picture.
I'm sure cociu is a hardworking guy and I have no bad feelings towards him. However, I think we should have higher standards as a community (meaning LET, LES & HT since we are all the same group just split across different boards) and given the history of this particular provider, I think the show has been allowed to go on long enough.
The excuse of "ITS CHEAP" only goes so far, especially if HS doesn't recover and the money people have added in credits (wrongly or rightly) disappears overnight.
Amitz, a very stable genius (it's true!) and Grand Rectumfier of the official LESLOS® (LES League of Shitposters).
Certified braindead since 1974 and still perfectly happy.
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural. It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
@sweatbar said:
Still no response to the tickets and the vps is down so im giving up on that
Don't sweat it and set the bar high.
I've been watching you
A la la la la long
A la la la la long long li long long long
Come on
A la la la la long
A la la la la long long li long long long
PeekaBoo at https://handyhost.net
(Not affiliated with the Russian Handyhost Website in any way.)
Just a typical LET thread the way it is playing out. As for @jsg, his mother needs to put some limits on his internet time. He is just that annoying kid that must always have the last word no matter what and will never realise he doesn't have an audience for his shit.
@vyas said:
Dont know what is scarier in the LET thread on the topic.
Jsg rambles or jsg declaring he is @seriesn fanboy.
And I thought Jay was cool
Not so long ago, jsg claimed that he would blacklist any provider that advertised on LES.
I argued with him about this, saying that it was a silly thing to do, but he stubbornly insisted that any such provider deserved to be blacklisted.
So much for his meaning what he said.
"A single swap file or partition may be up to 128 MB in size. [...] [I]f you need 256 MB of swap, you can create two 128-MB swap partitions." (M. Welsh & L. Kaufman, Running Linux, 2e, 1996, p. 49)
@vyas said:
Dont know what is scarier in the LET thread on the topic.
Jsg rambles or jsg declaring he is @seriesn fanboy.
And I thought Jay was cool
Not so long ago, jsg claimed that he would blacklist any provider that advertised on LES.
I argued with him about this, saying that it was a silly thing to do, but he stubbornly insisted that any such provider deserved to be blacklisted.
So much for his meaning what he said.
you should try to point this out to him over there. gonna be fun drama and I'll bet he will find arguments why you obviously misunderstood and he never said that. oh, and don't forget to provide a link, if you seriously want to discuss 😂
@vyas said:
Dont know what is scarier in the LET thread on the topic.
Jsg rambles or jsg declaring he is @seriesn fanboy.
And I thought Jay was cool
Not so long ago, jsg claimed that he would blacklist any provider that advertised on LES.
I argued with him about this, saying that it was a silly thing to do, but he stubbornly insisted that any such provider deserved to be blacklisted.
So much for his meaning what he said.
you should try to point this out to him over there. gonna be fun drama and I'll bet he will find arguments why you obviously misunderstood and he never said that. oh, and don't forget to provide a link, if you seriously want to discuss 😂
Ag then we will get to witness “what I type versus what you read” sermon
@Falzo said: you should try to point this out to him over there. gonna be fun drama and I'll bet he will find arguments why you obviously misunderstood and he never said that. oh, and don't forget to provide a link, if you seriously want to discuss 😂
Already happened, he starting caveating it with things like it depending on the date the provider started advertising on LES and other weak BS.
Comments
@seriesn the Mathematician...
As seen on OGF.
Did he just poke holes into the Cociu story?
blog archives
Ikea and WalMart.
Nexus Bytes Ryzen Powered NVMe VPS | NYC|Miami|LA|London|Netherlands| Singapore|Tokyo
Storage VPS | LiteSpeed Powered Web Hosting + SSH access | Switcher Special |
I don't know. I just wanted to make sense out of the whole thing.
Nexus Bytes Ryzen Powered NVMe VPS | NYC|Miami|LA|London|Netherlands| Singapore|Tokyo
Storage VPS | LiteSpeed Powered Web Hosting + SSH access | Switcher Special |
Cociu message is more confused than usual. I didn't even try to make sense of it let alone try to work out the numbers.
To be honest, it sounds unbelievable.
He's probably using these: https://www.ebay.com/itm/HP-Proliant-DL380e-Gen8-Storage-Server-Dual-8-Core-E5-2450L-P420-1GB-14x-LFF-Bay-/274431553423 (Correct me if I'm wrong)
These have 12 x 3.5" slots and 2 x 2.5" slots.
I would assume he uses the 3.5" ones for HDDs and 2.5" ones as OS disks.
These are 2U.
Let's say he uses 46U racks (rare but let's assume)
And let's say he only puts servers in these and don't use 1U for TOR switch.
Then he can load 23 x 2U servers in a rack.
23 * 12 = 276 3.5" HDDs.
So for 400 HDDs, he needs another rack.
So he had 1.4 racks worth of servers (33.3 total) failed with a power outage.
Again, 33.3 servers had RAID failure with a power outage at the same time.
He should be suing the power company and RAID card manufacturer along with the HDD manufacturer for damages.
I gotta wonder about what kind of power conditioning he might have been using. Is the claim actually that all those hdd's were fried?
The last power outage I did notice was 23 days ago, but that is a bit far away.
Free NAT KVM | Free NAT LXC
His claim:
In the day 12.05.2021 we have find 1 big storage netup down with+ 3000 clients involucrated in this so we have try to recuperate the data but unsucces.
unfortunatly from 400 hdd of 12tb each we have recuperate like 50 until now. he big problem in this period is buy new hdd due of the shit coin chia because the price in new hardware is almost 3x more exensive than normal so i have dedicated all my time infind a solution to have new hdd as rapid i can to have the posibility to migrate the data from the affected hdd.
What i think to do and hope is restablish the old service until the clients can migrate the data and offer a same service with 1 year for free to all clients involucrated in this (To note from all our services the issue is only in some storage plans , the rest is up an running so the compensation will be only for the clients from this Rack wich was fucked.)
I have not connect here to ask merci because i dont need it , i have connect to update a little the problem as i have been driving 3 times the way oradea to bucharest in 1 week.
We will came with a final decision Monday as i need 1 day for me because i am almost depressed due of this .
To me cociu comes across as someone who genuinely wants best for his clients, yet he's too nice in a sense that he under prices his stuff, which in turn causes him to get into financial trouble. He then tries to 'solve' this with some workarounds like double credit promos or shifting stuff (on paper) towards his Perfume company, but in the end that's just buying time and not tackling the actual problem: his prices are not sustainable.
I hope he recovers from this and, more importantly, learns from it. I wish him all the best, even though I have no services with him... but please stop lying about that magic 'faster than light' tunnel.
LinuxFreek.com — Thank you for your attention to this matter
https://hostloc.com/thread-847254-1-1.html
Free NAT KVM | Free NAT LXC
Anything interesting there to be translated @yoursunny ?
"The imitator dooms himself to hopeless mediocrity." — Ralph Waldo Emerson
Hostsolutions.ro
Has become
Hostsolutions Row (controversy/strife)
Or
Hostsolutions रो (Ro spelt in Devnagri, means “to cry” in most parts of India)
blog archives
I missed to mention that, there was nothing interesting.
Besides, chinese people did rage about the shitty english to chinese translation.
Free NAT KVM | Free NAT LXC
If its local storage, with SAN Storage it would be possible. But very expensive.
https://h20195.www2.hpe.com/v2/getpdf.aspx/c04607918.pdf
Thats HPs smalles/cheapest alternative (older 3Par) not expecting him to have a Nimble storage already (without warranty).
“Technology is best when it brings people together.” – Matt Mullenweg
I have a different guess. Let assume everything he says is actually true. Lets do this quick and dirty,,,,
Any type of 2u dual E5 server. (as cheap as possible) - 2u
(The main criteria for the 2u server is having both 2x 10g sfp+ or better networking and the option for a SAS HBA.)
10x CSE-847 847JBOD-14 ... those- are 45 disks in 4u -> 10 units in the rack=> 40u
gives you a total disk capacity of 45x10 -> 450 3,5" disks + up to 12x 3,5" disks from the server in one 42u rack.
then he probably runs 2x 10g lagg or something to one of his other racks where the vps nodes sit. and the rest is network storage mounts.
here our single points of failure are:
edit: he also stated he uses 12TB disks. so those are potetially SMR ... so a lot of fun when attempting to rebuild a raid array or zfs pool.
this is getting messsy
I bench YABS 24/7/365 unless it's a leap year.
Dl380 LFF has up to 14 Slots for 3,5 drive Slots.
As He is talking about a Rack there could be around 20 of These servers (even > @serverian said:
There is also a 14 x 3,5 Slot Version of the Dl380 wich we use for our Storage Solutions, but that is also not enough for 400 drives in one rack.
But I also can't believe what He told has happen. (I am nearly sure He install the OS on the same array or use a USB drive / SD card for)
We got 3 Power outages since I startet in 2013, but there were never more that a Hand full drives dead after that happens.
Are you saying that he's using 2 CPUs (low power and low core ones, too) to host thousands of VMs?
Very optimistic guess thinking he'd achieve that kind of cluster setup running fine.
They must be bare metal servers.
I’m only adding SAN storage as an option IF it actually was 400 disks that went tits up.
Honestly, I’m having trouble believing everything that @cociu ever said and now say about his infrastructure.
“Technology is best when it brings people together.” – Matt Mullenweg
When you are running that kind of bare budget set up, anything is possible I guess. History if nothing else shows us that budget hosts only need one thing to happen in order to bring them down. With HS there appears to be many of those one things ready and waiting to happen.
He mentioned some kind of 48 drive fiber storage back in 2017:
https://www.lowendtalk.com/discussion/comment/2055222/#Comment_2055222
Are you sure about that? There are no 12TB SMR drives on the market afaik.
Franscisco is enough with jsg
thank you for pointing that out. it appears i have been misinformed on that one.
forgot that that seems to end at 8tb
Poor mule
blog archives
I think the overselling makes sense.
I bench YABS 24/7/365 unless it's a leap year.
I like JSG. But, the part I didn't understand was, his two comments.
Nexus Bytes Ryzen Powered NVMe VPS | NYC|Miami|LA|London|Netherlands| Singapore|Tokyo
Storage VPS | LiteSpeed Powered Web Hosting + SSH access | Switcher Special |
I don't have anything constructive to add, so it's meme time
yabs daddy
There is a lot of hearsay and people making assumptions or guesses as to what has happened, but to me, the two hard facts below speak the loudest.
If you then consider previous downtimes, strange network configurations, unsustainable pricing, double credit promos and the potentially unconvincing explanation about this current downtime it starts to paint a pretty interesting picture.
I'm sure cociu is a hardworking guy and I have no bad feelings towards him. However, I think we should have higher standards as a community (meaning LET, LES & HT since we are all the same group just split across different boards) and given the history of this particular provider, I think the show has been allowed to go on long enough.
The excuse of "ITS CHEAP" only goes so far, especially if HS doesn't recover and the money people have added in credits (wrongly or rightly) disappears overnight.
Still no response to the tickets and the vps is down so im giving up on that
PeekaBoo at https://handyhost.net
(Not affiliated with the Russian Handyhost Website in any way.)
But, it's another lesson that needs to be learnt and losing money is the only way for many in the LE* arena to learn it.
Amitz, a very stable genius (it's true!) and Grand Rectumfier of the official LESLOS® (LES League of Shitposters).
Certified braindead since 1974 and still perfectly happy.
What is HT?
UpCloud free $25 through this aff link - Akamai, DigitalOcean and Vultr alternative, multiple location, IPv6.
HostedTalk.net by @jarland
Don't sweat it and set the bar high.
Involucrated by the bar?
Get the best deal on your next VPS or Shared/Reseller hosting from RacknerdTracker.com - The original aff garden.
Not a pretty sight!
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
I've been watching you
A la la la la long
A la la la la long long li long long long
Come on
A la la la la long
A la la la la long long li long long long
PeekaBoo at https://handyhost.net
(Not affiliated with the Russian Handyhost Website in any way.)
Dont know what is scarier in the LET thread on the topic.
Jsg rambles or jsg declaring he is @seriesn fanboy.
And I thought Jay was cool
blog archives
how about both.
and nexusbytes better not slip up ever , or prepare for a 10,000 word essay from @jsg
I bench YABS 24/7/365 unless it's a leap year.
Just a typical LET thread the way it is playing out. As for @jsg, his mother needs to put some limits on his internet time. He is just that annoying kid that must always have the last word no matter what and will never realise he doesn't have an audience for his shit.
Not so long ago, jsg claimed that he would blacklist any provider that advertised on LES.
I argued with him about this, saying that it was a silly thing to do, but he stubbornly insisted that any such provider deserved to be blacklisted.
So much for his meaning what he said.
"A single swap file or partition may be up to 128 MB in size. [...] [I]f you need 256 MB of swap, you can create two 128-MB swap partitions." (M. Welsh & L. Kaufman, Running Linux, 2e, 1996, p. 49)
isn't that's why he's so against InceptionHosting? whining about Anthony / Clouvider conspiracy and whatnot. Getting too high on donuts.
I bench YABS 24/7/365 unless it's a leap year.
you should try to point this out to him over there. gonna be fun drama and I'll bet he will find arguments why you obviously misunderstood and he never said that. oh, and don't forget to provide a link, if you seriously want to discuss 😂
Ag then we will get to witness “what I type versus what you read” sermon
blog archives
I'm sure all the providers are quaking at the thought of jsg's blacklist. Lol.
Already happened, he starting caveating it with things like it depending on the date the provider started advertising on LES and other weak BS.
In my mind, jsg is actually just cociu and he's convinced us all that he struggles with English to throw us off the scent.