CC spotted the mass exodus and decided to call a halt on the remaining! [Speculation]
I'm thinking some technical issues going on instead. My dedi came back online about 10 minutes ago, so I'm back up and running. Virmach site still appears to be down.
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural. It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
CC spotted the mass exodus and decided to call a halt on the remaining! [Speculation]
I'm thinking some technical issues going on instead. My dedi came back online about 10 minutes ago, so I'm back up and running. Virmach site still appears to be down.
We have a contingency plan. It's not perfect. Nor were we expecting it to go this far or be done in this way. I've been working on it the entire time, and have emergency replacement servers to some degree. I'm sending out some vague communication that's at the very least necessary to immediately inform customers of the risks at this point.
Our site is indirectly related. Luckily it's only the few static pages that happened to be on one of the servers facing issues. This isn't important right now, what I previously said is so we're going to be focusing on that.
I'm working with our developer today to pro-rate refund all dedicated servers where possible, for any remaining duration. We don't want to give people the wrong idea of what's going on and want to reduce any panic, at least on any end we can control. Then, first come first serve basis on emergency tickets created requesting a replacement server, we'll begin provisioning those today, as well as backup servers for anyone who requires it. Both of these would temporarily have no charge (as in, free for the time being.)
This should give everyone a few different choices given the scenario, so can they proceed with a purchase as necessary elsewhere, or to at the very least have some VPS or server to hold any backup or temporary home.
Let me know if you guys have any questions or suggestions but please keep in mind they'll have to be specific and direct.
@VirMach said:
I'm working with our developer today to pro-rate refund all dedicated servers where possible, for any remaining duration. We don't want to give people the wrong idea of what's going on and want to reduce any panic, at least on any end we can control. Then, first come first serve basis on emergency tickets created requesting a replacement server, we'll begin provisioning those today, as well as backup servers for anyone who requires it. Both of these would temporarily have no charge (as in, free for the time being.)
This should give everyone a few different choices given the scenario, so can they proceed with a purchase as necessary elsewhere, or to at the very least have some VPS or server to hold any backup or temporary home.
Let me know if you guys have any questions or suggestions but please keep in mind they'll have to be specific and direct.
@VirMach said:
I'm working with our developer today to pro-rate refund all dedicated servers where possible, for any remaining duration. We don't want to give people the wrong idea of what's going on and want to reduce any panic, at least on any end we can control. Then, first come first serve basis on emergency tickets created requesting a replacement server, we'll begin provisioning those today, as well as backup servers for anyone who requires it. Both of these would temporarily have no charge (as in, free for the time being.)
This should give everyone a few different choices given the scenario, so can they proceed with a purchase as necessary elsewhere, or to at the very least have some VPS or server to hold any backup or temporary home.
Let me know if you guys have any questions or suggestions but please keep in mind they'll have to be specific and direct.
is there any chance i can get my data back?
Are you on a dedicated server that's currently offline?
@VirMach said:
I'm working with our developer today to pro-rate refund all dedicated servers where possible, for any remaining duration. We don't want to give people the wrong idea of what's going on and want to reduce any panic, at least on any end we can control. Then, first come first serve basis on emergency tickets created requesting a replacement server, we'll begin provisioning those today, as well as backup servers for anyone who requires it. Both of these would temporarily have no charge (as in, free for the time being.)
Is it safe to assume any existing dedicated servers (I had two) are gone for good? No chance at getting any remaining data off them?
@VirMach said:
I'm working with our developer today to pro-rate refund all dedicated servers where possible, for any remaining duration. We don't want to give people the wrong idea of what's going on and want to reduce any panic, at least on any end we can control. Then, first come first serve basis on emergency tickets created requesting a replacement server, we'll begin provisioning those today, as well as backup servers for anyone who requires it. Both of these would temporarily have no charge (as in, free for the time being.)
This should give everyone a few different choices given the scenario, so can they proceed with a purchase as necessary elsewhere, or to at the very least have some VPS or server to hold any backup or temporary home.
Let me know if you guys have any questions or suggestions but please keep in mind they'll have to be specific and direct.
Relaying these questions on behalf of a friend:
Is it at all likely that any existing dedicated machines will come back up for an hour or so? We've got most of the data kept on off-site backups, but one VM could benefit from a rescue mission.
For servers that were bought as part of the 'migration special' on OGF - will the old pricing be honoured once that is ready if the old machines are cancelled?
Thanks for your time. You must be having an absolute nightmare at the moment - thoughts & prayers are with you.
@VirMach said:
I'm working with our developer today to pro-rate refund all dedicated servers where possible, for any remaining duration. We don't want to give people the wrong idea of what's going on and want to reduce any panic, at least on any end we can control. Then, first come first serve basis on emergency tickets created requesting a replacement server, we'll begin provisioning those today, as well as backup servers for anyone who requires it. Both of these would temporarily have no charge (as in, free for the time being.)
Is it safe to assume any existing dedicated servers (I had two) are gone for good? No chance at getting any remaining data off them?
@acidpuke said:
I have 2 dedicated servers Chicago and Los Angeles both down .. so is it safe to assume I'm F'ed
In our next meeting with the legal team, I'll attempt to get back some information for you.
We will of course pursue this to the best of our ability. I would not necessarily know what legal ramifications are available, if any, that would assist customers in at least getting some access to their data. If anyone is interested in getting more directly involved in these discussions let me know and I'll try to pass along your message and see if that's a possibility.
@Yorkie said: Relaying these questions on behalf of a friend:
Is it at all likely that any existing dedicated machines will come back up for an hour or so? We've got most of the data kept on off-site backups, but one VM could benefit from a rescue mission.
For servers that were bought as part of the 'migration special' on OGF - will the old pricing be honoured once that is ready if the old machines are cancelled?
We will of course continue to honor everything that's within our control. We are still planning on continuing as expected with the migration specials. If your friend would like to take that route, then just ensure they request a replacement server. If the old machine is cancelled, as a direct result of today's events, I will consider allowing them to continue the migration end of the special, although it's not currently set up to be processed in that manner.
The only changes we're offering are meant to be to the customer's benefit. We're not for example, going to cancel a special offer for a dedicated server at a good price if the customer wants to continue it on a new server with our new datacenter partner.
Thanks for your time. You must be having an absolute nightmare at the moment - thoughts & prayers.
Ryzen virtual servers are not going to be affected by this matter. SolusVM and WHMCS (VPS control, billing databases) are safe as well. We're wrapping up any leftovers today and looking at disaster recovery backups if necessary. Hopefully we have done a pretty good job with Ryzen migrations so far so this should be relatively rare.
@acidpuke said:
I just opened 2 network down support tickets one for each server, is that good enough? or should I open one specifically asking for a new servers?
Same question here, just opened a ticket for my 3 servers related to them being down, but wondering if I need to specify that I want new servers. This sucks, but I expected CC to pull something shady so I luckily have backups
@VirMach said:
I'm working with our developer today to pro-rate refund all dedicated servers where possible, for any remaining duration.
Ohh lordy... I have the "FreeMach" dedi sponsored by you guys. It's "prepaid" through 2030 and renews at like $500/yr. All I'm trying to say is please make sure your dev is aware so you don't send me $4k. Lol
Hoping for the best! Fingers crossed for a replacement, but I completely understand if we have to axe the project given the circumstances. Regardless of what happens, I (and all FreeMach users) sincerely appreciate the 4.5+ years of sponsorship and stellar service!
@VirMach said:
I'm working with our developer today to pro-rate refund all dedicated servers where possible, for any remaining duration.
Ohh lordy... I have the "FreeMach" dedi sponsored by you guys. It's "prepaid" through 2030 and renews at like $500/yr. All I'm trying to say is please make sure your dev is aware so you don't send me $4k. Lol
Hoping for the best! Fingers crossed for a replacement, but I completely understand if we have to axe the project given the circumstances. Regardless of what happens, I (and all FreeMach users) sincerely appreciate the 4.5+ years of sponsorship and stellar service!
Take your $4000 and order a replacement at $250/year?
Then you can continue the project until 2038.
No need to worry beyond 2038 because Unix timestamp will overflow, which ends the world.
We accept Karma donations for the last flan. 🍮 affbrr
@acidpuke said:
I just opened 2 network down support tickets one for each server, is that good enough? or should I open one specifically asking for a new servers?
Same question here, just opened a ticket for my 3 servers related to them being down, but wondering if I need to specify that I want new servers. This sucks, but I expected CC to pull something shady so I luckily have backups
We do have some tickets from before we sent out the emergency notice, and some people who aren't as caught up to the series of events that occurred so it's best to specifically ask for new server details and mention your requirements if any.
@VirMach said:
I'm working with our developer today to pro-rate refund all dedicated servers where possible, for any remaining duration.
Ohh lordy... I have the "FreeMach" dedi sponsored by you guys. It's "prepaid" through 2030 and renews at like $500/yr. All I'm trying to say is please make sure your dev is aware so you don't send me $4k. Lol
Hoping for the best! Fingers crossed for a replacement, but I completely understand if we have to axe the project given the circumstances. Regardless of what happens, I (and all FreeMach users) sincerely appreciate the 4.5+ years of sponsorship and stellar service!
Honestly I'm just happy that people are empathetic. That in itself makes today a lot easier on me, as it would be insanely difficult to mentally get through the day if we were also being attacked and unable to defend ourselves. Of course there will be some of that but knowing the people who stick around the most understand what's happening is enough for me.
We're not going to use this situation to get out of any previous obligations, and that includes "FreeMach." You'll have a replacement. It'll be the same price, so free if it's free.
I haven't figured out how the credits will be done. I want to do it to payment method but that will likely be harder to code out. We'll probably do store credit then allow people to get it back to their original method in this instance if they contact us and it's possible. So I'm not sure if you were going to end up getting a boatload of credit in your unique situation but thanks for making me aware of that being a potential edge case, I'll pass it on to the developer.
I wonder if Virmach's neighbours can hear again. Because I know if after all the BS already happened / happening / still being sorted, this happened, there would lots of angry yelling.
I knew this whole thing would be a rough period from my experiences in the business, so many changes in so little time and so many issues to happen along the way even without stupid companies screwing you over. So I'm here for the long haul and understand the work you are doing to try and fix everything, which in the end will be a better setup.
Honestly I'm just happy that people are empathetic. That in itself makes today a lot easier on me, as it would be insanely difficult to mentally get through the day if we were also being attacked and unable to defend ourselves. Of course there will be some of that but knowing the people who stick around the most understand what's happening is enough for me.
I have two in Chicago that are offline. Short to medium term I know I can do without one and I'm going to try for both. It's not much but at least you know that's one more person who isn't going to lose their shit. We'll figure it out.
We're not going to use this situation to get out of any previous obligations, and that includes "FreeMach." You'll have a replacement. It'll be the same price, so free if it's free.
I haven't figured out how the credits will be done. I want to do it to payment method but that will likely be harder to code out. We'll probably do store credit then allow people to get it back to their original method in this instance if they contact us and it's possible. So I'm not sure if you were going to end up getting a boatload of credit in your unique situation but thanks for making me aware of that being a potential edge case, I'll pass it on to the developer.
100% for laughter value, I literally renewed Monday. I'm fine with store credits for now since I'll just be replacing existing servers eventually anyway.
Question: Are the emergency dedi's already racked and if so are they at multiple locations or all in LA for example? Tangential to that, how will location of replacements be handled?
Bloody hell, I called it right.
Best of luck sorting this mess out and somehow try to get some rest.
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural. It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
@corbpie said:
Damn that’s probably my longest serving service gone, had it since 2016
Bought my dedi on 08/15/2016, so same. I still have a few VPS with other providers that are older but definitely the longest dedicated server of mine by far.
@skorous said:
Question: Are the emergency dedi's already racked and if so are they at multiple locations or all in LA for example? Tangential to that, how will location of replacements be handled?
First batch is in NYC. Second batch SJC. They're all for the most part racked, even the batches we haven't received yet, but they just need to be set up. I had this written out, maybe I forgot to post it? Around 15% are immediately available, just waiting on us to sort it out and send details. Around 15% are being worked on immediately and most likely delivered to us today. Around 30% more have been prioritized to begin delivering tomorrow. These are all figures I'm vaguely quoting from our new provider.
The specifications will vary. IPv4 will be less for some people. Some will be 16GB, others 32GB (similar to what we provide.) They'll also be HDD and SSD setups. We were supposed to send in SSDs, this got delayed on my end (to try to make more of them SSD.)
@VirMach said:
We do have some tickets from before we sent out the emergency notice
Where/how are these being sent? I haven't received anything via email and I see nothing in the billing panel related to this.
Had I not checked the forums, I would have no idea of the severity of the outage.
I had to send these via WHMCS, which we usually don't do. It's listed as 100% sent on one screen, but I realized on another screen it's listed as "remaining" with "no failed emails." None of it makes sense, it's possible these actually didn't send which would be horrible as the emails were very important and I did my best to send them quickly.
Checking further to see what happened in the logs.
[Emergency] Potential Dedicated Server Service Disruption
Dear VirMach Customers,
We are sending this emergency notification to let you know of a potential service disruption in regards to your dedicated serve0(s.) We will discuss additional details when possible, but at this time, it is important to ensure your backups are up to date.
Please ensure you maintain a copy of all important files in a secondary location, outside of your dedicated server. This is always best practice, but there is currently an elevated risk of service disruption.
If there is an unexpected interruption, please contact us in the "priority" department and mention in the subject that the matter is regarding your dedicated server with us. We will continue to provide SLA credit per our terms of service. We have also prepared replacement servers which we will be distributing on an emergency basis. If you would like to more immediately switch your dedicated server to a server located with our new datacenter partners, please create a ticket in the "priority" department called "Dedicated Server Migration" and provide [1] your existing dedicated server credentials, [2] your minimum requirements for your new dedicated server, [3] your desired location(s) for your new dedicated server, if available.
Thank you.
Oh man if this actually didn't send after listing as sent ... that really sucks. Because it would have been helpful.
Shows up as such on WHMCS, but checking email logs definitely doesn't show anything sent around that time. This is quite upsetting.
Sending report shows as:
Client Failure Reason
There are no failed emails
But also shows as:
184 Remaining
Looks like we're going to have to just create individual tickets at this point, I don't know if wee can trust the email tool to re-try at this point. I know all other emails are sending though so this definitely has to do with the specific bulk email tool.
Forced the cron through again, same status. Forcing other crons, nothing. I had to specifically force the --ProcessEmailQueue cron after forcing the --EmailCampaigns cron. They have a cron that updates the status without actually doing it, and then another cron that processes them.
It might still not send but now it's listed as 25 Sent / 159 Remaining (and on other screen still says 100% done.)
It's actually just stuck at 25 now. I might have to force it 6 more times?
Edit -- 30 sent, 34 sent... yeah I actually have to force it once per 25 emails because it apparently does max batches of 25.
Edit -- Third batch being forced now. At this point it's completely my fault for not expecting this.
Edit -- Fourth batch being forced.
Edit -- Fifth batch being forced.
Edit -- Sixth batch being forced.
Edit - Seventh batch being forced.
Edit - Eight batch being forced.
Edit -- All showing as sent on WHMCS but it's basically showing the same thing 184/184 [100%] so I'm going through mail logs and recounting them.
Man, it never seems to stop for you! Good luck with it all, and remember to rest and sleep in the midst of this. At the end of the day, they're only servers, your health is more important.
@VirMach said: We're not going to use this situation to get out of any previous obligations, and that includes "FreeMach." You'll have a replacement. It'll be the same price, so free if it's free.
@VirMach said: Edit -- Third batch being forced now. At this point it's completely my fault for not expecting this.
That's actually so crushing knowing those didn't go out 5-6 hours ago.
Ouch, failed by yet another thing not really in your control. Did you like smash some mirrors or walk under a truck full of ladders or something lately?
I can only hope that karma pays you back for all of this hell you've been goign through with many years of things working well. And maybe like a bathtub full of whiskey or something after this..
@Jab said:
Jesus this WHMCS, I am shocked that they don't have a competition...
I wrote out this insane rant about them the other day, and then I realized how insane it sounded and deleted it (I think.) It was essentially a 10 paragraph huge clump of text. This was after a period of not sleeping for several days. But the shortest way I can re-iterate the rant is that it is the single piece of software I despise the most, not because of the DRM or bad coding, but simply based on how ridiculous the logic is behind each action.
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural. It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
@VirMach said:
We will of course continue to honor everything that's within our control. We are still planning on continuing as expected with the migration specials. If your friend would like to take that route, then just ensure they request a replacement server. If the old machine is cancelled, as a direct result of today's events, I will consider allowing them to continue the migration end of the special, although it's not currently set up to be processed in that manner.
The only changes we're offering are meant to be to the customer's benefit. We're not for example, going to cancel a special offer for a dedicated server at a good price if the customer wants to continue it on a new server with our new datacenter partner.
First, wanted to echo the sentiment of understanding, I know you've got to be WAY more stressed about the situation than I am. So, I'm happy to wait for a replacement server in another datacenter, and don't need access to the old data, I've got recent enough backups. I'd rather not end up with a refund / canceled server from all of this, should I just create a priority ticket and explain the same there or how are you wanting people to handle this? I'd like to make it the least work on you / your team, since I know you've got plenty of it already.
all the best Virmach. personally started off with very low expectations on the quality of your VPSes especially with heavy MJJ userbase and previous association with cuntcroissant.
seeing your struggle over a year (or more) and persistence on fixing all these endless problems at such a huge scale with all these naysayers and backstabbers, i am now rooting for you to win.
not exactly a happy customer right now, but i want you to win and win big so i can be a happy customer on dirt cheap ryzen VPSes.
P.S. do plan on longer term support levels when this over but who knows when it will be
@VirMach said:
I'm working with our developer today to pro-rate refund all dedicated servers where possible, for any remaining duration. We don't want to give people the wrong idea of what's going on and want to reduce any panic, at least on any end we can control. Then, first come first serve basis on emergency tickets created requesting a replacement server, we'll begin provisioning those today, as well as backup servers for anyone who requires it. Both of these would temporarily have no charge (as in, free for the time being.)
This should give everyone a few different choices given the scenario, so can they proceed with a purchase as necessary elsewhere, or to at the very least have some VPS or server to hold any backup or temporary home.
Let me know if you guys have any questions or suggestions but please keep in mind they'll have to be specific and direct.
is there any chance i can get my data back?
Are you on a dedicated server that's currently offline?
Successes usually come to those who are too busy to be looking for it. Ordinary people are satisfied with their short term achievements, but an extraordinary person continues working hard and accomplishes goals one after another. You are one of the extraordinary people who is worthy of their success. Best wishes for VirMach's continued success in the future.
some may claim milllions lost, others will say billions gained thanks to patience.
on a lighter note, we will celebrate with a thread full of bus images once this is over. If @FrankZ consents and participates.
@VirMach my buffalo server since 2018 went down today had been very satisfied since so I certainly hope to get a replacement.
Its a bummer but oh well nothing mission critical since I have my backups.
@VirMach said:
Hivelocity's the one that acquired Incero right? I don't know why they didn't come to mind (HV) when I was thinking about Dallas. We can't do it right now but there's absolutely no way we can stay with Flexential at this state, WTF.
You've got dozens of options in Dallas. The "go-to" is Carrier-1 but there are many others, DallasColo is another example, or Stack.
Where you are now, the network reliability is crap. The whole connection goes down every week or two and takes out VirMach, DediPath, Nexril, etc. I literally do have better network reliability at my house.
@Daevien said:
Yeah, know anyone in Dallas with a rack in their house? They'd be more useful & have better uptime probably.
This is actually true.
$ uptime
01:15:06 up 229 days, 13:53, 1 user, load average: 0.00, 0.00, 0.00
$ ping -c 5 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=117 time=4.87 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=117 time=2.95 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=117 time=4.92 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=117 time=3.05 ms
64 bytes from 8.8.8.8: icmp_seq=5 ttl=117 time=5.84 ms
--- 8.8.8.8 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4012ms
rtt min/avg/max/mdev = 2.957/4.330/5.848/1.139 ms
Comments
I'm thinking some technical issues going on instead. My dedi came back online about 10 minutes ago, so I'm back up and running. Virmach site still appears to be down.
yabs daddy
If they spotted it now after 1-1,5months is surely sounds like CC
I'm still waiting for IPv6 from them, maybe they hear about it next year or 2030
Yeah, I was being (trying to be) facetious, guys.
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
I wouldn't be able to comment on this.
We have a contingency plan. It's not perfect. Nor were we expecting it to go this far or be done in this way. I've been working on it the entire time, and have emergency replacement servers to some degree. I'm sending out some vague communication that's at the very least necessary to immediately inform customers of the risks at this point.
Our site is indirectly related. Luckily it's only the few static pages that happened to be on one of the servers facing issues. This isn't important right now, what I previously said is so we're going to be focusing on that.
I'm working with our developer today to pro-rate refund all dedicated servers where possible, for any remaining duration. We don't want to give people the wrong idea of what's going on and want to reduce any panic, at least on any end we can control. Then, first come first serve basis on emergency tickets created requesting a replacement server, we'll begin provisioning those today, as well as backup servers for anyone who requires it. Both of these would temporarily have no charge (as in, free for the time being.)
This should give everyone a few different choices given the scenario, so can they proceed with a purchase as necessary elsewhere, or to at the very least have some VPS or server to hold any backup or temporary home.
Let me know if you guys have any questions or suggestions but please keep in mind they'll have to be specific and direct.
We accept Karma donations for the last flan. 🍮 affbrr
is there any chance i can get my data back?
Are you on a dedicated server that's currently offline?
Is it safe to assume any existing dedicated servers (I had two) are gone for good? No chance at getting any remaining data off them?
I have 2 dedicated servers Chicago and Los Angeles both down .. so is it safe to assume I'm Fu*k'ed
Relaying these questions on behalf of a friend:
Is it at all likely that any existing dedicated machines will come back up for an hour or so? We've got most of the data kept on off-site backups, but one VM could benefit from a rescue mission.
For servers that were bought as part of the 'migration special' on OGF - will the old pricing be honoured once that is ready if the old machines are cancelled?
Thanks for your time. You must be having an absolute nightmare at the moment - thoughts & prayers are with you.
In our next meeting with the legal team, I'll attempt to get back some information for you.
We will of course pursue this to the best of our ability. I would not necessarily know what legal ramifications are available, if any, that would assist customers in at least getting some access to their data. If anyone is interested in getting more directly involved in these discussions let me know and I'll try to pass along your message and see if that's a possibility.
We will of course continue to honor everything that's within our control. We are still planning on continuing as expected with the migration specials. If your friend would like to take that route, then just ensure they request a replacement server. If the old machine is cancelled, as a direct result of today's events, I will consider allowing them to continue the migration end of the special, although it's not currently set up to be processed in that manner.
The only changes we're offering are meant to be to the customer's benefit. We're not for example, going to cancel a special offer for a dedicated server at a good price if the customer wants to continue it on a new server with our new datacenter partner.
Thanks, means a lot.
Just throwing this out there as well:
Ryzen virtual servers are not going to be affected by this matter. SolusVM and WHMCS (VPS control, billing databases) are safe as well. We're wrapping up any leftovers today and looking at disaster recovery backups if necessary. Hopefully we have done a pretty good job with Ryzen migrations so far so this should be relatively rare.
I just opened 2 network down support tickets one for each server, is that good enough? or should I open one specifically asking for new servers?
All my data is backed up so I don't need anything from the old servers
Ticket #319037
Ticket #140230
Same question here, just opened a ticket for my 3 servers related to them being down, but wondering if I need to specify that I want new servers. This sucks, but I expected CC to pull something shady so I luckily have backups
Ohh lordy... I have the "FreeMach" dedi sponsored by you guys. It's "prepaid" through 2030 and renews at like $500/yr. All I'm trying to say is please make sure your dev is aware so you don't send me $4k. Lol
Hoping for the best! Fingers crossed for a replacement, but I completely understand if we have to axe the project given the circumstances. Regardless of what happens, I (and all FreeMach users) sincerely appreciate the 4.5+ years of sponsorship and stellar service!
yabs daddy
Take your $4000 and order a replacement at $250/year?
Then you can continue the project until 2038.
No need to worry beyond 2038 because Unix timestamp will overflow, which ends the world.
We accept Karma donations for the last flan. 🍮 affbrr
We do have some tickets from before we sent out the emergency notice, and some people who aren't as caught up to the series of events that occurred so it's best to specifically ask for new server details and mention your requirements if any.
Honestly I'm just happy that people are empathetic. That in itself makes today a lot easier on me, as it would be insanely difficult to mentally get through the day if we were also being attacked and unable to defend ourselves. Of course there will be some of that but knowing the people who stick around the most understand what's happening is enough for me.
We're not going to use this situation to get out of any previous obligations, and that includes "FreeMach." You'll have a replacement. It'll be the same price, so free if it's free.
I haven't figured out how the credits will be done. I want to do it to payment method but that will likely be harder to code out. We'll probably do store credit then allow people to get it back to their original method in this instance if they contact us and it's possible. So I'm not sure if you were going to end up getting a boatload of credit in your unique situation but thanks for making me aware of that being a potential edge case, I'll pass it on to the developer.
I wonder if Virmach's neighbours can hear again. Because I know if after all the BS already happened / happening / still being sorted, this happened, there would lots of angry yelling.
I knew this whole thing would be a rough period from my experiences in the business, so many changes in so little time and so many issues to happen along the way even without stupid companies screwing you over. So I'm here for the long haul and understand the work you are doing to try and fix everything, which in the end will be a better setup.
I have two in Chicago that are offline. Short to medium term I know I can do without one and I'm going to try for both. It's not much but at least you know that's one more person who isn't going to lose their shit. We'll figure it out.
100% for laughter value, I literally renewed Monday.
I'm fine with store credits for now since I'll just be replacing existing servers eventually anyway.
Question: Are the emergency dedi's already racked and if so are they at multiple locations or all in LA for example? Tangential to that, how will location of replacements be handled?
Bloody hell, I called it right.
Best of luck sorting this mess out and somehow try to get some rest.
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
Damn that’s probably my longest serving service gone, had it since 2016
Where/how are these being sent? I haven't received anything via email and I see nothing in the billing panel related to this.
Had I not checked the forums, I would have no idea of the severity of the outage.
Bought my dedi on 08/15/2016, so same. I still have a few VPS with other providers that are older but definitely the longest dedicated server of mine by far.
First batch is in NYC. Second batch SJC. They're all for the most part racked, even the batches we haven't received yet, but they just need to be set up. I had this written out, maybe I forgot to post it? Around 15% are immediately available, just waiting on us to sort it out and send details. Around 15% are being worked on immediately and most likely delivered to us today. Around 30% more have been prioritized to begin delivering tomorrow. These are all figures I'm vaguely quoting from our new provider.
The specifications will vary. IPv4 will be less for some people. Some will be 16GB, others 32GB (similar to what we provide.) They'll also be HDD and SSD setups. We were supposed to send in SSDs, this got delayed on my end (to try to make more of them SSD.)
I had to send these via WHMCS, which we usually don't do. It's listed as 100% sent on one screen, but I realized on another screen it's listed as "remaining" with "no failed emails." None of it makes sense, it's possible these actually didn't send which would be horrible as the emails were very important and I did my best to send them quickly.
Checking further to see what happened in the logs.
I'll provide a copy here, typo included.
Oh man if this actually didn't send after listing as sent ... that really sucks. Because it would have been helpful.
Shows up as such on WHMCS, but checking email logs definitely doesn't show anything sent around that time. This is quite upsetting.
Sending report shows as:
But also shows as:
Looks like we're going to have to just create individual tickets at this point, I don't know if wee can trust the email tool to re-try at this point. I know all other emails are sending though so this definitely has to do with the specific bulk email tool.
I never received that notification
Forced the cron through again, same status. Forcing other crons, nothing. I had to specifically force the --ProcessEmailQueue cron after forcing the --EmailCampaigns cron. They have a cron that updates the status without actually doing it, and then another cron that processes them.
It might still not send but now it's listed as 25 Sent / 159 Remaining (and on other screen still says 100% done.)
It's actually just stuck at 25 now. I might have to force it 6 more times?
Edit -- 30 sent, 34 sent... yeah I actually have to force it once per 25 emails because it apparently does max batches of 25.
Edit -- Third batch being forced now. At this point it's completely my fault for not expecting this.
Edit -- Fourth batch being forced.
Edit -- Fifth batch being forced.
Edit -- Sixth batch being forced.
Edit - Seventh batch being forced.
Edit - Eight batch being forced.
Edit -- All showing as sent on WHMCS but it's basically showing the same thing 184/184 [100%] so I'm going through mail logs and recounting them.
Edit -- Logs for the most part look correct.
Man, it never seems to stop for you! Good luck with it all, and remember to rest and sleep in the midst of this. At the end of the day, they're only servers, your health is more important.
Just got it, boss
yabs daddy
That's actually so crushing knowing those didn't go out 5-6 hours ago.
I just got notification also .. thank you
Priority Ticket Created #998050
Jesus this WHMCS, I am shocked that they don't have a competition...
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
Ouch, failed by yet another thing not really in your control. Did you like smash some mirrors or walk under a truck full of ladders or something lately?
I can only hope that karma pays you back for all of this hell you've been goign through with many years of things working well. And maybe like a bathtub full of whiskey or something after this..
CPANEL and WHMCS, two things I definitely don't miss about the hosting world lol
I wrote out this insane rant about them the other day, and then I realized how insane it sounded and deleted it (I think.) It was essentially a 10 paragraph huge clump of text. This was after a period of not sleeping for several days. But the shortest way I can re-iterate the rant is that it is the single piece of software I despise the most, not because of the DRM or bad coding, but simply based on how ridiculous the logic is behind each action.
When did WHMCS get logic? It was just random shit stacked higher and higher from what I remember lol
@Virmach GET SLEEP!
(Silly Sod)
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
Aha, email came through.
First, wanted to echo the sentiment of understanding, I know you've got to be WAY more stressed about the situation than I am. So, I'm happy to wait for a replacement server in another datacenter, and don't need access to the old data, I've got recent enough backups. I'd rather not end up with a refund / canceled server from all of this, should I just create a priority ticket and explain the same there or how are you wanting people to handle this? I'd like to make it the least work on you / your team, since I know you've got plenty of it already.
all the best Virmach. personally started off with very low expectations on the quality of your VPSes especially with heavy MJJ userbase and previous association with cuntcroissant.
seeing your struggle over a year (or more) and persistence on fixing all these endless problems at such a huge scale with all these naysayers and backstabbers, i am now rooting for you to win.
not exactly a happy customer right now, but i want you to win and win big so i can be a happy customer on dirt cheap ryzen VPSes.
P.S. do plan on longer term support levels when this over but who knows when it will be
I bench YABS 24/7/365 unless it's a leap year.
yes,in DAL and LA
Successes usually come to those who are too busy to be looking for it. Ordinary people are satisfied with their short term achievements, but an extraordinary person continues working hard and accomplishes goals one after another. You are one of the extraordinary people who is worthy of their success. Best wishes for VirMach's continued success in the future.
@VirMach,
Keep chugging-
some may claim milllions lost, others will say billions gained thanks to patience.
on a lighter note, we will celebrate with a thread full of bus images once this is over. If @FrankZ consents and participates.
blog archives
@vyas - I'm in and I am sure many others will join in the fun.
@VirMach my buffalo server since 2018 went down today had been very satisfied since so I certainly hope to get a replacement.
Its a bummer but oh well nothing mission critical since I have my backups.
All the best @VirMach
Was wondering if we should have a front page post separated from this thread so no "DEDPOOL" threads spring up?
HetrixTools - Highly recommended uptime monitor [aff]
You've got dozens of options in Dallas. The "go-to" is Carrier-1 but there are many others, DallasColo is another example, or Stack.
Where you are now, the network reliability is crap. The whole connection goes down every week or two and takes out VirMach, DediPath, Nexril, etc. I literally do have better network reliability at my house.
This is actually true.
I have a question, @VirMach, do you even sleep? You work damn hard from what I've gathered so far in this thread, jesus.
I can't even begin to fathom the volume of tickets that you've probably gotten so far.
I am a representative of Advin Servers
Have you tried turning it off and on again?