@acidpuke said:
I just opened 2 network down support tickets one for each server, is that good enough? or should I open one specifically asking for a new servers?
Same question here, just opened a ticket for my 3 servers related to them being down, but wondering if I need to specify that I want new servers. This sucks, but I expected CC to pull something shady so I luckily have backups
@VirMach said:
I'm working with our developer today to pro-rate refund all dedicated servers where possible, for any remaining duration.
Ohh lordy... I have the "FreeMach" dedi sponsored by you guys. It's "prepaid" through 2030 and renews at like $500/yr. All I'm trying to say is please make sure your dev is aware so you don't send me $4k. Lol
Hoping for the best! Fingers crossed for a replacement, but I completely understand if we have to axe the project given the circumstances. Regardless of what happens, I (and all FreeMach users) sincerely appreciate the 4.5+ years of sponsorship and stellar service!
@VirMach said:
I'm working with our developer today to pro-rate refund all dedicated servers where possible, for any remaining duration.
Ohh lordy... I have the "FreeMach" dedi sponsored by you guys. It's "prepaid" through 2030 and renews at like $500/yr. All I'm trying to say is please make sure your dev is aware so you don't send me $4k. Lol
Hoping for the best! Fingers crossed for a replacement, but I completely understand if we have to axe the project given the circumstances. Regardless of what happens, I (and all FreeMach users) sincerely appreciate the 4.5+ years of sponsorship and stellar service!
Take your $4000 and order a replacement at $250/year?
Then you can continue the project until 2038.
No need to worry beyond 2038 because Unix timestamp will overflow, which ends the world.
@acidpuke said:
I just opened 2 network down support tickets one for each server, is that good enough? or should I open one specifically asking for a new servers?
Same question here, just opened a ticket for my 3 servers related to them being down, but wondering if I need to specify that I want new servers. This sucks, but I expected CC to pull something shady so I luckily have backups
We do have some tickets from before we sent out the emergency notice, and some people who aren't as caught up to the series of events that occurred so it's best to specifically ask for new server details and mention your requirements if any.
@VirMach said:
I'm working with our developer today to pro-rate refund all dedicated servers where possible, for any remaining duration.
Ohh lordy... I have the "FreeMach" dedi sponsored by you guys. It's "prepaid" through 2030 and renews at like $500/yr. All I'm trying to say is please make sure your dev is aware so you don't send me $4k. Lol
Hoping for the best! Fingers crossed for a replacement, but I completely understand if we have to axe the project given the circumstances. Regardless of what happens, I (and all FreeMach users) sincerely appreciate the 4.5+ years of sponsorship and stellar service!
Honestly I'm just happy that people are empathetic. That in itself makes today a lot easier on me, as it would be insanely difficult to mentally get through the day if we were also being attacked and unable to defend ourselves. Of course there will be some of that but knowing the people who stick around the most understand what's happening is enough for me.
We're not going to use this situation to get out of any previous obligations, and that includes "FreeMach." You'll have a replacement. It'll be the same price, so free if it's free.
I haven't figured out how the credits will be done. I want to do it to payment method but that will likely be harder to code out. We'll probably do store credit then allow people to get it back to their original method in this instance if they contact us and it's possible. So I'm not sure if you were going to end up getting a boatload of credit in your unique situation but thanks for making me aware of that being a potential edge case, I'll pass it on to the developer.
I wonder if Virmach's neighbours can hear again. Because I know if after all the BS already happened / happening / still being sorted, this happened, there would lots of angry yelling.
I knew this whole thing would be a rough period from my experiences in the business, so many changes in so little time and so many issues to happen along the way even without stupid companies screwing you over. So I'm here for the long haul and understand the work you are doing to try and fix everything, which in the end will be a better setup.
Honestly I'm just happy that people are empathetic. That in itself makes today a lot easier on me, as it would be insanely difficult to mentally get through the day if we were also being attacked and unable to defend ourselves. Of course there will be some of that but knowing the people who stick around the most understand what's happening is enough for me.
I have two in Chicago that are offline. Short to medium term I know I can do without one and I'm going to try for both. It's not much but at least you know that's one more person who isn't going to lose their shit. We'll figure it out.
We're not going to use this situation to get out of any previous obligations, and that includes "FreeMach." You'll have a replacement. It'll be the same price, so free if it's free.
I haven't figured out how the credits will be done. I want to do it to payment method but that will likely be harder to code out. We'll probably do store credit then allow people to get it back to their original method in this instance if they contact us and it's possible. So I'm not sure if you were going to end up getting a boatload of credit in your unique situation but thanks for making me aware of that being a potential edge case, I'll pass it on to the developer.
100% for laughter value, I literally renewed Monday. I'm fine with store credits for now since I'll just be replacing existing servers eventually anyway.
Question: Are the emergency dedi's already racked and if so are they at multiple locations or all in LA for example? Tangential to that, how will location of replacements be handled?
@corbpie said:
Damn that’s probably my longest serving service gone, had it since 2016
Bought my dedi on 08/15/2016, so same. I still have a few VPS with other providers that are older but definitely the longest dedicated server of mine by far.
@skorous said:
Question: Are the emergency dedi's already racked and if so are they at multiple locations or all in LA for example? Tangential to that, how will location of replacements be handled?
First batch is in NYC. Second batch SJC. They're all for the most part racked, even the batches we haven't received yet, but they just need to be set up. I had this written out, maybe I forgot to post it? Around 15% are immediately available, just waiting on us to sort it out and send details. Around 15% are being worked on immediately and most likely delivered to us today. Around 30% more have been prioritized to begin delivering tomorrow. These are all figures I'm vaguely quoting from our new provider.
The specifications will vary. IPv4 will be less for some people. Some will be 16GB, others 32GB (similar to what we provide.) They'll also be HDD and SSD setups. We were supposed to send in SSDs, this got delayed on my end (to try to make more of them SSD.)
@VirMach said:
We do have some tickets from before we sent out the emergency notice
Where/how are these being sent? I haven't received anything via email and I see nothing in the billing panel related to this.
Had I not checked the forums, I would have no idea of the severity of the outage.
I had to send these via WHMCS, which we usually don't do. It's listed as 100% sent on one screen, but I realized on another screen it's listed as "remaining" with "no failed emails." None of it makes sense, it's possible these actually didn't send which would be horrible as the emails were very important and I did my best to send them quickly.
Checking further to see what happened in the logs.
[Emergency] Potential Dedicated Server Service Disruption
Dear VirMach Customers,
We are sending this emergency notification to let you know of a potential service disruption in regards to your dedicated serve0(s.) We will discuss additional details when possible, but at this time, it is important to ensure your backups are up to date.
Please ensure you maintain a copy of all important files in a secondary location, outside of your dedicated server. This is always best practice, but there is currently an elevated risk of service disruption.
If there is an unexpected interruption, please contact us in the "priority" department and mention in the subject that the matter is regarding your dedicated server with us. We will continue to provide SLA credit per our terms of service. We have also prepared replacement servers which we will be distributing on an emergency basis. If you would like to more immediately switch your dedicated server to a server located with our new datacenter partners, please create a ticket in the "priority" department called "Dedicated Server Migration" and provide [1] your existing dedicated server credentials, [2] your minimum requirements for your new dedicated server, [3] your desired location(s) for your new dedicated server, if available.
Thank you.
Oh man if this actually didn't send after listing as sent ... that really sucks. Because it would have been helpful.
Shows up as such on WHMCS, but checking email logs definitely doesn't show anything sent around that time. This is quite upsetting.
Sending report shows as:
Client Failure Reason
There are no failed emails
But also shows as:
184 Remaining
Looks like we're going to have to just create individual tickets at this point, I don't know if wee can trust the email tool to re-try at this point. I know all other emails are sending though so this definitely has to do with the specific bulk email tool.
Forced the cron through again, same status. Forcing other crons, nothing. I had to specifically force the --ProcessEmailQueue cron after forcing the --EmailCampaigns cron. They have a cron that updates the status without actually doing it, and then another cron that processes them.
It might still not send but now it's listed as 25 Sent / 159 Remaining (and on other screen still says 100% done.)
It's actually just stuck at 25 now. I might have to force it 6 more times?
Edit -- 30 sent, 34 sent... yeah I actually have to force it once per 25 emails because it apparently does max batches of 25.
Edit -- Third batch being forced now. At this point it's completely my fault for not expecting this.
Edit -- Fourth batch being forced.
Edit -- Fifth batch being forced.
Edit -- Sixth batch being forced.
Edit - Seventh batch being forced.
Edit - Eight batch being forced.
Edit -- All showing as sent on WHMCS but it's basically showing the same thing 184/184 [100%] so I'm going through mail logs and recounting them.
Man, it never seems to stop for you! Good luck with it all, and remember to rest and sleep in the midst of this. At the end of the day, they're only servers, your health is more important.
@VirMach said: We're not going to use this situation to get out of any previous obligations, and that includes "FreeMach." You'll have a replacement. It'll be the same price, so free if it's free.
@VirMach said: Edit -- Third batch being forced now. At this point it's completely my fault for not expecting this.
That's actually so crushing knowing those didn't go out 5-6 hours ago.
Ouch, failed by yet another thing not really in your control. Did you like smash some mirrors or walk under a truck full of ladders or something lately?
I can only hope that karma pays you back for all of this hell you've been goign through with many years of things working well. And maybe like a bathtub full of whiskey or something after this..
@Jab said:
Jesus this WHMCS, I am shocked that they don't have a competition...
I wrote out this insane rant about them the other day, and then I realized how insane it sounded and deleted it (I think.) It was essentially a 10 paragraph huge clump of text. This was after a period of not sleeping for several days. But the shortest way I can re-iterate the rant is that it is the single piece of software I despise the most, not because of the DRM or bad coding, but simply based on how ridiculous the logic is behind each action.
@VirMach said:
We will of course continue to honor everything that's within our control. We are still planning on continuing as expected with the migration specials. If your friend would like to take that route, then just ensure they request a replacement server. If the old machine is cancelled, as a direct result of today's events, I will consider allowing them to continue the migration end of the special, although it's not currently set up to be processed in that manner.
The only changes we're offering are meant to be to the customer's benefit. We're not for example, going to cancel a special offer for a dedicated server at a good price if the customer wants to continue it on a new server with our new datacenter partner.
First, wanted to echo the sentiment of understanding, I know you've got to be WAY more stressed about the situation than I am. So, I'm happy to wait for a replacement server in another datacenter, and don't need access to the old data, I've got recent enough backups. I'd rather not end up with a refund / canceled server from all of this, should I just create a priority ticket and explain the same there or how are you wanting people to handle this? I'd like to make it the least work on you / your team, since I know you've got plenty of it already.
all the best Virmach. personally started off with very low expectations on the quality of your VPSes especially with heavy MJJ userbase and previous association with cuntcroissant.
seeing your struggle over a year (or more) and persistence on fixing all these endless problems at such a huge scale with all these naysayers and backstabbers, i am now rooting for you to win.
not exactly a happy customer right now, but i want you to win and win big so i can be a happy customer on dirt cheap ryzen VPSes.
P.S. do plan on longer term support levels when this over but who knows when it will be
@VirMach said:
I'm working with our developer today to pro-rate refund all dedicated servers where possible, for any remaining duration. We don't want to give people the wrong idea of what's going on and want to reduce any panic, at least on any end we can control. Then, first come first serve basis on emergency tickets created requesting a replacement server, we'll begin provisioning those today, as well as backup servers for anyone who requires it. Both of these would temporarily have no charge (as in, free for the time being.)
This should give everyone a few different choices given the scenario, so can they proceed with a purchase as necessary elsewhere, or to at the very least have some VPS or server to hold any backup or temporary home.
Let me know if you guys have any questions or suggestions but please keep in mind they'll have to be specific and direct.
is there any chance i can get my data back?
Are you on a dedicated server that's currently offline?
Successes usually come to those who are too busy to be looking for it. Ordinary people are satisfied with their short term achievements, but an extraordinary person continues working hard and accomplishes goals one after another. You are one of the extraordinary people who is worthy of their success. Best wishes for VirMach's continued success in the future.
some may claim milllions lost, others will say billions gained thanks to patience.
on a lighter note, we will celebrate with a thread full of bus images once this is over. If @FrankZ consents and participates.
Comments
Same question here, just opened a ticket for my 3 servers related to them being down, but wondering if I need to specify that I want new servers. This sucks, but I expected CC to pull something shady so I luckily have backups
Ohh lordy... I have the "FreeMach" dedi sponsored by you guys. It's "prepaid" through 2030 and renews at like $500/yr. All I'm trying to say is please make sure your dev is aware so you don't send me $4k. Lol
Hoping for the best! Fingers crossed for a replacement, but I completely understand if we have to axe the project given the circumstances. Regardless of what happens, I (and all FreeMach users) sincerely appreciate the 4.5+ years of sponsorship and stellar service!
Humble janitor of LES
Proud papa of YABS
Take your $4000 and order a replacement at $250/year?
Then you can continue the project until 2038.
No need to worry beyond 2038 because Unix timestamp will overflow, which ends the world.
We do have some tickets from before we sent out the emergency notice, and some people who aren't as caught up to the series of events that occurred so it's best to specifically ask for new server details and mention your requirements if any.
Honestly I'm just happy that people are empathetic. That in itself makes today a lot easier on me, as it would be insanely difficult to mentally get through the day if we were also being attacked and unable to defend ourselves. Of course there will be some of that but knowing the people who stick around the most understand what's happening is enough for me.
We're not going to use this situation to get out of any previous obligations, and that includes "FreeMach." You'll have a replacement. It'll be the same price, so free if it's free.
I haven't figured out how the credits will be done. I want to do it to payment method but that will likely be harder to code out. We'll probably do store credit then allow people to get it back to their original method in this instance if they contact us and it's possible. So I'm not sure if you were going to end up getting a boatload of credit in your unique situation but thanks for making me aware of that being a potential edge case, I'll pass it on to the developer.
I wonder if Virmach's neighbours can hear again. Because I know if after all the BS already happened / happening / still being sorted, this happened, there would lots of angry yelling.
I knew this whole thing would be a rough period from my experiences in the business, so many changes in so little time and so many issues to happen along the way even without stupid companies screwing you over. So I'm here for the long haul and understand the work you are doing to try and fix everything, which in the end will be a better setup.
I have two in Chicago that are offline. Short to medium term I know I can do without one and I'm going to try for both. It's not much but at least you know that's one more person who isn't going to lose their shit. We'll figure it out.
100% for laughter value, I literally renewed Monday.
I'm fine with store credits for now since I'll just be replacing existing servers eventually anyway.
Question: Are the emergency dedi's already racked and if so are they at multiple locations or all in LA for example? Tangential to that, how will location of replacements be handled?
Bloody hell, I called it right.
Best of luck sorting this mess out and somehow try to get some rest.
lowendinfo.com had no interest.
Damn that’s probably my longest serving service gone, had it since 2016
Where/how are these being sent? I haven't received anything via email and I see nothing in the billing panel related to this.
Had I not checked the forums, I would have no idea of the severity of the outage.
Bought my dedi on 08/15/2016, so same. I still have a few VPS with other providers that are older but definitely the longest dedicated server of mine by far.
First batch is in NYC. Second batch SJC. They're all for the most part racked, even the batches we haven't received yet, but they just need to be set up. I had this written out, maybe I forgot to post it? Around 15% are immediately available, just waiting on us to sort it out and send details. Around 15% are being worked on immediately and most likely delivered to us today. Around 30% more have been prioritized to begin delivering tomorrow. These are all figures I'm vaguely quoting from our new provider.
The specifications will vary. IPv4 will be less for some people. Some will be 16GB, others 32GB (similar to what we provide.) They'll also be HDD and SSD setups. We were supposed to send in SSDs, this got delayed on my end (to try to make more of them SSD.)
I had to send these via WHMCS, which we usually don't do. It's listed as 100% sent on one screen, but I realized on another screen it's listed as "remaining" with "no failed emails." None of it makes sense, it's possible these actually didn't send which would be horrible as the emails were very important and I did my best to send them quickly.
Checking further to see what happened in the logs.
I'll provide a copy here, typo included.
Oh man if this actually didn't send after listing as sent ... that really sucks. Because it would have been helpful.
Shows up as such on WHMCS, but checking email logs definitely doesn't show anything sent around that time. This is quite upsetting.
Sending report shows as:
But also shows as:
Looks like we're going to have to just create individual tickets at this point, I don't know if wee can trust the email tool to re-try at this point. I know all other emails are sending though so this definitely has to do with the specific bulk email tool.
I never received that notification
Forced the cron through again, same status. Forcing other crons, nothing. I had to specifically force the --ProcessEmailQueue cron after forcing the --EmailCampaigns cron. They have a cron that updates the status without actually doing it, and then another cron that processes them.
It might still not send but now it's listed as 25 Sent / 159 Remaining (and on other screen still says 100% done.)
It's actually just stuck at 25 now. I might have to force it 6 more times?
Edit -- 30 sent, 34 sent... yeah I actually have to force it once per 25 emails because it apparently does max batches of 25.
Edit -- Third batch being forced now. At this point it's completely my fault for not expecting this.
Edit -- Fourth batch being forced.
Edit -- Fifth batch being forced.
Edit -- Sixth batch being forced.
Edit - Seventh batch being forced.
Edit - Eight batch being forced.
Edit -- All showing as sent on WHMCS but it's basically showing the same thing 184/184 [100%] so I'm going through mail logs and recounting them.
Edit -- Logs for the most part look correct.
Man, it never seems to stop for you! Good luck with it all, and remember to rest and sleep in the midst of this. At the end of the day, they're only servers, your health is more important.
Just got it, boss
Humble janitor of LES
Proud papa of YABS
That's actually so crushing knowing those didn't go out 5-6 hours ago.
I just got notification also .. thank you
Priority Ticket Created #998050
Jesus this WHMCS, I am shocked that they don't have a competition...
Ouch, failed by yet another thing not really in your control. Did you like smash some mirrors or walk under a truck full of ladders or something lately?
I can only hope that karma pays you back for all of this hell you've been goign through with many years of things working well. And maybe like a bathtub full of whiskey or something after this..
CPANEL and WHMCS, two things I definitely don't miss about the hosting world lol
I wrote out this insane rant about them the other day, and then I realized how insane it sounded and deleted it (I think.) It was essentially a 10 paragraph huge clump of text. This was after a period of not sleeping for several days. But the shortest way I can re-iterate the rant is that it is the single piece of software I despise the most, not because of the DRM or bad coding, but simply based on how ridiculous the logic is behind each action.
When did WHMCS get logic? It was just random shit stacked higher and higher from what I remember lol
@Virmach GET SLEEP!
(Silly Sod)
lowendinfo.com had no interest.
Aha, email came through.
First, wanted to echo the sentiment of understanding, I know you've got to be WAY more stressed about the situation than I am. So, I'm happy to wait for a replacement server in another datacenter, and don't need access to the old data, I've got recent enough backups. I'd rather not end up with a refund / canceled server from all of this, should I just create a priority ticket and explain the same there or how are you wanting people to handle this? I'd like to make it the least work on you / your team, since I know you've got plenty of it already.
all the best Virmach. personally started off with very low expectations on the quality of your VPSes especially with heavy MJJ userbase and previous association with cuntcroissant.
seeing your struggle over a year (or more) and persistence on fixing all these endless problems at such a huge scale with all these naysayers and backstabbers, i am now rooting for you to win.
not exactly a happy customer right now, but i want you to win and win big so i can be a happy customer on dirt cheap ryzen VPSes.
P.S. do plan on longer term support levels when this over but who knows when it will be
I bench YABS 24/7/365 unless it's a leap year.
yes,in DAL and LA
Successes usually come to those who are too busy to be looking for it. Ordinary people are satisfied with their short term achievements, but an extraordinary person continues working hard and accomplishes goals one after another. You are one of the extraordinary people who is worthy of their success. Best wishes for VirMach's continued success in the future.
Certainement qui est en droit de vous rendre absurde, est en droit de vous rendre injuste - Voltaire
@VirMach,
Keep chugging-
some may claim milllions lost, others will say billions gained thanks to patience.
on a lighter note, we will celebrate with a thread full of bus images once this is over. If @FrankZ consents and participates.
VPS reviews | | MicroLXC | English is my nth language.