My message above was essentially what I failed to post earlier. I have some additional information to share as well.
Here's some updates since then:
We've been working hard in loading in disaster recovery backups and completing any remaining virtual servers affected by yesterday's issues.
We finally set up Chicago Hivelocity. Since Chicago servers already got moved, a lot of people will be going there from former Buffalo location but it'll also be available on Ryzen button.
Tokyo got a few extra servers in that have been stuck in Miami for some time. One is having some weird problems related to the setup (nothing major hardware wise) but we were able to get another other online.
Miami has the servers and network equipment and everything there and should be set up very soon, possibly today.
Dedicated server specific update:
I organized all the servers we have available and sorted through the requests.
I wanted to provision these sooner but ran into problems trying to set up the new module, as well as performing any functions on the old. It looks like I have to essentially strip away the old module while trying to maintain some of the information and then re-insert it but what I'm saying doesn't even make sense. Short version, WHMCS is being super weird. I can't even load invoices associated with the old module without it throwing errors and I spent way too much time on this.
With that said, we're going to just skip over to dealing with that portion later and I'm going to begin assigning the new services without further delay. These are happening now. It won't look pretty at the beginning.
Website update:
The whole website missing thing has been spooking a lot of people out. I originally wanted to leave it this way for some time but it seems like it would actually significant reduce ticket load at this point in time if we fix the website.
@VirMach said: We finally set up Chicago Hivelocity. Since Chicago servers already got moved, a lot of people will be going there from former Buffalo location but it'll also be available on Ryzen button.
You likely have lots of other stuff on your mind right now, but will we get the Ryzen migration button enabled for services that have been auto-migrated to a (more or less) arbitrary location eventually?
Location: NYC Metro. No other location currently immediately available. Future Locations: San Jose, Los Angeles. Potential Other Locations: Dallas, Las Vegas, Denver, Phoenix, Jacksonville.
Processors: E3. Mostly v2. Some may be substituted for Dual E5 as we run out.
RAM: 16GB, 32GB.
Disk Configurations: 2TB HDD, 240GB SSD, 1TB SSD, 2TB SSD, 4TB HDD, 500GB SSD Note: 2TB means 1x2TB or 2x1TB, and so on.
IPv4: /29 IPv4 Default.
Currently available are all: 16GB RAM, 2TB HDD, NYC Metro, /29 IPv4 with CentOS6/7 installed. Some have IPMI, some do not. At least one or two have no OS but IPMI. One doesn't have a main IP but has IPMI, so there's definitely some issues that need to be ironed out on a few.
In this case, I probably can wait as I prefer West coast DC like San Jose or Los Angeles for my replacement Dedi. Hope all these issues are sorted out, resolved so we can return back as normal again.
@bula said:
In this case, I probably can wait as I prefer West coast DC like San Jose or Los Angeles for my replacement Dedi. Hope all these issues are sorted out, resolved so we can return back as normal again.
Same for me, I went ahead and submitted a ticket so that no one wastes time/energy setting up something in NYC for me so they can go to people who need something ASAP.
@vyas said:
Quality difference in response between here and LET to the situation . As expected, lot more PMS there
I don't think VirMach has posted over there since this CC issue happened so there's probably a bunch of people who don't use LES going "omg zero communication!!"
@vyas said:
Quality difference in response between here and LET to the situation . As expected, lot more PMS there
I don't think VirMach has posted over there since this CC issue happened so there's probably a bunch of people who don't use LES going "omg zero communication!!"
Shit... you're right. Won't be long until we get invaded worse than when this thread first started
@soulchief said:
I don't think VirMach has posted over there since this CC issue happened so there's probably a bunch of people who don't use LES going "omg zero communication!!"
Someone had been updating the basic points last i checked over there but not the official Virmach account. I'm pretty sure he prob doesn't want to deal with getting bombarded and side tracked more by their insanity, it makes my head hurt going there for anything.
@vyas said:
Quality difference in response between here and LET to the situation . As expected, lot more PMS there
I don't think VirMach has posted over there since this CC issue happened so there's probably a bunch of people who don't use LES going "omg zero communication!!"
I think the "I received the mail at xxxx hours while my vps has been down for over 10 hours... yadaa yadaa" messages can be read, so there is some chatter.
@vyas said:
Quality difference in response between here and LET to the situation . As expected, lot more PMS there
I don't think VirMach has posted over there since this CC issue happened so there's probably a bunch of people who don't use LES going "omg zero communication!!"
A couple of guys have been posting a few bits of the information from here on OGF, so I guess they know a bit of what's going on.
I don't think it's just this issue, it seems VirMach has been a lot freer with explanations of what's been going on with the migrations here for weeks and people here are generally supportive, but on OGF it's just full of people moaning non-stop, so I can see why he's less keen to poke his head up there.
@Advin said:
I have a question, @VirMach, do you even sleep? You work damn hard from what I've gathered so far in this thread, jesus.
I can't even begin to fathom the volume of tickets that you've probably gotten so far.
Nah virmach just pretends to be working, I am trying to find new ip to which my vps was migrated to still the services detail page does not open, i get timeout after waiting too long for it to open?
I have been unable to use that vps for more than a month still I see no resolution,
Heck even boomer fixes his pannel in a week
I've not closely followed the migration process, but one thing I've noticed is that a few people seem unsure of their new IP following migration. Was the new IP not notified prior to the migration with some notice?
I understand IPs change, and with rental being more common now and pricing becoming more competitive its likely to occur more frequency, but surely some notice should be provided for this.
@VirMach said: This is something we haven't addressed yet. I'm trying to think of every way we can bring it down to a meaningful level without just auto crediting and closing the tickets as it will of course upset a lot of people.
I hesitate to levy criticism as you're clearly working extremely hard and are going through a bad time, but has an alternative been considered to automation that often breaks things and auto-closing tickets which annoys people and probably makes them open even more tickets?
For example: automation that works and covers all/most cases, temporary increase in support to go through the tickets (even briefly), etc.
@adly said: I've not closely followed the migration process, but one thing I've noticed is that a few people seem unsure of their new IP following migration. Was the new IP not notified prior to the migration with some notice?
Sometimes no, sometimes yes. Can't say what % - if you clicked "Ryzen migrate" manually you were getting 2nd IP 'reserved' and it was likely used for migration. If your server ended in different location that Ryzen migrate button - you had no idea about IP [or sometimes you had - both were shown in panel, while server migrating, sometiems not]. If you got emergency migration - you had no idea about IP. Current batch is people from CC servers... so totally emergency migration, so no IPs known
@VirMach said: We finally set up Chicago Hivelocity. Since Chicago servers already got moved, a lot of people will be going there from former Buffalo location but it'll also be available on Ryzen button.
You likely have lots of other stuff on your mind right now, but will we get the Ryzen migration button enabled for services that have been auto-migrated to a (more or less) arbitrary location eventually?
This was already supposed to be enabled but there's so many segmentation of packages and locations that there's obviously some cases where it's not going through. I'm going to see if we can change the filter today to make it more of a free-for-all.
@vyas said:
Quality difference in response between here and LET to the situation . As expected, lot more PMS there
I don't think VirMach has posted over there since this CC issue happened so there's probably a bunch of people who don't use LES going "omg zero communication!!"
Shit... you're right. Won't be long until we get invaded worse than when this thread first started
Location: NYC Metro. No other location currently immediately available. Future Locations: San Jose, Los Angeles. Potential Other Locations: Dallas, Las Vegas, Denver, Phoenix, Jacksonville.
Processors: E3. Mostly v2. Some may be substituted for Dual E5 as we run out.
RAM: 16GB, 32GB.
Disk Configurations: 2TB HDD, 240GB SSD, 1TB SSD, 2TB SSD, 4TB HDD, 500GB SSD Note: 2TB means 1x2TB or 2x1TB, and so on.
IPv4: /29 IPv4 Default.
Currently available are all: 16GB RAM, 2TB HDD, NYC Metro, /29 IPv4 with CentOS6/7 installed. Some have IPMI, some do not. At least one or two have no OS but IPMI. One doesn't have a main IP but has IPMI, so there's definitely some issues that need to be ironed out on a few.
In this case, I probably can wait as I prefer West coast DC like San Jose or Los Angeles for my replacement Dedi. Hope all these issues are sorted out, resolved so we can return back as normal again.
I had to end up delaying the provisions as we had to have them re-provisioned due to it missing a key piece of information. That's been done now and we have a lot of San Jose servers so I'll be able to honor those requests now.
@soulchief said:
I don't think VirMach has posted over there since this CC issue happened so there's probably a bunch of people who don't use LES going "omg zero communication!!"
Someone had been updating the basic points last i checked over there but not the official Virmach account. I'm pretty sure he prob doesn't want to deal with getting bombarded and side tracked more by their insanity, it makes my head hurt going there for anything.
It's a lot more difficult being ambiguous there given the hostile environment. It's easier communicating here with more like-minded individuals. @ OGF, I feel like everything would be misinterpreted right now.
@adly said:
I've not closely followed the migration process, but one thing I've noticed is that a few people seem unsure of their new IP following migration. Was the new IP not notified prior to the migration with some notice?
For my vps7 migration from ColonCrossing ATL to Flexential ATL:
I pressed "Ryzen migrate to New York" button, it gave secondary IP but didn't migrate.
There was a notice that migration out of ColonCrossing may happen sometime, but not when.
One day, the machine went offline.
Billing panel shows new IP in Flexential ATL, which differs from the secondary IP assigned before.
Secondary IP disappears after several days.
For my vps1 migration from ColonCrossing LAX to DediPath SJC:
I pressed "Ryzen migrate to San Jose" button, it immediately migrated and showed IP, which worked.
There was a notice that some IPs in SJC may change, but not when.
One day, I rebooted the OS and it received different IP from DHCP. The previous IP worked up until this point.
Now I'm using the new IP from DHCP.
In vps1 step 3, I suppose the renumbering happened sometime ago but I'm unknowningly using somebody else's IP until I rebooted.
I was not told about when the renumbering would occur or what's the new IP, until I did the reboot myself.
@Advin said:
I have a question, @VirMach, do you even sleep? You work damn hard from what I've gathered so far in this thread, jesus.
I can't even begin to fathom the volume of tickets that you've probably gotten so far.
Nah virmach just pretends to be working, I am trying to find new ip to which my vps was migrated to still the services detail page does not open, i get timeout after waiting too long for it to open?
I have been unable to use that vps for more than a month still I see no resolution,
Heck even boomer fixes his pannel in a week
I've not closely followed the migration process, but one thing I've noticed is that a few people seem unsure of their new IP following migration. Was the new IP not notified prior to the migration with some notice?
I understand IPs change, and with rental being more common now and pricing becoming more competitive its likely to occur more frequency, but surely some notice should be provided for this.
We couldn't do the usual rounds of communication given the situation. We couldn't do a lot of things we would normally ensure in terms of communication and notices. A lot of it was being directly misinterpreted/quoted to paint a false picture as well. I don't know if that makes sense but it was easy for one party to go "you mentioned XX, therefore YY, and we're going to apply this to everything else as well." I'll probably be able to reference this fully in a couple years and clear things up.
We were operating in a very hostile and unstable environment and had to be careful in multiple ways. If we were too specific, hostile actors could take advantage of it. I do not believe we would have been able to move everyone in that case without being sabotaged.
@VirMach said: We were operating in a very hostile and unstable environment and had to be careful in multiple ways. If we were too specific, hostile actors could take advantage of it. I do not believe we would have been able to move everyone in that case without being sabotaged.
@VirMach said: This is something we haven't addressed yet. I'm trying to think of every way we can bring it down to a meaningful level without just auto crediting and closing the tickets as it will of course upset a lot of people.
I hesitate to levy criticism as you're clearly working extremely hard and are going through a bad time, but has an alternative been considered to automation that often breaks things and auto-closing tickets which annoys people and probably makes them open even more tickets?
For example: automation that works and covers all/most cases, temporary increase in support to go through the tickets (even briefly), etc.
I understand what you're saying here and for this to have worked, for the most part, we would have had to invest a significant amount of time in doing a lot of that beforehand.
Temporary increase in support was difficult to coordinate. It would have ended up with people receiving frustratingly bad advice. I'm still trying to come up with a plan in sorting tickets and providing actual responses to avoid auto closing everything again. It didn't work out last time but this time we have the added advantage of everything actually finally settling down instead of other changes being stacked on that cause further grievances.
@adly said:
I've not closely followed the migration process, but one thing I've noticed is that a few people seem unsure of their new IP following migration. Was the new IP not notified prior to the migration with some notice?
For my vps7 migration from ColonCrossing ATL to Flexential ATL:
I pressed "Ryzen migrate to New York" button, it gave secondary IP but didn't migrate.
There was a notice that migration out of ColonCrossing may happen sometime, but not when.
One day, the machine went offline.
Billing panel shows new IP in Flexential ATL, which differs from the secondary IP assigned before.
Secondary IP disappears after several days.
For my vps1 migration from ColonCrossing LAX to DediPath SJC:
I pressed "Ryzen migrate to San Jose" button, it immediately migrated and showed IP, which worked.
There was a notice that some IPs in SJC may change, but not when.
One day, I rebooted the OS and it received different IP from DHCP. The previous IP worked up until this point.
Now I'm using the new IP from DHCP.
In vps1 step 3, I suppose the renumbering happened sometime ago but I'm unknowningly using somebody else's IP until I rebooted.
I was not told about when the renumbering would occur or what's the new IP, until I did the reboot myself.
One major problem that caused a lot of these added issues was the fact that SVM or perhaps rather KVM virtualization in general operates in a way where several steps of reboots have to occur. When it's being done for a lot of services at once, things are bound to go wrong. We had to adjust timeouts based on each server's load and it wasn't always perfect.
We couldn't just set a 5 minute timeout per VM to ensure it goes 100% smoothly because if a hundred plus have to get done at once, that's a 10 hour queue.
If at any time this "broke" in any way, it left many servers in various partial states and there were multiple steps of adding and removing IPs. Sometimes one step would break but not another. We had to re-code everything over and over and over so depending on when it was done, and with the other added bugs, the situation was unpredictable. Now add in human error of having to keep track of these states over a hundred plus servers.
I didn't code out the IP change script for the migration without data, but when reviewing the code it looked fine. The multiple IP issue seems to be related to how SVM deals with timeouts and overlaps. Sometimes it'd assign an IP, but not really assign it in a way where it got locked away from it being re-assigned to another and the default behavior seems to be it'd just do it all over again, hence some people got multiple functional IPs.
Comments
My message above was essentially what I failed to post earlier. I have some additional information to share as well.
Here's some updates since then:
Dedicated server specific update:
Website update:
The whole website missing thing has been spooking a lot of people out. I originally wanted to leave it this way for some time but it seems like it would actually significant reduce ticket load at this point in time if we fix the website.
You likely have lots of other stuff on your mind right now, but will we get the Ryzen migration button enabled for services that have been auto-migrated to a (more or less) arbitrary location eventually?
Need a free NAT LXC? -> https://microlxc.net/
In this case, I probably can wait as I prefer West coast DC like San Jose or Los Angeles for my replacement Dedi. Hope all these issues are sorted out, resolved so we can return back as normal again.
Same for me, I went ahead and submitted a ticket so that no one wastes time/energy setting up something in NYC for me so they can go to people who need something ASAP.
Quality difference in response between here and LET to the situation . As expected, lot more PMS there
VPS reviews | | MicroLXC | English is my nth language.
I don't think VirMach has posted over there since this CC issue happened so there's probably a bunch of people who don't use LES going "omg zero communication!!"
Shit... you're right. Won't be long until we get invaded worse than when this thread first started
Humble janitor of LES
Proud papa of YABS
Lol, just found out about LES yesterday.
I knew about LES for a couple years, but every time I went to LES it always had the same thread names as LET so I thought it was just a backup of LET.
Someone had been updating the basic points last i checked over there but not the official Virmach account. I'm pretty sure he prob doesn't want to deal with getting bombarded and side tracked more by their insanity, it makes my head hurt going there for anything.
I noticed that, caught me off guard a bit as well.
I think the "I received the mail at xxxx hours while my vps has been down for over 10 hours... yadaa yadaa" messages can be read, so there is some chatter.
VPS reviews | | MicroLXC | English is my nth language.
A couple of guys have been posting a few bits of the information from here on OGF, so I guess they know a bit of what's going on.
I don't think it's just this issue, it seems VirMach has been a lot freer with explanations of what's been going on with the migrations here for weeks and people here are generally supportive, but on OGF it's just full of people moaning non-stop, so I can see why he's less keen to poke his head up there.
Virmach 2022 The Year Of The Perfect Storm.
lowendinfo.com had no interest.
Virmach's new partnership will be with a whiskey company after all of this to recover and relax
I've not closely followed the migration process, but one thing I've noticed is that a few people seem unsure of their new IP following migration. Was the new IP not notified prior to the migration with some notice?
I understand IPs change, and with rental being more common now and pricing becoming more competitive its likely to occur more frequency, but surely some notice should be provided for this.
I hesitate to levy criticism as you're clearly working extremely hard and are going through a bad time, but has an alternative been considered to automation that often breaks things and auto-closing tickets which annoys people and probably makes them open even more tickets?
For example: automation that works and covers all/most cases, temporary increase in support to go through the tickets (even briefly), etc.
Sometimes no, sometimes yes. Can't say what % - if you clicked "Ryzen migrate" manually you were getting 2nd IP 'reserved' and it was likely used for migration. If your server ended in different location that Ryzen migrate button - you had no idea about IP [or sometimes you had - both were shown in panel, while server migrating, sometiems not]. If you got emergency migration - you had no idea about IP. Current batch is people from CC servers... so totally emergency migration, so no IPs known
This was already supposed to be enabled but there's so many segmentation of packages and locations that there's obviously some cases where it's not going through. I'm going to see if we can change the filter today to make it more of a free-for-all.
Yeah, we're not going to use OGF for a while.
I had to end up delaying the provisions as we had to have them re-provisioned due to it missing a key piece of information. That's been done now and we have a lot of San Jose servers so I'll be able to honor those requests now.
It's a lot more difficult being ambiguous there given the hostile environment. It's easier communicating here with more like-minded individuals. @ OGF, I feel like everything would be misinterpreted right now.
I can't wait until I can go back to being an alcoholic. I haven't been able to have a serious drink in over 1-2 years.
For my vps7 migration from ColonCrossing ATL to Flexential ATL:
For my vps1 migration from ColonCrossing LAX to DediPath SJC:
In vps1 step 3, I suppose the renumbering happened sometime ago but I'm unknowningly using somebody else's IP until I rebooted.
I was not told about when the renumbering would occur or what's the new IP, until I did the reboot myself.
We couldn't do the usual rounds of communication given the situation. We couldn't do a lot of things we would normally ensure in terms of communication and notices. A lot of it was being directly misinterpreted/quoted to paint a false picture as well. I don't know if that makes sense but it was easy for one party to go "you mentioned XX, therefore YY, and we're going to apply this to everything else as well." I'll probably be able to reference this fully in a couple years and clear things up.
We were operating in a very hostile and unstable environment and had to be careful in multiple ways. If we were too specific, hostile actors could take advantage of it. I do not believe we would have been able to move everyone in that case without being sabotaged.
I love your positive attitude.... "were"
I understand what you're saying here and for this to have worked, for the most part, we would have had to invest a significant amount of time in doing a lot of that beforehand.
Temporary increase in support was difficult to coordinate. It would have ended up with people receiving frustratingly bad advice. I'm still trying to come up with a plan in sorting tickets and providing actual responses to avoid auto closing everything again. It didn't work out last time but this time we have the added advantage of everything actually finally settling down instead of other changes being stacked on that cause further grievances.
One major problem that caused a lot of these added issues was the fact that SVM or perhaps rather KVM virtualization in general operates in a way where several steps of reboots have to occur. When it's being done for a lot of services at once, things are bound to go wrong. We had to adjust timeouts based on each server's load and it wasn't always perfect.
We couldn't just set a 5 minute timeout per VM to ensure it goes 100% smoothly because if a hundred plus have to get done at once, that's a 10 hour queue.
If at any time this "broke" in any way, it left many servers in various partial states and there were multiple steps of adding and removing IPs. Sometimes one step would break but not another. We had to re-code everything over and over and over so depending on when it was done, and with the other added bugs, the situation was unpredictable. Now add in human error of having to keep track of these states over a hundred plus servers.
I didn't code out the IP change script for the migration without data, but when reviewing the code it looked fine. The multiple IP issue seems to be related to how SVM deals with timeouts and overlaps. Sometimes it'd assign an IP, but not really assign it in a way where it got locked away from it being re-assigned to another and the default behavior seems to be it'd just do it all over again, hence some people got multiple functional IPs.
Seems LET/LEB has gone completely vindictive on VirMach now.
I can surely understand why he is not posting over there at the moment.
https://lowendbox.com/blog/virmach-teeters-at-the-edge/
Certainement qui est en droit de vous rendre absurde, est en droit de vous rendre injuste - Voltaire
@VirMach I have sent you a message in Skype. I will help you sort the millions of tickets free of charge after my shifts end every day.
I'm so sorry to hear these news. You really don't deserve all this.