VirMach
VirMach
Comments
-
In our next meeting with the legal team, I'll attempt to get back some information for you. We will of course pursue this to the best of our ability. I would not necessarily know what legal ramifications are available, if any, that would assist customers in at least getting some access to their data. If anyone is…
-
Are you on a dedicated server that's currently offline?
-
I'm working with our developer today to pro-rate refund all dedicated servers where possible, for any remaining duration. We don't want to give people the wrong idea of what's going on and want to reduce any panic, at least on any end we can control. Then, first come first serve basis on emergency tickets created…
-
I wouldn't be able to comment on this. We have a contingency plan. It's not perfect. Nor were we expecting it to go this far or be done in this way. I've been working on it the entire time, and have emergency replacement servers to some degree. I'm sending out some vague communication that's at the very least necessary to…
-
"There's been a lot of cases of people doing their own network configurations due to issues with reconfigure button, but then they set the IP address to the node's main IP or a random IP in the subnet. This is considered IP stealing/network abuse, so I don't know if I have to provide a PSA here but definitely only use the…
-
Everything other than LAXA014 should be at the very least temporary resolved. I'm compiling a list of more permanent solutions to implement and still investigating some nodes to see what happened. Are you on Ryzen?
-
Message me the IP, I want to make sure it's not a node-wide issue.
-
Yeah, possibly IP conflict. > @cybertech said: It's actually a pretty hilarious paradox. They keep raising the rates, reducing the level of service they provide, and I highly doubt it translates over to the hands getting paid much. From what I've seen they basically end up making around 10-20% of the hourly rate you get…
-
Yeah, possibly IP conflict. Enabled protection on it. Reconfigure and try again. If it doesn't work, let me know. This could also potentially knock out the entire network for NYCB018 if we're extremely unlucky so I'll check back in 10 minutes.
-
All labeling ever did for us is cause MORE problems so I stopped doing it. Example A: * Nothing is labeled. * Techs set it up how they want, and maybe communicate it correctly if we're lucky. * Tech may be able to follow instructions for something in relation to RU9. Example B: * Everything is labeled. * Techs set it up…
-
I haven't been billing for these recently but they also take an outrageous amount of time to complete, so a fair bit of warning. This is one of the things I've tried and failed to fix over the last two years. Maybe we'll finally have time for reworking the process but until then you can realistically expect to wait weeks…
-
Finally fixed NYCB036. They had the LAN port plugged in as dedicated IPMI on the switch we use for IPMI, and labeled it as a port on the main switch when it wasn't. Took a fair bit of detective work to find since the whole setup is a mess at this point. I'm probably going to have to go down to New York and fix it one day…
-
This is one of the things where I don't even think it's their fault but they're on a blacklist UCEPROTECT which as @FrankZ mentioned should not be trusted but for some reason it's "industry standard" for a bunch of lazy sys admins to use it. I guess by literally having infinite false positives, it gets their job done of…
-
I got lucky with SJCZ004. After about 20 reboots and messing with the BIOS, when I finally gave up and was going to try recovering data, I selected virtual disk on BIOS and it booted into the OS instead (???) seems like it just wants to do the opposite of everything I tried. Anyway, something's definitely wrong with it and…
-
I've gone down these dozens of times at this point and checked for broken IP assignments, looks like I missed one on NYCB018 because I'm semi-dyslexic and it was .212 instead of .221 on a single VM. I've corrected that one now.
-
Announcement: We've added an addon for all the people over the last few months that have indicated their extreme displeasure with their service going offline, claiming that as a result they lost thousands. There have also been people that were highly sensitive to IP address changes, location changes, and were in a…
-
Yeah, I saw that as well as a few other things happen and at this point I'm trying to figure out if I actually need to just start a VirBot cloning program so we can ship me off to live in a cage at every datacenter.
-
Hivelocity's the one that acquired Incero right? I don't know why they didn't come to mind (HV) when I was thinking about Dallas. We can't do it right now but there's absolutely no way we can stay with Flexential at this state, WTF.
-
Holy crap, we're actually so doomed in Dallas @FrankZ they finally got back to us and said (Flexential) due to the "complexity" of the request, the request being: * Clear CMOS * Reset BMC I think maybe I made it sound so complicated by asking them to also verify at the end. I'll figure something out for this location, JFC.
-
Yeah that's already been established but at this point I've basically made myself step back and cool off, otherwise I feel like we're going to be migrating everyone around for the next 2 years before we can settle down with an appropriate set of partners that meet or exceed our bare minimum expectations. I had a whole…
-
Chicago going up today.
-
Finally got back the physical port numbers for the servers I needed, NYCB036 (formerly 101) should be fixed soon, just need to wrap up what I was working on and configure it in the switch. SJCZ004 unfortunately still down, I was going to fly in yesterday and just literally press the button then fly back but flight times…
-
Took it away while we do more migrations, we can't have it drastically change too much or it'll put the plans out of whack. It'll re-enable later tonight.
-
There's no way to differentiate between fully broken and partially broken nodes to display it only to those people.
-
Button updated. I can't get it to work but let's see if anyone else has any luck (edit -- looks like it did work for at least one person so far.) To be far I was trying to break it by being impatient and refreshing/closing it. Let me allow it to run for 5 minutes and see what happens instead. Oh wait I think it's because I…
-
I'd actually accept these kinds of request if it was possible to keep it clean but realistically that means you get a new VM and then the old one just kind of hangs around while it's offline and takes up space until we manually verify which ones are abandoned and clear up the space. For Phoenix though it's technically…
-
How do you propose a migration of an offline server?
-
Yeah I don't know how reliable this random site is that I found but:
-
Phoenix is completely trashed right now.
-
LA10GKVM14 was part of a massive PDU or power circuit event with them, where like a dozen plus servers went offline and online, and then offline, causing a bunch of power supplies to fail. A lot of times when we've had power issues like that, CC also moves them to another switch without telling us and then does not…
-
Honestly at this point I think 8.8.8.8 issue is actually not on any specific datacenter's end. We had it happen on our WHMCS for a little bit, and that one's not even hosted with any datacenter we use right now for these Ryzens. We've also had it reported on LAX which is QN and Phoenix which is PhoenixNAP IIRC. So it has…
-
NYC storage node, I had to do a lot of weird setups for this one to make it work as we faced problems such as a disk going missing, the wrong switch, and so on in the background. Right now it's LACP aggregated 3Gbps. Originally was supposed to have 36 disks, then had 31, then Amazon delayed it, then either one died or DC…
-
Very unlucky user or very lucky user based on preference, when it happened, etc. Maybe your services are getting migrated to Ryzen, maybe they're all broken.
-
I was going to get a flight to San Jose for tomorrow morning but I'm still waiting to see how we can even get DC access since it's the first time. I might still go on Tuesday and see if I can beat the DC hands to it for SJCZ004 at this point, only about an hour flight. Phoenix actually has some of our infrastructure on it…
-
I did also finally get some switch access, so you will either see more nodes up in NYC that we can use for migrations, or you'll see me break the networking.
-
All systems are existing systems that have been tested, some more than others. The newer nodes I've actually spent less time testing, not as a result of being careless, but in that a lot of the issues were already ironed out so it was a quicker fix and it's possible less of these core issues get passed onto those nodes. We…
-
Okay after working on this all day, outside of the two nodes somehow still down (SJCZ004 and NYCB036) I've counted these specifically and we've gone down from something like 2% of people after the most recent round of migrations to exactly 0.91% of people on Ryzen having a VPS with an incorrect IP or non-bootable. At this…
-
There's about 3-5% of the VMs offline, but the node as a whole isn't. Those are on our queue.
-
It's been renamed to NYCB036 and I've been painfully aware of it being down. I have no access to IPMI and no one seems to care. I've mentioned it at least 4 times and it hasn't even been marked as request put in to the DC yet. Also marked on a problem sheet and specifically marked as having customers on it and therefore…