@VirMach said:
Also aware of this, I think mobile version just uses the full size image, right? I just had the one old large logo and again, lazily used it for now.
Correct. Since you already have the template, I can provide help to bring the site back to a professional state in 24-48 hours (Fixing all the small layout bugs etc).
Me and @MikePT are just trying to help our favourite provider.
I can reply basic tickets while I'm watching "How I met your mother" if you want.
It’s great to see a few people wanting to help - I’m also available if @VirMach is interested. However, there is probably a better way to approach them about it, such as @MikePT having a personal/Skype contact.
I haven't been paying super close attention to the thread so I hope I'm not asking something that's already been discussed. Is some of the problem caused by VirMach being unable to get new equipment due to shortages? I.e., VirMach wanted to leave a certain WNY datacenter but there simply weren't enough Ryzen chips/mobos/servers on the market to get setup somewhere else? And now the SHTF...
@dedicados said:
hope my dedi server come back soon =(
I was actually thinking of you specifically when it happened since you had a close call with the previous termination situation that I helped you out with, I just hope you finally got some backups in.
yeah you helped me. thank you for that!
about server , it was a customer server and they did not take backup, so is not my problem anymore. but since they where good customers i am going to lose them. so bad.
@fluttershy said:
It’s great to see a few people wanting to help - I’m also available if @VirMach is interested. However, there is probably a better way to approach them about it, such as @MikePT having a personal/Skype contact.
For domain registrations, create an account at Dynadot (ref) and spend $9.99 within 48 hours to receive $5 DynaDollars!
Looking for cost-effective Managed/Anycast/DDoS-Protected/Geo DNS Services? Try ClouDNS (aff).
That'd be interesting if we allow people to request whatever they want, the only catch is we only do it when it's financially feasible. I'm sure inflation and energy costs will catch up to us eventually though so running 180W TDP in 2032 might end up costing $80 a month.
@fluttershy said:
It’s great to see a few people wanting to help - I’m also available if @VirMach is interested. However, there is probably a better way to approach them about it, such as @MikePT having a personal/Skype contact.
Also open to handling a few tickets if needed.
Yes but are you able to handle opening a few tickets?
@rockinmusicgv said: there simply weren't enough Ryzen chips/mobos/servers on the market to get setup somewhere else? And now the SHTF...
i remembered virmach said he can't get server chassis...
@rockinmusicgv said:
I haven't been paying super close attention to the thread so I hope I'm not asking something that's already been discussed. Is some of the problem caused by VirMach being unable to get new equipment due to shortages? I.e., VirMach wanted to leave a certain WNY datacenter but there simply weren't enough Ryzen chips/mobos/servers on the market to get setup somewhere else? And now the SHTF...
I'll try to tell the whole story again and add some more details.
What ended up happening is we had something like 200~ chassis with E3 processors and enough CPUs, SSDs, motherboards for them all. These were for dedicated server customers.
ASRock, the only brand that makes Ryzen motherboards and therefore Ryzen motherboard and chassis combos for datacenter use (the motherboards have built-in BMC chip, so IPMI) ran into huge supply chain issues. They initially quoted us let's say $X per chassis and motherboard combo and then ghosted us for a few months, no longer replying to our sales request. By the time we grabbed their attention and went through the entire process of contacting the same sales manager, he didn't seem very interested in making the deal (but was essentially forced to reply after we bothered a dozen people about it.) So he just punted it to one of their resellers instead, who quoted us a higher price than MSRP. This ended up being about double the original cost and was not within our budget.
It no longer made sense to buy the motherboard and chassis combo, since it was something like triple the cost of just buying the motherboard. The chassis combo came with a power supply, heatsink, fans, and appropriate brackets/cables.
At the same time, this was when gasoline prices were skyrocketing and there was a general shortage in workers in the US and globally. This meant we had to now try to individually source all these missing parts ourselves. There's only so many power supplies we could purchase, so many fans, and we also had to 3D print brackets because they weren't really for sale for a custom chassis being converted to fit what we were trying to achieve. Most of it was OK, except getting a bulk quantity of the same chassis was the more difficult part. We got lucky with the 200~ we had purchased earlier, and got them at a good price, but with shipping costs alone, it would have ended up costing around $50 per chassis just to get it shipped. Something weird happened around the same time where shipping via freight actually ended up being more expensive than just sending them per box, so we couldn't save any money either by purchasing them in bulk. Then the final issue was getting rails that fit the specific chassis we would have to purchase, and it essentially made it impossible to try to get 100+ chassis that were more or less the "same" to where we could CAD and 3D print parts for them. Most the "bulk" chassis available were with built-in I/O panels for specific boards so we would have had to cut out a square on every single one if we went that route.
Long story short, by the time we absolutely had to finish these builds, which at this point in time consisted of all of them being "done" except for placing them in the chassis, the only viable option was to use the chassis we had already built for the customer E3 dedicated servers. We also already had rails for them which were difficult to find on short notice in bulk quantities. This meant our plan for having replacements for E3 servers had to be delayed indefinitely. To add insult to injury, this meant we had to undo all the work we had already done for the E3 servers (putting in and testing the motherboards, labeling everything, putting in RAID controllers, SSD brackets, etc.) We did probably end up saving $50,000 in the process but this was $50,000 we hadn't planned on having to pay on top of everything in the first place so, again, it was the only option for various reasons. Plus even if we did pay the extra money to try to get the proper chassis and motherboard combos, it was still on a 2 month backorder.
So at least 100 chassis meant for customer E3 servers ended up being repurposed for our Ryzen builds that were going to be used for customer VPS services instead. And the other 100 or so left were a different type of servers where the rail kits were impossible to find for months and months, and those also happened to be the ones we hadn't finished building out yet as a result of having no rails (they were de-prioritized.) This meant on top of everything, we were now short on time. Even if we did figure out the rail kit situation with maybe using universal rails, we still had to redo the builds into the second half of chassis.
The backup plan was to just rent these servers instead, and we found a partner for those bulk quantity E3 rentals, we just ended up getting screwed over at the finish line with what abruptly happened last week.
As for customer Ryzen builds, we did have a smaller portion of the E3 purchases being upgraded by customers to Ryzen builds, I believe 3600X. We just couldn't do those because all the motherboards ended up being used for the VPS builds. The prices on those also went up and the only alternative required a KVM switch setup for IPMI which meant we had to more specifically plan out exactly which cabinets they went into and it ended up being a logistical nightmare with everything else going on. Again, this was just a small quantity of them so it's basically just a footnote to the whole scenario.
@VirMach said:
Also aware of this, I think mobile version just uses the full size image, right? I just had the one old large logo and again, lazily used it for now.
Correct. Since you already have the template, I can provide help to bring the site back to a professional state in 24-48 hours (Fixing all the small layout bugs etc).
Me and @MikePT are just trying to help our favourite provider.
I can reply basic tickets while I'm watching "How I met your mother" if you want.
It’s great to see a few people wanting to help - I’m also available if @VirMach is interested. However, there is probably a better way to approach them about it, such as @MikePT having a personal/Skype contact.
I saw your kind private message, as well as other offers by everyone. I'm still trying to think if there's a viable method of accepting everyone's offer of assistance. The closest idea I had was launching an official message board where people such as yourself could essentially have the role of official moderators and be equipped to deal with the majority of basic questions and requests which do take up the majority of the current ticket queue.
I'll try to finalize a plan, if any, over the weekend.
@VirMach said: The closest idea I had was launching an official message board where people such as yourself could essentially have the role of official moderators and be equipped to deal with the majority of basic questions and requests which do take up the majority of the current ticket queue.
Honestly, this kind of openness makes me respect you more as a provider. Maybe when the dust has settled a bit you should do a blog post, or maybe even offer raindog a proper article about these logistical issues rather than his speculations.
@VirMach said: The closest idea I had was launching an official message board where people such as yourself could essentially have the role of official moderators and be equipped to deal with the majority of basic questions and requests which do take up the majority of the current ticket queue.
An official discord would be neat.
Yes.
Can I have my own public channel? I would like to share flan and jelly photos. Everybody should know about Pollo a la brasa too.
@VirMach said: The closest idea I had was launching an official message board where people such as yourself could essentially have the role of official moderators and be equipped to deal with the majority of basic questions and requests which do take up the majority of the current ticket queue.
An official discord would be neat.
Agreed! I'm sort of surprised there isn't one already.
Yes but are you able to handle opening a few tickets?
I'll try my best to open as many tickets as possible, just for you /s
I'm happy to dust off my old helpdesk hat if you do need assistance, I have a feeling you aren't getting much sleep lately. Try not to burn out! Made that mistake a few times before.
I didn't mean to make you type all that out. I just remember seeing a post about a server that might have been dropped. I know from my own experience getting chips out of China/Taiwan/HK has been nearly impossible over the past few years. The last time I had a quote it was something on the order of $60/kg using a courier like FedEx. I couldn't imagine how much it must've cost to get servers chassis out.
If you need any logistics help. Let me know, I might be able to help out...
Too much to read and retain at the same time, even though it was quite a read!
Here's the audio version. I threw in some sound effects for good measure.
@VirMach said: I'm still trying to think if there's a viable method of accepting everyone's offer of assistance. The closest idea I had was launching an official message board where people such as yourself could essentially have the role of official moderators and be equipped to deal with the majority of basic questions and requests which do take up the majority of the current ticket queue.
My personal advice would be to accept and delegate some work (maybe aging or basic tickets?) so you can feed your focus 👌
For domain registrations, create an account at Dynadot (ref) and spend $9.99 within 48 hours to receive $5 DynaDollars!
Looking for cost-effective Managed/Anycast/DDoS-Protected/Geo DNS Services? Try ClouDNS (aff).
For virtual servers, we've located and organized disaster recovery backups for any remaining VMs. This is about 700 or 800 of them, and it'd be for any VMs that got marked as "locked" around Wednesday not yet migrated. A small quantity of VMs had corrupt backups, I think maybe 20 of them total so for those we'll recreate them at the end. This number isn't including any other nodes or transfers that previously had issues, it's only for the most recent issue. These were delegated to be done by someone else earlier on Wednesday but he unfortunately ran into issues and I needed to help so it was delayed.
For dedicated servers, we now have about 30% of them available for immediate provisioning. I fell behind on handing them out the last two days as several other tasks came up such as fixing ATLZ007 which went down again, as well as configuring several new nodes and troubleshooting the last dozen builds I need to send out and probably a dozen other smaller tasks, but we're on track to finish all the deliveries today.
For virtual servers, we've located and organized disaster recovery backups for any remaining VMs. This is about 700 or 800 of them, and it'd be for any VMs that got marked as "locked" around Wednesday not yet migrated.
I'm guessing that it would be in fact more work for you if some of us didn't care about the restore but more having the vps active again (ie: we rebuilt from scratch once they were configured on a host node)?
I know the two I have in this state have nothing critical on them and i can rebuild them pretty quickly but I assume that would involve more manual work to separate them from restore routines and such?
All this work and tragedy just because you are inclined to use Ryzen, a Desktop CPU. I do understand they are nice, cheap alternative to EPYC, people love the brand, etc. But what about other CPU's and type of builds? Maybe this process would have been less of an issue, specially the part of the mobos/chassis?
From what I gather ... you are a big provider, the numbers you are mentioning aren't small.
WebNX / GorillaServers (sister company) has like overstock of Ryzen servers. What about making a deal there, probably better than buying and colo'ing at this stage?
@VirMach said: Here's my philosophy on negative press: I like it because it helps ignorant customers stay away. It's mutually beneficial.
This is good to know, and I expect you are correct.
EDIT: Then it is also a good thing that LEB mods seem to be removing any positive comments from the previously mentioned blog post and just showing repeated negative comments by the same users.
@VirMach said: Here's my philosophy on negative press: I like it because it helps ignorant customers stay away. It's mutually beneficial.
This is good to know, and I expect you are correct.
EDIT: Then it is also a good thing that LEB mods seem to be removing any positive comments from the previously mentioned blog post and just showing repeated negative comments by the same users.
Yeah my comment got deleted, and a couple of the comment regarding justin got deleted.
@VirMach said: Here's my philosophy on negative press: I like it because it helps ignorant customers stay away. It's mutually beneficial.
This is good to know, and I expect you are correct.
EDIT: Then it is also a good thing that LEB mods seem to be removing any positive comments from the previously mentioned blog post and just showing repeated negative comments by the same users.
LEB has been a joke for a while, with LET closely behind particularly following the influx of the new ‘moderators’. Despite this, Jon seems to think it’s some extremely valuable resource people are failing to utilise, rather than the actuality of it having the reverse Midas touch (everything it touches turns to shit).
@VirMach said: Here's my philosophy on negative press: I like it because it helps ignorant customers stay away. It's mutually beneficial.
This is good to know, and I expect you are correct.
EDIT: Then it is also a good thing that LEB mods seem to be removing any positive comments from the previously mentioned blog post and just showing repeated negative comments by the same users.
Yeah my comment got deleted, and a couple of the comment regarding justin got deleted.
Funny how that works. If you're a big user of ColoCrossing than you're a hero. The greatestest provider in the universe.
But once you decide that CC is not for you anymore, you get ousted. Almost literally.
VirMach-FFME001-gateway is now DOWN
Target: 149.57.160.1[ping]
Noticed at: 2022-08-08 10:50:36 (UTC 01:00)
Encountered errors:
Amsterdam: Timeout (5 sec)
Frankfurt: Timeout (5 sec)
Warsaw: Timeout (5 sec)
Losing millions again
VirMach-FFME001-gateway is now UP
Downtime: 3 min
Target: 149.57.160.1[ping]
Noticed at: 2022-08-08 10:53:36 (UTC 01:00)
Millions has been restored also the SolusVM seems to be working now :P
aaaaaaand it's down again.
VirMach-FFME001-gateway is now DOWN
Target: 149.57.160.1[ping]
Noticed at: 2022-08-08 10:54:39 (UTC 01:00)
Encountered errors:
Amsterdam: Timeout (5 sec)
London: Timeout (5 sec)
Frankfurt: Timeout (5 sec)
Warsaw: Timeout (5 sec)
Millions lost again, VirMach doubling the rate of losing millions.
and it's back after another 3 minutes - SolusVM back to NOT working too. Operation Timed Out After 90001 Milliseconds With 0 Bytes Received. VirMach must be having fun day there.
Comments
It’s great to see a few people wanting to help - I’m also available if @VirMach is interested. However, there is probably a better way to approach them about it, such as @MikePT having a personal/Skype contact.
i dont think i can help in vps stuff but considering to buy some account credits
I migrate the server to another data center, but it gives me a ticket: "Migration Request, Awaiting Request". how long do I have to wait?
thanks.
I haven't been paying super close attention to the thread so I hope I'm not asking something that's already been discussed. Is some of the problem caused by VirMach being unable to get new equipment due to shortages? I.e., VirMach wanted to leave a certain WNY datacenter but there simply weren't enough Ryzen chips/mobos/servers on the market to get setup somewhere else? And now the SHTF...
yeah you helped me. thank you for that!
about server , it was a customer server and they did not take backup, so is not my problem anymore. but since they where good customers i am going to lose them. so bad.
hope everything gets ok for you.
Also open to handling a few tickets if needed.
VirMach's ticket list/queue:
For domain registrations, create an account at Dynadot (ref) and spend $9.99 within 48 hours to receive $5 DynaDollars!
Looking for cost-effective Managed/Anycast/DDoS-Protected/Geo DNS Services? Try ClouDNS (aff).
i remembered virmach said he can't get server chassis...
That'd be interesting if we allow people to request whatever they want, the only catch is we only do it when it's financially feasible. I'm sure inflation and energy costs will catch up to us eventually though so running 180W TDP in 2032 might end up costing $80 a month.
Yes but are you able to handle opening a few tickets?
I'll try to tell the whole story again and add some more details.
What ended up happening is we had something like 200~ chassis with E3 processors and enough CPUs, SSDs, motherboards for them all. These were for dedicated server customers.
ASRock, the only brand that makes Ryzen motherboards and therefore Ryzen motherboard and chassis combos for datacenter use (the motherboards have built-in BMC chip, so IPMI) ran into huge supply chain issues. They initially quoted us let's say $X per chassis and motherboard combo and then ghosted us for a few months, no longer replying to our sales request. By the time we grabbed their attention and went through the entire process of contacting the same sales manager, he didn't seem very interested in making the deal (but was essentially forced to reply after we bothered a dozen people about it.) So he just punted it to one of their resellers instead, who quoted us a higher price than MSRP. This ended up being about double the original cost and was not within our budget.
It no longer made sense to buy the motherboard and chassis combo, since it was something like triple the cost of just buying the motherboard. The chassis combo came with a power supply, heatsink, fans, and appropriate brackets/cables.
At the same time, this was when gasoline prices were skyrocketing and there was a general shortage in workers in the US and globally. This meant we had to now try to individually source all these missing parts ourselves. There's only so many power supplies we could purchase, so many fans, and we also had to 3D print brackets because they weren't really for sale for a custom chassis being converted to fit what we were trying to achieve. Most of it was OK, except getting a bulk quantity of the same chassis was the more difficult part. We got lucky with the 200~ we had purchased earlier, and got them at a good price, but with shipping costs alone, it would have ended up costing around $50 per chassis just to get it shipped. Something weird happened around the same time where shipping via freight actually ended up being more expensive than just sending them per box, so we couldn't save any money either by purchasing them in bulk. Then the final issue was getting rails that fit the specific chassis we would have to purchase, and it essentially made it impossible to try to get 100+ chassis that were more or less the "same" to where we could CAD and 3D print parts for them. Most the "bulk" chassis available were with built-in I/O panels for specific boards so we would have had to cut out a square on every single one if we went that route.
Long story short, by the time we absolutely had to finish these builds, which at this point in time consisted of all of them being "done" except for placing them in the chassis, the only viable option was to use the chassis we had already built for the customer E3 dedicated servers. We also already had rails for them which were difficult to find on short notice in bulk quantities. This meant our plan for having replacements for E3 servers had to be delayed indefinitely. To add insult to injury, this meant we had to undo all the work we had already done for the E3 servers (putting in and testing the motherboards, labeling everything, putting in RAID controllers, SSD brackets, etc.) We did probably end up saving $50,000 in the process but this was $50,000 we hadn't planned on having to pay on top of everything in the first place so, again, it was the only option for various reasons. Plus even if we did pay the extra money to try to get the proper chassis and motherboard combos, it was still on a 2 month backorder.
So at least 100 chassis meant for customer E3 servers ended up being repurposed for our Ryzen builds that were going to be used for customer VPS services instead. And the other 100 or so left were a different type of servers where the rail kits were impossible to find for months and months, and those also happened to be the ones we hadn't finished building out yet as a result of having no rails (they were de-prioritized.) This meant on top of everything, we were now short on time. Even if we did figure out the rail kit situation with maybe using universal rails, we still had to redo the builds into the second half of chassis.
The backup plan was to just rent these servers instead, and we found a partner for those bulk quantity E3 rentals, we just ended up getting screwed over at the finish line with what abruptly happened last week.
As for customer Ryzen builds, we did have a smaller portion of the E3 purchases being upgraded by customers to Ryzen builds, I believe 3600X. We just couldn't do those because all the motherboards ended up being used for the VPS builds. The prices on those also went up and the only alternative required a KVM switch setup for IPMI which meant we had to more specifically plan out exactly which cabinets they went into and it ended up being a logistical nightmare with everything else going on. Again, this was just a small quantity of them so it's basically just a footnote to the whole scenario.
I saw your kind private message, as well as other offers by everyone. I'm still trying to think if there's a viable method of accepting everyone's offer of assistance. The closest idea I had was launching an official message board where people such as yourself could essentially have the role of official moderators and be equipped to deal with the majority of basic questions and requests which do take up the majority of the current ticket queue.
I'll try to finalize a plan, if any, over the weekend.
An official discord would be neat.
Honestly, this kind of openness makes me respect you more as a provider. Maybe when the dust has settled a bit you should do a blog post, or maybe even offer raindog a proper article about these logistical issues rather than his speculations.
Yes.
Can I have my own public channel? I would like to share flan and jelly photos. Everybody should know about Pollo a la brasa too.
Agreed! I'm sort of surprised there isn't one already.
I'll try my best to open as many tickets as possible, just for you
/s
I'm happy to dust off my old helpdesk hat if you do need assistance, I have a feeling you aren't getting much sleep lately. Try not to burn out! Made that mistake a few times before.
I didn't mean to make you type all that out. I just remember seeing a post about a server that might have been dropped. I know from my own experience getting chips out of China/Taiwan/HK has been nearly impossible over the past few years. The last time I had a quote it was something on the order of $60/kg using a courier like FedEx. I couldn't imagine how much it must've cost to get servers chassis out.
If you need any logistics help. Let me know, I might be able to help out...
@VirMach Hello
https://microlxc.net/index.php?p=deploy
Too much to read and retain at the same time, even though it was quite a read!
Here's the audio version. I threw in some sound effects for good measure.
Click to listen to audio (opens in new window)
Audio: The Virmach Saga
Click the link to visit the audio page.
https://pod.co/gatha-podcasts/the-virmach-migration-sage
VPS reviews | | MicroLXC | English is my nth language.
@vyas Sage=saga?
lowendinfo.com had no interest.
50-50 Typo and Intentional
VPS reviews | | MicroLXC | English is my nth language.
My personal advice would be to accept and delegate some work (maybe aging or basic tickets?) so you can feed your focus 👌
For domain registrations, create an account at Dynadot (ref) and spend $9.99 within 48 hours to receive $5 DynaDollars!
Looking for cost-effective Managed/Anycast/DDoS-Protected/Geo DNS Services? Try ClouDNS (aff).
Update.
For virtual servers, we've located and organized disaster recovery backups for any remaining VMs. This is about 700 or 800 of them, and it'd be for any VMs that got marked as "locked" around Wednesday not yet migrated. A small quantity of VMs had corrupt backups, I think maybe 20 of them total so for those we'll recreate them at the end. This number isn't including any other nodes or transfers that previously had issues, it's only for the most recent issue. These were delegated to be done by someone else earlier on Wednesday but he unfortunately ran into issues and I needed to help so it was delayed.
For dedicated servers, we now have about 30% of them available for immediate provisioning. I fell behind on handing them out the last two days as several other tasks came up such as fixing ATLZ007 which went down again, as well as configuring several new nodes and troubleshooting the last dozen builds I need to send out and probably a dozen other smaller tasks, but we're on track to finish all the deliveries today.
Place your bets:
Is Jab's VPS one of those or not?!
I'm guessing that it would be in fact more work for you if some of us didn't care about the restore but more having the vps active again (ie: we rebuilt from scratch once they were configured on a host node)?
I know the two I have in this state have nothing critical on them and i can rebuild them pretty quickly but I assume that would involve more manual work to separate them from restore routines and such?
Depends on how much Jab has bugged Virmach lately?
All this work and tragedy just because you are inclined to use Ryzen, a Desktop CPU. I do understand they are nice, cheap alternative to EPYC, people love the brand, etc. But what about other CPU's and type of builds? Maybe this process would have been less of an issue, specially the part of the mobos/chassis?
From what I gather ... you are a big provider, the numbers you are mentioning aren't small.
WebNX / GorillaServers (sister company) has like overstock of Ryzen servers. What about making a deal there, probably better than buying and colo'ing at this stage?
This is good to know, and I expect you are correct.
EDIT: Then it is also a good thing that LEB mods seem to be removing any positive comments from the previously mentioned blog post and just showing repeated negative comments by the same users.
Certainement qui est en droit de vous rendre absurde, est en droit de vous rendre injuste - Voltaire
Yeah my comment got deleted, and a couple of the comment regarding justin got deleted.
>
good plan.
LEB has been a joke for a while, with LET closely behind particularly following the influx of the new ‘moderators’. Despite this, Jon seems to think it’s some extremely valuable resource people are failing to utilise, rather than the actuality of it having the reverse Midas touch (everything it touches turns to shit).
Funny how that works. If you're a big user of ColoCrossing than you're a hero. The greatestest provider in the universe.
But once you decide that CC is not for you anymore, you get ousted. Almost literally.
Losing millions again
Millions has been restored also the SolusVM seems to be working now :P
aaaaaaand it's down again.
Millions lost again, VirMach doubling the rate of losing millions.
and it's back after another 3 minutes - SolusVM back to NOT working too.
Operation Timed Out After 90001 Milliseconds With 0 Bytes Received. VirMach must be having fun day there.