My personal point of view (not sure if it's shared by most customers, or not):
1)
Only those who never do anything make no mistakes. The difference between the good and the bad providers is whether mistakes are something that happens regularly, or a rare exception. Also, it matters if measures are taken to prevent the same mistakes from re-occuring.
For some reason, many people / companies fail to test restoring from their backups. Similar goes for other safety measures. To me it's like cancelling an emergency evacuation exercise because of rain.
It's better learning from other people's mistakes, than from one's own, but only the really stubborn people don't learn from their own mistakes at least.
2)
Again, in my point of view: longer downtime is best announced, though explaining the reasons once it's already started is also fair.
Either way: it's better to have a longer downtime on occasions, knowing that things are fixed "systematically", for better functioning in the long run - than it is to have downtime (and/or data-loss) every now and then.
3)
Emergencies are wars, disasters, violent crimes - in IT, there's only poor planning - generally speaking.
Customers who need (near) 100% uptime should have failover website deployments in more than one geo-location. And backups - that's always necessary.
Still, many (most?) people are not reasonable. They don't want to spend the money it takes for the above mentioned, but they want to be convinced that a provider (whichever they choose) will provide 99.999% uptime.
One thing is having a TOS that says that backups are a convenience but no guarantees/responsibility are taken. With explanation that going below 99.99% gives a few dollars compensation (depending on the downtime duration), up to the price of monthly subscription.
Another thing is stating that clearly, on the front / sign up page. Don't think one would sell a lot of accounts with such policy.
Which all results in expectations too high.
4)
Amount of money spent on hosting makes no difference and is no guarantee. Not really.
Educational situation with highly reputable and rather expensive MDDhosting (TLDR: all servers down and all backups messed up, week of downtime): https://forums.mddhosting.com/topic/1582-major-outage-092118-09242018/
As a customer - I'm aware that I shouldn't expect really good service for dirt cheap. Aware that many people promise many things, not taking everyone's word for granted.
As a service provider - regardless of the price, I would require myself to offer what's promised. Think in the long run it's all about reputation and trust. Would probably never get rich, but if I were to sell crap (not claiming, nor implying that Smallweb is doing that), I'd state it up front. MXroute email provider is a great example of that. Their website clearly states, up front, something along the lines (paraphrasing from memory): "we offer cheap service that works well, BUT we cut the costs on support - you won't get much help with the basic stuff and for most questions will be lead to a community support forum, if you want more hand-holding, pay a lot more with other providers". That's fair and straight forward from the start. Though I'm sure it turns many customers away initially, the customers that do take the service don't get disappointed, quite the contrary. They could have promised all bells and whistles on the front page, but it would have backfired (one disappointed customer review can hardly make up for 1000 good reviews).
Whoever has read this till the end - come to Serbia, you've earned a bottle of home-made rakija to re-set your brains...
And for everyone who thought twice and decided to save the potential drive-by shitposts for some other thread. I think the real lowend spirit may shine a bit brighter here for your forbearance.
@SmallWeb I hope this serves to confirm a solid level of support from your customers and colleagues alike. Re5pekt! Maybe, just maybe you have earned enough of it already by "paying it forward" - I'm not sure exactly how, but that does seem to be what we are seeing here.
As a startup, and as a solo practitioner, you may often feel like you're way out on a limb over thin ice without a net. So when something goes wrong, a sense of urgency verging on panic can easily lead down a slippery slope to an avalanche of errors. If you know in those situations that you can reach out for advice - technical and otherwise - then that's a much better place to be in.
(I'm about to murder more metaphors but will stop short of suggesting to take a deep breath when the shit hits the fan.)
Really glad to see the Los Angeles location is now rising from the ashes. Will happily leave you to it now. Cheers!
Suggest to make backups a beta feature , free out of goodwill, instead of part of advertised service. It will save Ur own ass and manage £1 expectations as well .
Some great advice and lessons to be learned. Thank you all. See you in Serbia @bikegremlin
@isunbejo By the looks of it that was the £11.11 for 10 years promotion. All plans have been converted to their yearly equivalent.
I just wanted to post the aftermath email that should currently be on the way to affected customers.
On 09/01/2020 we suffered a catastrophic data loss of all services and service data in Los Angeles (Server: Shared-LAX1) due to my own poor decision making and the lack of testing of the script we use to transfer backups remotely. Upon realisation, affected customers were sent an email describing the incident and an abrupt closure notice for Shared-LAX1. I deeply regret the decisions I made during this incident which caused the loss of data, services and your time and efforts.
Shared-LAX1 was rebuilt from scratch on 10/01/2020 and all “active” services were recreated and account emails were sent. By “active” I mean accounts that had not already received a refund or a replacement service. I will be communicating with these customers directly. The events of this loss made it clear that stricter processes and verifications were needed and I wanted to inform you on some of the things we’re implementing to increase the integrity of our service.
BACKUP VERIFICATION: After this unfortunate event we immediately verified the backup script for all other locations. Servers run an automatic task twice per day which backs up website, SQL and email data for active (non suspended) accounts. The backup files are compressed and transferred to a remote server every 24 hours.
DRIVE SNAPSHOTS: The majority of our hosting services are provided using “Cloud” infrastructure. We lease portions of hardware from respectable network providers around the world, enabling us more freedom in our ability to expand our operations. Many of these providers offer the ability to automatically or manually create snapshots of the entire volume drive in case of a system disaster and store them independently. This disaster recovery option has been enabled for services in Los Angeles & Germany.
DATA REDUNDANCY: We have deployed additional servers to act as mirrors of our backup data. A server in Las Vegas contains mirrored copies of USA shared hosting data and a server in Luxembourg contains mirrored copies of EUR shared hosting data.
SQL BACKUPS: Seperate to user backups, each server also contains SQL database backups which are also compressed and remotely transferred every 24 hours.
Understandably there is no more room for mistakes and I will be continuously working on improving our infrastructure so that this can never happen again.
For users who have had a service re-provisioned, you are still welcome to a refund if you wish to cancel. Additionally, all LAX services may be extended by one year upon request.
I once again sincerely apologise for my actions and understand that nothing can truly fix this but if there's anything I can help you with please let me know.
@SmallWeb said:
Some great advice and lessons to be learned. Thank you all. See you in Serbia @bikegremlin
@isunbejo By the looks of it that was the £11.11 for 10 years promotion. All plans have been converted to their yearly equivalent.
I just wanted to post the aftermath email that should currently be on the way to affected customers.
On 09/01/2020 we suffered a catastrophic data loss of all services and service data in Los Angeles (Server: Shared-LAX1) due to my own poor decision making and the lack of testing of the script we use to transfer backups remotely. Upon realisation, affected customers were sent an email describing the incident and an abrupt closure notice for Shared-LAX1. I deeply regret the decisions I made during this incident which caused the loss of data, services and your time and efforts.
Shared-LAX1 was rebuilt from scratch on 10/01/2020 and all “active” services were recreated and account emails were sent. By “active” I mean accounts that had not already received a refund or a replacement service. I will be communicating with these customers directly. The events of this loss made it clear that stricter processes and verifications were needed and I wanted to inform you on some of the things we’re implementing to increase the integrity of our service.
BACKUP VERIFICATION: After this unfortunate event we immediately verified the backup script for all other locations. Servers run an automatic task twice per day which backs up website, SQL and email data for active (non suspended) accounts. The backup files are compressed and transferred to a remote server every 24 hours.
DRIVE SNAPSHOTS: The majority of our hosting services are provided using “Cloud” infrastructure. We lease portions of hardware from respectable network providers around the world, enabling us more freedom in our ability to expand our operations. Many of these providers offer the ability to automatically or manually create snapshots of the entire volume drive in case of a system disaster and store them independently. This disaster recovery option has been enabled for services in Los Angeles & Germany.
DATA REDUNDANCY: We have deployed additional servers to act as mirrors of our backup data. A server in Las Vegas contains mirrored copies of USA shared hosting data and a server in Luxembourg contains mirrored copies of EUR shared hosting data.
SQL BACKUPS: Seperate to user backups, each server also contains SQL database backups which are also compressed and remotely transferred every 24 hours.
Understandably there is no more room for mistakes and I will be continuously working on improving our infrastructure so that this can never happen again.
For users who have had a service re-provisioned, you are still welcome to a refund if you wish to cancel. Additionally, all LAX services may be extended by one year upon request.
I once again sincerely apologise for my actions and understand that nothing can truly fix this but if there's anything I can help you with please let me know.
Regards,
Michael from SmallWeb Ltd
Thank you.
No problem, my data backup still exists in a safe place.
Lax Smallweb is up. Reborn though not reincarnated.
From mail received:
We've been busy cooking up your recent SmallWeb order and it's now active! Here's what you need to know.
Login details yada yadaa
This is good for anyone that uses the WordPress UpdraftPlus plugin for backups. I reckon that this is the most popular (perhaps not the "best') backup plugin, but it only restores to the original host. If a host deadpools, the UpdraftPlus backup, stored at Dropbox, Google Drive, etc is useless. Only the paid version supports migration between hosts.
So good that LA is back up, and I hope that UpdraftPlus restores work for those impacted. I use the NYC datacenter and working great.
Lax Smallweb is up. Reborn though not reincarnated.
From mail received:
We've been busy cooking up your recent SmallWeb order and it's now active! Here's what you need to know.
Login details yada yadaa
This is good for anyone that uses the WordPress UpdraftPlus plugin for backups. I reckon that this is the most popular (perhaps not the "best') backup plugin, but it only restores to the original host. If a host deadpools, the UpdraftPlus backup, stored at Dropbox, Google Drive, etc is useless. Only the paid version supports migration between hosts.
So good that LA is back up, and I hope that UpdraftPlus restores work for those impacted. I use the NYC datacenter and working great.
Lax Smallweb is up. Reborn though not reincarnated.
From mail received:
We've been busy cooking up your recent SmallWeb order and it's now active! Here's what you need to know.
Login details yada yadaa
This is good for anyone that uses the WordPress UpdraftPlus plugin for backups. I reckon that this is the most popular (perhaps not the "best') backup plugin, but it only restores to the original host. If a host deadpools, the UpdraftPlus backup, stored at Dropbox, Google Drive, etc is useless. Only the paid version supports migration between hosts.
So good that LA is back up, and I hope that UpdraftPlus restores work for those impacted. I use the NYC datacenter and working great.
What is your best WP backup scenario
I consider using a plugin for backup a poor solution. Not expecting an application to back itself up.
Prefer to set backups with the hosting server - to a remote storage of choice.
Static site generators are better because all your contents are safely on GitHub before it even reaches your hosting server. Your local git clone and the GitHub repository are backups of each other.
Don't forget to commit NGINX configure file into git as well.
If your host deadpools, rebuild the site from git. It only takes 30 minutes.
Static site generators are better because all your contents are safely on GitHub before it even reaches your hosting server. Your local git clone and the GitHub repository are backups of each other.
Don't forget to commit NGINX configure file into git as well.
If your host deadpools, rebuild the site from git. It only takes 30 minutes.
It depends on the use case - type of site, user's knowledge, experience...
@bikegremlin said:
It depends on the use case - type of site, user's knowledge, experience...
@Clouvider said:
Yeah, I can easily imagine a newbie figuring out how to commit and clone for more than 30 minutes ;-)
When I build a site for a client, I let them email me every article they want to post, and then charge them per thousand characters and per picture upload. If they want pictures inserted in the middle of article instead of at the end, it's a surcharge.
This was 2006. There was no git. I login to my CMS and paste the Word file, then insert each picture. The CMS generates static HTML files, and uploads them to another FTP server (the web server of another client) for backup.
The client could use this CMS themselves, but they'd rather pay the per article charge.
I'm sure they aren't using this CMS anymore. I wrote the CMS and sold them a 10-year license that expired in 2018.
Comments
My personal point of view (not sure if it's shared by most customers, or not):
1)
Only those who never do anything make no mistakes. The difference between the good and the bad providers is whether mistakes are something that happens regularly, or a rare exception. Also, it matters if measures are taken to prevent the same mistakes from re-occuring.
For some reason, many people / companies fail to test restoring from their backups. Similar goes for other safety measures. To me it's like cancelling an emergency evacuation exercise because of rain.
It's better learning from other people's mistakes, than from one's own, but only the really stubborn people don't learn from their own mistakes at least.
2)
Again, in my point of view: longer downtime is best announced, though explaining the reasons once it's already started is also fair.
Either way: it's better to have a longer downtime on occasions, knowing that things are fixed "systematically", for better functioning in the long run - than it is to have downtime (and/or data-loss) every now and then.
3)
Emergencies are wars, disasters, violent crimes - in IT, there's only poor planning - generally speaking.
Customers who need (near) 100% uptime should have failover website deployments in more than one geo-location. And backups - that's always necessary.
Still, many (most?) people are not reasonable. They don't want to spend the money it takes for the above mentioned, but they want to be convinced that a provider (whichever they choose) will provide 99.999% uptime.
One thing is having a TOS that says that backups are a convenience but no guarantees/responsibility are taken. With explanation that going below 99.99% gives a few dollars compensation (depending on the downtime duration), up to the price of monthly subscription.
Another thing is stating that clearly, on the front / sign up page. Don't think one would sell a lot of accounts with such policy.
Which all results in expectations too high.
4)
Amount of money spent on hosting makes no difference and is no guarantee. Not really.
Educational situation with highly reputable and rather expensive MDDhosting (TLDR: all servers down and all backups messed up, week of downtime):
https://forums.mddhosting.com/topic/1582-major-outage-092118-09242018/
Gitlabs experience with not having tested restoration from backups prior to trying it in production, when it was needed:
https://about.gitlab.com/blog/2017/02/10/postmortem-of-database-outage-of-january-31/
As a customer - I'm aware that I shouldn't expect really good service for dirt cheap. Aware that many people promise many things, not taking everyone's word for granted.
As a service provider - regardless of the price, I would require myself to offer what's promised. Think in the long run it's all about reputation and trust. Would probably never get rich, but if I were to sell crap (not claiming, nor implying that Smallweb is doing that), I'd state it up front. MXroute email provider is a great example of that. Their website clearly states, up front, something along the lines (paraphrasing from memory): "we offer cheap service that works well, BUT we cut the costs on support - you won't get much help with the basic stuff and for most questions will be lead to a community support forum, if you want more hand-holding, pay a lot more with other providers". That's fair and straight forward from the start. Though I'm sure it turns many customers away initially, the customers that do take the service don't get disappointed, quite the contrary. They could have promised all bells and whistles on the front page, but it would have backfired (one disappointed customer review can hardly make up for 1000 good reviews).
Whoever has read this till the end - come to Serbia, you've earned a bottle of home-made rakija to re-set your brains...
BikeGremlin I/O
Mostly WordPress ™
Lax Smallweb is up. Reborn though not reincarnated.
From mail received:
Login details yada yadaa
VPS reviews | | MicroLXC | English is my nth language.
@yoursunny and @bikegremlin thanks for adding those thoughtful comments!
And
for everyone who thought twice and decided to save the potential drive-by shitposts for some other thread. I think the real lowend spirit may shine a bit brighter here for your forbearance. 
@SmallWeb I hope this serves to confirm a solid level of support from your customers and colleagues alike. Re5pekt! Maybe, just maybe you have earned enough of it already by "paying it forward" - I'm not sure exactly how, but that does seem to be what we are seeing here.
As a startup, and as a solo practitioner, you may often feel like you're way out on a limb over thin ice without a net. So when something goes wrong, a sense of urgency verging on panic can easily lead down a slippery slope to an avalanche of errors. If you know in those situations that you can reach out for advice - technical and otherwise - then that's a much better place to be in.
(I'm about to murder more metaphors but will stop short of suggesting to take a deep breath when the shit hits the fan.)
Really glad to see the Los Angeles location is now rising from the ashes. Will happily leave you to it now. Cheers!
HS4LIFE (+ (* 3 4) (* 5 6))
Suggest to make backups a beta feature , free out of goodwill, instead of part of advertised service. It will save Ur own ass and manage £1 expectations as well .
I bench YABS 24/7/365 unless it's a leap year.
Hosting Package: ½GB
Domain: *****
Package Price: £3.00 / Year
from £11.00 / Year to £3.00 / Year
@MikePT this correct ?
Thank you
@SmallWeb :-)
VPS reviews | | MicroLXC | English is my nth language.
typo
on single day 11.11 does anyone remember the promo ?
is this for 1 year? how many years for package ½GB, sorry I forgot
https://www.lowendtalk.com/discussion/comment/3040912/#Comment_3040912
https://www.lowendtalk.com/discussion/comment/3040916/#Comment_3040916
https://www.lowendtalk.com/discussion/comment/3041003/#Comment_3041003
I think these were the two three offers.
p.s: I read the entire thread and wow, some good offers in that.
VPS reviews | | MicroLXC | English is my nth language.
Wow amazing, for package ½GB (£1.11/Year ) = 10 year (£11.11),
Thank you
Some great advice and lessons to be learned. Thank you all. See you in Serbia @bikegremlin
@isunbejo By the looks of it that was the £11.11 for 10 years promotion. All plans have been converted to their yearly equivalent.
I just wanted to post the aftermath email that should currently be on the way to affected customers.
Thank you.
Michael
No problem, my data backup still exists in a safe place.
production -> ftp / rsync <-storage box -> gdrive -> myhome storage
now it has been restored and running well
Thanks.
I just want to thank all of the members of the forum who reached out to help Smallweb. That's what makes a community work and LES a nice place to be.
.> @vyas said:
This is good for anyone that uses the WordPress UpdraftPlus plugin for backups. I reckon that this is the most popular (perhaps not the "best') backup plugin, but it only restores to the original host. If a host deadpools, the UpdraftPlus backup, stored at Dropbox, Google Drive, etc is useless. Only the paid version supports migration between hosts.
So good that LA is back up, and I hope that UpdraftPlus restores work for those impacted. I use the NYC datacenter and working great.
What is your best WP backup scenario
@SmallWeb proud to see you back mate, I knew you'd do it.
I consider using a plugin for backup a poor solution. Not expecting an application to back itself up.
Prefer to set backups with the hosting server - to a remote storage of choice.
DirectAdmin backup.
cPanel backup with JetBackup (to supercheap Backblaze B2 storage).
BikeGremlin I/O
Mostly WordPress ™
Don't use WordPress.
Static site generators are better because all your contents are safely on GitHub before it even reaches your hosting server. Your local git clone and the GitHub repository are backups of each other.
Don't forget to commit NGINX configure file into git as well.
If your host deadpools, rebuild the site from git. It only takes 30 minutes.
It depends on the use case - type of site, user's knowledge, experience...
BikeGremlin I/O
Mostly WordPress ™
Yeah, I can easily imagine a newbie figuring out how to commit and clone for more than 30 minutes ;-)
Clouvider Limited - True HA KVM OnApp VPS from £9.50/m - Our LES Exclusive Unmetered Dedicated Server Offers
When I build a site for a client, I let them email me every article they want to post, and then charge them per thousand characters and per picture upload. If they want pictures inserted in the middle of article instead of at the end, it's a surcharge.
This was 2006. There was no git. I login to my CMS and paste the Word file, then insert each picture. The CMS generates static HTML files, and uploads them to another FTP server (the web server of another client) for backup.
The client could use this CMS themselves, but they'd rather pay the per article charge.
I'm sure they aren't using this CMS anymore. I wrote the CMS and sold them a 10-year license that expired in 2018.
There are two kinds of people: those who have broken prod and those who haven't yet.