Dear Plex User,
We want you to be aware of an incident involving your Plex account information yesterday. While we believe the actual impact of this incident is limited, we want to ensure you have the right information and tools to keep your account secure.
What happened
Yesterday, we discovered suspicious activity on one of our databases. We immediately began an investigation and it does appear that a third-party was able to access a limited subset of data that includes emails, usernames, and encrypted passwords. Even though all account passwords that could have been accessed were hashed and secured in accordance with best practices, out of an abundance of caution we are requiring all Plex accounts to have their password reset. Rest assured that credit card and other payment data are not stored on our servers at all and were not vulnerable in this incident.
What we're doing
We've already addressed the method that this third-party employed to gain access to the system, and we're doing additional reviews to ensure that the security of all of our systems is further hardened to prevent future incursions. While the account passwords were secured in accordance with best practices, we're requiring all Plex users to reset their password.
What you can do
Long story short, we kindly request that you reset your Plex account password immediately. When doing so, there's a checkbox to "Sign out connected devices after password change." This will additionally sign out all of your devices (including any Plex Media Server you own) and require you to sign back in with your new password. This is a headache, but we recommend doing so for increased security. We have created a support article with step-by-step instructions on how to reset your password here.
We'd also like to remind you that no one at Plex will ever reach out to you to ask for a password or credit card number over email. For further account protection, we also recommend enabling two-factor authentication on your Plex account if you haven't already done so.
Lastly, we sincerely apologize to you for any inconvenience this situation may cause. We take pride in our security system and want to assure you that we are doing everything we can to swiftly remedy this incident and prevent future incidents from occurring. We are all too aware that third-parties will continue to attempt to infiltrate IT infrastructures around the world, and rest assured we at Plex will never be complacent in hardening our security and defenses.
For step-by-step instructions on how to reset your password, visit: https://support.plex.tv/articles/account-requires-password-reset
Thank you,
The Plex Security Team

just the received this message from Hosthatch:

Hello,

This email concerns your server named host.com

In the late evening hours of yesterday, we were alerted to an issue affecting one of our nodes in Amsterdam.

Unfortunately even with our best efforts, we were not able to restore it to it's previous working state.

We have deployed a new VM for you with the same operating system and IPv4/IPv6 address. We ask that you restore your VM from backups at this time.

For the inconvenience, we will provide compensation equivalent to 3 months of service. This will be automatically added to your account in the next 48 hours.

We sincerely apologize for the inconvenience caused and stand by for any questions that you may have.

Kindest Regards,
Your HostHatch team

I've noticed that my VM went down last night and was in state offline. I could turn it on but it didn't have network connectivity. Data was still accessible, though, so I didn't worry too much. And now this. I didn' really have any important data on it, but setting it up from scratch is a bit annoying. Would have been nice if they could still make the old VM available to transfer any remaining data..

This was a 5TB Black Friday deal from last year. Anybody else affected by this?

Cheers,
Michael

https://techcrunch.com/2022/06/20/cloudflare-outage-knocks-popular-services-offline/amp/

🙄

Annual Domain Transfer for Profit

Since my first domain name in 2006, I have purchased several domain names for my various websites.
A few years ago, I discovered a secret in the domain registration business: many registrars offer a cheaper price for domain transfer than domain renewal, as a means to attract new customers.
Therefore, if I transfer my domain every year to a different registrar, I would pay less than renewing the domain at the same registrar.

DNS services for a domain used to be associated with the registrar.
When I transfer a domain away, the DNS server of the old registrar would stop responding to queries regarding my domain, and the DNS server of the new registrar does not yet have any records about the IP addresses of my web server.
Therefore, a domain transfer would usually cause the website to become inaccessible for a day or two.
Typically, I post a tweet when a domain transfer is about to happen, so that my readers could know why my website is down.

Nowadays, I'm using Cloudflare DNS for most of my domains.
Cloudflare DNS server is independent from the domain registrar, so that my website continues to resolve correctly throughout a domain transfer, as long as neither registrars modify the name server delegation records.
In case the new registrar automatically updates the delegation records to their DNS servers, I have to quickly login to the control panel and change it back to Cloudflare, which would then cause a brief downtime of the website.
Having done so for many years, I am accustomed to this process.

Transfer of ndn.today

I registered ndn.today domain name in 2020, to host several of my personal projects related to Named Data Networking, which include the popular NDN push-ups page.
Later that year, I transferred this domain from NameCheap to Porkbun.
After entering the Auth-Code at the new registrar and accepting the transfer request at the old registrar, the domain moved over, and Cloudflare continues to resolve the domain so that there's no website downtime.

Many domain registrars offer free WHOIS privacy services, which conceal my name, street address, and email from the public WHOIS database.
I do not consider WHOIS privacy essential because my information is public, but I kept it enabled so that I could receive fewer spam email.
During the above domain transfer, WHOIS information changed from Withheld for Privacy ehf to Private by Design, LLC, which are the WHOIS privacy services of NameCheap and Porkbun respectively.
Despite the change, I am still the domain owner as recorded in the registrar's control panel.

Fast forward to March 30, 2022, it's less than two months before the expiration date of ndn.today, so it's time to yo-yo the domain again.
Tldes.com indicates that one.com has the cheapest domain transfer offer for .today at $2.22, and they are an IANA-accredited registrar.
Following the usual procedure, I unlocked the domain, entered the Auth-Code, paid the invoice, and accepted the transfer request.
The domain moved over, and I went to bed.

The next morning, I received an alert:

Hi,

The monitor web RSpec (https://rspec.ndn.today) is currently DOWN (Connection Timeout).

UptimeRobot will alert you when it is back up.

Usually, such an alert indicates a problem with the hosting server.
I SSH'ed into the server, but did not find any issues.
Since I was busy with coding that day, I ignored the alert thinking it would resolve by itself in a few hours.

In the evening of March 31, I recalled this UptimeRobot alert, and discovered that I cannot open the NDN push-ups website anymore.

I started to realize, the alert was caused by a DNS problem, not a server issue.
I queried the domain with two online WHOIS lookup tools, one shows the name server delegation to be Cloudflare as expected, the other one shows:

Updated Date: 2022-03-31T13:03:53Z
Creation Date: 2020-05-25T19:09:17Z
Registry Expiry Date: 2023-05-25T19:09:17Z
Registrar: One.com A/S
Registrar IANA ID: 1462
Domain Status: transferPeriod https://icann.org/epp#transferPeriod
Registry Registrant ID: REDACTED FOR PRIVACY
Registrant Name: REDACTED FOR PRIVACY
Registrant Organization: Private by Design, LLC
Name Server: ns01.one.com
Name Server: ns02.one.com
DNSSEC: signedDelegation

Apparently, one.com automatically changed the name server delegation to their own.
Since I haven't entered any DNS records in one.com DNS control panel, their DNS server cannot resolve my website IP address.

Domain Owner: Private by Design, LLC

I haven't encountered an automatic name server update since 2014, but I know how to deal with it: login to the new registrar's control panel, and change the name server back to Cloudflare.
I filled up the form, clicked submit, and then received a message:

You have name server changes pending approval.
A request has been sent to the registrant of the domain for approval.
If not approved within three days, the request will fail.

Looking through my email inbox, I received nothing.
After poking around, I found the issue: according to one.com, the domain is now owned by Private by Design, LLC, Porkbun's WHOIS privacy service.
Therefore, they sent the request to whoisprivacy@porkbun.com, but I have no way to access that mailbox.

I attempted to modify the domain contact.
However, that form also triggers an automated email to whoisprivacy@porkbun.com that I cannot access.
I contacted Porkbun chat support, and they told me the validation code contained in one of the emails.
I submitted this validation code at one.com, but it did not allow me to immediately update the domain contact.

Road to Recovery

One.com is the current registrar and they have the authority and responsibility to resolve the problem.
I contacted their chat support, Support Robot gave a 7-minute estimate for a support agent, but nobody picked up after 15 minutes.
I tried again after a few hours and got to someone, who asked me to fill up a general owner change PDF form and submit by email.
Seeing that the form should be accompanied by a photo ID such as my driver's license, I felt uneasy to send sensitive documents via unencrypted email.
Upon raising my concern, the support agent Angel said that I can also use the ticket system, much better!

I prepared and signed the form, scanned my driver's license, and submitted both via the ticket system.
The same Angel replied to the ticket, saying that she's going to forward the form to their Hostmasters, who would process the form within 24 hours.
Sigh.

The next morning, on April 1, I received another email:

Owner details update for domain ndn.today

We have declined your request of Mar 31, 2022 to update the owner details for your domain ndn.today.

I started to panic PMS: I submitted the PDF form according to instructions, and why is it declined?
A different chat support agent checked the status of my ticket, and assured me that the PDF form is still being processed, and the decline message came from my earlier attempt in the control panel.

DNSSEC Thwarts the Temporary Measure

As I'm waiting for the hostmaster to process the "owner change", I thought about a temporary solution: I can enter server IP addresses in the control panel, so that one.com DNS server could resolve the records.
I have been using Cloudflare CDN for some of my sites under this domain, mainly for the convenience of TLS termination.
Moving them off the CDN means that I have to do TLS termination on my own server, but I'm experienced with this: Caddy server has automatic TLS and can obtain certificates automatically.

I inserted A and AAAA records in one.com DNS control panel, and configured reverse proxies in the Caddyfile:

https://pushups.ndn.today {
  reverse_proxy https://pushups.netlify.app
}

Certificate Transparency notifications started rolling in, suggesting that my reverse proxy has successfully obtained TLS certificates for the subdomains.
I couldn't access the websites myself at that time, but I'm confident that DNS propagation delay would eventually resolve itself.
Certificates are issued by ZeroSSL instead of the usual Let's Encrypt, but it's not a matter of concern.

It's now April 2, two days after the initial UptimeRobot alert, I still cannot access my websites.
I queried my domain on DNS lookup & Propagation Check, and it gave mixed results: some DNS servers can resolve the domain and some cannot.
While DNS does have negative caching, such caching is normally short-lived, so that DNS caching is not the only one to blame.

The reason of failed DNS resolution lies in DNSSEC.
Following a recommendation from Cloudflare, I enabled DNSSEC for my domain, so that unauthorized DNS servers cannot respond to queries with bogus responses.
Setting up DNSSEC is a two-step process:

1. Cloudflare DNS server generated a signing key pair, and would use it to sign every response under my domain.
2. Using a form on the Porkbun control panel, I submitted the digest of the public key to the .today registry, in what's called a DS record.

When a DNS resolver receives the DS record, it would retrieve the public key from the delegated name server (i.e. Cloudflare), and check that the public key matches the digest and the records are signed by this public key.
Right now, the name server delegation is pointing to ns01.one.com, but the DS record contains the digest of Cloudflare's public key.
Since one.com does not own the corresponding private key, it would not be able to come up with a valid signature.
Consequently, DNS resolvers would reject its responses and refuse to resolve my domain.

VeriSign Labs DNSSEC Analyzer confirms my suspicion:

Not every DNS resolver validates DNSSEC signatures, which explains why some DNS servers can resolve the domain and some cannot.
I checked Caddy server logs, and it suggests that Let's Encrypt could not issue certificates due to DNS resolution failure, so that Caddy automatically switched to ZeroSSL as a fallback.
This implies that Let's Encryption is using a DNSSEC-validating resolver, while ZeroSSL is using a non-validating resolver.

Now, my domain is in limbo.
If a viewer is using a non-validating DNS resolver, they can visit my website and see my push-ups.
If a viewer is using a DNSSEC-validating resolver, there would be no push-ups for them.

Summary & Final Words

I regularly transfer domains between registrars to take advantage of lower pricing.
This time, I transferred a domain from Porkbun to one.com without disabling WHOIS privacy service.
The new registrar considered Private by Design, LLC, Porkbun's WHOIS privacy service, to be the domain owner, and restricted me from accessing most features in the control panel.
Paperwork for re-assigning the domain owner to me is still being processed after two days.

The new registrar automatically changed name server delegation to their own.
I inserted DNS records to one.com DNS server as a temporary measure, but these records are being rejected by DNSSEC-validating resolvers because one.com does not possess the signing keys.
Consequently, ndn.today domain has been inaccessible for three days and counting, and half of the world population is unable to watch my push-ups.

UPDATE 2022-04-05:
Shortly after this article was first published, one.com approved the domain owner change request and re-assigned the domain under my own name.
I updated the name server back to Cloudflare right away, and my websites are fully recovered as of this writing.
According to UptimeRobot, total downtime was about 77 hours and 49 minutes.
Needless to say, I wasted so much time trying to fix my domain, the time that I could otherwise spend doing push-ups.

@hosthatch - it seems there is some weird error making the storage VPS not boot.

Error displayed at boot: "XZ-compressed data is corrupt. System halted."

The hubiC offers as they currently exist will stop soon. We want to thank you for your trust and loyalty during these past several years and hope that hubiC has given you a satisfying experience.
A new service will soon emerge, resolutely enriched and turned toward the future, with different conditions.
At this time, we need to know if you want to keep your account until the closing of the service, or if we can delete it now.
[...]
The hubiC teams thank you once again for your trust and loyalty.

I don't have any important thing there, just curious about what happens

edit location

No one responds to support tickets

Online chat says they are online but they never answer

Leaving a message for online chat and pressing Send shows: SMTP invalid

Trying to boot the VPS from the portal says: Command failed, create support ticket

The network status always says no issues but this has happened many times in the last months, but then it starts to work again.

Is there any issues or information about Oneil Online??? They were advertised on LowEndBox.

Customer websites are down.
Emails are not delivering.

"What's absolutely nuts about this is we can not change our NameServers due to eNom itself is down (going on 24 hours now). Never in my life have I been in a situation where I can't route around network issue at the root domain records level. We are totally trapped!"
eNom customer

One of the affected websites is Certitudes, a geocaching puzzle checker, which creates great difficulty for geocachers around the world.

@martijnk could you let us know what is going on?

It was NOT a BGP error , it was a node error with Hetzner where this forum is hosted.
All should be back to normal again.

Hope Dejan pulls it back up soon~~

Update : Fixed and working perfectly now.

@cociu said:

Guys , a little update : Today the police was come to our (still here) DC and powered off our main router due of some infantil videochat activity. This is totaly prohibided here always i have specified this but seems some "smart" guy was escapate with this. NOTE we have not received any complain of this issue before of powerdown or any other notification. I have talk with the inspector and only the interest is this client. Working out with the to identif the server and isolated in order to put all back online. No eta for the moment. After all this shit we will see how we can compenste this mess. Really sorry.

EDIT : we are working in order to put at least our website asap online to have a comunication with our clients.

I am experienced cant open soluz cp @ https://vpsadmin.inceptionhosting.com/, anyone has the same issue? sorry if not in the right forum categories

Client Area occasionally throwing Error 524. I don't see anything on Network Status.

Tagging
@seriesn

https://www.bleepingcomputer.com/news/security/spamcop-anti-spam-service-suffers-an-outage-after-its-domain-expired/

https://www.cloudflarestatus.com/

" Cloudflare has identified the root cause as related to a database replication issue due to a configuration error. It was not related to any outside event or traffic. We are working on fully restoring the database cluster. In the interim, we have restored functionality to most services with replicas but users will see increased latency and intermittent issues in the dashboard that in most cases will be addressed with retry. We will provide another update in 60 minutes."

My tom@srvr.ovh email address seems to be able to receive but not to send.

I got hit with a Paypal dispute because a new customer didn't receive the emails Roundcube reported successfully sending.

Since I do not have access to the logs on the OVH MTA, I can't directly see what was happening.

Actually it's a lot of fun to run my own mailserver for my non-OVH domains. It's nice to be able to see all the headers and all the logs and to mess with the configurations.

I'm going just to wait a bit, since OVH will fix the issue. Meanwhile, please send me a PM here if you need to be in touch.

srvr.ovh servers seem unaffected. It seems to be just OVH hosted email that's down.

Thanks!

https://www.cloudflarestatus.com/

EDIT: analysis after it was resolved:
https://blog.cloudflare.com/analysis-of-todays-centurylink-level-3-outage/

Also after bringing it back up via the control panel, proxy domains aren't working. Deleted my domain and added it again, still not working.

Private ip is 102.11

Anybody have the same issue?

Looks like nearly entire Europe is affected. I will laugh if it turns out to be a clock overflow in their software. Anyone willing to shed some light? Maybe how it affected you, especially you, dear providers?

https://app.fing.com/internet/outages

contacts while being down: mail inbox@mr VM . net (except the spaces)

So I checked with some online tools to see if it is just for me or is worldwide.

So, no chance to be tempted to buy something

As per the notification email:

Dear Customer,

Earlier today the server hosting Los Angeles website data suffered a networking failure which prevented it from connecting to a network and performing standard tasks. Attempts to restore the network were made but ultimately after the server being offline for such a prolonged period of time we attempted to re-install the server and possibly migrate backups elsewhere.

However it was discovered that our system backups for this location were not suitable to be restored and thus unfortunately we are announcing the immediate closure of Shared-LAX1.

I am extremely disappointed in myself for having to deliver this news to you and this is not how I wanted 2020 to start but I would like to learn from this experience going forward to improve our other available locations.

Because we can no longer deliver your service you are entitled to a full refund. To claim a refund, please respond to this email or open a support ticket. Your refund will be processed within 7 days and your LAX service will no longer be accessible through your client area.

If you have any questions, please let me know. I am extremely sorry for this disastrous event.

Regards,

Michael

SmallWeb Ltd.

Apparently a typo in a backup script - and not enough testing to catch it.

Seems like a painful lesson learned - hopefully not too many bridges burned.

I'm personally more inclined to offer my condolences etc rather than get too bent out of shape over a glitch in a £5 per year service. And I can't help but appreciate the generally direct and transparent communication from @Mic-hael so far (compare with for example a similar situation at Gandi) ... Hopefully we'll enjoy more supportive discussion and suggestions for end-user backup best practices here. (Or whatever.)

Might also suggest service in Melbourne may be an interesting alternative for some customers, if that's a viable option. Though I'll miss the LA location - life goes on ...