vps is compromised

seenuseenu
edited July 2021 in Help

Hello,

i have a vps which i am not able to ssh, when i use vnc and change ssh config and login with password...
i see that ~/.ssh/authorized_keys doesn't have my keys but someone else's key (Misha@asus-laptop) ....so i realize someone hacked into vps.

i have some data in it so i don't want to wipe it off and start from scratch again and also this is an opportunity for me to learn about security etc things. (if needed i can wipe it off and restore from backup)

It had some staging wordpress site which i deleted just now

and looking at cat \etc\passwd shows

root:x:0:0:root:/root:/usr/bin/zsh
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
sync:x:4:65534:sync:/bin:/bin/sync
games:x:5:60:games:/usr/games:/usr/sbin/nologin
man:x:6:12:man:/var/cache/man:/usr/sbin/nologin
lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin
mail:x:8:8:mail:/var/mail:/usr/sbin/nologin
news:x:9:9:news:/var/spool/news:/usr/sbin/nologin
uucp:x:10:10:uucp:/var/spool/uucp:/usr/sbin/nologin
proxy:x:13:13:proxy:/bin:/usr/sbin/nologin
www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin
backup:x:34:34:backup:/var/backups:/usr/sbin/nologin
list:x:38:38:Mailing List Manager:/var/list:/usr/sbin/nologin
irc:x:39:39:ircd:/var/run/ircd:/usr/sbin/nologin
gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/usr/sbin/nologin
nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin
systemd-network:x:100:102:systemd Network Management,,,:/run/systemd:/usr/sbin/nologin
systemd-resolve:x:101:103:systemd Resolver,,,:/run/systemd:/usr/sbin/nologin
systemd-timesync:x:102:104:systemd Time Synchronization,,,:/run/systemd:/usr/sbin/nologin
messagebus:x:103:106::/nonexistent:/usr/sbin/nologin
syslog:x:104:110::/home/syslog:/usr/sbin/nologin
_apt:x:105:65534::/nonexistent:/usr/sbin/nologin
tss:x:106:111:TPM software stack,,,:/var/lib/tpm:/bin/false
uuidd:x:107:112::/run/uuidd:/usr/sbin/nologin
tcpdump:x:108:113::/nonexistent:/usr/sbin/nologin
landscape:x:109:115::/var/lib/landscape:/usr/sbin/nologin
pollinate:x:110:1::/var/cache/pollinate:/bin/false
sshd:x:111:65534::/run/sshd:/usr/sbin/nologin
systemd-coredump:x:999:999:systemd Core Dumper:/:/usr/sbin/nologin
lxd:x:998:100::/var/snap/lxd/common/lxd:/bin/false
gitlab-runner:x:997:997:GitLab Runner:/home/gitlab-runner:/bin/bash
mysql:x:112:120:MySQL Server,,,:/nonexistent:/bin/false
lsadm:x:996:996:lsadm:/:/sbin/nologin
redis:x:113:121::/var/lib/redis:/usr/sbin/nologin

and i don't feel all those users list belongs to me.

Also, what other folders/files i should check to confirm vps is compromised?

thank you so much.

ThemeForest Profile

Comments

  • chocolateshirtchocolateshirt OG

    try using last -a command, if the users and ip address different from yours then you could say it compromised.

    ⭕ A simple uptime dashboard using UptimeRobot API https://upy.duo.ovh
    ⭕ Currently using VPS from BuyVM, GreenCloudVPS, Gullo's, Hetzner, HostHatch, InceptionHosting, LetBox, MaxKVM, MrVM, VirMach.

  • chimichurrichimichurri

    alternatively, assuming it was a low end hacker (that did not care to cover up the traces left behind), you could try taking a look at /var/log/syslog and/or /var/log/secure (grep for ssh or sshd) to see if there were any suspicious ssh connections

    Thanked by (1)ehab

    Contribute your idling VPS/dedi (link), Android (link) or iOS (link) devices to medical research

  • TheDPTheDP OG

    Also check /etc/shadow to see if there are accounts with passwords set.

    Thanked by (1)Ympker

    For domain registrations, create an account at Dynadot (ref) and spend $9.99 within 48 hours to receive $5 DynaDollars!
    Looking for cost-effective Managed/Anycast/DDoS-Protected/Geo DNS Services? Try ClouDNS (aff).

  • seenuseenu

    Thanks everyone for your response...highly appreciated :)

    @chocolateshirt said:
    try using last -a command, if the users and ip address different from yours then you could say it compromised.

    it shows ips mostly from my isp range.

    @chimichurri said: alternatively, assuming it was a low end hacker (that did not care to cover up the traces left behind), you could try taking a look at /var/log/syslog and/or /var/log/secure (grep for ssh or sshd) to see if there were any suspicious ssh connections

    first command shows full system log and second file doesn't exists

    @thedp said: Also check /etc/shadow to see if there are accounts with passwords set.

    show this

    root:$6$gTWfJcI0IqH8EKMO$sfsd.modified.puI3UWI2OM2gkeZHCG.KFEZaij0:18821:0:99999:7:::
daemon:*:18375:0:99999:7:::
bin:*:18375:0:99999:7:::
sys:*:18375:0:99999:7:::
sync:*:18375:0:99999:7:::
games:*:18375:0:99999:7:::
man:*:18375:0:99999:7:::
lp:*:18375:0:99999:7:::
mail:*:18375:0:99999:7:::
news:*:18375:0:99999:7:::
uucp:*:18375:0:99999:7:::
proxy:*:18375:0:99999:7:::
www-data:*:18375:0:99999:7:::
backup:*:18375:0:99999:7:::
list:*:18375:0:99999:7:::
irc:*:18375:0:99999:7:::
gnats:*:18375:0:99999:7:::
nobody:*:18375:0:99999:7:::
systemd-network:*:18375:0:99999:7:::
systemd-resolve:*:18375:0:99999:7:::
systemd-timesync:*:18375:0:99999:7:::
messagebus:*:18375:0:99999:7:::
syslog:*:18375:0:99999:7:::
_apt:*:18375:0:99999:7:::
tss:*:18375:0:99999:7:::
uuidd:*:18375:0:99999:7:::
tcpdump:*:18375:0:99999:7:::
landscape:*:18375:0:99999:7:::
pollinate:*:18375:0:99999:7:::
sshd:*:18475:0:99999:7:::
systemd-coredump:!!:18475::::::
lxd:!:18475::::::
gitlab-runner:!:18596::::::
mysql:!:18598:0:99999:7:::
lsadm:!:18609::::::
redis:*:18609:0:99999:7:::

    thanks.

    ThemeForest Profile

  • AlwaysSkintAlwaysSkint OG

    history | less

    lowendinfo.com had no interest.

  • seenuseenu

    @AlwaysSkint said: history | less

    Thanks for that command, learned something new :)

    i have gone through whole history...kinda interesting to see what i have done :D

    i don't see anything suspicious.

    i doubt if that key is from provider (wishosting) they moved vps from one DC to another one last time.

    so they might put their key.

    ThemeForest Profile

  • chimichurrichimichurri
    edited July 2021

    Hmmm..
    So if there was no /var/log/secure I guess it's not a CentOS - how about /var/log/auth.log? (assuming it's Debian or Ubuntu)
    Also, were there no traces of ssh connections inside the syslog? (I was expecting something like this: https://www.ibm.com/support/pages/sshd-can-use-unix-syslog-facilities-logging )

    You could also try finding files (using the last modified/accessed filters) - this perhaps could also show some suspicious activity (assuming you're aware of what's usually happening on the server) - command examples: https://www.cyberciti.biz/faq/howto-finding-files-by-date/

    Contribute your idling VPS/dedi (link), Android (link) or iOS (link) devices to medical research

  • MulderMulder

    Misha = Michael so that's probably wishosting's keys

  • seenuseenu

    @chimichurri said:
    Hmmm..
    So if there was no /var/log/secure I guess it's not a CentOS - how about /var/log/auth.log? (assuming it's Debian or Ubuntu)
    Also, were there no traces of ssh connections inside the syslog? (I was expecting something like this: https://www.ibm.com/support/pages/sshd-can-use-unix-syslog-facilities-logging )

    oops sorry, i should have mentioned it is ubuntu 20

    /var/log/auth.log has too much logs even at this minute...someone trying with random ports with failed password

    the contents are too big, i am just pasting recent log

    Jul 13 23:53:28 ubuntu sshd[5974]: Failed password for root from 111.19.157.70 port 59632 ssh2
Jul 13 23:53:29 ubuntu sshd[5974]: Received disconnect from 111.19.157.70 port 59632:11: Bye Bye [preauth]
Jul 13 23:53:29 ubuntu sshd[5974]: Disconnected from authenticating user root 111.19.157.70 port 59632 [preauth]
Jul 13 23:54:56 ubuntu sshd[5984]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=124.47.36.58  user=root
Jul 13 23:54:58 ubuntu sshd[5984]: Failed password for root from 124.47.36.58 port 56496 ssh2
Jul 13 23:55:00 ubuntu sshd[5984]: Received disconnect from 124.47.36.58 port 56496:11: Bye Bye [preauth]
Jul 13 23:55:00 ubuntu sshd[5984]: Disconnected from authenticating user root 124.47.36.58 port 56496 [preauth]
Jul 13 23:55:01 ubuntu CRON[5986]: pam_unix(cron:session): session opened for user root by (uid=0)
Jul 13 23:55:01 ubuntu CRON[5986]: pam_unix(cron:session): session closed for user root
Jul 13 23:55:18 ubuntu sshd[5989]: Invalid user serverpilot from 165.22.249.19 port 36742
Jul 13 23:55:18 ubuntu sshd[5989]: pam_unix(sshd:auth): check pass; user unknown
Jul 13 23:55:18 ubuntu sshd[5989]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=165.22.249.19
Jul 13 23:55:20 ubuntu sshd[5989]: Failed password for invalid user serverpilot from 165.22.249.19 port 36742 ssh2
Jul 13 23:55:22 ubuntu sshd[5989]: Received disconnect from 165.22.249.19 port 36742:11: Bye Bye [preauth]
Jul 13 23:55:22 ubuntu sshd[5989]: Disconnected from invalid user serverpilot 165.22.249.19 port 36742 [preauth]
Jul 13 23:55:56 ubuntu sshd[5993]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=111.19.157.70  user=root
Jul 13 23:55:59 ubuntu sshd[5993]: Failed password for root from 111.19.157.70 port 22101 ssh2
Jul 13 23:56:00 ubuntu sshd[5993]: Received disconnect from 111.19.157.70 port 22101:11: Bye Bye [preauth]
Jul 13 23:56:00 ubuntu sshd[5993]: Disconnected from authenticating user root 111.19.157.70 port 22101 [preauth]
Jul 13 23:56:06 ubuntu sshd[5995]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=124.47.36.58  user=root
Jul 13 23:56:08 ubuntu sshd[5995]: Failed password for root from 124.47.36.58 port 34526 ssh2
Jul 13 23:56:09 ubuntu sshd[5995]: Received disconnect from 124.47.36.58 port 34526:11: Bye Bye [preauth]
Jul 13 23:56:09 ubuntu sshd[5995]: Disconnected from authenticating user root 124.47.36.58 port 34526 [preauth]

    ThemeForest Profile

  • seenuseenu

    @Mulder said:
    Misha = Michael so that's probably wishosting's keys

    if thats true then i am bit relieved :D

    ThemeForest Profile

  • chimichurrichimichurri
    edited July 2021

    /var/log/auth.log has too much logs even at this minute...someone trying with random ports with failed password

    the contents are too big, i am just pasting recent log

    You're not supposed to just read it :D
    Looks like they (people bruteforcing boxes with password auth) got you - this is definitely too much to go line by line, the last idea I'm able to come up with ATM is to grep the shit out of that ssh log to pin point the user / IP combination that successfully logged in (assuming this indeed was a hack and not the VPS provider as suggested above, lol :P), and then perhaps take a look at /var/log/messages (everything that is in there shortly after the first successful attempt) - although I'm not sure whether you'd be able to find anything meaningful there ><'

    Also maybe try finding files accessed/modified shortly after the attacker's ssh session successfully started (and rinse and repeat for all the times after subsequent logins as root, that were made no longer using the password, but the seemingly newly set ssh key)

    For future reference:

    • if this is a setup you're using frequently (or at least will be using again after the reinstall), and cannot change it to one using ssh keys no matter what, I'd strongly recommend a provider with some kind of software firewall available - so that you limit the incoming connection(s) to trusted IPs only.. Hetzner Cloud is perhaps one of the cheapest options I can think of offering this, you can start and stop the VMs as you wish, hourly billing FTW!; alternatively, at least try setting up the OS firewall (although this is most likely much more of a hassle)
    • for making investigations like this (=getting to know what was modified by the attacker) more efficient next time, consider setting up a file integrity checker like https://aide.github.io/ (or Deep Security, if your pockets are deep ;)
    Thanked by (1)seenu

    Contribute your idling VPS/dedi (link), Android (link) or iOS (link) devices to medical research

  • deankdeank OG
    edited July 2021

    .... When you copy and pasted the log, you should have inspected it?

    ♻ Amitz day is October 21.
    ♻ Join Nigh sect by adopting my avatar. Let us spread the joys of the end.

  • seenuseenu

    @chimichurri said:

    /var/log/auth.log has too much logs even at this minute...someone trying with random ports with failed password

    the contents are too big, i am just pasting recent log

    You're not supposed to just read it :D
    Looks like they (people bruteforcing boxes with password auth) got you - this is definitely too much to go line by line, the last idea I'm able to come up with ATM is to grep the shit out of that ssh log to pin point the user / IP combination that successfully logged in (assuming this indeed was a hack and not the VPS provider as suggested above, lol :P), and then perhaps take a look at /var/log/messages (everything that is in there shortly after the first successful attempt) - although I'm not sure whether you'd be able to find anything meaningful there ><'

    Also maybe try finding files accessed/modified shortly after the attacker's ssh session successfully started (and rinse and repeat for all the times after subsequent logins as root, that were made no longer using the password, but the seemingly newly set ssh key)

    For future reference:

    • if this is a setup you're using frequently (or at least will be using again after the reinstall), and cannot change it to one using ssh keys no matter what, I'd strongly recommend a provider with some kind of software firewall available - so that you limit the incoming connection(s) to trusted IPs only.. Hetzner Cloud is perhaps one of the cheapest options I can think of offering this, you can start and stop the VMs as you wish, hourly billing FTW!; alternatively, at least try setting up the OS firewall (although this is most likely much more of a hassle)
    • for making investigations like this (=getting to know what was modified by the attacker) more efficient next time, consider setting up a file integrity checker like https://aide.github.io/ (or Deep Security, if your pockets are deep ;)

    Thanks for great explanation.

    i have created a ticket to know whether it is really himself/team.

    if its not himself, then i will dig deeper into it,
    definitely learned few things with this incident.

    thanks.

    ThemeForest Profile

  • Not_OlesNot_Oles Retired

    @seenu

    A few random thoughts:

    You say that your key was removed from .ssh/authorized_keys and that an unknown Misha key is present.

    You do not mention anything else which seems really broken or strange. I am not at all an expert, but the files you posted, /etc/passwd and /etc/shadow seem basically sane to me.

    Does the authorized_keys file in your most recent backup have your key?

    Does the authorized_keys file in your most recent backup have the Misha key?

    What is the backup authorized_keys file's modification date?

    What is the modification date on the presently in use .ssh/authorized_keys?

    When is the most recent day and time that you logged in using your key?

    Look at auth.log on the date and prior to the time authorized_keys was last modified.

    I vote for having ssh access firewall-restricted to specific IPs. Make sure to have more than one allowed IP in case you lose access to an allowed IP.

    Honeypots

    Maybe you might be interested in this paper: https://github.com/security-union/going-fishing-with-my-raspberry-pi/blob/master/going_fishing_raspberry_pi.pdf

    The guy who wrote the above paper also has a very cool Youtube video:

    Intrusion Recovery via Selective Re-execution

    From 2010, Intrusion Recovery Using Selective Re-execution (pdf)

    From 2015, Intrusion Recovery Using Selective Re-execution

    Thanked by (2)seenu Falzo

    Tom. 穆坦然. Not Oles. Happy New York City guy visiting Mexico! How is your 文言文?
    The MetalVPS.com website runs very speedily on MicroLXC.net! Thanks to @Neoon!

  • seenuseenu

    thanks for your time @Not_Oles

    unfortunately i don't have a backup of authorized_keys, when i am not able to login...i login through vnc and found that my key is not there so i just removed misha key and placed mine....after few moments...i realized...i should have seen last modified time :trollface:

    I want to wait till i get response from provider before i want to investigate this one further.

    thanks :)

    Thanked by (1)Not_Oles

    ThemeForest Profile

  • seenuseenu

    I am finally relieved.

    Just got a mail from support saying it is their key :D

    Mods... please close this thread.

    thanks everyone, i learned something new in this process.

    Thanked by (2)kuroneko23 Not_Oles

    ThemeForest Profile

  • mikhomikho AdministratorHosting ProviderOG

    Closed per request.

    Thanked by (1)seenu

    https://clients.mrvm.net

This discussion has been closed.

© LowEndSpirit 2024

ipv6 ready

This Site is currently in maintenance mode.
Please check back here later.

→ Site Settings