Email spammer on my server

[Moose]

My server (Ubuntu 12.04) has recently been unable to send emails as it's IP has been blocked due to it being reported for email spam. I decided to investigate and was not pleased by what I discovered!

A wireshark capture revealed an email attempted to be sent about once every 10 seconds, further investigation seemed to show that sshd sessions were being initiated which were sending tons of emails, the sshd sessions also appeared to be connected to other ips who were presumably logged in?

The sshd sessions are called "sshd: root" so they are logged in as root, first thing I did was change the root password and remove all the keys.

Still sshd connections are being made and are sending emails! What can I do?

 

[Athlon2K15]

hit the power button?

 

[W1zzard]

look in the logs, check what's happening, fix it

 

[Aquinus]

look in the logs, check what's happening, fix it

+1: But I would still kill sshd until he figures it out.

Still sshd connections are being made and are sending emails! What can I do?

A: disable sshd if you can work locally.
(sudo /etc/init.d/sshd stop)

B: Disable password authentication (biggest vulnerability in a *nix system IMHO.)
@ /etc/ssh/sshd_config
You want:

PasswordAuthentication no
RSAAuthentication yes
PubkeyAuthentication yes
#AuthorizedKeysFile     %h/.ssh/authorized_keys
PermitRootLogin no

C: Enable shared key auth (and only shared key auth,) and generate a public/private RSA key pair.
(ssh-keygen -b 4096)

D: Allowing SSH into root is also dangerous. I would disable root login in then sshd config.

E: Copy your public key somewhere and enable sshd and you should be all set. That way the only way a hacker can get in through SSH is if they have your private key.

One of the more common reasons that mail fails (not initially, but over time) is when DNS is not properly setup. Maybe you're missing or have a bad MX or PTR record and the email server keeps retrying. That will make mail servers reject your email very quickly after a little while.

 

[W1zzard]

I'm not even sure that SSH is the source of his problems

 

[qubit]

I'd do a format and reinstall if it looks like the box has been rooted. Otherwise, fix the leak and monitor activity like a hawk.

 

[W1zzard]

I'd do a format and reinstall if it looks like the box has been rooted. Otherwise, fix the leak and monitor activity like a hawk.

+1, rooted = reinstall

 

[Aquinus]

I'm not even sure that SSH is the source of his problems

I know a couple of people who have been compromised because SSH was open and it allowed password authentication. Always use a key-pair when ever possible and if you can, require it. I agree though, there could be a problem elsewhere but that doesn't mean you shouldn't fix a potential problem before it happens if it wasn't SSH.

I'd do a format and reinstall if it looks like the box has been rooted. Otherwise, fix the leak and monitor activity like a hawk.

+1, rooted = reinstall

This. Fixing the problem is only a stop-gap measure. If they're in root they can make it very easy to get back in short of you turning the machine off or taking it off the network. Take it off the network, back it up and nuke it. After you re-install though, make sure to not go too lenient on the security settings for things like SSH though. Don't need this happening again. Make sure to change your password that you use on this box as well, for all accounts that had sudo and the password for root.

Occasionally a connection from China will try to make its way into my network. You may want to consider blocking IP ranges that you know that should never contact your server.

 

[W1zzard]

Always use a key-pair when ever possible and if you can, require it

The fun starts when you lose your private key due to fuckup, HDD crash or similar. Also trojan on your system could steal the private key (just like a keylogger can steal your typed password).

SSH password logins are perfectly safe and probably 90% of unix systems run with it. Weak passwords are not.

We moved SSH to another port on our servers to get rid of random (chinese) people trying to bruteforce it.

 

[Aquinus]

We moved SSH to another port on our servers to get rid of random (chinese) people trying to bruteforce it.

+1: Always a good choice. My personal favorite is 60031.

 

[Moose]

The fun starts when you lose your private key due to fuckup, HDD crash or similar. Also trojan on your system could steal the private key (just like a keylogger can steal your typed password).

SSH password logins are perfectly safe and probably 90% of unix systems run with it. Weak passwords are not.

We moved SSH to another port on our servers to get rid of random (chinese) people trying to bruteforce it.

I'm trying to work out how anyone could get my ssh password and I don't think they could, 10 digits long random letters and numbers to anyone but me. More likely is someone stole the key off my pc with a trojan, but still not very likely.

Is there anyway for me to get rid of this thing? What logs would tell me which processes are responsible? Because there must be something running as root that is letting them in now after I have changed the password and key. Btw the server is in a datacenter.

 

[W1zzard]

use the "last" command, "top", "ps", check /var/log/messages

documentation for these commands can be found by running "man last" or "man top" etc

 

[Moose]

Well the good thing is "last" command shows that my ip and :1 are the only ones to login to the server as any user including root for the past month.

Using top and ps x and ps aux, nothing struck me as being an obvious problem except the 2-4 "sshd: root" processes running and the 2-4 "sshd: root@notty" processes running (but apparently neither are being logged into?!)

 

[W1zzard]

find out what process is sending those emails, find out how it got on your system

 

[Moose]

Well the process is "sshd: root" it's the one connecting to mail servers.

I have stopped it doing it by changing sshd port, but that isn't a very good fix as they shouldn't be able to do it on any port!

 

[qubit]

Does it look to you like the server has been rooted ie running malware? If so, format and reinstall, don't waste your time trying to clean it up.

 

[Moose]

I reinstalled on a new server new IP helped and was cheaper, took hours of time though.

 

[qubit]

Yeah, ya just gotta go clean with it sometimes. I know what you mean about spending hours on it, lol.