Solved Linode Abuse - DoS Attack Originating from my server?

[Nicolas]

Hello,

I use Cloudron in part because managing a VPS is outside of my usual comfort zone. I was gravely concerned when I received an email today from Linode suggesting my VPS had been used to attempt some sort of brute force/DoS attack targeted at another site. I rely on Cloudron to ensure the general security of the VPS, and my root password is quite secure (generated from a password manager and only saved there). I've changed the root password, but am not sure of what next steps I should take, or how I can prevent this in the future.

[nebulon]

Did linode provide any additional information? There are two things which could have been compromised. The host system itself or an individual app. Generally you should probably change from password based login via SSH to key based auth. This removes one possible angle. Also make sure no other key was placed there already.

[Nicolas]

@nebulon Thanks for the quick reply!
They provided logs (which I'll leave at the end of this reply), but they don't seem to give enough information if it's a host system compromise or an app.
/root/.ssh does not have any authorized keys listed (which I take as a good sign).

we detected a DOS attack from your network.

Below the logs.
172.105.104.122 - - [23/Nov/2020:15:43:33 +0100] "GET /wp-login.php HTTP/1.0" 200 1296 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 172.105.104.122 - - [23/Nov/2020:15:43:35 +0100] "POST /wp-login.php HTTP/1.0" 200 1659 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 172.105.104.122 - - [23/Nov/2020:15:43:36 +0100] "GET /wp-login.php HTTP/1.0" 200 1296 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 172.105.104.122 - - [23/Nov/2020:15:43:37 +0100] "POST /wp-login.php HTTP/1.0" 200 1669 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 172.105.104.122 - - [23/Nov/2020:15:43:38 +0100] "POST /xmlrpc.php HTTP/1.0" 403 212 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 172.105.104.122 - - [23/Nov/2020:16:10:33 +0100] "GET /wp-login.php HTTP/1.0" 200 4386 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 172.105.104.122 - - [23/Nov/2020:16:10:34 +0100] "POST /wp-login.php HTTP/1.0" 200 4386 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 172.105.104.122 - - [23/Nov/2020:16:10:35 +0100] "GET /wp-login.php HTTP/1.0" 200 4386 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 172.105.104.122 - - [23/Nov/2020:16:10:36 +0100] "POST /wp-login.php HTTP/1.0" 200 4386 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 172.105.104.122 - - [23/Nov/2020:16:10:37 +0100] "GET /wp-login.php HTTP/1.0" 200 4386 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 172.105.104.122 - - [23/Nov/2020:16:10:37 +0100] "POST /wp-login.php HTTP/1.0" 200 4386 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 172.105.104.122 - - [23/Nov/2020:16:10:38 +0100] "GET /wp-login.php HTTP/1.0" 200 4386 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 172.105.104.122 - - [23/Nov/2020:16:10:39 +0100] "POST /wp-login.php HTTP/1.0" 200 4386 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 172.105.104.122 - - [23/Nov/2020:16:10:39 +0100] "GET /wp-login.php HTTP/1.0" 200 4386 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 172.105.104.122 - - [23/Nov/2020:16:10:40 +0100] "POST /wp-login.php HTTP/1.0" 200 4386 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 172.105.104.122 - - [23/Nov/2020:16:10:41 +0100] "GET /wp-login.php HTTP/1.0" 200 4386 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 172.105.104.122 - - [23/Nov/2020:16:10:41 +0100] "POST /wp-login.php HTTP/1.0" 200 4386 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 172.105.104.122 - - [23/Nov/2020:16:10:42 +0100] "GET /wp-login.php HTTP/1.0" 200 4386 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 172.105.104.122 - - [23/Nov/2020:16:10:43 +0100] "POST /wp-login.php HTTP/1.0" 200 4386 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 172.105.104.122 - - [23/Nov/2020:16:10:44 +0100] "POST /xmlrpc.php HTTP/1.0" 403 212 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"

Please take action as soon as possible. The destination IP of the attack from your network is: web.mercurio.vhosting-it.com

[nebulon]

@nicolas hm not sure I can make out much from those logs, besides that something from your ip tries to request some wordpress login on that other server (which appears to be some plesk installation). Maybe to prevent this further, we should consider having outbound request limits?

[mehdi]

@nicolas Can you list which apps are installed on your Cloudron? It may be some app that got hacked on your server, and which is in turned used as a relay for such attacks.

[girish]

FWIW, we got similar reports from DigitalOcean. It look us a bit of time to realize this was coming from our demo cloudron instance where people were installing the torrent apps (so we fixed it, by not allowing those apps to be installed anymore in the demo instance).

[Nicolas]

@mehdi Have two wordpress apps, a webmail app, and two LAMP apps running Xenforo.

[iamthefij]

@nicolas Are your wordpress instances up to date? Do you have plugins installed in those instances?

That's where my mind goes immediately.

[Nicolas]

@iamthefij Thanks - it seems to be have been one of the wordpress instances (this memory spike corresponds to the same times the CPU spiked for the attack).
However, it's a Managed wordpress instance and so autoupdates are disabled (as cloudron does these updates) - I have no plugins installed on it.

[marcusquinn]

@Nicolas and anyone with a Wordpress issue and not access to developers I recommend the following steps:

1. If your site doesn't change much, it is usually a good idea to roll back to the oldest backup you have that is still what you want. You might be lucky and get to a point before any breach.
2. Change all Admin user passwords immediately. Might not be that but it's a process of elimination. Most hacks I've seen keep getting back in because they have a compromised Admin account one way or another.
3. Run updates for core and all plugins.
4. Deactivate and remove all plugins that don't update.
5. Run the free version of this: https://wordpress.org/plugins/gotmls/
6. Run the free version of this: https://wordpress.org/plugins/wp-cerber/
7. Run the free version of this: https://wordpress.org/plugins/wordfence/
8. This plugin is old but still works for evaluating any plugin you are unsure of the code quality for, maybe run on a clone just to be safe: https://wordpress.org/plugins/plugin-inspector/

GOTMLS should catch anything and is deep and fast to do and see results, it's the one I've seen most managed Wordpress vendors use and recommend as 1st thing to do when contacting them. It used to scan the core for free but honestly, it's worth the donation for the quality, speed and updates they offer.

Cerber & Wordfence I've had mixed results and speed with but they can't do any harm, and are safe to leave installed and running on the free versions.

[marcusquinn]

@nicolas I have seen legit plugins behave like malware too, hence Plugin Inspector can be very revealing.

Basically, with WP plugins, you have to be very, very fussy. If in doubt, create a post with links to any plugin you want experienced opinions on in here as there's a lot of experience that may help and many of us have further contacts if unsure.

[marcusquinn]

@nicolas Also note, your theme is also effectively a plugin, so Theme Check is good for reviewing the quality of that:

• https://wordpress.org/plugins/theme-check/

I also like Simple History for auditing recent events:

• https://wordpress.org/plugins/simple-history/

[robi]

Looking at the logs they sent, this is either a spoofed IP or someone found a way to proxy from your IP.

I'm assuming you don't run X11 on your VPS, hence it cannot be a browser doing that from the host OS.

While user agents can be spoofed easily, the one in the logs is an odd choice for blending in, hence likely the actual user agent from the Linux desktop user apparently bouncing through your system.

There are amplification attacks that change the low level packet source address of the payload and hence just make it look like your IP and thousands of others are attacking the one destination. Can't do much there.

Check your WP and Cloudron logs or post them here during the CPU spike to see if it's leaning on spoof traffic or exploited bounce traffic.

[ruihildt]

I have to double check, but I did have yesterday a WordPress test instance I seldom use that took a lot of resource on the server and was contacting tons of website.

I just deleted it without inspecting further but it matches a DDOS or bruteforce behavior.

I probably still have the app as part of a backup, so should be able to revive it for research purpose.

[robi]

@ruihildt renaming and stopping the App might be a better practice before deleting.

[Nicolas]

@marcusquinn Thanks for this. I deactivated even the add-ons that felt trustworthy and moved to a default theme (since that wordpress is really more of an archive than an active site). I also ran the scan you suggested and found that one add-on that I deactivated, and a theme, were both had scripts in them with known issues.

I wonder if a suggestion for the cloudron managed wordpress package is rate limiting external calls?

[girish]

@nicolas There's a feature request to monitor network traffic at the app level. If we had that, we could have narrowed this down more quickly.