Our server is hacked: foreign addresses in china, finland, france etc

[ZeZaung]

@BrutalBirdie
How do I tell which version of cloudron we're running?

[ZeZaung]

@BrutalBirdie Also, which system logs would be most appropriate?

[BrutalBirdie]

@ZeZaung like I said, a full disk image for forensic analysis would be best.

You can determine the Cloudron version from the latest backup made, since the newest backup included the box_X.Y.Z.tar.gz

If you can provide everything that you have from that system that would be best.

[andreasdueren]

Piggybacking on this thread: I was wondering for a while about enabling 2fa for SSH. would this interfere with cloudron?

[jdaviescoates]

@andreasdueren said in Our server is hacked: foreign addresses in china, finland, france etc:

I was wondering for a while about enabling 2fa for SSH. would this interfere with cloudron?

I shouldn't think so.

Personally I never have login with a password enabled and only have login via SHA public/ private keys enabled. Not sure what benefit 2FA would have in that case, but I'm presuming you've got a root password?

[BrutalBirdie]

@andreasdueren I am running all cloudron servers with root on complete lockdown.
This even blocks default support access for the staff, if you enabled it.

I also use normal ssh keys for the designated sudo user and also ed25519-sk and ecdsa-sk ssh keys for 2FA via Yubikey.
This works with no issues.

[LoudLemur]

@BrutalBirdie Does Cloudron have a "click to create disk image" button?

[BrutalBirdie]

@LoudLemur a disk image is a complete bit to bit copy to a file of the whole drive.
There is no way a software running on the same drive and at the same time that can create such an image, since it would recursively write it self into infinity.

For that you need a live boot and create it yourself with dd or a tool like clonzilla.

For more details: https://wiki.archlinux.org/title/disk_cloning

[privsec]

@andreasdueren said in Our server is hacked: foreign addresses in china, finland, france etc:

Piggybacking on this thread: I was wondering for a while about enabling 2fa for SSH. would this interfere with cloudron?

@jdaviescoates said in Our server is hacked: foreign addresses in china, finland, france etc:

@andreasdueren said in Our server is hacked: foreign addresses in china, finland, france etc:

I was wondering for a while about enabling 2fa for SSH. would this interfere with cloudron?

I shouldn't think so.

Personally I never have login with a password enabled and only have login via SHA public/ private keys enabled. Not sure what benefit 2FA would have in that case, but I'm presuming you've got a root password?

I have all accounts require username, password, SSH with a key and 2FA

[LoudLemur]

Thanks. For a cloned disk image to be useful, wouldn't it also have to be very recent, too? Repeatedly cloning the entire disk and then archiving these images would be resource intensive.

Perhaps information from btrfs or ZFS might be useful, if they were the file system.

@BrutalBirdie said in Our server is hacked: foreign addresses in china, finland, france etc:

@LoudLemur a disk image is a complete bit to bit copy to a file of the whole drive.
There is no way a software running on the same drive and at the same time that can create such an image, since it would recursively write it self into infinity.

For that you need a live boot and create it yourself with dd or a tool like clonzilla.

For more details: https://wiki.archlinux.org/title/disk_cloning

[BrutalBirdie]

@LoudLemur yes full disk images are regularly as big as the disk it self.
So if you got a 1TB disk the Disk Image will be 1TB.
Yes you could cut empty space which is unused.

When a system is compromised, normally you shutdown the system, create a disk image and then format the server and setup new.
Then this disk image can be used for later analytics.

[humptydumpty]

@BrutalBirdie on a vps, is a server snapshot the equivalent to a full disk image and can it be used for analytics?

[BrutalBirdie]

@humptydumpty Should be.
If you can create a server from it or attach it to a different server, then sure.

[privsec]

@BrutalBirdie You are the bomb.com

It might be worthwhile to write up a guide or refer to a already created guide on what to do in this exact type of an instance for future cases like this.

[andreasdueren]

@BrutalBirdie said in Our server is hacked: foreign addresses in china, finland, france etc:

@andreasdueren I am running all cloudron servers with root on complete lockdown.
This even blocks default support access for the staff, if you enabled it.
I also use normal ssh keys for the designated sudo user and also ed25519-sk and ecdsa-sk ssh keys for 2FA via Yubikey.
This works with no issues.

Would you mind sharing the steps you took? I'd like to try that out on a test server

[BrutalBirdie]

@andreasdueren

For everyone interested we (we as in, my company and me) also offer Cloudron hosting as as a service.
So I can't reveal my whole hand so please be understanding

---

But what you are asking about is pretty simple:

Create a user with sudo permissions and add your ssh public key to that user (don't lose the password for that user since you will need it for sudo)

I also disable all ssh access with password, since this only opens the window for brute force attempts

Depends on the lock down wanted, you can also disable the root login via /etc/passwd by setting the login shell to /sbin/nologin looks something like this:

root:x:0:0:root:/root:/sbin/nologin

Then, even if you try a sudo su - you get this:

This account is currently not available.

But since you can edit the /etc/passwd with sudo access (unless you lock down the system even further) this can be a bit snake oily.

---

There is much more going on in my servers, but since we deploy everything via. Ansible I don't need to keep track of ever single detail, since its infrastructure as a code, I can just read up.
Login tracking, Log Tracking, Monitoring yada yada yada.

If a system farts, I get a message.

I hope this shares some insights.
A step by step guide on how to lock down the root user would simply be me copy pasting google searches.
When it's about Linux security you can do sooooo much: https://wiki.archlinux.org/title/security
there is also a good section on "restricting root"

---

EDIT:

Maybe I can do a step by step guide in the forum when I got some spare time.
But right now its a bit late and I am lazy

[andreasdueren]

@BrutalBirdie I never had password access enabled on my servers in the first place but I'm interested in locking it down a little more that's why I asked. Thank you!

[scooke]

@BrutalBirdie said in Our server is hacked: foreign addresses in china, finland, france etc:

step by step guide in the forum when I got some spare time.

I don't think you "need" to do this. There are sooo many tutorials out there, easily findable, about how to secure and lockdown a server. What you described is even the most base-level steps, but so many people don't even do that! All anyone has to do is google those terms, "secure and lock down a server" and then choose the most reliable results. Anyone who needs this can find the info and put the steps into place way before you will find time to do a write-up.

[andreasdueren]

@scooke I'm not so much concerned as to achieve hardened security on my server but more not having those measure interfere with cloudron and create problems. For example I would like to add 2FA with Yubikey but that involves installing packages which we are discouraged from doing. That's why I asked for his setup.

[scooke]

@andreasdueren I'd be cautious about implementing it then. Cloudron hardens your server enough - doing more by installing more software, which is NOT recommended, will only lead to problems, especially if you don't already have a deep enough understanding of what is happening. It seems to be that @BrutalBirdie's gang knows their stuff (they're using Ansible to install Cloudron??? Yeah, that is next level coding there). Of course, they may also be paying for the Enterprise level of service (I'm not asking btw, no need to respond to that @BrutalBirdie ) so if they have hassles then I suppose it's fine for them to get help beyond typical Cloudron support, especially if they are doing more to their servers than what Cloudron themselves advise.