Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


    Cloudron Forum

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular

    Solved Our server is hacked: foreign addresses in china, finland, france etc

    Support
    security
    9
    29
    512
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Z
      ZeZaung last edited by girish

      Hello there

      Evidently, we got hacked.

      Our instance provider has let us know that we have abuse traffic emanating from our machine. We've monitored foreign addreses and know, for sure, the instance is compromised.

      Question for the cloudron's so-called turnkey security team: What can we do to protect ourselves? This is especially important to get an answer from you since we're incapable of installing modest network or Os-level security provisions within the instance without interfering with cloudron itself.

      We understand Cloudron literature suggests the following

      Security is a core feature of Cloudron and we continue to push out updates to tighten the Cloudron firewall's security policy. Our goal is that Cloudron users should be able to rely on Cloudron being secure out of the box without having to do manual configuration.
      

      Can the turnkey security team give us some advice on software we can install (that still works with cloudron) to deal with this? A lot of our preferred security & hardening measures are not doable without interfering with cloudron itself

      Before our team goes through the onerous process of creating a new instance and backing up these apps individually or resetting them all together, we want to know what security measures can be implemented on the second time around, so we aren't wasting time after the fact.

      A simple netstat and awk monitoring program shows unique hits from foreign addresses in countries such as China, finland, and France, all on strange ports certainly not associated with our deployed applications.

      Considering the restrictive nature of 3rd party installations of software on a OS level for the master control plane instance we are hoping cloudron's turnkey security team can get back to us promptly. the current state of things suggests we can't rely on cloudron as a platform for production grade loads or traffic.

      BrutalBirdie 1 Reply Last reply Reply Quote 0
      • BrutalBirdie
        BrutalBirdie Staff @ZeZaung last edited by

        @ZeZaung
        Could you please explain what got hacked?
        Because the statement:

        foreign addresses in china, finland, france etc

        can be expected for a public server.

        Our instance provider has let us know that we have abuse traffic emanating from our machine. We've monitored foreign addreses and know, for sure, the instance is compromised.

        What exactly was happening? This is all so vague.

        Can the turnkey security team give us some advice on software we can install (that still works with cloudron) to deal with this? A lot of our preferred security & hardening measures are not doable without interfering with cloudron itself

        To give advise or investigate it would be tremendously helpful to know what happened.
        Do you have:

        • logs
        • statements from the provider

        Did the server get compromised on the root level, was custom software running?
        An App level?


        Did you perhaps run an the adguard app and did not secure it properly, and it got used for a reflection attack?

        🤷
        Please share more details.

        Like my work? Consider donating a beer 🍻 Cheers!

        Z 1 Reply Last reply Reply Quote 5
        • Z
          ZeZaung @BrutalBirdie last edited by

          @BrutalBirdie

          Answers to your question.

          Could you please explain what got hacked?

          The VPS itself has been hacked and an unknown process emanating from this IP has been detected by our cloud provider

          can be expected for a public server.

          I am telling you that connections from all these different hosts is unconnected and not explainable, no/

          Our instance provider has let us know that we have abuse traffic emanating from our machine. We've monitored foreign addreses and know, for sure, the instance is compromised.

          What exactly was happening? This is all so vague.

          See below

          Do you have:

          • logs
          • statements from the provider
            See below
          Hello,
          
          Here is a sample of the traffic that was detected per your request.
          
          -- Traffic excerpt below --
          
          Reported-From: root@dascos.info
          Report-ID: 1676447666@jeeg
          Category: abuse
          Report-Type: login-attack
          Service: 404trap
          User-Agent: csf v14.17
          Date: 2023-02-15T08:54:26+0100
          Source:(OUR.IP.WAS.HERE)
          Source-Type: ipv4
          Attachment: text/plain
          Schema-URL: https://download.configserver.com/abuse_login-attack_0.2.json
          
          (OUR.IP.WAS.HERE) 172.16.0.50 nolodapria.it [15/Feb/2023:08:54:17 +0100] GET /?author=161 HTTP/2.0 404
          (OUR.IP.WAS.HERE) 172.16.0.50 nolodapria.it [15/Feb/2023:08:54:18 +0100] GET /?author=162 HTTP/2.0 404
          (OUR.IP.WAS.HERE) 172.16.0.50 nolodapria.it [15/Feb/2023:08:54:19 +0100] GET /?author=163 HTTP/2.0 404
          (OUR.IP.WAS.HERE).16.0.50 nolodapria.it [15/Feb/2023:08:54:20 +0100] GET /?author=164 HTTP/2.0 404
          
          -- Traffic excerpt above --
          
          Please review and resolve the abuse.
          
          We will be forced to power down this instance if we do not receive an update within 24 hours.
          

          This was one of their warnings. We have done some preliminary investigation via netstat and rootkit searches and can provide logs to you. Who do we email?

          Can you help us differentiate the traffic we are seeing to discern what processes are responsible for this and where the security compromise is arising from within the operating system? A lot of our diagnostic tools we can not install without interfering with cloudron.

          Did you perhaps run an the adguard app and did not secure it properly, and it got used for a reflection attack?
          Sorry no

          Please let us know who to send this info to. We will capture logs over hte past 20 mins or so in a few files.

          BrutalBirdie 1 Reply Last reply Reply Quote 0
          • BrutalBirdie
            BrutalBirdie Staff @ZeZaung last edited by

            @ZeZaung
            please provide as many details as possible to support@cloudron.io

            Like my work? Consider donating a beer 🍻 Cheers!

            Z 1 Reply Last reply Reply Quote 1
            • Z
              ZeZaung @BrutalBirdie last edited by

              thanks, we'll be in toch

              BrutalBirdie 1 Reply Last reply Reply Quote 0
              • girish
                girish Staff last edited by girish

                Yes, please send a mail to support@cloudron.io with more information. I am not really sure what was compromised or how things were compromised from your report. There is nothing inherently wrong with a server contacting china/finland/france depending on what software you have installed. If you can tell us what additional security measure you want to put in place , we can give suggestions there as well , so that it doesn't interfere with Cloudron.

                1 Reply Last reply Reply Quote 0
                • BrutalBirdie
                  BrutalBirdie Staff @ZeZaung last edited by

                  @ZeZaung also I highly doubt you made a disk image of the server for forensic analytics, right?

                  Like my work? Consider donating a beer 🍻 Cheers!

                  Z L 2 Replies Last reply Reply Quote 0
                  • Z
                    ZeZaung @BrutalBirdie last edited by

                    @BrutalBirdie Would a copy of a file generated at /var/backups suffice?

                    BrutalBirdie 1 Reply Last reply Reply Quote 0
                    • BrutalBirdie
                      BrutalBirdie Staff @ZeZaung last edited by

                      @ZeZaung Please provide as many information as possible to support@cloudron.io

                      • VPS Provider
                      • OS Used (Version)
                      • was ssh for root enabled with a weak password
                      • what cloudron version was running
                      • what apps where running
                      • logs of the apps
                      • logs of the system

                      Provide as many details as possible.

                      Like my work? Consider donating a beer 🍻 Cheers!

                      Z 2 Replies Last reply Reply Quote 2
                      • Z
                        ZeZaung @BrutalBirdie last edited by

                        @BrutalBirdie
                        How do I tell which version of cloudron we're running?

                        1 Reply Last reply Reply Quote 0
                        • Z
                          ZeZaung @BrutalBirdie last edited by

                          @BrutalBirdie Also, which system logs would be most appropriate?

                          BrutalBirdie 1 Reply Last reply Reply Quote 0
                          • BrutalBirdie
                            BrutalBirdie Staff @ZeZaung last edited by

                            @ZeZaung like I said, a full disk image for forensic analysis would be best.

                            You can determine the Cloudron version from the latest backup made, since the newest backup included the box_X.Y.Z.tar.gz

                            If you can provide everything that you have from that system that would be best.

                            Like my work? Consider donating a beer 🍻 Cheers!

                            1 Reply Last reply Reply Quote 0
                            • andreasdueren
                              andreasdueren last edited by

                              Piggybacking on this thread: I was wondering for a while about enabling 2fa for SSH. would this interfere with cloudron?

                              jdaviescoates BrutalBirdie P 3 Replies Last reply Reply Quote 2
                              • jdaviescoates
                                jdaviescoates @andreasdueren last edited by

                                @andreasdueren said in Our server is hacked: foreign addresses in china, finland, france etc:

                                I was wondering for a while about enabling 2fa for SSH. would this interfere with cloudron?

                                I shouldn't think so.

                                Personally I never have login with a password enabled and only have login via SHA public/ private keys enabled. Not sure what benefit 2FA would have in that case, but I'm presuming you've got a root password?

                                I use Cloudron with Gandi & Hetzner

                                1 Reply Last reply Reply Quote 1
                                • BrutalBirdie
                                  BrutalBirdie Staff @andreasdueren last edited by BrutalBirdie

                                  @andreasdueren I am running all cloudron servers with root on complete lockdown.
                                  This even blocks default support access for the staff, if you enabled it.

                                  I also use normal ssh keys for the designated sudo user and also ed25519-sk and ecdsa-sk ssh keys for 2FA via Yubikey.
                                  This works with no issues.

                                  Like my work? Consider donating a beer 🍻 Cheers!

                                  andreasdueren 1 Reply Last reply Reply Quote 3
                                  • L
                                    LoudLemur @BrutalBirdie last edited by

                                    @BrutalBirdie Does Cloudron have a "click to create disk image" button?

                                    BrutalBirdie 1 Reply Last reply Reply Quote 0
                                    • BrutalBirdie
                                      BrutalBirdie Staff @LoudLemur last edited by BrutalBirdie

                                      @LoudLemur a disk image is a complete bit to bit copy to a file of the whole drive.
                                      There is no way a software running on the same drive and at the same time that can create such an image, since it would recursively write it self into infinity.

                                      For that you need a live boot and create it yourself with dd or a tool like clonzilla.

                                      For more details: https://wiki.archlinux.org/title/disk_cloning

                                      Like my work? Consider donating a beer 🍻 Cheers!

                                      L 1 Reply Last reply Reply Quote 0
                                      • P
                                        privsec @andreasdueren last edited by privsec

                                        @andreasdueren said in Our server is hacked: foreign addresses in china, finland, france etc:

                                        Piggybacking on this thread: I was wondering for a while about enabling 2fa for SSH. would this interfere with cloudron?

                                        @jdaviescoates said in Our server is hacked: foreign addresses in china, finland, france etc:

                                        @andreasdueren said in Our server is hacked: foreign addresses in china, finland, france etc:

                                        I was wondering for a while about enabling 2fa for SSH. would this interfere with cloudron?

                                        I shouldn't think so.

                                        Personally I never have login with a password enabled and only have login via SHA public/ private keys enabled. Not sure what benefit 2FA would have in that case, but I'm presuming you've got a root password?

                                        I have all accounts require username, password, SSH with a key and 2FA

                                        1 Reply Last reply Reply Quote 1
                                        • L
                                          LoudLemur @BrutalBirdie last edited by

                                          Thanks. For a cloned disk image to be useful, wouldn't it also have to be very recent, too? Repeatedly cloning the entire disk and then archiving these images would be resource intensive.

                                          Perhaps information from btrfs or ZFS might be useful, if they were the file system.

                                          @BrutalBirdie said in Our server is hacked: foreign addresses in china, finland, france etc:

                                          @LoudLemur a disk image is a complete bit to bit copy to a file of the whole drive.
                                          There is no way a software running on the same drive and at the same time that can create such an image, since it would recursively write it self into infinity.

                                          For that you need a live boot and create it yourself with dd or a tool like clonzilla.

                                          For more details: https://wiki.archlinux.org/title/disk_cloning

                                          BrutalBirdie 1 Reply Last reply Reply Quote 0
                                          • BrutalBirdie
                                            BrutalBirdie Staff @LoudLemur last edited by

                                            @LoudLemur yes full disk images are regularly as big as the disk it self.
                                            So if you got a 1TB disk the Disk Image will be 1TB.
                                            Yes you could cut empty space which is unused.

                                            When a system is compromised, normally you shutdown the system, create a disk image and then format the server and setup new.
                                            Then this disk image can be used for later analytics.

                                            Like my work? Consider donating a beer 🍻 Cheers!

                                            humptydumpty 1 Reply Last reply Reply Quote 3
                                            • humptydumpty
                                              humptydumpty @BrutalBirdie last edited by

                                              @BrutalBirdie on a vps, is a server snapshot the equivalent to a full disk image and can it be used for analytics?

                                              BrutalBirdie 1 Reply Last reply Reply Quote 1
                                              • BrutalBirdie
                                                BrutalBirdie Staff @humptydumpty last edited by

                                                @humptydumpty Should be.
                                                If you can create a server from it or attach it to a different server, then sure.

                                                Like my work? Consider donating a beer 🍻 Cheers!

                                                P 1 Reply Last reply Reply Quote 1
                                                • P
                                                  privsec @BrutalBirdie last edited by

                                                  @BrutalBirdie You are the bomb.com

                                                  It might be worthwhile to write up a guide or refer to a already created guide on what to do in this exact type of an instance for future cases like this.

                                                  1 Reply Last reply Reply Quote 2
                                                  • andreasdueren
                                                    andreasdueren @BrutalBirdie last edited by

                                                    @BrutalBirdie said in Our server is hacked: foreign addresses in china, finland, france etc:

                                                    @andreasdueren I am running all cloudron servers with root on complete lockdown.
                                                    This even blocks default support access for the staff, if you enabled it.
                                                    I also use normal ssh keys for the designated sudo user and also ed25519-sk and ecdsa-sk ssh keys for 2FA via Yubikey.
                                                    This works with no issues.

                                                    Would you mind sharing the steps you took? I'd like to try that out on a test server

                                                    BrutalBirdie 1 Reply Last reply Reply Quote 0
                                                    • BrutalBirdie
                                                      BrutalBirdie Staff @andreasdueren last edited by BrutalBirdie

                                                      @andreasdueren

                                                      For everyone interested we (we as in, my company and me) also offer Cloudron hosting as as a service.
                                                      So I can't reveal my whole hand 😉 so please be understanding ❤


                                                      But what you are asking about is pretty simple:

                                                      Create a user with sudo permissions and add your ssh public key to that user (don't lose the password for that user since you will need it for sudo)

                                                      I also disable all ssh access with password, since this only opens the window for brute force attempts

                                                      Depends on the lock down wanted, you can also disable the root login via /etc/passwd by setting the login shell to /sbin/nologin looks something like this:

                                                      root:x:0:0:root:/root:/sbin/nologin
                                                      

                                                      Then, even if you try a sudo su - you get this:

                                                      This account is currently not available.
                                                      

                                                      94fca282-560f-4d7d-bfc4-fb2c4816d6e8-image.png
                                                      But since you can edit the /etc/passwd with sudo access (unless you lock down the system even further) this can be a bit snake oily.


                                                      There is much more going on in my servers, but since we deploy everything via. Ansible I don't need to keep track of ever single detail, since its infrastructure as a code, I can just read up.
                                                      Login tracking, Log Tracking, Monitoring yada yada yada.

                                                      If a system farts, I get a message.

                                                      I hope this shares some insights.
                                                      A step by step guide on how to lock down the root user would simply be me copy pasting google searches.
                                                      When it's about Linux security you can do sooooo much: https://wiki.archlinux.org/title/security
                                                      there is also a good section on "restricting root" 😉


                                                      EDIT:

                                                      Maybe I can do a step by step guide in the forum when I got some spare time. 🙂
                                                      But right now its a bit late and I am lazy 💤

                                                      Like my work? Consider donating a beer 🍻 Cheers!

                                                      andreasdueren scooke 2 Replies Last reply Reply Quote 8
                                                      • andreasdueren
                                                        andreasdueren @BrutalBirdie last edited by

                                                        @BrutalBirdie I never had password access enabled on my servers in the first place but I'm interested in locking it down a little more that's why I asked. Thank you!

                                                        1 Reply Last reply Reply Quote 0
                                                        • scooke
                                                          scooke @BrutalBirdie last edited by scooke

                                                          @BrutalBirdie said in Our server is hacked: foreign addresses in china, finland, france etc:

                                                          step by step guide in the forum when I got some spare time.

                                                          I don't think you "need" to do this. There are sooo many tutorials out there, easily findable, about how to secure and lockdown a server. What you described is even the most base-level steps, but so many people don't even do that! All anyone has to do is google those terms, "secure and lock down a server" and then choose the most reliable results. Anyone who needs this can find the info and put the steps into place way before you will find time to do a write-up.

                                                          A life lived in fear is a life half-lived

                                                          andreasdueren 1 Reply Last reply Reply Quote 1
                                                          • andreasdueren
                                                            andreasdueren @scooke last edited by

                                                            @scooke I'm not so much concerned as to achieve hardened security on my server but more not having those measure interfere with cloudron and create problems. For example I would like to add 2FA with Yubikey but that involves installing packages which we are discouraged from doing. That's why I asked for his setup.

                                                            scooke 1 Reply Last reply Reply Quote 2
                                                            • scooke
                                                              scooke @andreasdueren last edited by

                                                              @andreasdueren I'd be cautious about implementing it then. Cloudron hardens your server enough - doing more by installing more software, which is NOT recommended, will only lead to problems, especially if you don't already have a deep enough understanding of what is happening. It seems to be that @BrutalBirdie's gang knows their stuff (they're using Ansible to install Cloudron??? Yeah, that is next level coding there). Of course, they may also be paying for the Enterprise level of service (I'm not asking btw, no need to respond to that @BrutalBirdie ) so if they have hassles then I suppose it's fine for them to get help beyond typical Cloudron support, especially if they are doing more to their servers than what Cloudron themselves advise.

                                                              A life lived in fear is a life half-lived

                                                              1 Reply Last reply Reply Quote 1
                                                              • Topic has been marked as a question  girish girish 
                                                              • Topic has been marked as solved  girish girish 
                                                              • First post
                                                                Last post
                                                              Powered by NodeBB