-
Having received an abuse report (now solved) from the hosting company where my Cloudron is hosted, I realised there is little I can do to react to such an event.
And little I can do proactively to detect and prevent it.The standard (and excellent / justified) advice is : don't install anything else other than Cloudron and its official apps (or custom at your own risk).
That's fair enough, and I've not doubted it for a second.
Until now.If the standard advice is to be a defendible position, then we really need some utilities which are "built into" the standard Cloudron installation 'out of the box'.
If we're not supposed to install such things, then Cloudron (with respect) should provide them.I'm not sure what apps exactly, but thoughts start with :
- virus and malware checking
- easily accessible system-wide log (not app-specific logs) for system events like installing, removing apps
- ditto for examining outbound traffic that apps are generating (across the whole system)
- some kind of tripwire alert system ?
- some kind of local firewall (ufw or better ?) to block and selectively allow outbound traffic to strange ports or addresses, or indeed inbound
- some kind of anti-netscan system block
Does any of this exist and I have missed it ?
I'm sure these suggestions can be improved on, so please do so !
I think this kind of platform/system improvement is far more urgently needed than other platform improvements in the pipeline.
-
G girish referenced this topic on
-
This is a good discussion to have. Let's first try to get the attack vector before we put analyzing tools in place.
From Cloudron point of view, we seal off all ports (by default). Only SSH & HTTPS ports are open. As you add email and other apps like git, you have to open up more ports.
-
We can safely assume that SSH is not unsafe unless the user set a weak password or something. Maybe we can do something here? I don't know, usually we just leave it to the deployer to tackle this and provide suggestions like https://docs.cloudron.io/security/#securing-ssh-access
-
We have the cloudron dashboard. For this discussion, let's assume this is secure. We have 2FA support and the dashboard itself provides no SSH access. You can access the web terminal but you are sandboxed to the app.
-
Which then leads us to apps. If app is "hacked", a hacker can damage the app. Most apps are basically locked and can't do much since they run on readonly file system (it's the whole reason we have this, you can't run random scripts if your compromise the app).
The BIG issue here is apps like WordPress (or say coding apps like n8n, directus?). These are "platforms" where you can install plugins to do whatever. You can install or write a plugin to DoS some site easily. I am not sure what can be done here. Our approach is to lock down apps but WordPress is a market where people want things to be so flexible that it's impossible for us to manage easily. It requires some user knowledge. WP Developer exists because of market demand and not because we want it. Before that, everyone was like "I can do this easily in my wp provider xxxx....".
-
-
Having received an abuse report (now solved) from the hosting company where my Cloudron is hosted, I realised there is little I can do to react to such an event.
And little I can do proactively to detect and prevent it.The standard (and excellent / justified) advice is : don't install anything else other than Cloudron and its official apps (or custom at your own risk).
That's fair enough, and I've not doubted it for a second.
Until now.If the standard advice is to be a defendible position, then we really need some utilities which are "built into" the standard Cloudron installation 'out of the box'.
If we're not supposed to install such things, then Cloudron (with respect) should provide them.I'm not sure what apps exactly, but thoughts start with :
- virus and malware checking
- easily accessible system-wide log (not app-specific logs) for system events like installing, removing apps
- ditto for examining outbound traffic that apps are generating (across the whole system)
- some kind of tripwire alert system ?
- some kind of local firewall (ufw or better ?) to block and selectively allow outbound traffic to strange ports or addresses, or indeed inbound
- some kind of anti-netscan system block
Does any of this exist and I have missed it ?
I'm sure these suggestions can be improved on, so please do so !
I think this kind of platform/system improvement is far more urgently needed than other platform improvements in the pipeline.
@timconsidine said in Cloudron Instance Platform Check App(s):
virus and malware checking
easily accessible system-wide log (not app-specific logs) for system events like installing, removing apps
ditto for examining outbound traffic that apps are generating (across the whole system)
some kind of tripwire alert system ?
some kind of local firewall (ufw or better ?) to block and selectively allow outbound traffic to strange ports or addresses, or indeed inbound
some kind of anti-netscan system blockUsually, these systems have to be built outside Cloudron. You can't build things in the server itself. If the server is compromised, an attacker can compromise any of the tools above.
We actually recommend using a cloud firewall - https://docs.cloudron.io/security/#cloud-firewall . Unfortunately, most server providers don't have a cloud firewall or the above monitoring tools (well, maybe they provide it a price).
-
G girish moved this topic from Feature Requests on
-
@warg 's suggestion from https://forum.cloudron.io/post/69563
No clue what could have caused it in your case as I have too less experience with Cloudron. What I noticed is that sometimes I find the standard settings not secure enough (standard hardcoded passwords, apps being public accessible after installation, no hardened configurations, etc.). For example maybe someone accessed one of the applications with a standard password that Cloudron provides and it was forgotten to change it after installation.
This is a good point. Maybe people just install apps and forget about it. This leaves an installation with the standard hardcoded password. We did this for ease of use but maybe we should go ahead and start generating random passwords on startup. Might be inconvenient but I guess it makes things a bit more secure from abuse.
apps being public accessible after installation
I guess we see this as a major feature
If apps aren't public, most people won't use Cloudron since data won't be accessible easily. There is a plan to build vpn/wireguard into Cloudron to protect apps. Maybe that will solve this?no hardened configurations
We so harden the configuration of each app. For example, things like secrets are always generated. Registration is kept open, but only because without registration you cannot create a user in many apps
Or maybe you mean something else? -
This is a good discussion to have. Let's first try to get the attack vector before we put analyzing tools in place.
From Cloudron point of view, we seal off all ports (by default). Only SSH & HTTPS ports are open. As you add email and other apps like git, you have to open up more ports.
-
We can safely assume that SSH is not unsafe unless the user set a weak password or something. Maybe we can do something here? I don't know, usually we just leave it to the deployer to tackle this and provide suggestions like https://docs.cloudron.io/security/#securing-ssh-access
-
We have the cloudron dashboard. For this discussion, let's assume this is secure. We have 2FA support and the dashboard itself provides no SSH access. You can access the web terminal but you are sandboxed to the app.
-
Which then leads us to apps. If app is "hacked", a hacker can damage the app. Most apps are basically locked and can't do much since they run on readonly file system (it's the whole reason we have this, you can't run random scripts if your compromise the app).
The BIG issue here is apps like WordPress (or say coding apps like n8n, directus?). These are "platforms" where you can install plugins to do whatever. You can install or write a plugin to DoS some site easily. I am not sure what can be done here. Our approach is to lock down apps but WordPress is a market where people want things to be so flexible that it's impossible for us to manage easily. It requires some user knowledge. WP Developer exists because of market demand and not because we want it. Before that, everyone was like "I can do this easily in my wp provider xxxx....".
-
-
@warg 's suggestion from https://forum.cloudron.io/post/69563
No clue what could have caused it in your case as I have too less experience with Cloudron. What I noticed is that sometimes I find the standard settings not secure enough (standard hardcoded passwords, apps being public accessible after installation, no hardened configurations, etc.). For example maybe someone accessed one of the applications with a standard password that Cloudron provides and it was forgotten to change it after installation.
This is a good point. Maybe people just install apps and forget about it. This leaves an installation with the standard hardcoded password. We did this for ease of use but maybe we should go ahead and start generating random passwords on startup. Might be inconvenient but I guess it makes things a bit more secure from abuse.
apps being public accessible after installation
I guess we see this as a major feature
If apps aren't public, most people won't use Cloudron since data won't be accessible easily. There is a plan to build vpn/wireguard into Cloudron to protect apps. Maybe that will solve this?no hardened configurations
We so harden the configuration of each app. For example, things like secrets are always generated. Registration is kept open, but only because without registration you cannot create a user in many apps
Or maybe you mean something else?@girish I learnt my lesson about open registrations with Kutt !
As far as I am aware, I don't have any apps which have easy open login or self-registration.
Apps which are publicly accessible are things like websites.The more I think about it, the more I think my recent problem was a Wordpress instance. I don't know if the user loaded anything, I just killed it immediately without asking, and it seemed to solve the problem.
-
@girish I learnt my lesson about open registrations with Kutt !
As far as I am aware, I don't have any apps which have easy open login or self-registration.
Apps which are publicly accessible are things like websites.The more I think about it, the more I think my recent problem was a Wordpress instance. I don't know if the user loaded anything, I just killed it immediately without asking, and it seemed to solve the problem.
@timconsidine said in Cloudron Instance Platform Check App(s):
@girish I learnt my lesson about open registrations with Kutt !
We have been brainstorming locally on what we can do to help admins not forget. Maybe a post install checklist that has to be checked off in the UI. Until then, it will show some warning indicator saying "xx tasks pending to be completed" .
-
@timconsidine said in Cloudron Instance Platform Check App(s):
virus and malware checking
easily accessible system-wide log (not app-specific logs) for system events like installing, removing apps
ditto for examining outbound traffic that apps are generating (across the whole system)
some kind of tripwire alert system ?
some kind of local firewall (ufw or better ?) to block and selectively allow outbound traffic to strange ports or addresses, or indeed inbound
some kind of anti-netscan system blockUsually, these systems have to be built outside Cloudron. You can't build things in the server itself. If the server is compromised, an attacker can compromise any of the tools above.
We actually recommend using a cloud firewall - https://docs.cloudron.io/security/#cloud-firewall . Unfortunately, most server providers don't have a cloud firewall or the above monitoring tools (well, maybe they provide it a price).
@girish said in Cloudron Instance Platform Check App(s):
Usually, these systems have to be built outside Cloudron.
Understood.
I didn't mean to suggest that they are "in Cloudron", just that the Cloudron deployment installs e.g. virus/malware checker.
Is it ok if I investigate adding one "alongside" Cloudron ?
Normally that is against advice.
But I struggle to justify e.g. to my hosting company "sorry, I can't check for malware"I have had 3 abuse reports over time, one was because Kutt was open registration (solved), and the other 2 both seem to be about netscans (probably Syncthing and Wordpress : solved by uninstalling). Is it possible to install something which prevents outbound netscans ?
EDIT : I think Cloudron uses iptables : what is the feeling about using something like FirewallD or Shorewall to manage the iptables config ? Still investigating them.
-
Perhaps something like NetData could be baked into Cloudron?
-
This is a good discussion to have. Let's first try to get the attack vector before we put analyzing tools in place.
From Cloudron point of view, we seal off all ports (by default). Only SSH & HTTPS ports are open. As you add email and other apps like git, you have to open up more ports.
-
We can safely assume that SSH is not unsafe unless the user set a weak password or something. Maybe we can do something here? I don't know, usually we just leave it to the deployer to tackle this and provide suggestions like https://docs.cloudron.io/security/#securing-ssh-access
-
We have the cloudron dashboard. For this discussion, let's assume this is secure. We have 2FA support and the dashboard itself provides no SSH access. You can access the web terminal but you are sandboxed to the app.
-
Which then leads us to apps. If app is "hacked", a hacker can damage the app. Most apps are basically locked and can't do much since they run on readonly file system (it's the whole reason we have this, you can't run random scripts if your compromise the app).
The BIG issue here is apps like WordPress (or say coding apps like n8n, directus?). These are "platforms" where you can install plugins to do whatever. You can install or write a plugin to DoS some site easily. I am not sure what can be done here. Our approach is to lock down apps but WordPress is a market where people want things to be so flexible that it's impossible for us to manage easily. It requires some user knowledge. WP Developer exists because of market demand and not because we want it. Before that, everyone was like "I can do this easily in my wp provider xxxx....".
@girish said in Cloudron Instance Platform Check App(s):
We can safely assume that SSH is not unsafe unless the user set a weak password or something.
I think by default there should be a bruteforce protection for SSH. I see no reason why it's not installed by default if it's suggested (same with having phpmyadmin enabled by default for LAMP setups if it's suggested to have it only active if you need to).
Is there any log for failed log-ins towards Cloudron UI? I found nothing and I tried 10+ trials. Still the specific account is not locked nor the IP address banned.
@girish said in Cloudron Instance Platform Check App(s):
Which then leads us to apps. If app is "hacked", a hacker can damage the app. Most apps are basically locked and can't do much since they run on readonly file system (it's the whole reason we have this, you can't run random scripts if your compromise the app).
This is not given for LAMP apps in general so I think there should be an improvement for that part (same issue as for WP instances).
@girish said in Cloudron Instance Platform Check App(s):
I guess we see this as a major feature
If apps aren't public, most people won't use Cloudron since data won't be accessible easily.It can be public but not by default. I don't see a reason why specific apps should be public by default especially if the default settings aren't hardened or default passwords being used. An admin must decide actively which should be public. If he/she set it to public with bad practices, it's a self-made issue (although Cloudron advertises itself as a security by default solution where I have a few doubts like outlined).
-
-
@girish said in Cloudron Instance Platform Check App(s):
We can safely assume that SSH is not unsafe unless the user set a weak password or something.
I think by default there should be a bruteforce protection for SSH. I see no reason why it's not installed by default if it's suggested (same with having phpmyadmin enabled by default for LAMP setups if it's suggested to have it only active if you need to).
Is there any log for failed log-ins towards Cloudron UI? I found nothing and I tried 10+ trials. Still the specific account is not locked nor the IP address banned.
@girish said in Cloudron Instance Platform Check App(s):
Which then leads us to apps. If app is "hacked", a hacker can damage the app. Most apps are basically locked and can't do much since they run on readonly file system (it's the whole reason we have this, you can't run random scripts if your compromise the app).
This is not given for LAMP apps in general so I think there should be an improvement for that part (same issue as for WP instances).
@girish said in Cloudron Instance Platform Check App(s):
I guess we see this as a major feature
If apps aren't public, most people won't use Cloudron since data won't be accessible easily.It can be public but not by default. I don't see a reason why specific apps should be public by default especially if the default settings aren't hardened or default passwords being used. An admin must decide actively which should be public. If he/she set it to public with bad practices, it's a self-made issue (although Cloudron advertises itself as a security by default solution where I have a few doubts like outlined).
@warg said in Cloudron Instance Platform Check App(s):
I think by default there should be a bruteforce protection for SSH. I see no reason why it's not installed by default if it's suggested
Are you referring to something like fail2ban here? I know people are "bothered" by ssh logins but if you have a proper key login and password login disabled, there's nothing the attacker can do -https://blog.codinghorror.com/brute-force-key-attacks-are-for-dummies/ . The logs are in
journalctl -u sshif you want to inspect those anyway. If you really want to save on network traffic best bet is to do this at the network level in your VPS provider's firewall.(same with having phpmyadmin enabled by default for LAMP setups if it's suggested to have it only active if you need to).
true, I will disable it in the next package update by default.
It can be public but not by default.
Are you thinking of some wireguard/openvpn kind of setup here? How would I connect to apps?
-
@warg said in Cloudron Instance Platform Check App(s):
I think by default there should be a bruteforce protection for SSH. I see no reason why it's not installed by default if it's suggested
Are you referring to something like fail2ban here? I know people are "bothered" by ssh logins but if you have a proper key login and password login disabled, there's nothing the attacker can do -https://blog.codinghorror.com/brute-force-key-attacks-are-for-dummies/ . The logs are in
journalctl -u sshif you want to inspect those anyway. If you really want to save on network traffic best bet is to do this at the network level in your VPS provider's firewall.(same with having phpmyadmin enabled by default for LAMP setups if it's suggested to have it only active if you need to).
true, I will disable it in the next package update by default.
It can be public but not by default.
Are you thinking of some wireguard/openvpn kind of setup here? How would I connect to apps?
@girish said in Cloudron Instance Platform Check App(s):
Are you referring to something like fail2ban here?
I think @warg is possibly referring to the changing port stuff mentioned here:
-
@girish said in Cloudron Instance Platform Check App(s):
Are you referring to something like fail2ban here?
I think @warg is possibly referring to the changing port stuff mentioned here:
@jdaviescoates Ah , you mean move ssh to port 202 by default for all installations ? Oof, that won't be a popular decision
I remember we put some basic password restrictions in the initial Cloudron releases years ago and we were basically "strong armed" into removing any restrictions wrt setting password length or characters etc!
-
@jdaviescoates Ah , you mean move ssh to port 202 by default for all installations ? Oof, that won't be a popular decision
I remember we put some basic password restrictions in the initial Cloudron releases years ago and we were basically "strong armed" into removing any restrictions wrt setting password length or characters etc!
@girish said in Cloudron Instance Platform Check App(s):
@jdaviescoates Ah , you mean move ssh to port 202 by default for all installations ?
I'm guessing that is perhaps what @warg meant, yes.
-
I'd add email sending rate limiting defaults to the protections wishes:
- https://forum.cloudron.io/topic/4006/add-a-setting-for-sendmail-limits-per-mailbox-per-day
- https://forum.cloudron.io/topic/8792/basic-outbound-email-limits-per-email-account-at-smtp-level-for-ip-protection
- https://forum.cloudron.io/topic/4691/email-improvement-ability-to-rate-limit-outgoing-messages-from-the-server-cap-messages-sent-per-domain-in-a-given-timeframe
-
@warg said in Cloudron Instance Platform Check App(s):
I think by default there should be a bruteforce protection for SSH. I see no reason why it's not installed by default if it's suggested
Are you referring to something like fail2ban here? I know people are "bothered" by ssh logins but if you have a proper key login and password login disabled, there's nothing the attacker can do -https://blog.codinghorror.com/brute-force-key-attacks-are-for-dummies/ . The logs are in
journalctl -u sshif you want to inspect those anyway. If you really want to save on network traffic best bet is to do this at the network level in your VPS provider's firewall.(same with having phpmyadmin enabled by default for LAMP setups if it's suggested to have it only active if you need to).
true, I will disable it in the next package update by default.
It can be public but not by default.
Are you thinking of some wireguard/openvpn kind of setup here? How would I connect to apps?
@girish said in Cloudron Instance Platform Check App(s):
Are you thinking of some wireguard/openvpn kind of setup here? How would I connect to apps?
htaccess protection, ldap or similar, etc. Just anything so that apps without log-in by default are protected.
@jdaviescoates said in Cloudron Instance Platform Check App(s):
I think @warg is possibly referring to the changing port stuff mentioned here:
Nope, changing the port by default would be weird although it reduces the attacks on the port. I refered to https://docs.cloudron.io/security/#fail2ban which is fine to use and makes sense but isn't active by default. So as long as pubkey auth isn't default, fail2ban should be default.
-
TBH I'd like to see the my.controlpanel.domain all Apps behind a VPN by default, with something like Firezone:
So you launch them all privately, and set to be public if or when desired. Harder to hack what you don't know exists.
-
@jdaviescoates Ah , you mean move ssh to port 202 by default for all installations ? Oof, that won't be a popular decision
I remember we put some basic password restrictions in the initial Cloudron releases years ago and we were basically "strong armed" into removing any restrictions wrt setting password length or characters etc!
@girish said in Cloudron Instance Platform Check App(s):
@jdaviescoates Ah , you mean move ssh to port 202 by default for all installations ? Oof, that won't be a popular decision
I remember we put some basic password restrictions in the initial Cloudron releases years ago and we were basically "strong armed" into removing any restrictions wrt setting password length or characters etc!
Though, as you mentioned it, with proper key login and disabling password there's no reason for not sleeping at night, let alone to move ssh to port 202 or else.
-
Thank you all, for this wonderful conversation. I really enjoyed reading that. What I take from it, is there is many things you can do, but the question remains: what should be done, also in terms of the product Cloudron itself.
Since I've been confronted with a bunch of abuse-report myself that started yesterday, I checked all the apps and their stats, respectively for an abnormal behavior. The traffic had surged to roughly 1gb per hour, which could be seen in the providers traffic-reports, but the system itself incl. every single app I checked, the network-io for the past 7 days was totally normal (read: low to non-existent).
That way, I was NOT able to find the app in question. Also, I missed a feature, where I could overlay ALL active apps in one graph to sort out, which one of them is the most active or verbose one, in regard of network-traffic. So that would be a nice addition to the admin-panel, e.g. the System-Info page, where it shows some graphs, even lists all apps and their corresponding disk-usage, but a networking information would be helpful here, I guess.
Out of despair, I allowed myself to install nethogs and bmon via apt as root on the server, against the advise but figured it would not hinder or hurt the installation, as it usually is not creating any conflicts. Please advise, if I'm wrong on that for any reason.
But the traffic that occurred was not captured on the cloudron network-io graph for reasons that I can not explain. What I found was a compromised Wordpress-installation that had malicious scripts on them. I de-activated the app in question and wonder if that fixes the issue.
The traffic has been not constant up, but 3-4 hours at 1-2 GB / hour and then nothing again, for another 3-5 hours. So it is harder to track and needs some monitoring and observation.
Additional information: I had a provider-based firewall active, so only ICMP was allowed, as well as port 80/443. ssh was blocked (I turn it on, when I need it). All outgoing connections are allowed, obviously. All Email functionality is set to outbound only, with the logs not showing any weird behavior as well.
tldr - open questions:
- How can the network graph not show the traffic, that was generated (if it was the Wordpress in question, that is still unclear)
- Please add a multi-app network monitoring graph (or even more stats) to compare apps with another and find those, that take more resources
- Is there a way to monitor network-usage of the system itself, that are not from the apps?
- How much impact can be expected, if little monitoring helpers are installed via apt (as root) on a Cloudron that could hurt future maintenance?
Thank you so much for your attention.
-
There is a new HTTPS/2 DDoS attack that is cheap to use. It doesn't affect apps since it only goes as far as the Nginx reverse proxy for which we have no stats from Cloudron.
Since your WP instance was compromised, it could have been the one that leaked your IP/subdomain to the botnet which then tried to use it unsuccessfully.
