Thank you all, for this wonderful conversation. I really enjoyed reading that. What I take from it, is there is many things you can do, but the question remains: what should be done, also in terms of the product Cloudron itself.
Since I've been confronted with a bunch of abuse-report myself that started yesterday, I checked all the apps and their stats, respectively for an abnormal behavior. The traffic had surged to roughly 1gb per hour, which could be seen in the providers traffic-reports, but the system itself incl. every single app I checked, the network-io for the past 7 days was totally normal (read: low to non-existent).
That way, I was NOT able to find the app in question. Also, I missed a feature, where I could overlay ALL active apps in one graph to sort out, which one of them is the most active or verbose one, in regard of network-traffic. So that would be a nice addition to the admin-panel, e.g. the System-Info page, where it shows some graphs, even lists all apps and their corresponding disk-usage, but a networking information would be helpful here, I guess.
Out of despair, I allowed myself to install nethogs and bmon via apt as root on the server, against the advise but figured it would not hinder or hurt the installation, as it usually is not creating any conflicts. Please advise, if I'm wrong on that for any reason.
But the traffic that occurred was not captured on the cloudron network-io graph for reasons that I can not explain. What I found was a compromised Wordpress-installation that had malicious scripts on them. I de-activated the app in question and wonder if that fixes the issue.
The traffic has been not constant up, but 3-4 hours at 1-2 GB / hour and then nothing again, for another 3-5 hours. So it is harder to track and needs some monitoring and observation.
Additional information: I had a provider-based firewall active, so only ICMP was allowed, as well as port 80/443. ssh was blocked (I turn it on, when I need it). All outgoing connections are allowed, obviously. All Email functionality is set to outbound only, with the logs not showing any weird behavior as well.
tldr - open questions:
- How can the network graph not show the traffic, that was generated (if it was the Wordpress in question, that is still unclear)
- Please add a multi-app network monitoring graph (or even more stats) to compare apps with another and find those, that take more resources
- Is there a way to monitor network-usage of the system itself, that are not from the apps?
- How much impact can be expected, if little monitoring helpers are installed via apt (as root) on a Cloudron that could hurt future maintenance?
Thank you so much for your attention.
