Cloudron Forum

@girish Hi Girish! So I could totally be misreading this, but after some testing, it really does look like CPU and Mem stats do come through from the underlying host... Here is what I tried to confirm this. Let me know if I am way off base and I'm missing something crucial.

First, just base observations:

• Prometheus reports back 4GB of total system memory, much more than the 512MB I have allocated for the Prometheus app container itself.
• Most of the time, the CPU utilization of my host VPS hovers around 2-3% as reported by Prometheus data, whereas the (according to Top) CPU utilization of the container is 1-2.5%
• Graphs on my VPS provider's control panel roughly correlate to the CPU utilization observed in Prometheus data.

Second, based on stress-testing

• Using Stress to stress-test individual CPUs or all CPUs to 100% in the host system correctly registers corresponding observations in Prometheus data.
• Using Stress -vm within the Prometheus app container to take up 256MB of RAM correctly registers a ~256MB bump over existing baseline memory observations from Prometheus data, and not 50+% utilization as would be expected if the container stats were being reported (container has 512MB allocated to it).
• Using Stress -vm out on the host system to chew up 2 GB of memory correctly shows a ~2GB increase in memory utilization reported by Prometheus data.

So far CPU, Mem, and Network stats are all that I have played with, and Network is the only one so far that I can confirm positively does only report the container stats instead of the host system.

Not having correct network stats isn't a deal-breaker for me (though it would be nice, but not if doing so would put the system at risk of a container breach). I thought I'd at least report my findings thus far for anyone else that stumbles upon this.

Anyway, even if I am wrong, I have had a real great time playing with this! Thanks for making it all possible!

@girish Hmm. That is strange. When I ran a stress test on my host system, Prometheus reported back 100% of one core utilized just a I expected, even though I was not running anything within the container. Same results when I ran a memory test on the host using stress-ng. I've also correlated the CPU usage with my VPS host's graphs on occasion when other containers are eating up small amounts of CPU.

I suppose it is possible that what I'm seeing is the host system stealing resources from the Prometheus container, but I'm not sure if that would appear as actual resource usage from within the container...

I suppose more testing is in order... Thanks for your response.

Additional info: The official node_exporter documentation also has good information regarding running node_exporter within a Docker container.

Hello! Just wanted to check if this is by design (i.e. if security issue exposing host network stats) or a potential bug.

I've determined after load testing that node_exporter running in the Prometheus container is able to report CPU, Memory and other core performance metrics of the host sytem just fine, however network stats such as node_network_receive_bytes_total and node_network_transmit_bytes_total only report data from the container.

I tested in 2 ways, first by running a download from another container running on the same host (same cloudron). My VPS host (hetzner) graphs reported roughly 1.5 MB/s download, while prometheus reported back only in the low single KB/s. The second test is a correlation between letting my connected Grafana dashboard idle, then checking it after a long period of time, which caues a sudden spike in reported network activity, leading me to believe prometheus is only reporting activity from its container.

Could this be a potential bug, or is this done by design (If for example mapping the host's /proc and/or /sys and/or other needed directories read-only to the container were to pose a security risk)?

I've seen in other posts that mounting these directories from the host might resolve the network reporting issue when running in a container, however I don't know enough about docker at this point to know whether this would indeed work, or whether it might pose risks. Here are some examples that I've seen elsewhere:

docker run -d -v /proc/:/host/proc/:ro -v /sys/:/host/sys/:ro

Then in options when we run node_exporter would consist of:
node_exporter --path.sysfs=/host/sys --path.procfs=/host/proc

This is only cobbled together from other threads I've seen elsewhere, so I have no idea whether it would work, or whether it is a risk or not, or if indeed this might even disrupt metrics that already are working such as CPU metrics, etc.

Hope this all makes sense. Thanks!

@girish Pefect. As always, thank you for the quick response!

Hi, Cloudron community!

Hope you can help.
I am attempting to update my prometheus.yml file in /app/data/config, however the changes to not seem to be working. I noticed that there is another prometheus.yml in /app/code, however this one is read-only. I gather from this topic that the /app/data/config version is the one to go with, but I am surprised that the one in /app/code does not update after reboot.

I also tried editing /app/code/prometheus.yml directly in Recovery mode, only to have the changes there wiped out after app restart.

Is there somewhere else I should be making these changes, or is /app/data/config the correct one and I'm barking up the wrong tree assuming this is the source of my issues?

Background if anyone needs it: I have installed node_exporter on my host VPS and am attempting to add the following to scrape_config:

  - job_name: "node"
    static_configs:
     - targets: ["localhost:9100"]

Thanks!

@BrutalBirdie said in Prometheus + Grafana actually working:

ps: up the retention_time in your /app/data/prometheus_variables.sh

#!/bin/bash

# https://prometheus.io/docs/prometheus/latest/storage/#operational-aspects
export retention_time="180d"

Default is 15 days which is... well useless for long term analytics.
https://prometheus.io/docs/prometheus/latest/storage/#operational-aspects

Just FYI, I noticed that /app/data/prometheus_variables.sh did not exist anymore - and in fact the app at startup will actually take prometheus_variables.sh if it DOES exist, and mv (replace) over /app/data/env.sh, which isn't good...

You can edit /app/data/env.sh itself though, and update the storage.tsdb.retention.time CLI option. Here is my env.sh for reference:

#!/bin/bash

# https://prometheus.io/docs/prometheus/latest/storage/#operational-aspects
# edit this variable to change startup options
export cli_options="--storage.tsdb.retention.time=180d --storage.tsdb.path=/app/data/storage"

Thanks

I use Mega for backups. I just pre-encrypt using duplicity with the megav2 backend (uses megacmd command line tool) and it uploads directly to Mega after encryption so I don't even have to worry whether it truly is "zero knowledge" or not

I figured out the issue - posting here for anyone who encounters the same...
I had not whitelisted my new server's IP address with my registrar (Namecheap). Once I did this, installs and uninstalls are now working.

The error was actually an external error passed from Namecheap, not an internal Cloudron error message

Hi, everyone! Hoping you can help...

I recently migrated my Cloudron to a new VPS provider, and for the most part things went smoothly, and all of my apps are running fine except for one, which I am attempting to restore from a manually downloaded backup.

I am recieving the error "Invalid request IP", with my NEW IP address in the error, and this happens whenever I:

• Attempt to import the backup using the "Filesystem" option

• Attempt to uninstall the app (the app is stuck in an error state and will not uninstall)

• Attempt to install any new app

I have tried manually switching my DNS records between the old IP and the new IP over the last 24 hours, with no difference in the error.

I feel like I am missing something obvious, like something in the global configuration for my Cloudron, but I cannot seem to find anything in particular that would cause all app operations to fail in this mode.

Please let me know if you need any additional information - hopefully someone in this great community can shed some light on what I'm doing wrong...

Thanks!

As an Oracle database and APEX developer as part of my day job, I would LOVE to see this integrated into Cloudron,
Unfortunately I very much doubt Oracle would allow its free products to be integrated into another commercial product without a very one-sided data-sharing agreement. Much of Oracle's profits come from it's data-brokerage and ad tech businesses.

@girish Good morning!
It appears to be working perfectly! Tested renewing cert and Android’s Private DNS function is still happy to connect to AGH via DoT. That’s amazing - thank you for implementing that so quickly!

I have to say, after using Cloudron for a while now, what you guys have built here is nothing short of amazing - with so much automation - app installation, VPS security patches, managing certs, auto DNS config + providing LDAP, quick access to Cron, integration with registrars, custom apps - I can go on for days.

This really is bringing self-host to the masses…

Thank you so much

@girish Hi!
Sorry for the late response…. Last few days have been busy…
As of package version 1.4.6 it seems it may be working, at least surviving a reboot. I will perform additional testing in the morning including a manual renew of the cert and will report back asap.

Thanks!

@girish Yes, that is exactly right. That matches with what I found as well.

Sure, I think that would make sense, if the AdGuardHome package could remove the X3 cert at start - perhaps going as far as making it an option exposed via checkbox for the user to include the X3 cert to support "legacy" Android devices, based on preference...

Ultimately I believe this to be an Android issue, in the way that they have rigidly implemented DNS over TLS, but I doubt a change to Android will be anywhere near a quick fix to get implemented

Let me know your thoughts...

Thanks again

@girish Awesome! Thanks for sharing the name of the code file. I will look through and see if there is any way to implement something similar to certbot's '--preferred-chain' option and will share here if I can find any workaround.

Thanks!

Hi! I am submitting this here because it’s not really a bug - at least not for Cloudron.

I have encountered the same situation with Android clients in my implementation of AdGuard Plus that is described here: Let’s Encrypt and DNS over TLS on Android.

Long story short, Let’s Encrypt’s expired X3 certificate causes problems for Android clients on DNS-over-TLS.

The issue can be mitigated by invoking the option —preferred-chain “ISRG Root X1” when renewing Let’s Encrypt certificates, which excludes the expired certificate from the cert chain.

It's in no way a Cloudron bug, but since Cloudron handles certificate renewal, it seems to me to be the best place to insert a solution.

I believe Cloudron uses the ACME API in order to request renewals from Let’s Encrypt, and I would love to have the ability to specify the above option (or perhaps the ability to request other options as well might also be useful to others) within the Cloudron UI or API as part of the certificate setup/renewal.

Thanks for your time, and please let me know if you need any clarification at all.

Thanks!