Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


Skip to content

Prometheus

15 Topics 150 Posts
  • Prometheus - Package Updates

    Pinned
    88
    1 Votes
    88 Posts
    6k Views
    nebulonN

    [1.31.0]

    Update Prometheus to 2.52.0 Full changelog [CHANGE] TSDB: Fix the predicate checking for blocks which are beyond the retention period to include the ones right at the retention boundary. #9633 [FEATURE] Kubernetes SD: Add a new metric prometheus_sd_kubernetes_failures_total to track failed requests to Kubernetes API. #13554 [FEATURE] Kubernetes SD: Add node and zone metadata labels when using the endpointslice role. #13935 [FEATURE] Azure SD/Remote Write: Allow usage of Azure authorization SDK. #13099 [FEATURE] Alerting: Support native histogram templating. #13731 [FEATURE] Linode SD: Support IPv6 range discovery and region filtering. #13774 [ENHANCEMENT] PromQL: Performance improvements for queries with regex matchers. #13461 [ENHANCEMENT] PromQL: Performance improvements when using aggregation operators. #13744 [ENHANCEMENT] PromQL: Validate label_join destination label. #13803 [ENHANCEMENT] Scrape: Increment prometheus_target_scrapes_sample_duplicate_timestamp_total metric on duplicated series during one scrape. #12933 [ENHANCEMENT] TSDB: Many improvements in performance. #13742 #13673 #13782 [ENHANCEMENT] TSDB: Pause regular block compactions if the head needs to be compacted (prioritize head as it increases memory consumption). #13754 [ENHANCEMENT] Observability: Improved logging during signal handling termination. #13772 [ENHANCEMENT] Observability: All log lines for drop series use "num_dropped" key consistently. #13823 [ENHANCEMENT] Observability: Log chunk snapshot and mmaped chunk replay duration during WAL replay. #13838 [ENHANCEMENT] Observability: Log if the block is being created from WBL during compaction. #13846 [BUGFIX] PromQL: Fix inaccurate sample number statistic when querying histograms. #13667 [BUGFIX] PromQL: Fix histogram_stddev and histogram_stdvar for cases where the histogram has negative buckets. #13852 [BUGFIX] PromQL: Fix possible duplicated label name and values in a metric result for specific queries. #13845 [BUGFIX] Scrape: Fix setting native histogram schema factor during scrape. #13846 [BUGFIX] TSDB: Fix counting of histogram samples when creating WAL checkpoint stats. #13776 [BUGFIX] TSDB: Fix cases of compacting empty heads. #13755 [BUGFIX] TSDB: Count float histograms in WAL checkpoint. #13844 [BUGFIX] Remote Read: Fix memory leak due to broken requests. #13777 [BUGFIX] API: Stop building response for /api/v1/series/ when the API request was cancelled. #13766 [BUGFIX] promtool: Fix panic on promtool tsdb analyze --extended when no native histograms are present. #13976
  • 1 Votes
    2 Posts
    100 Views
    girishG

    @agitk this is the Cloudron forum . Here we discsus the Promotheus package of Cloudron . Are you using Cloudron for hosting Prometheus?

    But regardless, app questions are probably best asked in https://github.com/prometheus/prometheus/discussions (I think).

  • Adding remote targets in prometheus.yml

    5
    1 Votes
    5 Posts
    578 Views
    BrutalBirdieB

    please post your yaml with removed / redacted sensitive information in a code block and not plain.
    You can also always use

    e.g. :

    job_name: <SYSTEM-NAMEN> scrape_interval: 5s static_configs: - targets: ['<MY-IP:9100']

    yamllint is also a great to to check if you yaml file is valid.

  • Network statistics report container instead of host with node_exporter

    5
    1 Votes
    5 Posts
    242 Views
    L

    @girish Hi Girish! So I could totally be misreading this, but after some testing, it really does look like CPU and Mem stats do come through from the underlying host... Here is what I tried to confirm this. Let me know if I am way off base and I'm missing something crucial.

    First, just base observations:

    Prometheus reports back 4GB of total system memory, much more than the 512MB I have allocated for the Prometheus app container itself. Most of the time, the CPU utilization of my host VPS hovers around 2-3% as reported by Prometheus data, whereas the (according to Top) CPU utilization of the container is 1-2.5% Graphs on my VPS provider's control panel roughly correlate to the CPU utilization observed in Prometheus data.

    Second, based on stress-testing

    Using Stress to stress-test individual CPUs or all CPUs to 100% in the host system correctly registers corresponding observations in Prometheus data. Using Stress -vm within the Prometheus app container to take up 256MB of RAM correctly registers a ~256MB bump over existing baseline memory observations from Prometheus data, and not 50+% utilization as would be expected if the container stats were being reported (container has 512MB allocated to it). Using Stress -vm out on the host system to chew up 2 GB of memory correctly shows a ~2GB increase in memory utilization reported by Prometheus data.

    So far CPU, Mem, and Network stats are all that I have played with, and Network is the only one so far that I can confirm positively does only report the container stats instead of the host system.

    Not having correct network stats isn't a deal-breaker for me (though it would be nice, but not if doing so would put the system at risk of a container breach). I thought I'd at least report my findings thus far for anyone else that stumbles upon this.

    Anyway, even if I am wrong, I have had a real great time playing with this! Thanks for making it all possible!

  • Prometheus.yml file does not appear to work and other config is read-only

    Solved
    4
    1 Votes
    4 Posts
    258 Views
    L

    @girish Pefect. As always, thank you for the quick response!

  • Prometheus + Grafana actually working

    Solved
    10
    1 Votes
    10 Posts
    815 Views
    girishG

    @lcd_official indeed, prometheus_variables.sh is obsolete by now. Use env.sh instead.

  • Change feature flag of prometheus

    3
    0 Votes
    3 Posts
    262 Views
    girishG

    I have adjusted the latest package to make CLI args customizable (since there's no way to do this via env vars).

  • Basic auth for incoming requests

    2
    0 Votes
    2 Posts
    142 Views
    girishG

    @JLX89 prometheus app already supports proxyAuth . Can you check with a new installation ? We even have basic auth turned on.

  • Cloudron System Backup Blocked By Prometheus

    Solved
    13
    0 Votes
    13 Posts
    860 Views
    girishG

    @Jan-Macenka yeah, there's some bug in the fs module we use . We are replacing it next release, so maybe this bug goes away.

  • Track installed apps

    2
    2 Votes
    2 Posts
    169 Views
    Jan MacenkaJ

    Blocking /metrics and /health endpoints for Apps by default

    I'd also like to track Jellyfin which I love to run as an cloudron App. It comes with pre installed Prometheus endpoint according to the documentation which I'd like to use. The default Cloudron-location for this file is /app/data/jellyfin/config/system.xml
    However I dont want to expose this endpoint to the public Internet. Any Ideas how I could prevent this?

    If we find a good solution, I think it would be best to have the /metrics and /health endpoints blocked by default because Cloudron-Admins most likely dont want to expose these to the Internet anyway.

    @girish would you have an Idea where I could apply such a config change?

    Best regards and thanks for the answer in advance 🙂

    PS: Will gladly contribute to the Documentation as we found a good solution. I just fell down the rabbit hole to a certain degree in advancing my home-lab more towards a real data-center 🙂

  • Prometheus - Cannot login

    Solved
    3
    0 Votes
    3 Posts
    162 Views
    vishnumadhusoodananV

    Thanks @girish and sorry for the late reply

  • Adding local targets to Prometheus

    2
    0 Votes
    2 Posts
    214 Views
    girishG

    @cortex said in Adding local targets to Prometheus:

    How do I add the other Cloudron apps to the targets to monitor them?

    I don't have much idea here, maybe @eriktad or @BrutalBirdie knows. In general, what is needed to add a target to Prometheus? IP Address?

  • Add option for users to add starting parameters

    Solved
    5
    0 Votes
    5 Posts
    497 Views
    nebulonN

    @brutalbirdie thanks, I merged it and made a new package release.

  • Configure prometheus.yml file

    Solved
    3
    0 Votes
    3 Posts
    567 Views
    K

    @fbartels it works, thanks!

  • Ownership of the /app/data directory not set to cloudron:cloudron

    Solved
    3
    0 Votes
    3 Posts
    359 Views
    girishG

    @christophermag This should be fixed now.