Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


Skip to content
Brand Logo

Cloudron Forum
  1. Cloudron Forum
  2. Prometheus

Prometheus

15 Topics 150 Posts
  • girishG

    Prometheus - Package Updates

    Pinned
    88
    1 Votes
    88 Posts
    5k Views
    nebulonN

    [1.31.0]

    Update Prometheus to 2.52.0 Full changelog [CHANGE] TSDB: Fix the predicate checking for blocks which are beyond the retention period to include the ones right at the retention boundary. #9633 [FEATURE] Kubernetes SD: Add a new metric prometheus_sd_kubernetes_failures_total to track failed requests to Kubernetes API. #13554 [FEATURE] Kubernetes SD: Add node and zone metadata labels when using the endpointslice role. #13935 [FEATURE] Azure SD/Remote Write: Allow usage of Azure authorization SDK. #13099 [FEATURE] Alerting: Support native histogram templating. #13731 [FEATURE] Linode SD: Support IPv6 range discovery and region filtering. #13774 [ENHANCEMENT] PromQL: Performance improvements for queries with regex matchers. #13461 [ENHANCEMENT] PromQL: Performance improvements when using aggregation operators. #13744 [ENHANCEMENT] PromQL: Validate label_join destination label. #13803 [ENHANCEMENT] Scrape: Increment prometheus_target_scrapes_sample_duplicate_timestamp_total metric on duplicated series during one scrape. #12933 [ENHANCEMENT] TSDB: Many improvements in performance. #13742 #13673 #13782 [ENHANCEMENT] TSDB: Pause regular block compactions if the head needs to be compacted (prioritize head as it increases memory consumption). #13754 [ENHANCEMENT] Observability: Improved logging during signal handling termination. #13772 [ENHANCEMENT] Observability: All log lines for drop series use "num_dropped" key consistently. #13823 [ENHANCEMENT] Observability: Log chunk snapshot and mmaped chunk replay duration during WAL replay. #13838 [ENHANCEMENT] Observability: Log if the block is being created from WBL during compaction. #13846 [BUGFIX] PromQL: Fix inaccurate sample number statistic when querying histograms. #13667 [BUGFIX] PromQL: Fix histogram_stddev and histogram_stdvar for cases where the histogram has negative buckets. #13852 [BUGFIX] PromQL: Fix possible duplicated label name and values in a metric result for specific queries. #13845 [BUGFIX] Scrape: Fix setting native histogram schema factor during scrape. #13846 [BUGFIX] TSDB: Fix counting of histogram samples when creating WAL checkpoint stats. #13776 [BUGFIX] TSDB: Fix cases of compacting empty heads. #13755 [BUGFIX] TSDB: Count float histograms in WAL checkpoint. #13844 [BUGFIX] Remote Read: Fix memory leak due to broken requests. #13777 [BUGFIX] API: Stop building response for /api/v1/series/ when the API request was cancelled. #13766 [BUGFIX] promtool: Fix panic on promtool tsdb analyze --extended when no native histograms are present. #13976
  • A

    Is there a reason why Prometheus chunk file size getting increased overtime?

    2
    1 Votes
    2 Posts
    68 Views
    girishG

    @agitk this is the Cloudron forum . Here we discsus the Promotheus package of Cloudron . Are you using Cloudron for hosting Prometheus?

    But regardless, app questions are probably best asked in https://github.com/prometheus/prometheus/discussions (I think).

  • F

    Adding remote targets in prometheus.yml

    5
    1 Votes
    5 Posts
    413 Views
    BrutalBirdieB

    please post your yaml with removed / redacted sensitive information in a code block and not plain.
    You can also always use

    e.g. :

    job_name: <SYSTEM-NAMEN> scrape_interval: 5s static_configs: - targets: ['<MY-IP:9100']

    yamllint is also a great to to check if you yaml file is valid.

  • L

    Network statistics report container instead of host with node_exporter

    5
    1 Votes
    5 Posts
    194 Views
    L

    @girish Hi Girish! So I could totally be misreading this, but after some testing, it really does look like CPU and Mem stats do come through from the underlying host... Here is what I tried to confirm this. Let me know if I am way off base and I'm missing something crucial.

    First, just base observations:

    Prometheus reports back 4GB of total system memory, much more than the 512MB I have allocated for the Prometheus app container itself. Most of the time, the CPU utilization of my host VPS hovers around 2-3% as reported by Prometheus data, whereas the (according to Top) CPU utilization of the container is 1-2.5% Graphs on my VPS provider's control panel roughly correlate to the CPU utilization observed in Prometheus data.

    Second, based on stress-testing

    Using Stress to stress-test individual CPUs or all CPUs to 100% in the host system correctly registers corresponding observations in Prometheus data. Using Stress -vm within the Prometheus app container to take up 256MB of RAM correctly registers a ~256MB bump over existing baseline memory observations from Prometheus data, and not 50+% utilization as would be expected if the container stats were being reported (container has 512MB allocated to it). Using Stress -vm out on the host system to chew up 2 GB of memory correctly shows a ~2GB increase in memory utilization reported by Prometheus data.

    So far CPU, Mem, and Network stats are all that I have played with, and Network is the only one so far that I can confirm positively does only report the container stats instead of the host system.

    Not having correct network stats isn't a deal-breaker for me (though it would be nice, but not if doing so would put the system at risk of a container breach). I thought I'd at least report my findings thus far for anyone else that stumbles upon this.

    Anyway, even if I am wrong, I have had a real great time playing with this! Thanks for making it all possible!

  • L

    Prometheus.yml file does not appear to work and other config is read-only

    Solved
    4
    1 Votes
    4 Posts
    179 Views
    L

    @girish Pefect. As always, thank you for the quick response!

  • cadornC

    Prometheus + Grafana actually working

    Solved
    10
    1 Votes
    10 Posts
    711 Views
    girishG

    @lcd_official indeed, prometheus_variables.sh is obsolete by now. Use env.sh instead.

  • gobenizzleG

    Change feature flag of prometheus

    3
    0 Votes
    3 Posts
    214 Views
    girishG

    I have adjusted the latest package to make CLI args customizable (since there's no way to do this via env vars).

  • J

    Basic auth for incoming requests

    2
    0 Votes
    2 Posts
    104 Views
    girishG

    @JLX89 prometheus app already supports proxyAuth . Can you check with a new installation ? We even have basic auth turned on.

  • BrutalBirdieB

    Cloudron System Backup Blocked By Prometheus

    Solved
    13
    0 Votes
    13 Posts
    738 Views
    girishG

    @Jan-Macenka yeah, there's some bug in the fs module we use . We are replacing it next release, so maybe this bug goes away.

  • Jan MacenkaJ

    Track installed apps

    2
    2 Votes
    2 Posts
    127 Views
    Jan MacenkaJ

    Blocking /metrics and /health endpoints for Apps by default

    I'd also like to track Jellyfin which I love to run as an cloudron App. It comes with pre installed Prometheus endpoint according to the documentation which I'd like to use. The default Cloudron-location for this file is /app/data/jellyfin/config/system.xml
    However I dont want to expose this endpoint to the public Internet. Any Ideas how I could prevent this?

    If we find a good solution, I think it would be best to have the /metrics and /health endpoints blocked by default because Cloudron-Admins most likely dont want to expose these to the Internet anyway.

    @girish would you have an Idea where I could apply such a config change?

    Best regards and thanks for the answer in advance 🙂

    PS: Will gladly contribute to the Documentation as we found a good solution. I just fell down the rabbit hole to a certain degree in advancing my home-lab more towards a real data-center 🙂

  • vishnumadhusoodananV

    Prometheus - Cannot login

    Solved
    3
    0 Votes
    3 Posts
    140 Views
    vishnumadhusoodananV

    Thanks @girish and sorry for the late reply

  • C

    Adding local targets to Prometheus

    2
    0 Votes
    2 Posts
    179 Views
    girishG

    @cortex said in Adding local targets to Prometheus:

    How do I add the other Cloudron apps to the targets to monitor them?

    I don't have much idea here, maybe @eriktad or @BrutalBirdie knows. In general, what is needed to add a target to Prometheus? IP Address?

  • BrutalBirdieB

    Add option for users to add starting parameters

    Solved
    5
    0 Votes
    5 Posts
    430 Views
    nebulonN

    @brutalbirdie thanks, I merged it and made a new package release.

  • K

    Configure prometheus.yml file

    Solved
    3
    0 Votes
    3 Posts
    501 Views
    K

    @fbartels it works, thanks!

  • ChristopherMagC

    Ownership of the /app/data directory not set to cloudron:cloudron

    Solved
    3
    0 Votes
    3 Posts
    323 Views
    girishG

    @christophermag This should be fixed now.