Prometheus - Package Updates

[girish]

[1.24.0]

• Move storage directory to /app/data/storage
• CLI options can now be set in /app/data/env.sh

[nebulon]

[1.25.0]

• Update Prometheus to 2.46.0
• Full changelog
• [FEATURE] Promtool: Add PromQL format and label matcher set/delete commands to promtool. #11411
• [FEATURE] Promtool: Add push metrics command. #12299
• [ENHANCEMENT] Promtool: Read from stdin if no filenames are provided in check rules. #12225
• [ENHANCEMENT] Hetzner SD: Support larger ID's that will be used by Hetzner in September. #12569
• [ENHANCEMENT] Kubernetes SD: Add more labels for endpointslice and endpoints role. #10914
• [ENHANCEMENT] Kubernetes SD: Do not add pods to target group if the PodIP status is not set. #11642
• [ENHANCEMENT] OpenStack SD: Include instance image ID in labels. #12502
• [ENHANCEMENT] Remote Write receiver: Validate the metric names and labels. #11688
• [ENHANCEMENT] Web: Initialize prometheus_http_requests_total metrics with code label set to 200. #12472
• [ENHANCEMENT] TSDB: Add Zstandard compression option for wlog. #11666
• [ENHANCEMENT] TSDB: Support native histograms in snapshot on shutdown. #12258
• [ENHANCEMENT] Labels: Avoid compiling regexes that are literal. #12434
• [BUGFIX] Histograms: Fix parsing of float histograms without zero bucket. #12577
• [BUGFIX] Histograms: Fix scraping native and classic histograms missing some histograms. #12554
• [BUGFIX] Histograms: Enable ingestion of multiple exemplars per sample. 12557
• [BUGFIX] File SD: Fix path handling in File-SD watcher to allow directory monitoring on Windows. #12488
• [BUGFIX] Linode SD: Cast InstanceSpec values to int64 to avoid overflows on 386 architecture. #12568
• [BUGFIX] PromQL Engine: Include query parsing in active-query tracking. #12418
• [BUGFIX] TSDB: Handle TOC parsing failures. #10623

[girish]

Alertmanager [1.6.0]

• Update alertmanaget to 0.25.0
• Full changelog
• [CHANGE] Change the default parse_mode value from MarkdownV2 to HTML for Telegram. #2981
• [CHANGE] Make api_url field optional for Telegram. #2981
• [CHANGE] Use CanonicalMIMEHeaderKey instead of TitleCasing for email headers. #3080
• [CHANGE] Reduce the number of notification logs broadcasted between peers by expiring them after (2 * repeat interval). #2982
• [FEATURE] Add proxy_url support for OAuth2 in HTTP client configuration. #3010
• [FEATURE] Reload TLS certificate and key from disk when updated. #3168
• [FEATURE] Add Discord integration. #2948
• [FEATURE] Add Webex integration. #3132

[girish]

[1.7.0]

• Update alertmanager to 0.26.0
• Full changelog
• [CHANGE] Telegram Integration: api_url is now optional. #2981
• [CHANGE] Telegram Integration: ParseMode default is now HTML instead of MarkdownV2. #2981
• [CHANGE] Webhook Integration: url is now marked as a secret. It will no longer show up in the logs as clear-text. #3228
• [CHANGE] Metrics: New label reason for alertmanager_notifications_failed_total metric to indicate the type of error of the alert delivery. #3094 #3307
• [FEATURE] Clustering: New flag --cluster.label, to help to block any traffic that is not meant for the cluster. #3354
• [FEATURE] Integrations: Add Microsoft Teams as a supported integration. #3324

[girish]

[1.26.0]

• Update Prometheus to 2.47.0
• Full changelog
• [FEATURE] Web: Add OpenTelemetry (OTLP) Ingestion endpoint. #12571 #12643
• [FEATURE] Scraping: Optionally limit detail on dropped targets, to save memory. #12647
• [ENHANCEMENT] TSDB: Write head chunks to disk in the background to reduce blocking. #11818
• [ENHANCEMENT] PromQL: Speed up aggregate and function queries. #12682
• [ENHANCEMENT] PromQL: More efficient evaluation of query with timestamp(). #12579
• [ENHANCEMENT] API: Faster streaming of Labels to JSON. #12598
• [ENHANCEMENT] Agent: Memory pooling optimisation. #12651
• [ENHANCEMENT] TSDB: Prevent storage space leaks due to terminated snapshots on shutdown. #12664
• [ENHANCEMENT] Histograms: Refactoring and optimisations. #12352 #12584 #12596 #12711 #12054
• [ENHANCEMENT] Histograms: Add histogram_stdvar and histogram_stddev functions. #12614
• [ENHANCEMENT] Remote-write: add http.resend_count tracing attribute. #12676

[nebulon]

[1.26.1]

• Update Prometheus to 2.47.1
• Full changelog
• [BUGFIX] Fix duplicate sample detection at chunk size limit #12874

[girish]

[1.8.0]

• Update base image to 4.2.0

[nebulon]

[1.26.2]

• Update Prometheus to 2.47.2
• Full changelog

[nebulon]

[1.27.0]

• Update Prometheus to 2.48.0
• Full changelog
• [CHANGE] Remote-write: respect Retry-After header on 5xx errors. #12677
• [FEATURE] Alerting: Add AWS SigV4 authentication support for Alertmanager endpoints. #12774
• [FEATURE] Promtool: Add support for histograms in the TSDB dump command. #12775
• [FEATURE] PromQL: Add warnings (and annotations) to PromQL query results. #12152 #12982 #12988 #13012
• [FEATURE] Remote-write: Add Azure AD OAuth authentication support for remote write requests. #12572
• [ENHANCEMENT] Remote-write: Add a header to count retried remote write requests. #12729
• [ENHANCEMENT] TSDB: Improve query performance by re-using iterator when moving between series. #12757
• [ENHANCEMENT] UI: Move /targets page discovered labels to expandable section #12824
• [ENHANCEMENT] TSDB: Optimize WBL loading by not sending empty buffers over channel. #12808
• [ENHANCEMENT] TSDB: Reply WBL mmap markers concurrently. #12801
• [ENHANCEMENT] Promtool: Add support for specifying series matchers in the TSDB analyze command. #12842
• [ENHANCEMENT] PromQL: Prevent Prometheus from overallocating memory on subquery with large amount of steps. #12734
• [ENHANCEMENT] PromQL: Add warning when monotonicity is forced in the input to histogram_quantile. #12931
• [ENHANCEMENT] Scraping: Optimize sample appending by reducing garbage. #12939
• [ENHANCEMENT] Storage: Reduce memory allocations in queries that merge series sets. #12938
• [ENHANCEMENT] UI: Show group interval in rules display. #12943
• [ENHANCEMENT] Scraping: Save memory when scraping by delaying creation of buffer. #12953
• [ENHANCEMENT] Agent: Allow ingestion of out-of-order samples. #12897
• [ENHANCEMENT] Promtool: Improve support for native histograms in TSDB analyze command. #12869
• [ENHANCEMENT] Scraping: Add configuration option for tracking staleness of scraped timestamps. #13060
• [BUGFIX] SD: Ensure that discovery managers are properly canceled. #10569
• [BUGFIX] TSDB: Fix PostingsForMatchers race with creating new series. #12558
• [BUGFIX] TSDB: Fix handling of explicit counter reset header in histograms. #12772
• [BUGFIX] SD: Validate HTTP client configuration in HTTP, EC2, Azure, Uyuni, PuppetDB, and Lightsail SDs. #12762 #12811 #12812 #12815 #12814 #12816
• [BUGFIX] TSDB: Fix counter reset edgecases causing native histogram panics. #12838
• [BUGFIX] TSDB: Fix duplicate sample detection at chunk size limit. #12874
• [BUGFIX] Promtool: Fix errors not being reported in check rules command. #12715
• [BUGFIX] TSDB: Avoid panics reported in logs when head initialization takes a long time. #12876
• [BUGFIX] TSDB: Ensure that WBL is repaired when possible. #12406
• [BUGFIX] Storage: Fix crash caused by incorrect mixed samples handling. #13055
• [BUGFIX] TSDB: Fix compactor failures by adding min time to histogram chunks. #13062

[girish]

[1.27.1]

• Update Prometheus to 2.48.1
• Full changelog
• [BUGFIX] TSDB: Make the wlog watcher read segments synchronously when not tailing. #13224
• [BUGFIX] Agent: Participate in notify calls (fixes slow down in remote write handling introduced in 2.45). #13223

[girish]

[1.28.0]

• Update Prometheus to 2.49.0
• Full changelog
• [FEATURE] Promtool: Add --run flag promtool test rules command. #12206
• [FEATURE] SD: Add support for NS records to DNS SD. #13219
• [FEATURE] UI: Add heatmap visualization setting in the Graph tab, useful histograms. #13096 #13371
• [FEATURE] Scraping: Add scrape_config.enable_compression (default true) to disable gzip compression when scraping the target. #13166
• [FEATURE] PromQL: Add a promql-experimental-functions feature flag containing some new experimental PromQL functions. #13103 NOTE: More experimental functions might be added behind the same feature flag in the future. Added functions:
• [FEATURE] SD: Add __meta_linode_gpus label to Linode SD. #13097
• [FEATURE] API: Add exclude_alerts query parameter to /api/v1/rules to only return recording rules. #12999
• [FEATURE] TSDB: --storage.tsdb.retention.time flag value is now exposed as a prometheus_tsdb_retention_limit_seconds metric. #12986
• [FEATURE] Scraping: Add ability to specify priority of scrape protocols to accept during scrape (e.g. to scrape Prometheus proto format for certain jobs). This can be changed by setting global.scrape_protocols and scrape_config.scrape_protocols. #12738

[girish]

[1.28.1]

• Update Prometheus to 2.49.1
• Full changelog
• [BUGFIX] TSDB: Fixed a wrong q= value in scrape accept header #13313

[nebulon]

[1.29.0]

• Update Prometheus to 2.50.0
• Full changelog
• [CHANGE] Remote Write: Error storage.ErrTooOldSample is now generating HTTP error 400 instead of HTTP error 500. #13335
• [FEATURE] Remote Write: Drop old inmemory samples. Activated using the config entry sample_age_limit. #13002
• [FEATURE] Experimental: Add support for ingesting zeros as created timestamps. (enabled under the feature-flag created-timestamp-zero-ingestion). #12733 #13279
• [FEATURE] Promtool: Add analyze histograms command. #12331
• [FEATURE] TSDB/compaction: Add a way to enable overlapping compaction. #13282 #13393 #13398
• [FEATURE] Add automatic memory limit handling. Activated using the feature flag. auto-gomemlimit #13395
• [ENHANCEMENT] Promtool: allow specifying multiple matchers in promtool tsdb dump. #13296
• [ENHANCEMENT] PromQL: Restore more efficient version of NewPossibleNonCounterInfo annotation. #13022
• [ENHANCEMENT] Kuma SD: Extend configuration to allow users to specify client ID. #13278
• [ENHANCEMENT] PromQL: Use natural sort in sort_by_label and sort_by_label_desc. This is experimental. #13411
• [ENHANCEMENT] Native Histograms: support native_histogram_min_bucket_factor in scrape_config. #13222
• [ENHANCEMENT] Native Histograms: Issue warning if histogramRate is applied to the wrong kind of histogram. #13392
• [ENHANCEMENT] TSDB: Make transaction isolation data structures smaller. #13015
• [ENHANCEMENT] TSDB/postings: Optimize merge using Loser Tree. #12878
• [ENHANCEMENT] TSDB: Simplify internal series delete function. #13261
• [ENHANCEMENT] Agent: Performance improvement by making the global hash lookup table smaller. #13262
• [ENHANCEMENT] PromQL: faster execution of metric functions, e.g. abs(), rate() #13446
• [ENHANCEMENT] TSDB: Optimize label values with matchers by taking shortcuts. #13426
• [ENHANCEMENT] Kubernetes SD: Check preconditions earlier and avoid unnecessary checks or iterations in kube_sd. #13408
• [ENHANCEMENT] Promtool: Improve visibility for promtool test rules with JSON colored formatting. #13342
• [ENHANCEMENT] Consoles: Exclude iowait and steal from CPU Utilisation. #9593
• [ENHANCEMENT] Various improvements and optimizations on Native Histograms. #13267, #13215, #13276 #13289, #13340
• [BUGFIX] Scraping: Fix quality value in HTTP Accept header. #13313
• [BUGFIX] UI: Fix usage of the function time() that was crashing. #13371
• [BUGFIX] Azure SD: Fix SD crashing when it finds a VM scale set. #13578

[nebulon]

[1.29.1]

• Update Prometheus to 2.50.1
• Full changelog
• [BUGFIX] API: Fix metadata API using wrong field names. #13633

[girish]

[1.9.0]

• Update alertmanager to 0.27.0
• Full changelog
• [CHANGE] Discord Integration: Enforce max length in message. #3597
• [CHANGE] API: Removal of all api/v1/ endpoints. These endpoints now log and return a deprecation message and respond with a status code of 410. #2970
• [FEATURE] UTF-8 Support: Introduction of support for any UTF-8 character as part of label names and matchers. Please read more below. #3453, #3483, #3567, #3570
• [FEATURE] Metrics: Introduced the experimental feature flag --enable-feature=receiver-name-in-metrics to include the receiver name in the following metrics: #3045
• [FEATURE] Metrics: Introduced a new gauge named alertmanager_inhibition_rules that counts the number of configured inhibition rules. #3681
• [FEATURE] Metrics: Introduced a new counter named alertmanager_alerts_supressed_total that tracks muted alerts, it contains a reason label to indicate the source of the mute. #3565

[nebulon]

[1.30.0]

• Update Prometheus to 2.51.0
• Full changelog

[girish]

[1.30.1]

• Update Prometheus to 2.51.1
• Full changelog
• [BUGFIX] PromQL: Re-instate validation of label_join destination label #13803
• [BUGFIX] Scraping (experimental native histograms): Fix handling of the min bucket factor on sync of targets #13846
• [BUGFIX] PromQL: Some queries could return the same series twice (library use only) #13845

[girish]

[1.30.2]

• Update Prometheus to 2.51.2
• Full changelog
• Notifier: could hang when using relabeling on alerts

[nebulon]

[1.31.0]

• Update Prometheus to 2.52.0
• Full changelog
• [CHANGE] TSDB: Fix the predicate checking for blocks which are beyond the retention period to include the ones right at the retention boundary. #9633
• [FEATURE] Kubernetes SD: Add a new metric prometheus_sd_kubernetes_failures_total to track failed requests to Kubernetes API. #13554
• [FEATURE] Kubernetes SD: Add node and zone metadata labels when using the endpointslice role. #13935
• [FEATURE] Azure SD/Remote Write: Allow usage of Azure authorization SDK. #13099
• [FEATURE] Alerting: Support native histogram templating. #13731
• [FEATURE] Linode SD: Support IPv6 range discovery and region filtering. #13774
• [ENHANCEMENT] PromQL: Performance improvements for queries with regex matchers. #13461
• [ENHANCEMENT] PromQL: Performance improvements when using aggregation operators. #13744
• [ENHANCEMENT] PromQL: Validate label_join destination label. #13803
• [ENHANCEMENT] Scrape: Increment prometheus_target_scrapes_sample_duplicate_timestamp_total metric on duplicated series during one scrape. #12933
• [ENHANCEMENT] TSDB: Many improvements in performance. #13742 #13673 #13782
• [ENHANCEMENT] TSDB: Pause regular block compactions if the head needs to be compacted (prioritize head as it increases memory consumption). #13754
• [ENHANCEMENT] Observability: Improved logging during signal handling termination. #13772
• [ENHANCEMENT] Observability: All log lines for drop series use "num_dropped" key consistently. #13823
• [ENHANCEMENT] Observability: Log chunk snapshot and mmaped chunk replay duration during WAL replay. #13838
• [ENHANCEMENT] Observability: Log if the block is being created from WBL during compaction. #13846
• [BUGFIX] PromQL: Fix inaccurate sample number statistic when querying histograms. #13667
• [BUGFIX] PromQL: Fix histogram_stddev and histogram_stdvar for cases where the histogram has negative buckets. #13852
• [BUGFIX] PromQL: Fix possible duplicated label name and values in a metric result for specific queries. #13845
• [BUGFIX] Scrape: Fix setting native histogram schema factor during scrape. #13846
• [BUGFIX] TSDB: Fix counting of histogram samples when creating WAL checkpoint stats. #13776
• [BUGFIX] TSDB: Fix cases of compacting empty heads. #13755
• [BUGFIX] TSDB: Count float histograms in WAL checkpoint. #13844
• [BUGFIX] Remote Read: Fix memory leak due to broken requests. #13777
• [BUGFIX] API: Stop building response for /api/v1/series/ when the API request was cancelled. #13766
• [BUGFIX] promtool: Fix panic on promtool tsdb analyze --extended when no native histograms are present. #13976

[girish]

[1.32.0]

• Update Prometheus to 2.53.0
• Full changelog
• [FEATURE] Rules: Add new option query_offset for each rule group via rule group configuration file and rule_query_offset as part of the global configuration to have more resilience for remote write delays. #14061 #14216 #14273
• [ENHANCEMENT] Rules: Add rule_group_last_restore_duration_seconds metric to measure the time it takes to restore a rule group. #13974
• [ENHANCEMENT] OTLP: Improve remote write format translation performance by using label set hashes for metric identifiers instead of string based ones. #14006 #13991
• [ENHANCEMENT] TSDB: Optimize querying with regexp matchers. #13620