Add option to configure Prometheus maximum storage capacity (#649)

Option to configure maximum storage capacity for the Prometheus.
This is addition to already configurable time retention.

Signed-off-by: Michal Pryc <mpryc@redhat.com>

Signed-off-by: Michal Pryc <mpryc@redhat.com>

Author: mpryc

charts/pelorus/Chart.yaml

@@ -14,7 +14,7 @@ type: application
 
 # This is the chart version. This version number should be incremented each time you make changes
 # to the chart and its templates, including the app version.
-version: 1.8.2
+version: 1.8.3
 
 dependencies:
   - name: exporters

charts/pelorus/templates/prometheus-cr.yaml

@@ -26,8 +26,11 @@ spec:
       name: thanos-objstore-config
   {{- end }}
   replicas: 2
-  retention: {{ .Values.prometheus_retention | default "1y" }}
   replicaExternalLabelName: ""
+  retention: {{ .Values.prometheus_retention | default "1y" }}
+  {{- if .Values.prometheus_retention_size }}
+  retentionSize: {{ .Values.prometheus_retention_size }}
+  {{- end }}
   ruleNamespaceSelector: {}
   ruleSelector: {}
   serviceAccountName: pelorus-prometheus

charts/pelorus/values.yaml

@@ -19,16 +19,19 @@ extra_prometheus_hosts:
 ## Persistent storage for Prometheus Operator
 # https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/user-guides/storage.md
 
-# Set Prometheus retention time. Older data values are deleted
-# prometheus_retention: 1y
-
 # Uncomment to use PVC for Prometheus
 # prometheus_storage: true
 
 # Additional configuration options for Prometheus PVC
 # prometheus_storage_pvc_capacity: 2Gi
 # prometheus_storage_pvc_storageclass: "gp2"
 
+# Set Prometheus retention time. Older data values are deleted
+# prometheus_retention: 1y
+# Set Prometheus retention size. If data grows over old values are deleted
+# This should be lower of what's defined in the prometheus_storage_pvc_capacity
+# prometheus_retention_size: 1GB
+
 exporters:
   instances:
   - app_name: deploytime-exporter

docs/Install.md

@@ -52,12 +52,20 @@ See [Configuring the Pelorus Stack](Configuration.md) for a full readout of all
 
 ### Configure Prometheus Retention
 
+For detailed information about planning Prometheus storage capacity and configuration options please refer to the [operational aspects](https://prometheus.io/docs/prometheus/latest/storage/#operational-aspects) of the Prometheus documentation.
+
 Prometheus is removing data older than 1 year, so if the metric you are interested in happened to be older than 1 year it won't be visible. This is configurable in the `values.yaml` file with the following option:
 
 ```yaml
 prometheus_retention: 1y
 ```
 
+Additionally user have option to configure maximum size of storage to be used by Prometheus. The oldest data will get removed first if over that limit. If the data is within retention time but over retention size it will also be removed:
+
+```yaml
+prometheus_retention_size: 1GB
+```
+
 ### Configure Prometheus Persistent Volume (Recommended)
 
 Unlike ephemeral volume that have a lifetime of a pod, persistent volume allows to withstand container restarts or crashes making Prometheus data resilient to such situations.