--- title: "Configure Prometheus" description: "Configure Calico Cloud Prometheus rules for denied-packet alerts and persistent storage on connected clusters by editing PrometheusRule and StorageClass resources." product: "Calico Cloud" version: "v20.4.0" section: "Operations" canonical_url: "https://docs.tigera.io/calico-cloud/next/operations/monitor/prometheus/configure-prometheus" --- # Configure Prometheus ## Updating denied packets rules This is an example of how to modify the sample rule created by the sample manifest. The process of updating rules is the same as for user created rules (documented below). - Save the current alert rule: ```bash kubectl -n tigera-prometheus get prometheusrule -o yaml > calico-prometheus-alert-rule-dp.yaml ``` - Make necessary edits to the alerting rules then apply the updated manifest. ```bash kubectl apply -f calico-prometheus-alert-rule-dp.yaml ``` Your changes should be applied in a few seconds by the prometheus-config-reloader container inside the prometheus pod launched by the prometheus-operator (usually named `prometheus-`). As an example, the range query in this Manifest is 10 seconds. ```yaml apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: calico namespace: tigera-prometheus labels: role: tigera-prometheus-rules prometheus: calico-node-prometheus spec: groups: - name: calico.rules rules: - alert: DeniedPacketsRate expr: rate(calico_denied_packets[10s]) > 50 labels: severity: critical annotations: summary: 'Instance {{$labels.instance}} - Large rate of packets denied' description: '{{$labels.instance}} with calico-node pod {{$labels.pod}} has been denying packets at a fast rate {{$labels.sourceIp}} by policy {{$labels.policy}}.' ``` To update this alerting rule, to say, execute the query with a range of 20 seconds modify the manifest to this: ```yaml apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: calico namespace: tigera-prometheus labels: role: tigera-prometheus-rules prometheus: calico-node-prometheus spec: groups: - name: calico.rules rules: - alert: DeniedPacketsRate expr: rate(calico_denied_packets[20s]) > 50 labels: severity: critical annotations: summary: 'Instance {{$labels.instance}} - Large rate of packets denied' description: '{{$labels.instance}} with calico-node pod {{$labels.pod}} has been denying packets at a fast rate {{$labels.sourceIp}} by policy {{$labels.policy}}.' ``` ## Creating a new alerting rule Creating a new alerting rule is straightforward once you figure out what you want your rule to look for. Check [alerting rules](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) and [Queries](https://prometheus.io/docs/querying/examples/) for more information. ### New alerting rule for monitoring Calico node To add the new alerting rule to our Prometheus instance, define a PrometheusRule manifest in the `tigera-prometheus` namespace with the labels `role: tigera-prometheus-rules` and `prometheus: calico-node-prometheus`. The labels should match the labels defined by the `ruleSelector` field of the Prometheus manifest. As an example, to fire a alert when a calico-node instance has been down for more than 5 minutes, save the following to a file, say `calico-node-down-alert.yaml`. ```yaml apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: calico-prometheus-calico-node-down namespace: tigera-prometheus labels: role: tigera-prometheus-rules prometheus: calico-node-prometheus spec: groups: - name: calico.rules rules: - alert: CalicoNodeInstanceDown expr: up == 0 for: 5m labels: severity: warning annotations: summary: 'Instance {{$labels.instance}} Pod: {{$labels.pod}} is down' description: '{{$labels.instance}} of job {{$labels.job}} has been down for more than 5 minutes' ``` Then create/apply this manifest in kubernetes. ```bash kubectl apply -f calico-node-down-alert.yaml ``` Your changes should be applied in a few seconds by the prometheus-config-reloader container inside the prometheus pod launched by the prometheus-operator (usually named `prometheus-`). ### New alerting rule for monitoring BGP peers Let’s look at an example of a new alerting rule to our Prometheus instance with respect to monitoring BGP peering health. Define a PrometheusRule manifest in the tigera-prometheus namespace with the labels `role: tigera-prometheus-rules` and `prometheus: calico-node-prometheus`. The labels should match the labels defined by the `ruleSelector` field of the Prometheus manifest. As an example, to fire an alert when the number of peering connections with a status other than “Established” is increasing at a non-zero rate in the cluster (over the last 5 minutes), save the following to a file, say `tigera-peer-status-not-established.yaml`. ```yaml apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: labels: prometheus: calico-node-prometheus role: tigera-prometheus-rules name: tigera-prometheus-peer-status-not-established namespace: tigera-prometheus spec: groups: - name: calico.rules rules: - alert: CalicoNodePeerStatusNotEstablished annotations: description: '{{$labels.instance}} has at least one peer connection that is no longer up.' summary: Instance {{$labels.instance}} has peer connection that is no longer up expr: rate(bgp_peers{status!~"Established"}[5m]) > 0 labels: severity: critical ``` Then create/apply this manifest in kubernetes. ```bash kubectl apply -f tigera-peer-status-not-established.yaml ``` Your changes should be applied in a few seconds by the prometheus-config-reloader container inside the prometheus pod launched by the prometheus-operator (usually named `prometheus-`). ## Additional Alerting Rules The Alerting Rules installed by the Calico Cloud install manifest is a simple one that fires an alert when the rate of denied packets denied by a policy on a node from a particular Source IP exceeds a certain packets per second threshold. The Prometheus query used for this (ignoring the threshold value 20) is: ```text rate(calico_denied_packets[10s]) ``` and this query will return results something along the lines of: ```text {endpoint="calico-metrics-port",instance="10.240.0.81:9081",job="calico-node-metrics",namespace="kube-system",pod="calico-node-hn0kl",policy="profile/k8s_ns.test/0/deny",service="calico-node-metrics",srcIP="192.168.167.129"} 0.6 {endpoint="calico-metrics-port",instance="10.240.0.84:9081",job="calico-node-metrics",namespace="kube-system",pod="calico-node-97m3g",policy="profile/k8s_ns.test/0/deny",service="calico-node-metrics",srcIP="192.168.167.175"} 0.2 {endpoint="calico-metrics-port",instance="10.240.0.84:9081",job="calico-node-metrics",namespace="kube-system",pod="calico-node-97m3g",policy="profile/k8s_ns.test/0/deny",service="calico-node-metrics",srcIP="192.168.252.157"} 0.4 {endpoint="calico-metrics-port",instance="10.240.0.81:9081",job="calico-node-metrics",namespace="kube-system",pod="calico-node-hn0kl",policy="profile/k8s_ns.test/0/deny",service="calico-node-metrics",srcIP="192.168.167.175"} 1 {endpoint="calico-metrics-port",instance="10.240.0.84:9081",job="calico-node-metrics",namespace="kube-system",pod="calico-node-97m3g",policy="profile/k8s_ns.test/0/deny",service="calico-node-metrics",srcIP="192.168.167.129"} 0.4 {endpoint="calico-metrics-port",instance="10.240.0.81:9081",job="calico-node-metrics",namespace="kube-system",pod="calico-node-hn0kl",policy="profile/k8s_ns.test/0/deny",service="calico-node-metrics",srcIP="192.168.167.159"} 0.4 {endpoint="calico-metrics-port",instance="10.240.0.81:9081",job="calico-node-metrics",namespace="kube-system",pod="calico-node-hn0kl",policy="profile/k8s_ns.test/0/deny",service="calico-node-metrics",srcIP="192.168.252.175"} 0.4 {endpoint="calico-metrics-port",instance="10.240.0.84:9081",job="calico-node-metrics",namespace="kube-system",pod="calico-node-97m3g",policy="profile/k8s_ns.test/0/deny",service="calico-node-metrics",srcIP="192.168.252.175"} 0.6 {endpoint="calico-metrics-port",instance="10.240.0.81:9081",job="calico-node-metrics",namespace="kube-system",pod="calico-node-hn0kl",policy="profile/k8s_ns.test/0/deny",service="calico-node-metrics",srcIP="192.168.252.157"} 0.6 {endpoint="calico-metrics-port",instance="10.240.0.84:9081",job="calico-node-metrics",namespace="kube-system",pod="calico-node-97m3g",policy="profile/k8s_ns.test/0/deny",service="calico-node-metrics",srcIP="192.168.167.159"} 0.6 ``` We can modify this query to find out all packets dropped by different policies on every node. ```text (sum by (instance,policy) (rate(calico_denied_packets[10s]))) ``` This query will aggregate the results from all different Source IPs, and preserve the `policy` and `instance` labels. Note that the `instance` label represents the calico node's IP Address and `PrometheusReporterPort`. This query will return results like so: ```text {instance="10.240.0.84:9081",policy="profile/k8s_ns.test/0/deny"} 2 {instance="10.240.0.81:9081",policy="profile/k8s_ns.test/0/deny"} 2.8 ``` To include the pod name in these results, add the label `pod` to the labels listed in the `by` expression like so: ```text (sum by (instance,pod,policy) (rate(calico_denied_packets[10s]))) ``` which will return the following results: ```text {instance="10.240.0.84:9081",pod="calico-node-97m3g",policy="profile/k8s_ns.test/0/deny"} 2 {instance="10.240.0.81:9081",pod="calico-node-hn0kl",policy="profile/k8s_ns.test/0/deny"} 2.8 ``` An interesting use case is when a rogue Pod is using tools such as nmap to scan a subnet for open ports. To do this, we have to execute a query that will aggregate across all policies on all instances while preserving the source IP address. This can be done using this query: ```text (sum by (srcIP) (rate(calico_denied_packets[10s]))) ``` which will return results, different source IP address: ```text {srcIP="192.168.167.159"} 1.0000000000000002 {srcIP="192.168.167.129"} 1.2000000000000002 {srcIP="192.168.252.175"} 1.4000000000000001 {srcIP="192.168.167.175"} 0.4 {srcIP="192.168.252.157"} 1.0000000000000002 ``` To use these queries as Alerting Rules, follow the instructions defined in the [Creating a new Alerting Rule](#creating-a-new-alerting-rule) section and create a ConfigMap with the appropriate query. ## Updating the scrape interval You may wish to modify the scrape interval (time between Prometheus polling each node for new denied packet information). Increasing the interval reduces load on Prometheus and the amount of storage required, but decreases the detail of the collected metrics. The scrape interval of endpoints (calico-node in our case) is defined as part of the ServiceMonitor manifest. To change the interval: - Save the current ServiceMonitor manifest: ```bash kubectl -n tigera-prometheus get servicemonitor calico-node-monitor -o yaml > calico-node-monitor.yaml ``` - Update the `interval` field under `endpoints` to desired settings and apply the updated manifest. ```bash kubectl apply -f calico-node-monitor.yaml ``` Your changes should be applied in a few seconds by the prometheus-config-reloader container inside the prometheus pod launched by the prometheus-operator (usually named `prometheus-`). As an example on what to update, the interval in this ServiceMonitor manifest is 5 seconds (`5s`). ```yaml apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: calico-node-monitor namespace: tigera-prometheus labels: team: network-operators spec: selector: matchLabels: k8s-app: calico-node namespaceSelector: matchNames: - kube-system endpoints: - port: calico-metrics-port interval: 5s ``` To update Calico Cloud Prometheus' scrape interval to 10 seconds modify the manifest to this: ```yaml apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: calico-node-monitor namespace: tigera-prometheus labels: team: network-operators spec: selector: matchLabels: k8s-app: calico-node namespaceSelector: matchNames: - kube-system endpoints: - port: calico-metrics-port interval: 10s ``` ## Troubleshooting Config Updates Check config reloader logs to see if they detected any recent activity. - For prometheus run: ```bash kubectl -n tigera-prometheus logs prometheus- prometheus-config-reloader ``` - For alertmanager run: ```bash kubectl -n tigera-prometheus logs alertmanager- config-reloader ``` The config-reloaders watch each pods file-system for updated config from ConfigMap's or Secret's and will perform steps necessary for reloading the configuration.