#kubernetes (2022-02)

kbrew is homebrew for Kubernetes

Collection of gadgets for debugging and introspecting Kubernetes applications using BPF

What this PR does / why we need it:

This PR Introduce a new flag --per-metric-labels-allowlist that its syntax works similar to --metric-labels-allowlist but instead of being a filter to add labels to kube_X_labels, it is a filter for K8S’ labels that will be added to each metric time series of a resource.

Motivation

Motivation for this change can be better described from a Platform Team POV, who is responsible for the observability stack and provides it to multiple teams and/or multiple tenants.

The goal is to make it easier to define queries, alerts, and rules without the need for complex joins. Making the barrier for smaller less experienced teams smaller. As well as alleviate pressure from Prometheus Server that’s constantly doing joins for each alert rule.

Use Case 1

1. A Development Team wants to create alerts for the multiple components of their applications.
2. Different Components have different alerts, severities, thresholds. (example, web pods, background consumers, different kinds of jobs) and as components live in the same namespace; filtering with a namespace is not feasible.
3. You’ll now have to use joins with kube_X_labels to filter for the specific resources. Complex queries become more complex, especially ones that had joins already.

Use Case 2

1. Platform Team defining general, default rules for every namespace.
2. Platform team expects resources to have a set of standard labels, that defines something like [alerting.xxx/severity](http://alerting.xxx/severity), [alerting.xxx/slack-channel](http://alerting.xxx/slack-channel) along side the [app.kubernetes.io/name](http://app.kubernetes.io/name), [app.kubernetes.io/component](http://app.kubernetes.io/component) ones.
3. Complex Rules are defined to join with these labels and generate alerts with variables based on labels defined by teams. Queries become even more complex as you want to join with more than one label.

Complex Queries Example

Deployment has not matched the expected number of replicas.

• Using only 1 label for joins

    (
     kube_deployment_spec_replicas{} * on (deployment) group_left(label_app_kubernetes_io_name) kube_deployment_labels{ label_app_kubernetes_io_name="emojiapp", namespace=~"emoji"}
     !=
     kube_deployment_status_replicas_available{} * on (deployment) group_left(label_app_kubernetes_io_name) kube_deployment_labels{ label_app_kubernetes_io_name="emojiapp", namespace=~"emoji"}
    ) 
    and 
    (
     changes(kube_deployment_status_replicas_updated{}[10m] ) * on (deployment) group_left(label_app_kubernetes_io_name) kube_deployment_labels{ label_app_kubernetes_io_name="emojiapp", namespace=~"emoji"}
     ==  0
    )
    
    

• Using 2 label for joins

    (
     kube_deployment_spec_replicas{} * on (deployment) group_left(label_app_kubernetes_io_name, label_alerting_severity) kube_deployment_labels{ label_app_kubernetes_io_name="emojiapp", label_alerting_severity="critical", namespace=~"emoji"}
     !=
     kube_deployment_status_replicas_available{} * on (deployment) group_left(label_app_kubernetes_io_name, label_alerting_severity) kube_deployment_labels{ label_app_kubernetes_io_name="emojiapp", label_alerting_severity="critical", namespace=~"emoji"}
    ) 
    and 
    (
     changes(kube_deployment_status_replicas_updated{}[10m] ) * on (deployment) group_left(label_app_kubernetes_io_name, label_alerting_severity) kube_deployment_labels{ label_app_kubernetes_io_name="emojiapp", label_alerting_severity="critical", namespace=~"emoji"}
     ==  0
    )
    
    

Same query but with labels as part of metric series:

•

    (
      kube_deployment_spec_replicas{label_app_kubernetes_io_name="emojiapp", namespace=~"emoji"} 
      !=
      kube_deployment_status_replicas_available{label_app_kubernetes_io_name="emojiapp", namespace=~"emoji"}
    ) 
    and 
    (
      changes(kube_deployment_status_replicas_updated{label_app_kubernetes_io_name="emojiapp", namespace=~"emoji"}[10m] )
    )
    
    

•

    (
      kube_deployment_spec_replicas{label_app_kubernetes_io_name="emojiapp", label_alerting_severity="critical", namespace=~"emoji"} 
      !=
      kube_deployment_status_replicas_available{label_app_kubernetes_io_name="emojiapp", label_alerting_severity="critical", namespace=~"emoji"}
    ) 
    and 
    (
      changes(kube_deployment_status_replicas_updated{label_app_kubernetes_io_name="emojiapp", label_alerting_severity="critical", namespace=~"emoji"}[10m] )
    )
    
    

Goal

• Making it easier for Platform and Development teams to create queries, rules, alerts, and dashboards for resources running in Kubernetes without the need for complex joins to filter resources. • Alleviate some pressure from the Prometheus Servers that are constantly running Joins for each alert rule.

Alternatives we tried

• Run recording rules to pre-compute the joins:
Having to have a recording rule for each metric series generated by KSM is cumbersome, and adds a dependency between teams and platform teams to add any recording rule. Not ideal especially in a multi-tenant environment.

How does this change affect the cardinality of KSM

(increases, decreases, or does not change cardinality)

• Unless explicitly using the new flag, this PR has no change to generated metrics cardinality. • Using the flag will add new labels for each metric series of each resource that has a label key that’s whitelisted. • Cardinality of the label values depends on how often these labels change for the same resource. • For the use-case behind this new feature, whitelisted labels are often labels that don’t change often. Admins should be cautious of what labels to whitelist.

Performance

• KSM already fetches each resource label during metric collection. • Prometheus Performance shouldn’t be affected as long as whitelisted labels are not changing constantly.

Misc

Which issue(s) this PR fixes:

• Fixes #1415

Relevant:

• Dynamic alert routing with Prometheus and Alertmanager

Notes:

• :warning: This is a Draft PR, the implementation is not final, PR is a working POC for pods resource, and is yet to be discussed.

Learn how to navigate the creating of a secure by default K8s cluster, avoid risky default settings and permissions, and listen to some live threat modeling of security EKS clusters. Join Lightspin CISO Jonathan Rau and Director of Security Research Gafnit Amiga to discuss hot topics and tips for leveling up your Kubernetes security knowledge. Questions and topics covered include: - Avoiding risky default settings in your Kubernetes clusters - Creating a secure by default Kubernetes cluster - Unique supply chain risks for Kubernetes Bring your questions and notepads to this live webinar!