#kubernetes (2024-06)

https://github.com/jonmosco/kube-ps1

For zsh in particular, I like https://github.com/romkatv/powerlevel10k It shows k8s contexts (active cluster name and namespace) as well as many many other useful bits of context info (like cloud provider account names, etc…)

I have a script that runs on every new terminal window setup in my zsh config to tell me what context I’m in, and I also run all my kubectl and Terraform commands through an alias that requires me to answer a Y/N prompt to confirm I really want to touch the Staging/Prod cluster (with Yellow/Red colouring respectively to really hit the point home)

I’ll grab it tomorrow when I login and stick it here

From my .zshrc file

Terraform/Terragrunt

for Terraform (we use Terragrunt so mine is geared towards that) where we have a staging and a production directory to separate the IaC which isn’t shared:

alias tg='AWS_PROFILE=<MYPROFILE> terragrunt'

tg_production() {
    if [[ "$PWD" == *production* ]]; then
        # Prompt for confirmation in red color
        echo -e "\033[31mYou are about to run the 'tg' command in a 'production' directory. Are you sure? (y/n)\033[0m"
        read -r response
        if [[ "$response" =~ ^[Yy]$ ]]; then
            tg "$@"
        else
            echo "Command cancelled."
        fi
    else
        echo "The current directory does not contain 'production' in its path."
    fi
}

tg_staging() {
    if [[ "$PWD" == *staging* ]]; then
        # Prompt for confirmation in yellow color
        echo -e "\033[33mYou are about to run the 'tg' command in a 'staging' directory. Are you sure? (y/n)\033[0m"
        read -r response
        if [[ "$response" =~ ^[Yy]$ ]]; then
            tg "$@"
        else
            echo "Command cancelled."
        fi
    else
        echo "The current directory does not contain 'staging' in its path."
    fi
}

This forces me to ensure I am running a command intended for production from within the production directory, and asks for extra confirmation. Same for staging.

Kubernetes

context="$(kubectl config current-context)"

case "$context" in
    *production*) echo -e "\033[31mWARNING!!! YOU ARE IN PRODUCTION KUBE CONTEXT\033[0m" ;;
    *staging*) echo -e "\033[33mWARNING!!! YOU ARE IN STAGING KUBE CONTEXT\033[0m" ;;
    *) echo "Safe!" ;;
esac

I also use Warp terminal which sets the prompt to the current k8s context whenever you type kubectl

Actually, setting resource limits is crucial. Without them, Kubernetes doesn’t know how much CPU/memory your app needs, making it hard to scale effectively. It’s like setting widths/heights in CSS for predictable layouts. Limits guide the scheduler, ensuring stability and efficient resource use.

Setting allocations is important. For CPU, it’s preferable to set requests instead of limits to prevent throttling. For memory, requests and limits should be set.

This post has a good explanation of the details: https://home.robusta.dev/blog/stop-using-cpu-limits

@Erik Osterman (Cloud Posse) but I am getting worried that the pod would crash when the code runs out of memory/cpu during high usage

@Nate McCurdy Ooooh I see! I’ll read the docu, thank you!

The deployment will start with 4 replicas, and the HPA will continuously evaluate the resource utilization. If the utilization is low, the HPA will scale down to the minimum of 2 replicas. The HPA dynamically adjusts the replica count to ensure optimal resource usage.

Ok, perfect, I’m glad my assumption was right. You’re a lifesaver, thanks for weighing in quickly!

I swear when I first implemented the HPA the Kubernetes docs had a dedicated section in their ‘Concepts’ section for HPAs but all their scaling stuff seems to have been moved into real-world-example-type docs now which isn’t as helpful to me as a reference

Yea, when things get so complicated as in Kubernetes, that helps a lot

Also, checkout Keda if you end up needing to do more specialized autoscaling

Forgot to reply yesterday but just wanted to say thanks for the heads up on Keda! - I’ve been testing new scaling configs today with it and I’m really impressed with the flexibility I now have without having to do much work on integration. Some of the metrics I can now easily scale by make me wonder: “Jeez why isn’t this supported by default?” Though of course the answer is because K8s development has lots of other priorities.

As a generic follow up question: Are there any other custom resources you’ve come across that have enhanced K8s in a way that shifted the way you work significantly like Keda looks like it will for me?

Don’t recommend. Kubernetes doesn’t natively support multitenancy and isolation. One bad actor can bring down cluster for everyone. Network latency will eat your SLO for breakfast.

Not to mention kubernetes version upgrades between nodes and control plane.

RBAC, Network Policy, Authentication, QoS and DNS setup will make you to lose sanity in this setup.

Hmm, so generally speaking, we’re talking about a single control plane per customer?

The only reason we want this is so data never leaves their account/perimeter. We just want our controllers, admission c, security services to ensure customer’s cannot do anything, except have exposure to the service/product we’re providing them.

I don’t develop such things, so my suggestions might not be what you look.

If you want to have your product installable on kubernetes, just let customers install it via manifests or helm chart on their own cluster. It’s up to them to manage cluster.

There is no good reason to take over control plane to install something on cluster.

We have that offering also, but it’s difficult to enforce things we need from an enterprise standpoint. Sharing the nodes with the customer, while retaining control over the control plane allows us to have a single point of entry for authz/authn, deploying product updates, updating customer configurations for their setup in near-realtime.

Everything you mentioned can be implemented with native kubernetes resources and application logic. I still don’t see you point ¯_(ツ)_/¯

Yeah enforcing license requirements in an isolated environment is not possible in our scenario. If the customer “cuts the cord” we can’t do anything about it. We won’t have any monitoring/metrics, etc. So this hybrid model came into being, we’re still flushing it all out.

Any clues from looking at the live-locobuzzing-api span that took ~6s? Is the span broken down further?

I take it you have reviewed the flame graph of the login service and done a profile of it to rule out the login service itself being the bottleneck?

I only ask because on your second image there are 4 times as many spans being indexed, so it is making me wonder whether between the two screenshots something has invalidated a cache your app relies on and it is having to rebuild that? Maybe a new pod of that service has been spun up from a scaling event and your containers don’t come with the cache prewarmed?

any url or logs?

I checked the Alb controller and argocd logs I don’t find any errors over there and i could see sync policy none in app details

is there an annotation related to this auth?

Yes we are using some annotations in ingress manifest

can you show the annotations?

guessing one of them may cause the issue

set retention policy to retain for the persistent volume used by the postgresql-ha statefulset (either manually or via helm chart values), remove postgres-ha helm deployment and reuse existing volume in postgresql chart alternatively, setup new postgres server with postgresql chart next to existing onw and use pg_dump and pg_restore to move the data. test either scenario on some dev environment before running this on production

Thank you @Piotr Pawlowski, I did disable the postgres-ha and instruct the Postgres chart to use the existing pvc , but when the db pod starts I get:

│ postgresql 15:22:40.22 INFO  ==> ** Starting PostgreSQL setup **                                                                                                             │
│ postgresql 15:22:40.24 INFO  ==> Validating settings in POSTGRESQL_* env vars..                                                                                              │
│ postgresql 15:22:40.25 INFO  ==> Loading custom pre-init scripts...                                                                                                          │
│ postgresql 15:22:40.26 INFO  ==> Initializing PostgreSQL database...                                                                                                         │
│ postgresql 15:22:40.28 INFO  ==> Custom configuration /opt/bitnami/postgresql/conf/postgresql.conf detected                                                                  │
│ postgresql 15:22:40.28 INFO  ==> Custom configuration /opt/bitnami/postgresql/conf/pg_hba.conf detected                                                                      │
│ postgresql 15:22:40.30 INFO  ==> Deploying PostgreSQL with persisted data...                                                                                                 │
│ postgresql 15:22:40.31 INFO  ==> Loading custom scripts...                                                                                                                   │
│ postgresql 15:22:40.32 INFO  ==> ** PostgreSQL setup finished! **                                                                                                            │
│                                                                                                                                                                              │
│ postgresql 15:22:40.34 INFO  ==> ** Starting PostgreSQL **                                                                                                                   │
│ 2024-06-25 15:22:40.400 GMT [1] FATAL:  could not access file "repmgr": No such file or directory                                                                            │
│ 2024-06-25 15:22:40.400 GMT [1] LOG:  database system is shut down                                                                                                           

Just an update, the only thing that worked is backup and restore, running the no ha pod on an hap pod’s volume is not working

Yup, k8s scheduler is not AZ aware when it comes to that sort of thing.

We always deploy 3 separate node pools, one per AZ

It also enables the scheduler to scale up specific AZs based on traffic.

Alternatively, you can use EFS whcih is not zone specific, but also not suitable for all kinds of workloads

We typically make EFS the default storage class

is “node pool” the same as “node group”?

yes… the terms can be used interchangeably

i do have efs already setup, but it’s not the default storage class. maybe i’ll try that

In our refarch, here’s our component https://github.com/cloudposse/terraform-aws-components/tree/23f29ccd8727cc1fbe8f19ab6b3b71dd37316a00/modules/eks/storage-class

Wouldn’t Karpenter be a good solution to working around this? You would just need a tiny node that is running anywhere to host the karpenter controller pod, then you can instruct Karpenter to only provision in a preferred AZ?

I’ve never used Karpenter, so I don’t really know?

@Jeremy G (Cloud Posse)

A few things here:

“Node Group” is an EKS term. As I understand it, it is defined as a collection of nodes sharing a single autoscaling group (ASG) and launch template (LT).

“Node Pool” is a Karpenter term, effectively the same kind of thing, but using Karpenter to launch nodes, rather than ASGs and LTs.

If you create an ASG that spans multiple Availability Zones (AZs), the K8s cluster autoscaler (or any autoscaler) does not have control over which AZ a new EC2 instance will be added to when the ASG scales up. At some point, the EC2 Autoscaler may see that there is a big imbalance and try to fix it by bringing up instances in one AZ and shutting them down in another, but again there is no way to proactively direct this.

Our eks/cluster component handles availability zone balance by creating one managed node group (MNG) per AZ. The K8s cluster autoscaler understands this configuration, and will keep the MNGs in balance across the AZs.

Furthermore, Kubernetes, the K8s cluster autoscaler, and Karpenter all understand that EBS-backed PVs are tied to a “topology zone” (generic term for what in AWS is an AZ), and will only schedule a Pod into the same zone as its PVC targets.

If there is not enough capacity in the zone with the PVC, K8s need to add a new Node in that AZ. Karpenter can always do this. The best the K8s cluster autoscaler can do is increase the desired capacity of an ASG. If the ASG is only in the target AZ, then it is guaranteed success, but if the ASG spans AZs, then scaling up the ASG may or may not increase capacity in the target zone, in which case I think the K8s cluster autoscaler just gives up.

EFS-backed EBS volumes solve the AZ problem, but at cost of 3x the $/MB price and severely limited IO bandwidth and IOPS limits. It’s appropriate for sharing configuration data and small amounts of logging. It gets overwhelmed by storing Prometheus’ scraped metrics unless you pay (dearly) for extra bandwidth.

The short answer, @loren, is that if you only intend to run one Node and you want a Persistent Volume, restrict your ASG to one AZ and use a gp3 EBS volume for your PV.

Thanks for that Jeremy! I’ll need to refactor things a fair bit to scale out the node group per az. Still working out some EFS issues on destroy, where the namespace gets locked up because the associated PVC has a claim but (I think) the access points got destroyed before the namespace, so K8S fails to remove the finalizers. Good times. Love K8S so so much.

i don’t think i ever would have thought about a node group per az, but what a difference! replacing updates on node groups now down to ~8min vs >40min! thanks again!