#random (2023-02)

@Scott Kaminski I’d check out https://www.fairwinds.com/insights. I’m unsure if they support doing the actual upgrade, but Insights does provide the deep type of governance + data into each cluster you’re managing, which I imagine is really your concern when governing at scale like that.

Happy to introduce you to their team if you’re interested to get a demo.

Take as much of the work away from AWS as possible. In our scenario we use ArgoCD application sets to automate the in cluster updates based on the cluster version.

Update the cluster via terraform, validate health in ArgoCD before cycling nodes using one or more managed node groups.

It’s still time consuming, but way more hands off after you do your first upgrade (which should be a test cluster ideally).

When you say “upgrades” do you mean the cluster themselves? the nodes? the applications running on top of it? all of them?

Upgrades in EKS go like this:

• update the cluster to the target version

• Update the dependent apps (kube-proxy, coredns, aws cni, and there are a few more now)

• Update all nodes to the target version Nodes must be within 1 version of the cluster version, but should not exceed the cluster version

You cannot revert a cluster upgrade, so have a way to test this separate from production.

Hi @Chris Dobbyn, apologies for jumping bit late into the conversation but I was looking for some specific topic regarding upgrades before asking on the channel and found this thread.

I wonder whether is possible to handle the control-plane EKS upgrade without automatically triggering the update on the data-plane.

I am using EKS terraform blueprints module and I am not able to find a way to specify different versions for control-plane and data-plane and I was wondering if this is even possible. I also couldn’t find any relevant information in AWS docs about if that’s possible.

Reasoning for doing this is that workloads are fairly sensitive (stateful) to restarts so I’d like to have full control over the rollout process for worker nodes, my idea was:

1. Update master nodes
2. Add additional node group with the newer version
3. Drain old nodes workloads so that they’re schedule in the new nodegroup
4. Decomission old nodegroup Does this makes sense?

Cluster upgrade and node upgrades are always separate and should be handled that way. So you would have to create a process for your own company. Some modules try to combine both for the sake of convenience and that’s a long-term mistake in my opinion.

I’m not super familiar with the EKS blueprints for managing the addons, but I think it’s in a better spot than it was a long time ago. Your order seems sane to me, I’d add a step between update control plane and adding a new node group. You need to ensure you’re upgrading any addons as required by the cluster upgrade (these are often documented in the EKS upgrade docs). So however you’re doing that.

But your upgrade path makes sense to me. To note though, the managed node group deployed by at least the cloudposse module (but I think this is an EKS thing) will scale up to double before slowly scaling down safely (cordon / drain).

Thanks a lot Chris, much clear now!

Even later to the party… but might still be useful.

I used to use: https://github.com/hellofresh/eks-rolling-update - for node upgrades. Now I’m using Karpenter with a node TTL of 3 days and run 100% spot across a diverse range of instance types and sizes. I use CDK for IaC (cluster and add-on versions are in there).

To get appropriate add-on versions:

aws eks describe-addon-versions --kubernetes-version 1.25 | \
    jq ".addons[] | (.addonName + \": \" + (.addonVersions[] | select(.compatibilities[].defaultVersion==true) | .addonVersion))" 

is a useful command.

Stuff on the cluster: I use helmfile (https://github.com/helmfile/helmfile) to manage helm releases and renovate (https://github.com/renovatebot/renovate) to create PRs with chart updates in (using this Github action: https://github.com/renovatebot/github-action)

Been using something similar https://github.com/dvorka/hstr

with this settings https://github.com/max-lobur/dotfiles/blob/master/dotfiles/.zshrc#L27-L29

Nice, it’s a bit different than the goal of my project though.

My goal is to have a fully configurable toolchain of commands that you can categorize, parameterize and execute.

I have started to rewrite this in native go, but it’s going to take some time…

Right now the mechanism is environment variables, but I want a more robust way of managing the configuration. Maybe even integrate with secret management tools such as teller

I see. Can you generate that map from a history?

I see. Can you generate that map from a history? No but it might be an idea for what I’m going to do. Not as a core feature though. I am fine with the zsh fzf plugin I use except for that it does not work with newlines

The main goal of what I want to do is have a configurable, reusable catalog. History search tools do this fine. So I don’t need to re-invent the wheel.

But maybe I could have an option to search the history using fzf and then when you select it, it’ll copy the YAML map into the clipboard, such that someone can add it to the catalog via their IDE or vim or whatever in another terminal / window.