#random (2023-02)

Non-work banter and water cooler conversation

A place for non-work-related flimflam, faffing, hodge-podge or jibber-jabber you’d prefer to keep out of more focused work-related channels.

Archive: https://archive.sweetops.com/random/

2023-02-02

2023-02-03

2023-02-04

2023-02-06

2023-02-07

Scott Kaminski avatar
Scott Kaminski

I have a large number of EKS deployments (50+), how does everyone deal with upgrades? I would like to make the process as hands off as possible, but its been a high touch scenario for us.

2
Matt Gowie avatar
Matt Gowie

@Scott Kaminski I’d check out https://www.fairwinds.com/insights. I’m unsure if they support doing the actual upgrade, but Insights does provide the deep type of governance + data into each cluster you’re managing, which I imagine is really your concern when governing at scale like that.

Happy to introduce you to their team if you’re interested to get a demo.

Fairwinds Insights | Kubernetes Governance Software
Fairwinds Insights is Kubernetes governance and guardrails software. Unite dev, sec and opsIdentify & remediate Kubernetes misconfigurationsReduce riskOptimize cloud spend
Chris Dobbyn avatar
Chris Dobbyn

Take as much of the work away from AWS as possible. In our scenario we use ArgoCD application sets to automate the in cluster updates based on the cluster version.

Update the cluster via terraform, validate health in ArgoCD before cycling nodes using one or more managed node groups.

Chris Dobbyn avatar
Chris Dobbyn

It’s still time consuming, but way more hands off after you do your first upgrade (which should be a test cluster ideally).

Tim Birkett avatar
Tim Birkett

When you say “upgrades” do you mean the cluster themselves? the nodes? the applications running on top of it? all of them?

Chris Dobbyn avatar
Chris Dobbyn

Upgrades in EKS go like this:

• update the cluster to the target version

• Update the dependent apps (kube-proxy, coredns, aws cni, and there are a few more now)

• Update all nodes to the target version Nodes must be within 1 version of the cluster version, but should not exceed the cluster version

You cannot revert a cluster upgrade, so have a way to test this separate from production.

j l avatar

Hi @Chris Dobbyn, apologies for jumping bit late into the conversation but I was looking for some specific topic regarding upgrades before asking on the channel and found this thread.

I wonder whether is possible to handle the control-plane EKS upgrade without automatically triggering the update on the data-plane.

I am using EKS terraform blueprints module and I am not able to find a way to specify different versions for control-plane and data-plane and I was wondering if this is even possible. I also couldn’t find any relevant information in AWS docs about if that’s possible.

Reasoning for doing this is that workloads are fairly sensitive (stateful) to restarts so I’d like to have full control over the rollout process for worker nodes, my idea was:

  1. Update master nodes
  2. Add additional node group with the newer version
  3. Drain old nodes workloads so that they’re schedule in the new nodegroup
  4. Decomission old nodegroup Does this makes sense?
Chris Dobbyn avatar
Chris Dobbyn

Cluster upgrade and node upgrades are always separate and should be handled that way. So you would have to create a process for your own company. Some modules try to combine both for the sake of convenience and that’s a long-term mistake in my opinion.

I’m not super familiar with the EKS blueprints for managing the addons, but I think it’s in a better spot than it was a long time ago. Your order seems sane to me, I’d add a step between update control plane and adding a new node group. You need to ensure you’re upgrading any addons as required by the cluster upgrade (these are often documented in the EKS upgrade docs). So however you’re doing that.

But your upgrade path makes sense to me. To note though, the managed node group deployed by at least the cloudposse module (but I think this is an EKS thing) will scale up to double before slowly scaling down safely (cordon / drain).

1
j l avatar

Thanks a lot Chris, much clear now!

Tim Birkett avatar
Tim Birkett

Even later to the party… but might still be useful.

I used to use: https://github.com/hellofresh/eks-rolling-update - for node upgrades. Now I’m using Karpenter with a node TTL of 3 days and run 100% spot across a diverse range of instance types and sizes. I use CDK for IaC (cluster and add-on versions are in there).

To get appropriate add-on versions:

aws eks describe-addon-versions --kubernetes-version 1.25 | \
    jq ".addons[] | (.addonName + \": \" + (.addonVersions[] | select(.compatibilities[].defaultVersion==true) | .addonVersion))" 

is a useful command.

Stuff on the cluster: I use helmfile (https://github.com/helmfile/helmfile) to manage helm releases and renovate (https://github.com/renovatebot/renovate) to create PRs with chart updates in (using this Github action: https://github.com/renovatebot/github-action)

1

2023-02-08

2023-02-09

2023-02-10

2023-02-21

2023-02-22

Alex Atkinson avatar
Alex Atkinson

What is the “law” in technology about system behaviors being depended upon by users so they can’t be changed….?

Alex Atkinson avatar
Alex Atkinson
3
1

2023-02-23

2023-02-24

Yonatan Koren avatar
Yonatan Koren

https://github.com/korenyoni/switchblade

Random side project I made on top of fzf and yq. Usually I get away with fzf on my zsh history, but sometimes you need something more structured and permanent. Has support for env var substitution (and preview).

korenyoni/switchblade

Tool for saving, finding and executing commands quickly (built using fzf).

4
1
7
Max Lobur (Cloud Posse) avatar
Max Lobur (Cloud Posse)

Been using something similar https://github.com/dvorka/hstr

korenyoni/switchblade

Tool for saving, finding and executing commands quickly (built using fzf).

Yonatan Koren avatar
Yonatan Koren

Nice, it’s a bit different than the goal of my project though.

My goal is to have a fully configurable toolchain of commands that you can categorize, parameterize and execute.

I have started to rewrite this in native go, but it’s going to take some time…

Right now the mechanism is environment variables, but I want a more robust way of managing the configuration. Maybe even integrate with secret management tools such as teller

tellerops/teller

Cloud native secrets management for developers - never leave your command line for secrets.

Max Lobur (Cloud Posse) avatar
Max Lobur (Cloud Posse)

I see. Can you generate that map from a history?

Yonatan Koren avatar
Yonatan Koren


I see. Can you generate that map from a history?
No but it might be an idea for what I’m going to do. Not as a core feature though. I am fine with the zsh fzf plugin I use except for that it does not work with newlines

#25 Feature: multi-line support

I managed to add support for multi-line blocks, and I want to share it.

What I did was to retrieve the number event of the selected command in fzf, and then use history expansion zle expand-or-complete with that number so zsh gets the correct block of code.

function fzf_history_search() {
  # get event number
  e_num=$(fc -l 0 | fzf --tac | awk '{print $1}')
  
  # HISTORY EXPANSION
  BUFFER="!$e_num"
  zle vi-fetch-history -n $BUFFER
  zle expand-or-complete
}

I modified the script to my needs, didn’t include some stuff, but maybe you like some of what i changed, here I leave it:

# do nothing if fzf is not installed
(( ! $+commands[fzf] )) && return

# Bind for fzf history search
(( ! ${+ZSH_FZF_HISTORY_SEARCH_BIND} )) &&
typeset -g ZSH_FZF_HISTORY_SEARCH_BIND='^r'

# Cursor to end-of-line
(( ! ${+ZSH_FZF_HISTORY_SEARCH_END_OF_LINE} )) &&
typeset -g ZSH_FZF_HISTORY_SEARCH_END_OF_LINE=''

# Include event numbers
(( ! ${+ZSH_FZF_HISTORY_SEARCH_EVENT_NUMBERS} )) &&
typeset -g ZSH_FZF_HISTORY_SEARCH_EVENT_NUMBERS=1

# Include full date timestamps in ISO8601 `yyyy-mm-dd hh:mm' format
(( ! ${+ZSH_FZF_HISTORY_SEARCH_DATES_IN_SEARCH} )) &&
typeset -g ZSH_FZF_HISTORY_SEARCH_DATES_IN_SEARCH=1

# Remove duplicate entries in history
(( ! ${+ZSH_FZF_HISTORY_SEARCH_REMOVE_DUPLICATES} )) &&
typeset -g ZSH_FZF_HISTORY_SEARCH_REMOVE_DUPLICATES=''

function fzf_history_search() {
	local cmd_col=1
	local fc_args=

	# SETUP DATES
	if [ -n "${ZSH_FZF_HISTORY_SEARCH_DATES_IN_SEARCH}" ]; then
		cmd_col=3
		fc_args+=' -i'
	fi

	local his_out=$(fc ${=fc_args} -l 0)

	# REMOVE DUPLICATES
	if [ -n "${ZSH_FZF_HISTORY_SEARCH_REMOVE_DUPLICATES}" ]; then
		his_out=$(printf '%s\n' "$his_out" | sort -k $((cmd_col+1)) | tac | uniq -f $((cmd_col)) | sort)
	fi

	# REMOVE NUMBER COLUMNS
	if ! [ -n "${ZSH_FZF_HISTORY_SEARCH_EVENT_NUMBERS}" ]; then
		his_out=$(printf '%s\n' "$his_out" | sed 's/^[[:blank:]]*[[:digit:]]*//')
	fi

	local e_num=$(printf '%s\n' "$his_out" | fzf --tac -q "$BUFFER" | awk '{print $1}')
	if [ -n "$e_num" ]; then
		BUFFER="!$e_num"
		zle vi-fetch-history -n $BUFFER
		zle expand-or-complete
		if [ -n "$ZSH_FZF_HISTORY_SEARCH_END_OF_LINE" ]; then
			zle end-of-buffer-or-history
		fi
	fi
	#zle reset-prompt
}
autoload fzf_history_search
zle -N fzf_history_search
bindkey $ZSH_FZF_HISTORY_SEARCH_BIND fzf_history_search
Yonatan Koren avatar
Yonatan Koren

The main goal of what I want to do is have a configurable, reusable catalog. History search tools do this fine. So I don’t need to re-invent the wheel.

But maybe I could have an option to search the history using fzf and then when you select it, it’ll copy the YAML map into the clipboard, such that someone can add it to the catalog via their IDE or vim or whatever in another terminal / window.

1
    keyboard_arrow_up