#kubernetes (2023-07)

need to disable Autopilot mode if privileged is required

is it possible to disable the autopilot mode?? or do i need to delete this cluster and recreate with standard mode??

https://cloud.google.com/kubernetes-engine/docs/concepts/choose-cluster-mode

When to use Standard instead of Autopilot

https://stackoverflow.com/questions/72615337/migration-existing-standard-gke-to-autopilot-gke

didn’t find the exact words mentioning no conversion though

So it looks as though the cluster-autoscaler is getting incorrect information. Has anyone seen this issue before?

If I hit the metrics server directly:

k get --raw /apis/metrics.k8s.io/v1beta1/nodes/ip-10-48-37-79.eu-west-1.compute.internal | jq '.usage'
{
  "cpu": "1854732210n",
  "memory": "2290572Ki"
}

This is an amd64 c5.xlarge SPOT instance with 4 vCPU

usage_nano=1854732210
capacity_milli=4000

usage_percentage=$(awk "BEGIN { print (($usage_nano / ($capacity_milli * 1000000)) * 100) }")
echo $usage_percentage
46.3683

they should be using the same metric, what is the value at 15:10 for instance?

Ah someone in the Kubernetes slack had this explanation:
cluster-autoscaler is looking at the amount of cpu requests vs node allocatable
This isn’t the amount of cpu time used, it’s the amount of cpu reserved by Pods

e.g. if you have a pod that has a cpu request of 3, and then it does nothing but sleep, cluster-autoscaler will see the node as 75% used, aws will see it as 0%
You can’t scale it down, because that Pod still needs to go somewhere that can satisfy the reservation

got it, differences between request and limit Thanks for sharing the details

Is there GPU being used in the instances?

If there is no GPU, from the codes, the utilization of Daemonsets and MirrorPods are counted by Autoscaler

How did you deploy autoscaler? Both utilizations of Daemonsets and MirrorPods can be disabled by --ignore-daemonsets-utilization and --ignore-mirror-pods-utilization

Then it should have the similar metrics as instance

Thanks for the blog, inspiring

This comparison is interesting, https://kcl-lang.io/docs/user_docs/getting-started/intro/

oh more interesting is why AntGroup invests into DevOps toolings lol

The quality of KCL’s codes and documents look good, and I’ve got some ideas of how it is used now

We are facing some challenges in cloud-native internally, so we have opened up our internal IaC and GitOps practices, Kusion and KCL, hoping to explore and solve common problems with the community. Nice to communicate with you.

How does KCL compare to something like Cuelang?

Hi @kallan.gerard Here are some blogs to explain it . https://kcl-lang.io/blog/2022-declarative-config-overview https://kcl-lang.io/docs/user_docs/getting-started/intro

It depends

I’d avoid thinking of this as strictly an SQS and Kubernetes problem. Queues with n workers are quite a common pattern and they’re part of a system as a whole

What’s producing the messages, what’s the queue size, whats the ordering, what is consuming messages and what is that thing doing with those messages, what is an acceptable amount of time for a message to remain in the queue, are the messages many aggregates with small lifecycles or few aggregates with long lifecycles, etc etc etc

I’d probably just start with one and see what happens

Thanks for the input. Are you scaling workers up and down? If so, based on what? Do you have an example (or two)?

In one example I can think of we use a custom metric with HPA which is based off the age of the oldest unacknowledged message in the queue

It’s a rough metric but it means if the ingesting workers start being overloaded, there will be more messages in the queue than what they can process, which means the message at the back of the queue will get older, which will cause another pod to be scheduled, which will help process messages faster,

But you’ll have to do the engineering for your particular requirements. Like I said these aren’t really kubernetes problems, these are application architecture problems. Don’t mix the two.

Well, I wouldn’t say I’m mixing something here Kubernetes is the tool doing the actual scaling based on metrics we provide. There is also pretty extensive documentation for the HPA like algorithms, scaling behaviour etc. so I’d say it’s definitely part of it of course you should also consider application requirements if there are any other than “please process my messages the fastest way possible”. Interesting metric for scaling. Thanks in any case.

Yes HPA would be whats being used, but Kubernetes is an implementation detail. This is a problem that’s at the application architectural level. Yes there needs to be feedback about what’s actually feasible and sensible with Kubernetes but it’s not what drives the decision.

It’s like deciding how much cereal you need to stow on a ship based on the size of the packets the cereal company sells.

There’s about a dozen questions that need to be decided first that have nothing to do with cereal boxes

I’ve been able to resolve this by adding fsGroup: 1337 to the securityContext as described in this article under “Check the pod and user group”. Even though we aren’t using IRSA on this cluster, and even though we are definitely past 1.19 (on 1.21) it seems to fix if I edit the deployment and put that value in there, in case anyone else is in the same boat!