SweetOps #kubernetes for September, 2023

2023-09-02

2023-09-03

idan levi

Hey all I started to face some wired issue today, somehow i cannot port-forward any service in my EKS cluster (thing that i was able to do earlier ). do someone maybe now how can i get some port-forward command logs ?

THanks !

Hao Wang

12:26:48 PM

Can you please describe the service/deployment being forwarded to?

1

venkata.mutyala

02:40:38 PM

Try a different machine? Maybe spin something up in EC2 and try to curl the service? I believe the API server gets used for the port forwarding so could be an issue there that requires AWS support.

2023-09-06

Xu Pengfei

01:16:46 PM

KCL Biweekly Newsletter (2023 08.24 - 09.06)

Kubernetes Operator, IDE Extensions and v0.5.6 are out! https://kcl-lang.io/blog/2023-09-06-biweekly-newsletter/ Welcome to feedback!

KCL Biweekly Newsletter (2023 08.24 - 09.06) | Kubernetes Operator, IDE Extensions and v0.5.6 are out! | KCL programming language.

KCL is an open-source, constraint-based record and functional language that enhances the writing of complex configurations, including those for cloud-native scenarios. With its advanced programming language technology and practices, KCL is dedicated to promoting better modularity, scalability, and stability for configurations. It enables simpler logic writing and offers ease of automation APIs and integration with homegrown systems.

2023-09-21

Sean

12:54:05 PM

FYI: Critical EKS bug. cgroup configuration is incorrect, which can cause resource issues on heavily loaded clusters:

• https://github.com/awslabs/amazon-eks-ami/issues/1436

#1436 Follow general guideline: containerd and kubelet should share the same kubereserved cgroup

The kubernetes General Guideline explains that the kubelet and the container runtime should share a cgroup slice.
The design proposal for node-allocatable resources also suggests that both kubelet and containerd are to be put into the same cgroup, and that kubelet should share it’s reserved resources with the container runtime.

Following is the recommended cgroup configuration for Kubernetes nodes. All OS system daemons are expected to be placed under a top level SystemReserved cgroup. Kubelet and Container Runtime are expected to be placed under KubeReserved cgroup. The reason for recommending placing the Container Runtime under KubeReserved is as follows:
A container runtime on Kubernetes nodes is not expected to be used outside of the Kubelet.
It’s resource consumption is tied to the number of pods running on a node.

Right now I’m seeing that this is not the case with the EKS AMI when running with default userdata: The cgroup tree is roughly like this:

/
├─runtime.slice
│   └─kubelet.service
└─system.slice
     └─containerd.service

While it should look like this instead:

/
└──runtime.slice
    ├─containerd.service
    └─kubelet.service

Sean

01:34:54 PM

And simple fix proposed: https://github.com/awslabs/amazon-eks-ami/pull/1437/files

#1436 Follow general guideline: containerd and kubelet should share the same kubereserved cgroup

The kubernetes General Guideline explains that the kubelet and the container runtime should share a cgroup slice.
The design proposal for node-allocatable resources also suggests that both kubelet and containerd are to be put into the same cgroup, and that kubelet should share it’s reserved resources with the container runtime.

Following is the recommended cgroup configuration for Kubernetes nodes. All OS system daemons are expected to be placed under a top level SystemReserved cgroup. Kubelet and Container Runtime are expected to be placed under KubeReserved cgroup. The reason for recommending placing the Container Runtime under KubeReserved is as follows:
A container runtime on Kubernetes nodes is not expected to be used outside of the Kubelet.
It’s resource consumption is tied to the number of pods running on a node.

Right now I’m seeing that this is not the case with the EKS AMI when running with default userdata: The cgroup tree is roughly like this:

/
├─runtime.slice
│   └─kubelet.service
└─system.slice
     └─containerd.service

While it should look like this instead:

/
└──runtime.slice
    ├─containerd.service
    └─kubelet.service