#kubernetes (2022-09)

kubernetes

Archive: https://archive.sweetops.com/kubernetes/

2022-09-01

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Can’t give anything too specific in this forum - but this is in active development. Will move to coming soon once I feel it aligns the roadmap categories defined here https://github.com/aws/containers-roadmap#faqs And yes, this will be an AWS API - and it will include CloudFormation support.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Jeremy G (Cloud Posse)

Can’t give anything too specific in this forum - but this is in active development. Will move to coming soon once I feel it aligns the roadmap categories defined here https://github.com/aws/containers-roadmap#faqs And yes, this will be an AWS API - and it will include CloudFormation support.

2022-09-05

Steve Chernyak avatar
Steve Chernyak

does anybody know where i can find details around how memory is throttled when k8s is running under cgroupsv2 with the feature enabled? I’m trying to wrap my head around what it means for memory to be treated as a “compressible” resource.

2022-09-07

2022-09-08

akhan4u avatar
akhan4u

Hey guys,

Facing an Issue running jenkins-operator on kubernetes cluster v1.22.11-eks with Datadog monitoring agent. The DD agent is injecting some ENV vars in the jenkins-instance when created ex: DD_AGENT_HOST & DD_ENTITY_ID. The above ENV vars are causing the operator to restart the jenkins-instance pod in a loop.

Did anyone of you have used jenkins-operator next to a monitoring agent like Datadog, Newrelic, etc?

1
akhan4u avatar
akhan4u

I found a suggestion from this page to use a different ENV var DATADOG_JENKINS_PLUGIN_TARGET_HOST. but that didn’t helped me either.

akhan4u avatar
akhan4u

Jenkins Operator Chart details

NAME                               	CHART VERSION	APP VERSION	DESCRIPTION
jenkinsci/jenkins-operator	0.6.2        	0.7.1      	Kubernetes native operator which fully manages ...

Datadog Chart details

NAME                               	CHART VERSION	APP VERSION	DESCRIPTION
datadog/datadog                    	2.36.6       	7          	Datadog Agent
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Hrmmm… could it be running out of memory when jenkins stats due to pod limits?

akhan4u avatar
akhan4u

I’ve updated the pod limits. But the problems seems to be related to only ENV var injection by DD.

Vinícius Azevedo avatar
Vinícius Azevedo

Can someone help me on understanding an anti-affinity policy I am creating for an application? I’m trying to guarantee each new replica will be deployed in a different AZ, so I came up with the following block:

affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      podAffinityTerm:
        labelSelector:
          matchExpressions:
          - key: app.kubernetes.io/name
            operator: In
            values:
            - ${app-name}
       topologyKey: topology.kubernetes.io/zone

So, my thinking is:

  1. I am creating an Anti-Affinity policy, meaning I want to avoid, if possible (thus the preferred element), assigning the pod to a node according to certain rules
  2. I am using podAffinityTerm and labelSelector, meaning I am looking for labels in pods that are already assigned to my cluster
  3. I am looking for pods with a label common to all my application pods (the scheduler will use the information in matchExpressions, key, operator, and values)
  4. I am telling the scheduler to look for a different zone (as defined in topologyKey) if the criteria is met
  5. I know that if all the zones are already filled with pods of my application, then the scheduler will be free to assign to any node (because no other rules are defined, and I defined a soft rule for the anti-affinity) So is my understanding correct?
Anirudh Ramanathan avatar
Anirudh Ramanathan

Hi folks, I’m Anirudh. I used to work on K8s core controllers in the past, and for the past 2y I’ve been working on a platform called Signadot to test microservices in K8s at scale. Rather than stamping out new copies of infrastructure, the approach we took is to make use of request-level tenancy and dynamic request routing to isolate environments. This makes it possible to get lightweight environments which can share resources with each other while isolating at the request level, similar to how the “copy-on-write” model works for memory. Just launched on Product Hunt and would love to get feedback if you have a few minutes to spare. TIA!

1

2022-09-09

2022-09-12

mr.shayv avatar
mr.shayv

Does anyone know how to troubleshoot a pod stuck on ContainerCreating? logs and events does not show anything that might be telling.. really weird

venkata.mutyala avatar
venkata.mutyala

What does your manifest contain? Have you checked other related objects for any details? Maybe check the deployment object for any issues/events?

mr.shayv avatar
mr.shayv

I have a keycloak gatekeeper configured in a configmap but i checked there’s nothing giving in the logs or events

venkata.mutyala avatar
venkata.mutyala

Huh. Just did a quick google search and saw this: https://serverfault.com/questions/728727/kubernetes-stuck-on-containercreating

It might be a little noisy but the command they share to get all events might be useful here.

Kubernetes stuck on ContainerCreating

A pod in my Kubernetes cluster is stuck on “ContainerCreating” after running a create. How do I see logs for this operation in order to diagnose why it is stuck? kubectl logs doesn’t seem to work s…

mr.shayv avatar
mr.shayv

Thanks I’ve seen it and tried unfortunately nothing helped.. i will say that my containers run on containerd (crictl) and not docker though

venkata.mutyala avatar
venkata.mutyala

Have you checked APi level logs? Maybe the scheduler logs?

venkata.mutyala avatar
venkata.mutyala

Also is this cloud managed? AWS/EKS/GCP/etc. support might be able to assist here

venkata.mutyala avatar
venkata.mutyala

Though i have a feeling the answer is just tucked away in logs you already have access too. Just not sure which ones…

venkata.mutyala avatar
venkata.mutyala

Please share when you figure it out.

2022-09-13

Gabriel avatar
Gabriel

I am trying to get the aws-ebs-csi-driver helm chart working on a EKS 1.23 cluster.

The message I am getting from PVC events

failed to provision volume with StorageClass "gp2": error generating accessibility requirements: no topology key found on CSINode

The CSI topology feature docs say that:

• The PluginCapability must support VOLUME_ACCESSIBILITY_CONSTRAINTS. • The plugin must fill in accessible_topology in NodeGetInfoResponse. This information will be used to populate the Kubernetes CSINode object and add the topology labels to the Node object. • During CreateVolume, the topology information will get passed in through CreateVolumeRequest.accessibility_requirements. I am not sure how to configure these points.

Gabriel avatar
Gabriel

I looked at the worker nodes (ec2) launch template / user data. The kubelet root path was not the standard /var/lib/kubelet. Instead it was a different one. I fixed the missing CSINode driver information by updating the volumes host paths with the correct kubelet root path.

2022-09-15

    keyboard_arrow_up