#kubernetes (2024-05)

There are a few options

First, understand how Service works, and it does not need to be of type load balancer. It can be a ClusterIP

That covers you for anything TCP related.

Now, if you need Ingress functionality, using ingress-nginx, with Service of type ClusterIP , and you now have a solution which does not depend on IaaS.

So with DNS would this work by pointing a new record to the cluster and making requests to the service that way?

Well, provided the ClusterIP is routable

Typically clusters are deployed on private CIDR ranges, so the Service will have a private ClusterIP

If you just want the DNS to work internally, that’s enough. Otherwise, you’ll need to ensure the ClusterIP is on a routable network, or configure port forwarding from some other service, like a firewall

Would have to know more about what your goal - the business objective.

Gotcha, seems easy enough.

i have a helm chart now that uses aws-load-balancer-controller and in order for this helm chart to be deployable “anywhere” (any cloud k8s) its just a matter of removing the IaaS deps

we were only targeting aws customers, but apparently a few customers in the pipeline have/use GCP - which i know nothing about

not even mentioning the cert-manager + istio work effort we’re pulling up on these next few sprints (in our org)

very limited experience with istio - but it too has an ingress, so likely we will go that way + cert-manager for x509 certs to support mTLS internally. we would give the customer an option, deploy cert-m if you don’t have one, just give us a CA to use, or we will gen self-signed. and istio to support it all

CAPACITY, ACCESS MODES, RECLAIM POLICY, STATUS CLAIM, STORAGECLASS, VOLUMEATTRIBUTESCLASS, REASON

• 1Gi, RWO, Delete, Bound, kafka-kraft/data-kafka-2, gp2,

• 1Gi, RWO, Delete, Bound, kafka-kraft/data-kafka-0, gp2,

• 1Gi, RWO, Delete, Bound, kafka-kraft/data-kafka-1, gp2,

@miko can you check what is retain policy

https://kubernetes.io/docs/concepts/storage/persistent-volumes/#retain

@Adi I run the following command kubectl get statefulset kafka -o yaml and I received this:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apps/v1","kind":"StatefulSet","metadata":{"annotations":{},"labels":{"app":"kafka-app"},"name":"kafka","namespace":"kafka-kraft"},"spec":{"replicas":3,"selector":{"matchLabels":{"app":"kafka-app"}},"serviceName":"kafka-svc","template":{"metadata":{"labels":{"app":"kafka-app"}},"spec":{"containers":[{"env":[{"name":"REPLICAS","value":"3"},{"name":"SERVICE","value":"kafka-svc"},{"name":"NAMESPACE","value":"kafka-kraft"},{"name":"SHARE_DIR","value":"/mnt/kafka"},{"name":"CLUSTER_ID","value":"gXh3X8A_SGCgyuF_lBqweA"}],"image":"<redacted>","name":"kafka-container","ports":[{"containerPort":9092},{"containerPort":9093}],"volumeMounts":[{"mountPath":"/mnt/kafka","name":"data"}]}],"imagePullSecrets":[{"name":"docker-reg-cred"}]}},"volumeClaimTemplates":[{"metadata":{"name":"data"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"1Gi"}}}}]}}
  creationTimestamp: "2024-04-18T07:01:03Z"
  generation: 4
  labels:
    app: kafka-app
  name: kafka
  namespace: kafka-kraft
  resourceVersion: "8108458"
  uid: b4179eec-d36d-41ce-aa07-d6bad10d884a
spec:
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain
    whenScaled: Retain
  podManagementPolicy: OrderedReady
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: kafka-app
  serviceName: kafka-svc
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: kafka-app
    spec:
      containers:
      - env:
        - name: REPLICAS
          value: "3"
        - name: SERVICE
          value: kafka-svc
        - name: NAMESPACE
          value: kafka-kraft
        - name: SHARE_DIR
          value: /mnt/kafka
        - name: CLUSTER_ID
          value: gXh3X8A_SGCgyuF_lBqweA
        image: <redacted>
        imagePullPolicy: IfNotPresent
        name: kafka-container
        ports:
        - containerPort: 9092
          protocol: TCP
        - containerPort: 9093
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /mnt/kafka
          name: data
      dnsPolicy: ClusterFirst
      imagePullSecrets:
      - name: docker-reg-cred
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      creationTimestamp: null
      name: data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi
      volumeMode: Filesystem
    status:
      phase: Pending
status:
  availableReplicas: 3
  collisionCount: 0
  currentReplicas: 3
  currentRevision: kafka-8dd896cf4
  observedGeneration: 4
  readyReplicas: 3
  replicas: 3
  updateRevision: kafka-8dd896cf4
  updatedReplicas: 3

It seems that the retain policy is Retain

spec:
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain
    whenScaled: Retain

I went to retrieve all the PVCs as well kubectl get pvc and kubectl describe pvc data-kafka-0 for which I received:

Name:          data-kafka-0
Namespace:     kafka-kraft
StorageClass:  gp2
Status:        Bound
Volume:        pvc-c9f45d3a-e1a7-4d3a-8e51-79a411411d43
Labels:        app=kafka-app
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: ebs.csi.aws.com
               volume.kubernetes.io/selected-node: <redacted>.<redacted>.compute.internal
               volume.kubernetes.io/storage-provisioner: ebs.csi.aws.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      1Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Used By:       kafka-0
Events:        <none>

@Adi idk what I’m missing but it seems to be saying Retain as its policy, perhaps it’s due to my Dockerfile path not the same with volumeMounts’ mounthPath from my StatefulSet?

volumeMounts:
            - name: data
              mountPath: /mnt/kafka

how about “kubectl get pv”?

volumeMode: Filesystem is set, so when the node is gone, the volume is gone with the node

Hi @Hao Wang ohh I’ll check how to change that, but I also lose the data when I upgrade the image version (I’m away from my laptop but when I get back I’ll run “kubectl get pv”)

oh I am wrong, filesystem is not file in docker

should be the retention policy in storageclass and pv

may need to create a new storageclass with reclaimpolicy as Retain

Oooh like change my EBS CSI driver?

Or explicitly define the storage class to use in my statefulset?

can try edit gp2 and set reclaim policy to Retain

@Hao Wang running kubectl get pv and I get this:

they were created 22 days ago

are they reused?

sorry dumb question but what do you mean by reused? what I’ve done so far is run my StatefulSet which I thought should be enough to get a persistent volume :C and yes I’ve run my StatefulSet many days ago but it’s just recently I realized that whenever I upgrade my StatefulSet kafka is I lose the messages inside of it

got it :+1: we can change the storageclass to Retain and it should fix the issue

Omg I checked the gp2 description and it’s policy is Delete

but how come :CCC why is the policy Delete? :C

it is default setting

kubectl describe storageclass gp2

Name:            gp2
IsDefaultClass:  Yes
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"},"name":"gp2"},"parameters":{"fsType":"ext4","type":"gp2"},"provisioner":"kubernetes.io/aws-ebs","volumeBindingMode":"WaitForFirstConsumer"}
,storageclass.kubernetes.io/is-default-class=true
Provisioner:           kubernetes.io/aws-ebs
Parameters:            fsType=ext4,type=gp2
AllowVolumeExpansion:  <unset>
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     WaitForFirstConsumer
Events:                <none>

this may be an old EKS, AWS got gp3

my cluster version is 1.29 but I deployed my AWS EKS via terraform along with the EBS CSI driver

ooh wait my EBS CSI driver I manually installed via ~Helm~ eksctl

eksctl may not touch storageclass

@Hao Wang I found the command to update gp2 reclaim policy, I’ll give it a try!

kubectl patch pv <your-pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

yeah, it is the one

oh this is for pv

kubectl edit sc gp2

Ohh I have to edit gp2 directly?

Ooowkie thank you!

np

wow I can’t edit reclaim policy The StorageClass "gp2" is invalid: reclaimPolicy: Forbidden: updates to reclaimPolicy are forbidden. would it be better if I try and migrate to gp3 and try to set default storageclass to gp3 for my future statefulset to always use gp3?

yeah, let us give it a try

Holy smokes… migrating to GP3 is a pain in the butt:

https://aws.amazon.com/blogs/containers/migrating-amazon-eks-clusters-from-gp2-to-gp3-ebs-volumes/

Since this cluster is not production I will just delete the resources in the following order: StatefulSet, PVC, PV and re-apply my StatefulSet again.

The quickest solution that doesn’t require deletion is to change the reclaim policy of PV and not StorageClasses level:

• DONT ◦ kubectl edit sc gp2 ◦ edit reclaim policy to Retain ◦ -> The StorageClass "gp2" is invalid: reclaimPolicy: Forbidden: updates to reclaimPolicy are forbidden. • DO ◦ kubectl patch pv pvc-646fef81-c677-46f4-8f27-9d394618f236 -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

What are you trying to measure/improvement exactly? Is it a web app where client requests are slower then you would like? Are you trying to ensure nothing like CPU throttling is happening on the app? or that the node/worker is not overloaded?

Mostly that there are no CPU throttling issues or overloads

So I’ve been using kube prometheus stack https://artifacthub.io/packages/helm/prometheus-community/kube-prometheus-stack

It has built in CPU throttling alerts already as I’ve seen them when my pods didn’t have enough requests set.

I’ve noticed it the default charts/graphs don’t do steel time. So i’ve been meaning to do a contribution for that. But if steel time is a concern for you (ex. you aren’t using dedicated/metal nodes), you may want to use this query:

sum by (instance, cpu) (rate(node_cpu_seconds_total{mode="steal"} [2m])) ref: https://stackoverflow.com/questions/76742560/how-to-measure-cpu-steal-time-with-prometheus-node-exporter-metrics#<i class="em em-~"</i>text=Node%20exporter%20exposes%20metric%20node_cpu_seconds_total,as%20part%20of%20cpu%20collector.&text=can%20show%20you%20how%20much,every%20CPU%20of%20every%20machine>.

^ Assumes you install kube prometheus stack first.

Also, i’m using EKS but that chart should work on any k8s distro

https://github.com/FairwindsOps/goldilocks

Something like https://github.com/asobti/kube-monkey

https://github.com/linki/chaoskube

Interesting… thanks for sharing

helm v4 is coming.

Interesting! I need to get up to speed on that.

Is it further along than it appears via issues?

https://github.com/helm/helm/issues/11331

there will be a blog coming sone.

it means you run some agent (service) on each node, most often as a daemonset, which is responsible for collecting logs from each of the container running on particular node, and forward them to centralised log storage

look here https://medium.com/kubernetes-tutorials/cluster-level-logging-in-kubernetes-with-fluentd-e59aa2b6093a

When it says to node specific it just simply means to deploy in the cluster right? Because control plane is on another machine

true for kubernetes as a service (EKS, AKS) where you do not have access to control plane nodes, for self-hosted k8s yopu should run them on CP too in order to collect logs from pods running on those nodes as well

@Piotr Pawlowski thank you so much!

What do you mean? The CSI driver?

Are you using the cloudposse module?