#kubernetes (2022-01)

kubernetes

Archive: https://archive.sweetops.com/kubernetes/

2022-01-28

rei avatar

Currently I am having issues deploying a new cluster with the latest TF EKS module cloudposse/eks-cluster/aws:0.45.0 I am getting the following “interesting” error after the nodes are up and then tf wants to set up the auth_config.

╷
│ Error: Post "<https://jsonplaceholder.typicode.com/api/v1/namespaces/kube-system/configmaps>": x509: certificate signed by unknown authority
│ 
│   with module.eks_cluster.kubernetes_config_map.aws_auth_ignore_changes[0],
│   on .terraform/modules/eks_cluster/auth.tf line 115, in resource "kubernetes_config_map" "aws_auth_ignore_changes":
│  115: resource "kubernetes_config_map" "aws_auth_ignore_changes" {
│ 

Any idea or hint what’s wrong. Google or StackOverflow does not help here…

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

looks like for some reason your system thinks that the cert is not valid on https://jsonplaceholder.typicode.com

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
terraform-aws-eks-cluster/variables.tf at master · cloudposse/terraform-aws-eks-cluster attachment image

Terraform module for provisioning an EKS cluster. Contribute to cloudposse/terraform-aws-eks-cluster development by creating an account on GitHub.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

you can try setting it to some other URL (or to null to test if it fixes the issue)

rei avatar

Will try thank you for the hint

2022-01-27

2022-01-26

btai avatar

anyone using EKS and replacing the AWS VPC CNI?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

for greater pod density?

steenhoven avatar
steenhoven

Listening. We hate the pod density.

Zach avatar

VPC CNI did get some changes to support increased density FYI

Zach avatar

the catch is that its only for pods that don’t need the ENI + SG attachments

Zach avatar

but, you can run both ‘modes’ together

steenhoven avatar
steenhoven

Thanks @ How do you specify the ENI requirements in a pod in that mixed scenario?

Zach avatar

been awhile since I looked at that aspect but here’s the blog post on it https://aws.amazon.com/blogs/containers/amazon-vpc-cni-increases-pods-per-node-limits/

Amazon VPC CNI plugin increases pods per node limits | Amazon Web Services attachment image

As of August 2021, Amazon VPC Container Networking Interface (CNI) Plugin supports “prefix assignment mode”, enabling you to run more pods per node on AWS Nitro based EC2 instance types. To achieve higher pod density, the VPC CNI plugin leverages a new VPC capability that enables IP address prefixes to be associated with elastic network […]

Zach avatar

the ‘ENI prefix’ part is the new feature, you have to allocate ENI prefixes in your vpc to use this

Zach avatar

the EKS lead on github told me though that you can use this feature + the pod ENI at the same time

Zach avatar

correct github link https://github.com/aws/containers-roadmap/issues/138#issuecomment-889208751

I asked about clarifying whether this new density stuff works with pod SGs.
Can pods getting branch interfaces/security groups co-exist with pods getting ENI prefix IP addresses in the same cluster and even on the same node? Yes

Is the prefix assignment launch related to the pod security groups feature at all, and does prefix assignment help increase pod density if you are only running pods with branch network interfaces/dedicated security groups? No

steenhoven avatar
steenhoven

Thanks a ton! Will dive into it

Zach avatar

It complicates the setup a bit, we played with it for a couple of days and didn’t quite get it working

steenhoven avatar
steenhoven

Hmm

Zach avatar

Admittedly I am not strong in networking

2022-01-24

Zachary Loeber avatar
Zachary Loeber

https://helmwave.github.io/ <– docker-compose for helm charts. Maybe this one was tossed out there already. Not certain if anyone has used it yet though (seems like something that could be accomplished with artful terraform depends_on constructs too))

Zachary Loeber avatar
Zachary Loeber

I’d be interested in an intersection between this and a generalized interface to multiple container runtimes for spinning up multi-service labs that involve kubernetes along with outside resources (so perhaps a vault service or dynamodb container external to a kind container cluster in the same network). It would be nice to have a declarative manifest for such situations that doesn’t end up devolving into bash script hackery

Andrew Nazarov avatar
Andrew Nazarov

I mentioned it once in the helmfile channel:) https://sweetops.slack.com/archives/CE5NGCB9Q/p1640848445057600

I’ve just come across this tool which seems to be an alternative to helmfile. https://github.com/helmwave/helmwave

Zachary Loeber avatar
Zachary Loeber

Ever end up using it?

Andrew Nazarov avatar
Andrew Nazarov

No, haven’t tried yet. And I don’t know if it happens since we put a lot of effort to embrace helmfile) Just for fun and for some side-projects maybe someday

2022-01-21

Mr.Devops avatar
Mr.Devops

Hi all - wondering if anyone can assist with the following error one of our pod

Readiness probe failed: Get "<http://x.x.x.x:9999/health?ready>": dial tcp x.x.x.x:9999: connect: connection refused

Readiness probe failed: HTTP probe failed with statuscode: 503
Zachary Loeber avatar
Zachary Loeber

seems self explanatory

1
Mr.Devops avatar
Mr.Devops

Yep the ip is itself and when I exec into the pod I can curl the endpoint

2022-01-19

steenhoven avatar
steenhoven

Hi, what kind of storage drivers do you use in EKS? The new efs-csi driver is not working for us due to chown-errors.

sheldonh avatar
sheldonh

CLI or Go SDK kubeconfig setup for remote cluster with K8 Service Account

Anyone willing to jump on this thread and help unblock me?

sheldonh avatar
sheldonh

• I have a kubernetes cluster running in Azure to connect to.

• I have no issues connecting for myself using the azure cli login workflow.

• Now I need to use a Kubernetes service account instead, which likely means I’d be using kubectl or the Go client api directly. I need this to setup the config so the remainder of my pipeline actions such as Pulumi can connect to the cluster correctly.

I’m not finding much clarity on how to (via cli) setup the connection to the kubernetes cluster using just the service account token secret I’ve already obtained (and if the ca.crt is required or optional for that). Most workflows assume already connected and have this in kubeconfig or use azure or aws cli to setup connnection. With Kubernetes service credentials directly setting up the connection… I’m finding a dearth of info.

Anyone willing to jump on this thread and help unblock me?

2022-01-15

2022-01-14

Mr.Devops avatar
Mr.Devops

has anyone run into this error in EKS Cannot enforce AppArmor: AppArmor is not enabled on the host we’re using aws linux 2 ami. I’ve read that appArmor (by default) is supported in ubuntu and not RHEL distribution. This is causing pods to fail with status blocked

Zach avatar

you can’t use app armor in AL2, its a centos derivative

Mr.Devops avatar
Mr.Devops

thx i just realize that after reading further into appArmor

2022-01-13

Franklin Lee avatar
Franklin Lee

Will a CKA without professional experience still be able to get a job?

Zachary Loeber avatar
Zachary Loeber

It is a start but I’d be prepared to know kube beyond the testing materials for the interview process

Zachary Loeber avatar
Zachary Loeber

knowing various ways to get workloads into a cluster (helm, kustomize, whatever) and knowing some common shared services one might deploy to a cluster (prometheus, grafana, et cetera) would be a good start

1
Zachary Loeber avatar
Zachary Loeber

try out a few ingress controllers as well and you will be cooking with gas

Zachary Loeber avatar
Zachary Loeber

good luck!

Franklin Lee avatar
Franklin Lee

Just want to also note I don’t have a developer background

Zachary Loeber avatar
Zachary Loeber

come from ops?

Franklin Lee avatar
Franklin Lee

No Ops background. Just familiar with Linux and Kubernetes

Franklin Lee avatar
Franklin Lee

And logged years of practice with Linux and K8s

1

2022-01-11

2022-01-10

Jakub avatar
Jakub

hi there!

anyone have experienced with EKS ? I have the problem with some nodes, on two nodes, in each 3 minutes, node is reporting that is unhealthy because of the PLEG. I don’t know if it can help in troubleshooting but I have problem with executing command “df -h” only on those nodes - it stuck. it looks like maybe some EFS has not been mounted properly or sth like that

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

are you mounting EFS directly to the nodes or using the EFS controller with pods?

Jakub avatar
Jakub

thanks for your reply. I use EFS controller but to be honest I dunno what happened because I restarted node and no more errors appear. I am afraid that it will happen in the future

1

2022-01-07

Mr.Devops avatar
Mr.Devops

hi there - does anyone know if there’s an approved terraform provider for creating the helm operator?

RB (Ronak) (Cloud Posse) avatar
RB (Ronak) (Cloud Posse)
GitHub - cloudposse/terraform-aws-helm-release: Create helm release and common aws resources like an eks iam role attachment image

Create helm release and common aws resources like an eks iam role - GitHub - cloudposse/terraform-aws-helm-release: Create helm release and common aws resources like an eks iam role

RB (Ronak) (Cloud Posse) avatar
RB (Ronak) (Cloud Posse)
Mr.Devops avatar
Mr.Devops

sweet i’ll check it out thx @RB (Ronak) (Cloud Posse)

RB (Ronak) (Cloud Posse) avatar
RB (Ronak) (Cloud Posse)
1
RB (Ronak) (Cloud Posse) avatar
RB (Ronak) (Cloud Posse)

np!

2022-01-05

    keyboard_arrow_up