#kubernetes (2020-11)

I have an ingress resource which was deployed via Helm. The helm chart has been uninstalled, but the ingress resource is still around. It was connected to the v1 alb-ingress-controller + external-dns.

A team member has been banging her head against this for a day or so now and she hasn’t been able to delete the resource. We’ve uninstalled our ingress controller and external-dns and neither of them seem to be the culprit that is leaving it around.

When we try to destroy the ingress using kubectl, the command succeeds, but the resource isn’t actually removed.

Definitely a problem specific to this single cluster as a helm uninstall of a similar chart in another cluster has no problems.

Really looking for advice on how folks would investigate the internals here — Not looking for a bullet proof solution to this specific issue.

Sometimes there is a finalizer that gets stuck, can happen for namespaces etc.

That ingress does have a finalizers entry in the metadata. How do you typically deal with that? I’ll start googling around that.

Ah I delete the finalizers block and that should do the trick?

Giving that a shot now.

And that did the trick — Awesome. Thank you @kskewes!

Oh awesome good job. Sorry was on mobile and juggling stuff.

We had some troubles with finalazers as well. Couldn’t remove the resource - it was recreated automatically right after the deletion. Cool, that you’ve already found a solution.

It was a third-party resource which troubled us and the solution was to patch it like:

kubectl patch crd/restoresessions.stash.appscode.com -p '{"metadata":{"finalizers":[]}}' --type=merge

Yeah, I did a very similar patch. It’s impressive because I had removed the underlying CRD which was responsible for finalizing it… and yet it still was sticking around.

is s3 strictly the requirement? or is the requirement to backup the PV

The reason I ask is native snapshots is a feature (beta) of 1.17

https://kubernetes.io/blog/2019/12/09/kubernetes-1-17-feature-cis-volume-snapshot-beta/

Not a requirement at all, just need a backup of the files off the cluster

There’s also Velero by VMware. https://github.com/vmware-tanzu/velero

(I haven’t tried either one)

Found stash but it had some weird licensing. We deploy clusters on demand and need some “I messed up” backups

So this will deploy kubernetes in your clouds, which deploy worker nodes, which deploy containers on nodes

Now we just need an automated deploy tool to deploy quovery lol

fwiw, kops is constantly snapshotting etcd and backing it up to S3

aha, you want to do PV snapshots, so ignore my comment.

https://kubernetes.io/blog/2019/12/09/kubernetes-1-17-feature-cis-volume-snapshot-beta/

See the PR from yesterday

https://github.com/cloudposse/terraform-datadog-monitor/pull/20

Ha got it. Thanks man.

the tags are only available when monitors types are metrics or event

for query type monitors, not available

That’s what I gathered as well, but wasn’t sure if ya’ll were enabling something I was missing. Glad I caught right after you learned the same thing.

This is new actually, finally coming to Kubernetes; https://kubernetes.io/docs/concepts/services-networking/service-topology/

That suggests that it’s not (easily) possible, currently.

Nah, it does seem a little cryptic but actually read on… https://kubernetes.io/docs/concepts/services-networking/service-topology/#prefer-node-local-zonal-then-regional-endpoints

The EndpointSlice Controller: This controller maintains EndpointSlices for Services and the Pods they reference. This is controlled by the EndpointSlice feature gate. It has been enabled by default since Kubernetes 1.18. So… 1.18 ? or maybe 1.19 is required … damn, now I’m unsure.
Kube-Proxy: When kube-proxy is configured to use EndpointSlices, it can support higher numbers of Service endpoints. This is controlled by the EndpointSliceProxying feature gate on Linux and WindowsEndpointSliceProxying on Windows. It has been enabled by default on Linux since Kubernetes 1.19

Have this on TODO, hadn’t yet tried it.

Well, this gives me good info to go on. Thanks. We’re at EKS 1.17 and I noticed that there are a few changes to get to 1.18 that I have to take a deeper look at. I’ll add this to the pile.

this seems like 1.18 is good to go

To enable service topology, enable the ServiceTopology and EndpointSlice feature gate for all Kubernetes components:

both of those are enabled default on 1.18

the second one I mentioned for kube-proxy (EndpointSliceProxying) isn’t mentioned as needed as far as I can tell.

A little more research led me to understand that, in my current config (load balancer -> service), the path to the service pod that handles the request is already optimized, since the ingress-nginx controller populates the LB with the service pod IP:PORT configs, so leaving the LB, the requests go right to the service pods on whatever node the ELB chooses. If my understanding is correct, then I don’t need to implement any additional affinity or routing. Does that sound right to yoU?

If the pods are immediately in the LB then I guess you’re fine :).

Take a look at this chart: https://gitlab.com/nicosingh/medium-deploy-eks-cluster-using-terraform-sample-app/-/tree/master/helm

It is a simple deployment with the whois docker container

Is it called echo-server?

It’s the whoami container: https://hub.docker.com/r/containous/whoami

You will something like this:

Hostname: sample-app-6c66c9c587-ssknb
IP: 127.0.0.1
IP: 10.aaa.aaa.aaa
RemoteAddr: 10.bbb.bbb.bbb:51158
GET / HTTP/1.1
Host: <insert-domain-here.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9,de;q=0.8
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
X-Amzn-Trace-Id: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
X-Forwarded-For: 10.ccc.ccc.ccc, 10.ddd.ddd.ddd
X-Forwarded-Host: <insert-domain-here.com>
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Original-Forwarded-For: 10.ccc.ccc.ccc
X-Real-Ip: 10.ccc.ccc.ccc
X-Request-Id: <some-id>
X-Scheme: https

Yeah that’s a good one, I remember that one too from way back. Thanks. Helps when you want to see how middleware, ingress or proxies behave.

We have used extrnal-dns with calico too

Here’s our #helmfile for it https://github.com/cloudposse/helmfiles/tree/master/releases/external-dns

thanks - how does this work? what IPs does the DNS point to?

it’s the IPs of the Service

and a service can be of type load balancer which is internal or external

the details really depend a lot on your setup

ahh got it, thank you

We have this type of setup running. Private eks cluster with internal NLB. NLB connects to ingress-nginx which forwards to the corresponding application ingress object. Cert-manager with dns01 and external-dns for pointing domains to the NLB ALIAS record

Based mostly on cloudposse Terraform modules and helmfiles. Although the helmfiles do need tweaking

• https://github.com/cloudposse/terraform-aws-vpc-peering.git?ref=tags/0.6.0
• https://github.com/cloudposse/terraform-aws-eks-cluster.git?ref=tags/0.29.0
• https://github.com/cloudposse/terraform-aws-tfstate-backend.git?ref=tags/0.26.1

The last module solves the chicken-egg problem to map serviceAccounts with IAM service roles and the corresponding policies

Wow thanks a lot. Really useful

I think for quality of life/debugging purposes, a query via namespaces will be a tad bit faster than query via label within the kube-system. depending on the cluster size, the initial amount of pods running in kube-system can already be pretty high. for example, I like that our logging provider I can just query via namespace:cert-manager as opposed to namespace:kube-system label:app=?? or service=?? where you possibly have to remember how each 3rd party helm chart labels their deployments etc.

This is also my thinking. I like more the idea of grouping everything into its own namespace and there is even no limit in the number .

What make think about this was the following descriptios for the cluster-autoscaler extra argument:

extraArgs:
  [...]
  # If true cluster autoscaler will never delete nodes with pods from kube-system (except for DaemonSet or mirror pods)
  skip-nodes-with-system-pods: "false"

Apparently the kube-system pods are actually handled differently…

@Jeremy G (Cloud Posse) what did we end up doing for this?

@Erik Osterman (Cloud Posse) We have not really found a solution, but we do what we can. We manually tag everything we create, and we sort of got around this related issue by specifying a custom launch template that tags everything it can.

doh, i was trying to avoid using a custom launch template but i suppose i could not be lazy and do that

thanks Jeremy and Erik

@joey If you use our terraform-aws-eks-node-group module to create your node group, it will make a launch template which propagates tags if you set the resources_to_tag variable.

thank you i got aws_launch_template working

although i’m peeved that you can’t use instance_market_options and spot with aws_launch_template

conveniently i’m primarily using this in a one-off use case where i have to use ondemand instances for now anyways, but i wanted to keep a dev environment that was nearly identical running with spot instances

You can try this:

export SECRET_NAME=$(kubectl get secrets -n namespace | grep pod-name- | cut -d ' ' -f 1)
kubectl describe secrets $SECRET_NAME -n namespace 

Drops bunch of tokens and jenkins-password: 10 bytes in bytes

https://pastebin.com/ZfnGaPkm

kubectl get secret shows up jenkins-password in plain text, however, Jenkins doesn’t digest password with the user profile name :) pastebin.com/65bMbuED

Got help from Bitnami folks, turns out base64 decoding is buggy on macOS 11.0.1 - have to fix it then. For the testing purposes, I used that suggested website for decoding a password string. Thanks for your help mate :)

“Hi @organicnz, I’m not really sure what is happening there. When the chart is deployed, some notes are shown with information about the commands that you need to execute to obtain the credentials. You can obtain those notes again by executing: helm get notes YOUR_INSTALLATION_NAME You will find a command like the following one: echo Password: $(kubectl get secret –namespace default YOUR_INSTALLATION_NAME -o jsonpath=“{.data.jenkins-password}” | base64 –decode) If you want to do what the command above is doing in a more manual way, you can also get the secret and output as yaml and you will get a plain-text string that is base64-encoded. You need to decode it to get the actual password. If the implementation of your base64 utility is buggy, just for testing purposes, you can use an online tool like this one: https://emn178.github.io/online-tools/base64_decode.html Again, this is just for testing purposes, you should never share your password on the internet.”

Make sure to evaluate the cost of ES. The amount of storage is limited by the instance size and it’s easy to get up to 500/mo for a cluster that barely stores a week of data and has HA. Then consider how many you want? You don’t want all logs going to the same cluster. Let’s say you have 4 accounts and 4 small clusters with HA. My point is it’s very easy to end up paying thousands of dollars a month for ES and the tight integration with APM and Metrics you get with Datadog is lacking. You’re also on the hook for developing and keeping it all up to date. I am curious if the cost/benefit really works out in favor of ES

Yeah, valid point. I believe I’m only going to run HA and more than a day of log retention in production environments. The client is single tenant environment per customer (enterprise customers), so really it’ll be expensive anyway we run it AFAICT.

The problem I / the client had with DataDog logs was that we have 3 clusters up right now and the logs cost alone for the month is $1K / cluster, which is pretty insane. I think anyway we do it should be cheaper than that.

I guess as long as you can make sure that indexes are granular it should make it easy to snapshot indexes to S3 buckets… and remove these afterwards… But then logs are not available for Kibana… so not sure if these will be of any use at all?

AWS has ultrawarm storage of ES now… but as far as I know you can’t snapshot your indexes while on utlrawarm - but you better to doublecheck I could be wrong on this one

I wonder if sending these directly to S3 and performing searches via Athena will make it a bit more reasonable in terms of pricing…? But I am not sure if QA and Dev teams will like it… Dear @Erik Osterman (Cloud Posse) have you had any experience with Athena being used for log queries instead of indexing in ES and having all nice features of Kibana?

I dont think so

What I usually do is update on console and bump terraform to that version

What is will pros/corn to use tf to build EKS and use EKS console to upgrade? Any unexpected side effect?

I’m not sure if EKS will allow to do that, but you should upgrade one minor version at a time, because control plain components are only backward-compatible with the previous minor version, so you can’t really jump two versions ahead. https://kubernetes.io/docs/setup/release/version-skew-policy/ Even if EKS allows to do that, they will for sure upgrade one minor version at a time under the hood.

Also, I would check if all of your cluster resources are compatible with the new version before upgrading.

What kind of cluster resources? How you check? Do your create new EKS or sandbox EKS to test upgrade work or not and then you do it on the main eks cluster?

Update: EKS won’t allow that.
Because Amazon EKS runs a highly available control plane, you can update only one minor version at a time. See Kubernetes Version and Version Skew Support Policy for the rationale behind this requirement. Therefore, if your current version is 1.16 and you want to upgrade to 1.18, then you must first upgrade your cluster to 1.17 and then upgrade it from 1.17 to 1.18. If you try to update directly from 1.16 to 1.18, then the update version command throws an error. from here: https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html

I have impression that EKS module will manage upgrade for us. i.e. Instead of running tf apply for 3 times. If we declare cluster_version = "1.18" then EKS module will do it same as 1.15 ==> 1.16 ==> 1.17 ==> 1.18 Do you think my assumption is correct?

Nope, I don’t think that it will do that for you.

You can always check source of module if there auto upgrade magic in it ( which I am mostly sure isn’t the case)

Thank you @aaratn & @Aleksandr Fofanov I think my assumption is wrong!!!

@Amit Karpe Regarding how to find resources which uses deprecated APIs before upgrading.

https://blog.doit-intl.com/kubernetes-how-to-automatically-detect-and-deal-with-deprecated-apis-f9a8fc23444c

Since you are jumping from 1.15 to 1.16 it’s worth checking if your cluster have any

And also there is pluto tool too

https://www.fairwinds.com/blog/kubernetes-easily-find-deprecated-api-versions-with-pluto

Pluto helped us in the 1.15 -> 1.16 migration. With Helm, this plugin helped too: https://github.com/hickeyma/helm-mapkubeapis - When you upgrade to 1.16 your running Kubernetes deployments will be transposed in-place and everything keeps running. The problem comes when you try to make a deployment (or rollback) and it fails halting that applications pipeline, which is of course, the end of the world.

im totally using that ami

So, it doesn’t impact the service of applications if you run mutiple instances on different nodes… But, what you’ll see is application restarts, nodes randomly become NotReady . If you have a fair bit of monitoring of things like replica counts, kube node status you’ll get a bit of noise. Along with perhaps, developers claiming that they should move back to Beanstalk, Fargate, Cloud Provider X as AWS should not have let a buggy AMI get out…

The latest AMI works fine: https://github.com/awslabs/amazon-eks-ami/releases/tag/v20201117

yeah i was bit by that bug but the 20201117 amis are fine though

Do you have a link to TF cloud operator?

I’m thinking of looking at crossplane. It looks pretty good.

Yep here is TF cloud operator link: https://github.com/hashicorp/terraform-k8s

I wonder if crossplane has anything similar to TF state… haven’t had a chance to look into documentation yet

pretty neat project, thanks for sharing

(Note the alb-ingress-controller is now the https://github.com/kubernetes-sigs/aws-load-balancer-controller)

Yup, that’s the one I’m using ^^

I think this usually happens if the subnets aren’t tagged properly

I used that combined with this guide to set it up https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/guide/controller/installation/

What should the subnets be tagged with? I have 3 public and 3 private subnets, attached to a single vpc that the eks cluster is a part of

Here’s the terraform used to create the vpcs / subnets

module "us-prod-eks-vpc" {
  source = "terraform-aws-modules/vpc/aws"

  providers = { aws = aws.us }
  name = "us-prod-eks-vpc"
  cidr = "10.0.0.0/16"

  azs             = ["us-east-1a", "us-east-1b", "us-east-1c"]
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]

  enable_nat_gateway = true
  enable_vpn_gateway = false

  tags = {
    Name = "us-prod-eks-vpc"
    Environment = "prod-us"
    Region = "us-east-1"
  }
}

See https://docs.aws.amazon.com/eks/latest/userguide/alb-ingress.html

I’ll take a look, thanks

Okay so I did the tags and they indeed were missing from the subnets, added those in, still results in the same error :s

  public_subnet_tags = {
    "kubernetes.io/role/elb" = "1"
  }

  private_subnet_tags = {
    "kubernetes.io/role/internal-elb" = "1"
  }

basically were added in

try adding this tag to your VPC

locals {
  tags = {   "kubernetes.io/cluster/${local.name}" = "shared"
  }
}

I confirm, using the above tag solved the issue for me too

If anyone wants to open up a PR to document this - that would probably save many others grief :-)

does something like https://github.com/heptiolabs/gangway help?

its OpenID based but maybe it can help with your use case

Thanks I will check…

teleport? https://goteleport.com/blog/kubernetes-sso-saml/

oh thats nice @loren

thanks foor sharing

one user can still access to your cluster, it’s the user you used for creation

for our part, it’s a dedicated user for terraform, as we apply from a CI/CD

Thank you. We able to revert the bad changes by EKS creator user

perfect

When we were using kiam on kops, we would do this, but haven’t carried it over to EKS.

If it would help, I can dig up the rules we used

No need Erik. Thanks.

I think the cleanest way is a metadata proxy.

If blocking metadata completely some apps might complain if they expect or need certain metadata. However. In most cases the needs are simple and providing an explicit AWS_REGION environment variable does the trick.

I think if I was to provide some form of multi tenant cluster with EC2 like promises (which I’m not really in the business of doing right now) I might be obliged to provide some emulation of the regular EC2 metadata.

Good question for office hours

Some ideas are the various operators for managing secrets like external secrets from SSM, ASM, Vault

The Reloader project from stakater to automatically roll replica sets when secrets or config maps change

#helmfile for managing configuration of helm releases

Dynamic config: LaunchDarkly, Split, or whatever hosted provider. I saw many re-implementations fail spectacularly

Deploying apps sharing the same config: beware creating a distributed monolith! I’d vote for helmfile / ParameterStore / shared ConfigMap.

Flagger is an open source alternative for launchdarkly https://github.com/weaveworks/flagger

Alternative-ish.

Flagger does Blue/Green in all its forms (canary, A/B, mirroring). Let’s take an example with 1% seeing the new feature. That is a random 1% of requests.

With LaunchDarkly (or whatever else feature flagging tool) you get to chose the 1%: instead of random you could show the feature to users whose email ends in [mycorp.com](http://mycorp.com), or the marketing team, or that specific QA team that will test the thing, or users who are on the free plan, or users who opt in to seeing beta features, or the client who this feature is developed for, and so on. You get a lot more control on risk allocation.

Thanks guys, a lot of good reading to look into

I’ll move it to #office-hours for any follow up

https://github.com/Unleash/unleash - has caught the attention of some developers I work with for feature flagging. I’d be keen to hear from anyone that might have used it? Or, share experience when I have gained it…

I used to work for the transport side of booking.com (rentalcars.com at the time) and they had an in-house A/B experiments framework in use on the frontend (horribly conditional JSPs). I think Optimizely (SaaS) is fairly popular in the front-end “which colour blue” arena.

https://www.youtube.com/watch?v=luZgWGhUqkU

nice!

This is what led me to checking out tailscale

Some interesting ideas in there for sure. But don’t think that they’ll be implemented too quickly. Maybe we’ll see something competitive in a couple years.

i consider that reasonably soon

Yeah solid point