#kubernetes (2024-04)

Yes, it should be possible to deploy nginx ingress on a private load balancer. Then deploy cloudflared and point it at the internal load balancer.

That way you get nginx capabilities, and the benefits of cloudflared.

Ultimately, you just point cloudflared at an endpoint. If it can reach it, it can forward the traffic.

https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/configure-tunnels/local-management/configuration-file/

Yes confirm cloudflared tunnels are great to avoid exposing your cluster and love their setup. Works fine but beware the doc is outdated and need a full refresh.

That’s probably a reasonable approach/ask. Ex. It’s a pre-requisite to installing your helm chart.

I’m not sure what your product does but if it can not have a dependency on vault then it might be easier for folks to roll out as your helm chart would be entirely self-contained at that point

Also, where do I go in the EKS console to see what the helm release did?

helm (https://helm.sh) calls itself a package manger for k8s. You can use it to deploy both third party apps using publicly published charts, or you can use custom charts to deploy your own apps.

What k8s resources get deployed depend on the individual chart. Typically a chart will create several k8s objects that enable an app to run properly. A deployment/statefulset/daemonset to create the pods where the docker container run, a service to allow connections into the pod and in ingress object to enable connections from outside the cluster are the most common. You may also have other things like secrets or configmaps.

In the EKS console you can see the k8s objects in a cluster by clicking the “Resources” tab once you click on a cluster.

Lots of wonderful docs here about how to use it, but nothing that actually says, “This is what helm is, this is what it does, this is how it works, the is why you want to use it.”

https://helm.sh/docs/

Ok… Do I need a functional node group to create “k8s resources” using the helm_release resource?

Yes. The node group defines the worker nodes and their config that will run your k8s workloads.

Got it, ok. Definitely a race condition then. The helm_release is starting before the node group is ready… I think I know how to fix that though… It waits for the cluster and its access entries, but it’s not waiting on the node group config.

that would definitely do it

it was weird, because sometimes it would work fine. but then sometimes it would all blow up

In terms of getting familiar with what a helm release does helm dashboard is nice. It’ll run a local webserver and let you browse the releases that are running in your cluster and see the resources that make up the release and the yaml manifests that created them.

Oh, it’s a plugin for helm: https://github.com/komodorio/helm-dashboard

lol yeah, i’m green enough here that i don’t have any basic tooling installed for connecting to or managing k8s in any way

if it’s not in terraform, i aint got it

That won’t last long

Ive never used flux so cannot say much about it. I do recall aws pushing Flux a lot when chatting with them.

I’ve used argocd for a few years and like it a lot. It’s easy, scalable, and has a nice interface and community. The hardest part about argocd (and probably flux) is setting up the yaml so it’s flexible enough to modify and add new apps too.

If there is a flux specific addon (like the opentofu controller) that you want to use, you can use it in argocd using the flux subsystem

https://github.com/flux-subsystem-argo/flamingo

IMHO argocd is more user-friendly and has a smoother experience (nice UI), but from my experience, managing lots of projects can be tough. flux is simpler, more reliable and has a better management than argo (CRDs), especially when you use helm + kustomize + OCI. so, depends on your project needs..

RE: FluxCD. It’s probably worth reading up on what happened with Weaveworks as well as who is taking over the project now just to make sure it’s still got a good future.

RE: ArgoCD

OCI is definitely a bit weird with argocd. It doesn’t auto populate values/etc like a standard helm chart. At least that’s been my experience.

Aside from that I’ve been using argocd with customers for over 2 years now and it’s been great. We use an Appset to manage all of our argocd applications that developers deploy themselves.

ArgoCD also has a UI which I believe flux is still missing or requires an add-on project.

The only other thing I don’t like about ArgoCD is it uses helm template rather than the full helm support which I believe Flux offers. For example if you want to do a “hook” you have to find the respective hook to use within ArgoCD. And i don’t believe it’s a 1:1 translation from helm to argocd either.

Note: I work at a company that sells a PaaS built ontop of ArgoCD + a bunch of other FOSS. So i’m probably a bit bias here.

thank you all for the info ill take notes from this discussion and discuss among our team,

@Jeremy G (Cloud Posse) are our recommendations documented publicly?

@Erik Osterman (Cloud Posse) We have recommendations in this post in a thread in this Slack channel, but somehow publishing it to our docs website fell off my to-do list.

@Zing I believe all of your “cluster-critical” components should be deployed to your MNG using on-demand (as opposed to spot) instances. Then you can deploy all or most of your other workloads to Spot instances managed by Karpenter. This is because you should be deploying your MNG such that you have a node in each Availability Zone, thus ensuring your cluster-critical components can be spread across AZs and that your cluster can stay relatively healthy even if Karpenter goes out of service for a while.

I am generally opposed to using Fargate when you have all of the Terraform tools we have available. Fargate is great for managing scale if you have a minimal EKS configuration you are managing with command-line tools. It is overpriced and underperforming if you have a full cluster with an autoscaler like Karpenter.

Much more detail and reasoning in the above linked post. LMK if you have more questions.

thanks for that. that was a good read. the reason i was considering fargate is that we’re currently looking at modernizing our fleet (half of it is using “old” modules, and the other half is using “even older” modules) with the addition of bottlerocket. we’re using managed node groups for our “old” stuff, but we’re actually using the aws_eks_workers cloudposse module (from 3 years ago!) for our “even older” fleet. that module doesn’t really play nice with bottlerocket in it’s current state, so i was debating just moving our base nodes to fargate instead of thinking about managed groups… but it sounds like that wouldn’t be the right call

we’re looking at revamping our architecture across the board quite soon, so i don’t want to make any drastic changes to our aws eks cluster modules, but it looks like it wouldn’t be a huge lift to just replace the aws_eks_workers module im currently using with the newest version of aws-eks-node-group (which seems to support bottlerocket?)

One advantage eks-workers have over MNG is that you can set max_instance_lifetime, ensuring the instances are refreshed every so often. If there is an issue with them and Bottlerocket, open a ticket on GitHub and post a link here and we will look into it (although no promises on how quickly).

it might just be that i’m using the version from 3 years ago

based on the thread you posted, it seems like you recommend that deployments that should have HA (non-spot instances) should be scheduled on MNGs? i was going to reserve that ONLY for kube-system components

curious why you wouldn’t just let the various end-user deployments live on karpenter-provisioned on-demand nodes

You absolutely can let the various end-user deployments live on karpenter-provisioned on-demand nodes. It’s just that spot nodes are cheaper, and your deployments should, in general, be designed to be interruptible and movable, so they should not need the extra expense that comes with the on-demand service.

gotcha, appreciate the insight

next problem to solve: using instance store instead of EBS

time to do some investigative work

instance store is ephemeral, not a replacement for EBS for persistent data.

yeah, historically we’ve used EBS and haven’t looked into instance store since it was new when we last looked, but we actually don’t need persistent data at all

when we do, we just use persistent volumes

so we’ve just been wasting money / limiting our performance

sadly, it’s a whole janky workaround to get it to work on bottlerocket

https://github.com/bottlerocket-os/bottlerocket/discussions/1991#discussioncomment-3265188

Are you able to run Docker without problems?

yes sir, Docker is operating fine

Are you trying to use use the built in support that ships with Docker for Desktop

yes sir i used the docker for desktop but it gave me an error related to default file, so i tried using virtual machine as the driver and it worked. But i am still figuring out why it didnt work with docker as a driver.

Hrmm… I’m not sure, but at least you found a workaround.

Also, Podman released support for kubernetes on the desktop now too

oh thanks sir. I will surely look and explore it

Issue might be with Minikube version, please check u r using correct architecture image.

Hi Shivam I downloaded latest minikube version only. It worked only when i ran the commands to download minikube on my powershell with virtual machine as device driver instead of downloading the .exe file and adding path to environment variable.

do you use Karpenter?

https://karpenter.sh/docs/concepts/disruption/

ASG and CA will be different

we use currently cluster autoscaler, but will try karpenter

got it, yeah, seems it is still something AWS is working on, https://github.com/hashicorp/terraform-provider-aws/issues/21692

What do you mean by a “project” in this context?

What do you mean that will work anywhere? Bare metal, GCP, EKS, AKS, DO, etc.

Yup! Just wondering if the cloud-native k8s could all still be deployed to without template changes or specific configurations for different cloud providers.

I know helm is just a wrapper around manifests, and manifests are k8s specific so theoretically it should be able to work anywhere

So, it really depends on what’s in the chart. That said, I would avoid trying to make a chart install any missing dependencies, and instead make those requirements.

E.g. I wouldn’t have a chart install an ingress controller.

I would start by taking a look at the most popular charts in the archived helm/charts repository, which was an attempt to do this.

ah - thank you very much. why was it just an attempt?

Hah, because it’s now archived

It’s hard. Annotations will be different on every cluster, for example. How to handle storage will be different. How to manage secrets.

E.g. while secrets are all Secrets, how do those get there?

Yeah my team will learn a lot about what works/ doesn’t work with our first customer, but I find it hard to be “prepared” with every scenario or at least trying to build that capability using our templates

thanks Erik I will look around at other helm projects as well. right now that’s all our team is comfortable with

Yea, also, it’s not worth to over think it. I would start with something that works. Then incrementally add k8s flavors as demand arises.

You’ll also need each flavor to test with, so it’s not easy to develop “on spec”

Haha, yeah try telling that to our product team

How would you like to “be told”?

haha, via a dashboard I guess would be ideal

but open to whatever the current “meta” is

I like the dashboards, only that “it gets complicated”

• How is authorization handled?

• How is it exposed?

• How is DNS managed Which leads to… more operators, more stuff.

I don’t have the answer. Just venting. We also used to help customers deploy kubecost. But over time stopped.

if you’re on eks, you might want to explore https://aws.amazon.com/about-aws/whats-new/2024/04/aws-split-cost-allocation-data-amazon-eks/, which launched recently

(i haven’t looked into it yet, so don’t have an opinion)

I like the dashboards, only that “it gets complicated”
• How is authorization handled?
• How is it exposed?
• How is DNS managed

Which leads to… more operators, more stuff.

I don’t have the answer. Just venting. We also used to help customers deploy kubecost. But over time stopped. yeah i hear you hard times

if you’re on eks, you might want to explore https://aws.amazon.com/about-aws/whats-new/2024/04/aws-split-cost-allocation-data-amazon-eks/, which launched recently wow that’s interesting

gonna take a look. it’s new new

hmm looks like the cost allocation stuff is using kubecost under the hood

it’s quite frustrating that it’s so hard to get some decent cost allocation tool in place :(

looks like I’m stuck with kubecost for now I haven’t tried the AMP integration, so might give that a shot. hopefully the costs aren’t insane