#kubernetes (2021-10)

HI, I’d say it really depends on what you want to deploy to your cluster: if it’s applications only, Pulumi might not be the best approach (and a bit “risky” in the sense that you could miss the fundamental concepts of Kubernetes by not taking this first YAML step), you might want to look first at YAML/Helm/Kustomize/… Yet, if you need to deploy infrastructure (create/configure the K8s cluster itself: control plane/node pools, DNS, ingress controller, certificate manager, etc. and also configure some cloud provider objects/services such as storage buckets), then Pulumi sounds nice as it’s advertised as infrastructure as code. There’s also cdk8s.

the yaml for a Deployment and a Service pointing to the Deployment is really simple

same for cronjobs and configmaps

We have non-infra folks edit them all the time to tweak env vars, with minimal kubernetes knowledge; the patterns are often self-explanatory within a file. If they can understand a docker-compose file, they can understand the stuff above.

Where things got complicated was:

• Helm (finally moved this to terraform)

• Ingress (nginx and now istio)

• any sort of rbac / service account setup… but we handle that all via terraform and just tell folks “here’s your login details” and “here’s the service account too use for your app” The “complicated stuff” lives in Terraform and rarely changes at this point. Easy stuff is still yamls (deployments, etc) and we recently moved to kustomize to make it a bit simpler if you’re just bumping an image version

So no one except me knows terraform on my team. We are consuming the cluster managed by cloud operations team so I’m focused on the namespace app definitions.

I’d love to use Pulumi in this scenario if it makes sense, but still not clear on how local development works if everyone wanted to deploy to minikube locally. Not sure if the state would be considered each as its own state remotely stored at that point or if local development testing is different.

I can use yaml but at this point I have folks who know Go more than DevOps tooling and was thinking this would be a good fit for pulumi.

I do not want a pile of terraform that only I know how to maintain

Is everyone on the team jointly managing the cluster?

I guarantee that if all the devs have to really worry about is their own app/namespace that they will be more productive and you will have less emergencies.

I’d still automate the whole deployment via tf and pipelines then maybe setup devspaces for the devs to use against it. Or if they want to use pulumi that would be their call at that point right? The scope of their Pulumi work would be at the kube namespace level and lower (plus maybe some other cloud resources they may depend upon)

We don’t manage the cluster. We are “consumers of it”. I’m now working at the application development level, not part of Cloud Operations team.

I was thinking of using Pulumi for this as it’s all in Go anyway.

We have nothing to do with the AKS cluster which is provided as a “service” to us by the Cloud Operations team. I used to be on that side of things so your advice makes sense there. I’m actually now in the application level part of things and can determine what to use to define the app directly.

I’m thinking Pulumi/Yaml, doesn’t really matter, but since Pulumi is Go I’ll have flexibility in more test automation in the near future with sticking with Pulumi. Does that make sense then? @Zachary Loeber

Pulumi started as typescript I thought? Either way, I’d likely scaffold out a yaml based deployment to then export and help the devs model in their own code if that is the level they would like to take things (using pulumi, AWS CDK, et cetera). The devs may appreciate not having to deal with any of the kube/app scaffolding (but usually golang coders are all in on coding the depths of everything though so I doubt you would be that lucky…)

I’m part of the Go development team now. I’m asking from that context. Is there any negative to just using Pulumi for the application level K8 definitions. I’m checking as my background was cloud operations which Pulumi might not always fit for. However, I’m now using the AKS stack provided by Cloud Operations and responsible for just the application level stack.

I figured Pulumi would be a legit use case for this since I’m working in Go with Go devs. I just wanted to double check and make sure that in context of K8 there weren’t any “negatives” that make using this for a new K8 project a problem.

I personally think Pulumi is the wrong fit if you are only worried about consuming/using k8s. Pulumi is really about giving infra maintainers the ability to use GPLs to help provision/manage their infrastructure and not be bound by DSLs that are the norm. In this case, k8s is already deployed, you likely have a namespace assigned (if not your own cluster), and all you care about is deploying k8s resources. By nature k8s is extremely declarative, and so something like Pulumi is just unnesisary overhead. My own advise would be to look at Helm and Helmcharts: its basically just pure Go templating which you will be well versed with already, so creating a bunch of go templates to standardize your yamls should be a breeze.

Keep in mind that even if you used something like Pulumi, there’s no way around having to learn those primitives to deploy to k8s. You’ll have to understand deployments, ingress, services, etc. Pulumi just lets you get more clever when it comes to generating those final resources.{

Just my 2cents

@justin.dynamicd perfect info. That’s the blend of info I was looking for. I’ve been mostly on the cloud operation side so Pulumi would have been a great fit for a full stack creation of vpcs and such with generated names etc. As a team Pulumi seems to fit for application level definitions with examples like this: https://www.pulumi.com/blog/deploy-kubernetes-and-apps-with-go/ that’s what got me interested, using it for app level.

It’s an interesting mix. To your case Pulumi is the wrong fit for the app only, but since the majority of devops teams I’ve seen aren’t doing application development, then using Pulumi/CDK which is meant to give access to the programming native languages is a mismatch since I’d assume the DevOps/Cloud Ops teams would be more comfortable with HCL over Go/Typescript projects.

That’s my gut reaction. I think the target for pulumi was meant to be developers managing their stack at any level, so your info is making me rethink my assumptions. Good input!

Coming from a Ops background myself, I found the biggest benefit with Pulumi was overcoming the limitations inherent in a declarative language (which is all the rage right now, hence the predominance of yaml). The more flexible you need the code to be, the less “declarative” it becomes, and the harder it is leverage all these yaml-happy DSLs. At it’s worst extreme look at Puppet’s DSL where you can’t even update variables: everything is declared and thus immutable forcing you jump through tons of hoops.

This is where Pulumi shines IMO: you get a full featured, powerful GPL to build your environment with so you are no longer shackled by declarative DSLs. So in go you could make components with as complex logic as you wanted. This is great because often you have to reuse/duplicate things … but the different environments are just different enough that it may cause a traditional DSL problems and in ways that arent necessarily easy to reflect in a simple param list via yaml/json.

My biggest problem with Pulumi: with great power comes great responsibility. Most DSLs are highly opinionated and have a very clear intended use. Hide that behind a GPL and suddenly its very easy to re-invent the wheel and invent a lot of needless work. “this logic will ensure deployments and pods are in place before my service is defined”, for example, is just wasted effort: k8s is designed to handle eventual consistency but its easy to get lost in those details “because you can”.

terraform provider doesn’t allow to use bottlerocket ami_type in aws_eks_node_group yet. filled https://github.com/hashicorp/terraform-provider-aws/issues/21548