#kubernetes (2022-03)

Thanks for this update

Could we do a demo on office hours sometime?

That would be great! I’d need to prep something up, but that would be fun to do. It’s still not written/talked about as much being so new to scene so the more out there shared the better!

I did a post on the secret part here: https://www.sheldonhull.com/pulumi-secrets/

@antonbabenko has also been looking into it

@sheldonh would love to see a demo of Pulumi on #office-hours I’ve been sticking my head in the sand about Pulumi because in all honesty… I don’t want it to be good haha.

Interesting bit on secrets though… Are you not a fan of sops? I believe that is cross-platform and is how I solve all of the work you just mentioned.

That’s a cool CLI. I hadn’t looked at that in a while. I think the tool is probably a great solution for many of these things but still adds more complexity for someone new to it. Think of the way Pulumi solves is per stack encryption for secrets with zero fuss as long as you already auth’d to Pulumi. Better? Maybe not, but quick and secure seems like it.

I starred sops to examine in future when I need to do this again. Thanks for the share!

Well I already had it starred lol. Just apparently didn’t dive into it at the time.

There is something to be said for the “per stack secrets” that you mentioned. It could be accomplished for sops + Terraform, but usually that fine grained of access hasn’t been something I’ve needed in my work. So that is more secure and I can get behind that idea. Then again… how does Pulumi deal with sharing secrets across many stacks?

Hmm. That’s another discussion I can look into if you have a more specific scenario. Just drop a comment on that post with what scenario you are looking at and when I can I’ll either update post or respond. I need some time + more detail to answer the scenario.

If targeting only kube how would Pulumi compare to something like cdk8s? (https://cdk8s.io/)

BTW, you are the first real person I’ve seen have any kind of real experience with Pulumi so thanks for forging this path

It’s more complex than yaml and mixed so far on how I feel about the random suffix for naming. It seems super useful to deploy uniquely, but it’s a lot more complex than I thought. I also can’t use a lot of the cool tools out of the box like Tilt as this branches into a more unique approach.

So far I’m slower to get new changes in, but the changes are solid, strongly typed, and I’m confident I built it right.

I can’t compare to CDK. I had to focus on one choice.

I think pulumi is pretty cool overall.

I don’t think I would have done it if they didn’t have a kube2pulumi or whatever that tool was. Gave me a great jump start.

I’m currently doing secrets management via a dedicated Hashi Vault enterprise deployment and using the kube CSI driver w/ Vault provider to declare required secrets (CRDS) for a deployment to avoid having secrets stored locally in kube and also not have to use the sidecar pattern for secrets injection

Also try their import command. Apparently now it can generate your code from deployments, so it might jump start you on existing stuff. Right now azure devops variables groups + kubernetes deployment are in pulumi and I’m plugging away as I work on new stuff to include it.

Haven’t touched automation api but looks intruiging

Cool! I had to use configmap mounting a config.yaml into container.

I want to explore something more robust like what you mentioned in the future.

It is easy in concept but becomes quite a bit of work when it comes to larger deployments with multiple owners (the dev owns the workload, their team may own the secrets, another team may own the kube clusters, yet another owns Vault…. et cetera)

In any case, I too would be interested in a demo if you are able to scratch one together, you always share such cool stuff

I bet.

I keep questioning myself on using pulumi with k8 as it’s really cool but not as common.

I keep coming back to, I can use yaml render and get back to templates if I need to, and in the meantime Kubernetes is complex already, it’s not like yaml makes it easier. In fact, I feel I have a better understanding of the resources and dependency chain due to using Go and requiring strongly typed dependencies and name ID resolution instead of just text name.

Overall I feel it was a reasonable decision as it was either helm or this.

I would love to if I can cobble together one. I appreciate the interest! I would love to share but have to prep a localized version or something.

I hear that, so much work that I do is not easily made externally presentable

In the meantime try playing with kube2pulumi or even pulumi import which supposedly can generate the source code + import into state. Maybe you’ll find some fun stuff with it

I will, I’ve already been poking at that cdk8s framework anyway

So pulumi and kubernetes have tons of overlap, but off the cuff, my first thought would be that using pulumi to drive your kubes gives you a proper programming language to work with instead of yaml and kubectl scripts or terraform modules. That means you get testing frameworks and all the other trimmings like integration/extensibility and package management.

I wrote a blog on my experiences a few months ago in case anyone is interested https://www.anitian.com/eureka-how-pulumi-brought-sanity-to-our-devops-team/

Great writeup! I do feel I still have a rube goldberg style machine with K8 + pulumi, but a strongly typed robust one. It’s hard though as it does require more complexity than pure yaml. However, once you compare to the same option as it expands in more templated yaml or other tools then I think you are picking your poison.

I’ll choose Go over a bunch of yaml tools anyday.

Another update related to this https://www.sheldonhull.com/using-randomization-for-pulumi-kubernetes-resources/

I’m pulling back from using Pulumi for Kubernetes primarily, but still using it for most other things. This is primarily because the path of least effort is now helm charts which are built now by another team and maintained. Even though Pulumi can run helm charts, I’m using things as is to benefit.

I still think it has a huge amount of promise, but it is a very different path to follow in the K8 ecosystem. I think the most benefit is found when you create components that wrap up a complete stack of resources with best practices applied. If you just use it for the equivalent of helm you aren’t gaining the full power.

In addition, tracking state for K8 deployments is a very foreign concept and makes things much tricker to integrate with other tooling. It feels sorta like you need to go all in or not at all with app deployments with Pulumi to benefit.

Cluster creation, I imagine is great, but app level I’m not certain I’d recommend unless you are going all in.

Thanks for the update!

yes, it appears so

Looks like others have gotten it working on kops which is technically “self managed”

https://dev.to/hshar7/how-to-get-an-alb-running-with-kops-in-2021-3jke

Shreyank, I think you’re missing a data point, which is the JVM heap size.

aws compute optimizer has been pretty easy to get going with… https://aws.amazon.com/compute-optimizer/

Are you asking if it’s possible to change the pod’s configuration upon restart?