#helmfile (2022-04)

Make sure to star the repo

And sponsor! https://github.com/sponsors/mumoshu

Yep, I think we’re doing this in some places with helmfile + argocd

we’ll have github actions run helmfile and generate the yaml manifests and commit them to an argocd deployment repo

Nice! We have exactly that. Good validation on our current workflow. Listening to office hours has been helpful. Thanks Erik!

i’d ask other maintainers if they would like to support it!

do you still use it?

So we were moving towards terraform-provider-helm, but there are some bad bugs we cannot over come. The worst of them is that believe it or not, a terraform plan can actually alter the terraform state of a chart deployed with the provider.

So our current plan is to make it a feature flag in our https://github.com/cloudposse/terraform-aws-helm-release module to use the helmfile provider.

We do have one problem preventing us from finishing it, and I think @Andriy Knysh (Cloud Posse) is going to open a PR for it.

There’s one other bug we have encountered with the hashicorp helm provider, is that it doesn’t handle the 3-way diff problem (that I think you) solved in the helm-diff plugin used by the helmfile and therefore the helmfile provider.

ah gotcha about the 3-way diff prob. afaik a terraform provider should generally let the terraform core diff it by comparing the old and the new values. there’s no easy way to do diff on the provider side

Yep, so not holding our breath for an easy fix in the helm provider. I think the helmfile provider would be a good enough stop gap for our use-case/problem.

um maybe.. i have a concern though. i’ve never tried using the helmfile provider on Terraform Cloud. have you?

I think it wouldn’t work well without using private hosted runners.

In our case, we’re using this only for the last-mile “root” modules in a customer environment where we can easily control for all the tools. For public/open source, it’s problematic since like you said it doesn’t work out-of-the-box everywhere.

terraform plan can actually alter the terraform state of a chart deployed with the provider. have you already reported it to the hashicorp repo? i thought this was impossible due to how tf provider plugin works(and how the core restricts writes to the state from any tf provider plugin functions related to terraform plan)

We thought it should be impossible as well. @Dylan do you have an open issue for this?

I guess the issue was that terraform updates the state, then the helm provider fails, but Tf does not update the state after the failure - the state says all is ok, but in reality nothing is deployed correctly

that makes sense to me! i see one of such issues is fixed here: https://github.com/hashicorp/terraform-provider-helm/pull/727 what was your version of the provider when you encountered it? in my experience, tf itself was quite solid so you won’t result in such state as long as (1)you correcntly surface the error, like done in https://github.com/hashicorp/terraform-provider-helm/pull/727 and (2)ensure the provider to update attributes only after the successful operation https://github.com/hashicorp/terraform-provider-helm/blob/e858cde60487a92f6dfe160f46c874350717df1c/helm/resource_release.go#L667-L676

Sorry for the slow reply, everyone! This is the canonical issue we keep referencing for this bug, @Erik Osterman (Cloud Posse): https://github.com/hashicorp/terraform-provider-helm/issues/472

@mumoshu What do you think of this?

@mumoshu thanks for @Brandon Miller

@Brandon Miller I somehow missed responding- Thank you so much for sharing this awesome logo! Do you mind if we used this as our official logo?

If so- I’m wondering how we could contribute back to you.

We’d certainly add credit to you next to the logo or the “acknowledgement” section in our documentation. If you run your own company would it help if we added a link to your company’s web site side by side with your name?

if you don’t mind, you could also submit a pull request to add the logo to the very beginning of our README yourself, so that your contribution is explicitly tracked in github (than we add it via our own pull request

@Brandon Miller @mumoshu

Thank you all for the kind words!

it is very good.

needs works okay when you need to express deployment requirements, but it’s a poor choice if you want to express runtime requirements. for example needs is useful when there is helmfile that deploys two charts: one comes with CRDs, secods uses these crds to deploys CRs (prometheus-operator and ServiceMonitor resources as example).

but you have proper microservices. there is no need to build deployment order (via needs). better look in how to implement readiness and liveness probes on service side, so they can check and wait until required services are up

otherwise you just building another monolith with needs as glue

One thing that may be of use to look up is a service mesh - I.E. Istio Gateway or something similar. Point being, try to avoid your services talking directly to one another as much as possible. Use some sort of intermediate such as AMQP, Kafka, etc, and let the mesh act as an authoritative source of federation. This has at least been my experience. It doesn’t always work out exactly this way, but the instances in which it doesn’t are normally fairly obvious and straightforward.

If there’s no way to use a broker as a mediator, there isn’t much you can do as far as guaranteeing that if one service breaks, it won’t break another, but a good start in that direction is to solidify your contracts at the service-level. gRPC Proto’s are a fantastic way of doing this.

@z0rc3r Thanks, that reinforces much of what I have discovered by reading about what others are doing.

@Brandon Miller Thanks for the pointer. My question was about more dependency management and I don’t have any hands-on experience with service meshes, but I’ll check it out.

Hmm, could you elaborate at a high level on what C is? The OP mentions it’s a service, so my immediate assumption I suppose was that C needs A because C is going to immediately need to communicate with A, directly, once it is deployed. The reason I mention the service mesh is that it is one of the many options you do have to use as the “glue” @z0rc3r mentioned above. Worth looking up at least to help make an informed decision about what’s best for your architecture :)

Depending on your implementation needs: might be an option. We pack some crds in charts and make release dependencies this way. However, this is only the deployment dependency. This approach also has been mentioned in the conversation above.

@Abhinav Goyal - are you using Helm to manage CRDs as Templates?

@Andrew Nazarov,@tim.j.birkett: Yes, I am managing CRD resources as a Helm chart. The CRD definitions are already a separate chart and specified as needs: for the CRD resources chart install. But where I have a problem is that the CRD resources chart itself has an order of deployment within a single chart. How do I take care of that?

Interesting @Abhinav Goyal - how do your CRDs have ordering or dependency in them? Does one CR contain the spec of another CR or something? Just trying to understand a bit more

@tim.j.birkett: Here is the ordered list of resources for install. Un-install order is reverse.

1. PodTemplate
2. TargetTypeDefinition
3. TargetInstanceConfig

Why do they have to be in order? Surely the whole point of using operators and CRDs is that everything is eventually consistent. Just as some special deployment ritual to get a distributed microservices based system running should be considered a “smell” of sorts, maybe having to install resources in Kubernetes in a specific order could also be classed as a code smell of sorts.

If the operator has some dependency, for example, the TargetInstanceConfig references a TargetTypeDefinition it could be possible to have the operator not validate that the TargetTypeDefinition exists but instead handle the error and make use of a status field on the TargetInstanceConfig CR to let the user know what is wrong or what state reconciliation is in.

Take the classic Deployment, Service, Ingress case. They all reference each other in some way yet you can install and uninstall them in any order. Kubernetes reconciliation should be treated and designed as eventually consistent.

Agree with Tim, I think the reconciliation loop should take care of it. Or if you really want to have the order control I don’t see the option other than to split this to separate charts which sounds a bit “counter-k8s”. For example Google’s Config Connector has many cross-referenced objects and the cool thing of k8s is that you just throw them into the cluster and controllers will do the rest.

@tim.j.birkett: I agree with your point but this was just done by another team and I don’t have control over it as of now. So in the current situation as @Andrew Nazarov says, is splitting the charts my only option now?

Afraid so @Abhinav Goyal - three separate charts each with one CRD installed in the special order. Work with the team to improve that if you can.

@tim.j.birkett: Sure, thanks for this.

@Leo Przybylski

that’s neat! hadn’t seen it

Try building an umbrella chart and add it in the templates folder