#helmfile (2022-04)

https://github.com/helmfile/helmfile

Questions and discussion around helmfile https://github.com/roboll/helmfile and https://github.com/cloudposse/helmfiles

Archive: https://archive.sweetops.com/helmfile/

2022-04-04

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Helmfile has officially forked!

https://github.com/helmfile/helmfile

helmfile/helmfile

Declaratively deploy your Kubernetes manifests, Kustomize configs, and Charts as Helm releases in one shot

5
bananadance1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Make sure to star the repo

helmfile/helmfile

Declaratively deploy your Kubernetes manifests, Kustomize configs, and Charts as Helm releases in one shot

2
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

2022-04-05

Christian avatar
Christian

Anyone uses helmfile just for templating?

Currently I have a monochart repository where all my apps depend on. I use argocd so I only use helmfile template --output-dir to achieve gitops. Just want to ask around as to how other people have been doing it!

Erik/Cloudposse has influenced me to move my standard helm charts (karpenter, external-dns, etc) to terraform helm. Helmfile is only for having a declarative config for templating my apps.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yep, I think we’re doing this in some places with helmfile + argocd

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we’ll have github actions run helmfile and generate the yaml manifests and commit them to an argocd deployment repo

Christian avatar
Christian

Nice! We have exactly that. Good validation on our current workflow. Listening to office hours has been helpful. Thanks Erik!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@mumoshu are you going to move the terraform-provider-helmfile to the new org?

mumoshu avatar
mumoshu

i’d ask other maintainers if they would like to support it!

mumoshu avatar
mumoshu

do you still use it?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

So we were moving towards terraform-provider-helm, but there are some bad bugs we cannot over come. The worst of them is that believe it or not, a terraform plan can actually alter the terraform state of a chart deployed with the provider.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

So our current plan is to make it a feature flag in our https://github.com/cloudposse/terraform-aws-helm-release module to use the helmfile provider.

cloudposse/terraform-aws-helm-release

Create helm release and common aws resources like an eks iam role

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We do have one problem preventing us from finishing it, and I think @Andriy Knysh (Cloud Posse) is going to open a PR for it.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

There’s one other bug we have encountered with the hashicorp helm provider, is that it doesn’t handle the 3-way diff problem (that I think you) solved in the helm-diff plugin used by the helmfile and therefore the helmfile provider.

mumoshu avatar
mumoshu

ah gotcha about the 3-way diff prob. afaik a terraform provider should generally let the terraform core diff it by comparing the old and the new values. there’s no easy way to do diff on the provider side

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yep, so not holding our breath for an easy fix in the helm provider. I think the helmfile provider would be a good enough stop gap for our use-case/problem.

mumoshu avatar
mumoshu

um maybe.. i have a concern though. i’ve never tried using the helmfile provider on Terraform Cloud. have you?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think it wouldn’t work well without using private hosted runners.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

In our case, we’re using this only for the last-mile “root” modules in a customer environment where we can easily control for all the tools. For public/open source, it’s problematic since like you said it doesn’t work out-of-the-box everywhere.

mumoshu avatar
mumoshu


terraform plan can actually alter the terraform state of a chart deployed with the provider.
have you already reported it to the hashicorp repo? i thought this was impossible due to how tf provider plugin works(and how the core restricts writes to the state from any tf provider plugin functions related to terraform plan)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We thought it should be impossible as well. @Dylan do you have an open issue for this?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

I guess the issue was that terraform updates the state, then the helm provider fails, but Tf does not update the state after the failure - the state says all is ok, but in reality nothing is deployed correctly

mumoshu avatar
mumoshu

that makes sense to me! i see one of such issues is fixed here: https://github.com/hashicorp/terraform-provider-helm/pull/727 what was your version of the provider when you encountered it? in my experience, tf itself was quite solid so you won’t result in such state as long as (1)you correcntly surface the error, like done in https://github.com/hashicorp/terraform-provider-helm/pull/727 and (2)ensure the provider to update attributes only after the successful operation https://github.com/hashicorp/terraform-provider-helm/blob/e858cde60487a92f6dfe160f46c874350717df1c/helm/resource_release.go#L667-L676

Dylan Bannon avatar
Dylan Bannon

Sorry for the slow reply, everyone! This is the canonical issue we keep referencing for this bug, @Erik Osterman (Cloud Posse): https://github.com/hashicorp/terraform-provider-helm/issues/472

The scenario is that a Helm release fails to deploy resources due to any error. Ie. Pod health check failure or misconfiguration of helm chart. The release updates the tf state and therefor wont redeploy on next attempt.

Terraform Version and Provider Version

Terraform v0.12.21
Helm v3.1.2

Provider Version

provider.helm v1.1.1

Affected Resource(s)

• helm_release

Expected Behavior

If any failure occurs during deployment of the release it shouldn’t mark the deployment as done

Actual Behavior

No changes are detected between new release and previous release so terraform doesn’t trigger a new deployment.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Helmfile needs a logo! I’m always struggling for something on our slides.

This is my crude hack.

5

2022-04-06

2022-04-07

2022-04-09

tom avatar

Hi thanks for the lib! I need to use the output of a.yaml.gotmpl in b.yaml.gotmpl. How can I do that? The “layering” feature seems promising but I fail to use it even for official examples (https://github.com/roboll/helmfile/issues/2128)

2022-04-11

Brandon Miller avatar
Brandon Miller

I do love me some helmfile! Quick mockup in adobe illustrator :p

Brandon Miller avatar
Brandon Miller

grr, I’ll throw it in photoshop later and get that white outline out of the way lol

Brandon Miller avatar
Brandon Miller

this might work…

3
yxxhero avatar
yxxhero

@mumoshu What do you think of this?

Brandon Miller avatar
Brandon Miller

weird, it isn’t showing in illustrator lol. Oh well, I’ve shown enough of my noobness for one day

yxxhero avatar
yxxhero

it is very good.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I like it!

1
Brandon Miller avatar
Brandon Miller

Thanks! I just watched your “Effortless Helm Deployments” talk for codefresh! Helped a lot!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Hahah wow that was like 3 years

Brandon Miller avatar
Brandon Miller

Yeah man, I haven’t been able to find a ton of recent k8’s-related learning material. Probably just not googling the right stuff. The cloudposse repo is a good SoT, i’ve found, if you’re looking for a quick “best practices” reference.

Brandon Miller avatar
Brandon Miller

I am curious though; what are some of the new/exciting things in k8’s-land in 2022? I was looking into Cluster API - that’s super exciting.

2022-04-12

observer avatar
observer

Hi folks, thanks for creating helmfile - it’s a great tool to manage multiple charts! I had a question about the needs parameter for specifying dependencies. The documentation recommends arranging helmfiles by microservice or by team. In our case we have multiple teams, with each team owning multiple microservices. The needs param works fine when service A depends (or needs) service B and both A and B are owned by the same team. If A needs B it’s simple to use needs in A’s release declaration in the same helmfile.

However, let’s say there is service C which is owned by another team, and C depends on A’s APIs. The needs param does not work across helmfiles (even when the kubecontext/namespace prefix is specified)

Whats is the recommended layout in such cases? One not so great approach would be a giant helmfile with everything inside it.

z0rc3r avatar

needs works okay when you need to express deployment requirements, but it’s a poor choice if you want to express runtime requirements. for example needs is useful when there is helmfile that deploys two charts: one comes with CRDs, secods uses these crds to deploys CRs (prometheus-operator and ServiceMonitor resources as example).

but you have proper microservices. there is no need to build deployment order (via needs). better look in how to implement readiness and liveness probes on service side, so they can check and wait until required services are up

z0rc3r avatar

otherwise you just building another monolith with needs as glue

Brandon Miller avatar
Brandon Miller

One thing that may be of use to look up is a service mesh - I.E. Istio Gateway or something similar. Point being, try to avoid your services talking directly to one another as much as possible. Use some sort of intermediate such as AMQP, Kafka, etc, and let the mesh act as an authoritative source of federation. This has at least been my experience. It doesn’t always work out exactly this way, but the instances in which it doesn’t are normally fairly obvious and straightforward.

Brandon Miller avatar
Brandon Miller

If there’s no way to use a broker as a mediator, there isn’t much you can do as far as guaranteeing that if one service breaks, it won’t break another, but a good start in that direction is to solidify your contracts at the service-level. gRPC Proto’s are a fantastic way of doing this.

observer avatar
observer

@z0rc3r Thanks, that reinforces much of what I have discovered by reading about what others are doing.

observer avatar
observer

@Brandon Miller Thanks for the pointer. My question was about more dependency management and I don’t have any hands-on experience with service meshes, but I’ll check it out.

Brandon Miller avatar
Brandon Miller

Hmm, could you elaborate at a high level on what C is? The OP mentions it’s a service, so my immediate assumption I suppose was that C needs A because C is going to immediately need to communicate with A, directly, once it is deployed. The reason I mention the service mesh is that it is one of the many options you do have to use as the “glue” @z0rc3r mentioned above. Worth looking up at least to help make an informed decision about what’s best for your architecture :)

2022-04-13

2022-04-18

Abhinav Goyal avatar
Abhinav Goyal

wave Hello, team! I am trying to use Helmfile to install resources based on some CRDs that we created. But the CRD kind resources themselves have a specific order of installation and un-install is just reverse. How can I have a specific the order of install and un-install with Helmfile?

Andrew Nazarov avatar
Andrew Nazarov

Depending on your implementation needs: might be an option. We pack some crds in charts and make release dependencies this way. However, this is only the deployment dependency. This approach also has been mentioned in the conversation above.

tim.j.birkett avatar
tim.j.birkett

@Abhinav Goyal - are you using Helm to manage CRDs as Templates?

Abhinav Goyal avatar
Abhinav Goyal

@Andrew Nazarov,@tim.j.birkett: Yes, I am managing CRD resources as a Helm chart. The CRD definitions are already a separate chart and specified as needs: for the CRD resources chart install. But where I have a problem is that the CRD resources chart itself has an order of deployment within a single chart. How do I take care of that?

tim.j.birkett avatar
tim.j.birkett

Interesting @Abhinav Goyal - how do your CRDs have ordering or dependency in them? Does one CR contain the spec of another CR or something? Just trying to understand a bit more

Abhinav Goyal avatar
Abhinav Goyal

@tim.j.birkett: Here is the ordered list of resources for install. Un-install order is reverse.

  1. PodTemplate
  2. TargetTypeDefinition
  3. TargetInstanceConfig
tim.j.birkett avatar
tim.j.birkett

Why do they have to be in order? Surely the whole point of using operators and CRDs is that everything is eventually consistent. Just as some special deployment ritual to get a distributed microservices based system running should be considered a “smell” of sorts, maybe having to install resources in Kubernetes in a specific order could also be classed as a code smell of sorts.

If the operator has some dependency, for example, the TargetInstanceConfig references a TargetTypeDefinition it could be possible to have the operator not validate that the TargetTypeDefinition exists but instead handle the error and make use of a status field on the TargetInstanceConfig CR to let the user know what is wrong or what state reconciliation is in.

Take the classic Deployment, Service, Ingress case. They all reference each other in some way yet you can install and uninstall them in any order. Kubernetes reconciliation should be treated and designed as eventually consistent.

Andrew Nazarov avatar
Andrew Nazarov

Agree with Tim, I think the reconciliation loop should take care of it. Or if you really want to have the order control I don’t see the option other than to split this to separate charts which sounds a bit “counter-k8s”. For example Google’s Config Connector has many cross-referenced objects and the cool thing of k8s is that you just throw them into the cluster and controllers will do the rest.

Abhinav Goyal avatar
Abhinav Goyal

@tim.j.birkett: I agree with your point but this was just done by another team and I don’t have control over it as of now. So in the current situation as @Andrew Nazarov says, is splitting the charts my only option now?

tim.j.birkett avatar
tim.j.birkett

Afraid so @Abhinav Goyal - three separate charts each with one CRD installed in the special order. Work with the team to improve that if you can.

1
Abhinav Goyal avatar
Abhinav Goyal

@tim.j.birkett: Sure, thanks for this.

2022-04-20

Ion avatar

Hi guys! Does anyone know if everything helmfile provides is also possible with flux2 ? Mainly speaking in terms of templating, orchestrating…

sohaibahmed98 avatar
sohaibahmed98
FairwindsOps/reckoner

Declaratively install and manage multiple Helm chart releases

sohaibahmed98 avatar
sohaibahmed98
FairwindsOps/nova

Find outdated or deprecated Helm charts running in your cluster.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Leo Przybylski (Cloud Posse)

FairwindsOps/nova

Find outdated or deprecated Helm charts running in your cluster.

sohaibahmed98 avatar
sohaibahmed98
FairwindsOps/goldilocks

Get your resource requests “Just Right”

sohaibahmed98 avatar
sohaibahmed98

any one using https://github.com/FairwindsOps/reckoner instead of helmfile?

FairwindsOps/reckoner

Declaratively install and manage multiple Helm chart releases

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

that’s neat! hadn’t seen it

FairwindsOps/reckoner

Declaratively install and manage multiple Helm chart releases

2022-04-25

2022-04-26

mr.shayv avatar
mr.shayv

Hey guys I’m trying to embed my daemonset yaml file into the public filebeat helm chart from github but I’m struggling. Can anyone assist me?

robert.foerster avatar
robert.foerster

Try building an umbrella chart and add it in the templates folder

2022-04-27

    keyboard_arrow_up