#helmfile (2021-01)

It seems that variable tillerless if is set to true has this side effect… but I’m still investigating…

I thought it will disable tiller but helm 3 does not have tiller anymore. So it should be false as it is by default. I hope this may help somebody :)

Ok, just found it few lines upper. Thanks

Mighty @mumoshu, can You take a look at https://github.com/variantdev/vals/pull/39 ? Thanks

thanks for the reminder! merged / released. i’ll downstream it to helmfile soon

Depends on the value of the wait: param

https://github.com/roboll/helmfile

--wait                         if set, will wait until all Pods, PVCs, Services, and minimum number of Pods of a Deployment, StatefulSet, or ReplicaSet are in a ready state before marking the release as successful. It will wait for as long as --timeout

https://helm.sh/docs/helm/helm_install/

also present on helm upgrade which I think is the one Helmfile actually uses (helm upgrade --install)

@roth.andy If wait is not enabled, it’ll show deployed as soon as helm install returns ? Thanks for the resources, I completely forgot about this option in helmfile.

Also, if the wait timeouts, will the status be pending-install or failed ?

If wait is not enabled, it’ll show deployed as soon as helm install returns ? yep
Also, if the wait timeouts, will the status be pending-install or failed ? not sure what you mean by status. Helm/Helmfile will return with a non-zero exit code

I meant will it update my release status . ohh ! so the sync itself would fail. got it ! Thanks!

I created feature request https://github.com/mumoshu/terraform-provider-helmfile/issues/51

I shelled out to kubectl with exec and called it a day Still interested to see if there are any other ideas or mechanisms I should be using though!!

well, you may try to introduce some helmfile state variable and use it in all the places you need as long as all the releases are managed via helmfile

Interesting!! Thanks! I have no context for what a helmfile state variable is / how to use them / what things can be put into them - do you have an example perhaps? I have seen them mentioned in the docs, but it wasn’t clear to me how to use them

helmfile.yaml:

environments:
  dev:
    values:
      - domain: example.org

kibana.yaml.gotmpl:

  hosts:
    - "kibana.{{ .Values.domain }}"

Ahhh I see, thanks!! So it still won’t be able to get a dynamically generated value off of the kubernetes objects (like an IP, let’s say) once created. This is a useful tool to keep in mind, though, thanks!

well, this will not work with generated values on the k8s side. It’s better to move generation to terraform/helmfile side to keep it “static” instead of dynamic

I would love to do that and generally agree. I will probably come back to it and see if there is a better solution (or if our app can determine the values at runtime rather than put them in configuration).

Basically, right now I need the nfs-server-provisioner’s service IP (because kubelet does not resolve the NFS service’s domain name), and the nfs-server-provisioner’s export path (which is a GUID and does not seem to be configurable) to do “sub-mounts”. There may be a way to statically define these things, but I am unfortunately not aware of them

because kubelet does not resolve the NFS service’s domain name I think you need to fix it. k8s depends on working resolver highly.

It’s an outstanding (yet closed) bug in some kubernetes implementations: https://github.com/kubernetes/kubernetes/issues/44528#issuecomment-745438475

ah, issue with integrated storage part of k8s

@cole how did you shelled out that? I would like to get few info from a running cluster and not use the envs

So you can do it either in helmfile.yaml.gotmpl or a values.yaml.gotmpl file. (In this case, we are doing the latter)

 {{ $nfsIP := exec "kubectl" (list "get" "svc" "nfs-nfs-server-provisioner" "--output=jsonpath={.spec.clusterIP}") }}
 {{ $nfsVolume := exec "kubectl" (list "get" "pvc" "ha-storage" "--output=jsonpath={.spec.volumeName}" ) }}

then use the variables in the template like:

{{ $nfsIP }}
{{ $nfsVolume }}

I hope that helps! We abuse this exec pattern a bit more than we probably should, but it works

yeah that helped a lot. thanks

got this error : [exec: “kustomize”: executable file not found in $PATH

problem found missing helm-x plugin

https://github.com/mumoshu/helm-x/tree/master/examples/helmfile

if I run with

$ helmfile --helm-binary ~/.helm/plugins/helm-x/bin/helm-x --log-level debug apply

I get error:

 panic: exec: "": executable file not found in $PATH

any idea ?

could go forward , but still not working

If I use –helm-binary and points to helm-x binary as mentioned in doc I got the following error:

F0112 11:08:07.843810  217384 main.go:59] unknown command "dependency" for "helm-x"]

withouth helm–binary option different error came

[exec: "kustomize": executable file not found in $PATH

any advice ? thanks in advance

plugin works… remain problem: I see it patches the files,but I don’t see the pod deployed….

how can I double check that helmfile generated patched yaml with kustomize?

Please not have

{{`{{"{{ $labels.target }}"}}`}}

as the solution. (reference: https://archive.sweetops.com/helmfile/2020/12/)

Have you tried \$ ? I think I used that for a . once and it worked well!

That doesn’t work for gotemplate

So the key in your values or a variable has a $ in it? Something like .[Values.my](http://Values.my)$thing? Does quoting work? Something like .Values.'my$thing'? I’ll probably need a reproducible example of a helmfile and what you’re trying to accomplish for me to be any help Maybe someone else will know top-of-mind

This doesn’t work because the actual values is a gotemplate, so you have gotemplate in a gotemplate. I used

{{ printf "{{ $label.target }}" }}

to shield the value string so that it doesn’t get interpolated. The alternative was to escape it, using:

{{`{{"{{ $labels.target }}"}}`}}

The printf method was more readable to me.

Kinda. It depends on what you mean by helmfiles in helmfiles:) See helmfiles: for example

https://github.com/roboll/helmfile/blob/master/docs/shared-configuration-across-teams.md

https://github.com/roboll/helmfile/issues/1649

OMG. Indentation error, /wrist. I wish there was better output to tell me what the error is… nuts.

Have you figured it out?

Indentation error, seems so

$ helmfile -l name=$RELEASE_NAME ...

Yes, more separation more headache, but allow you have CHANGELOG.md and proper tags/versions. You want to have stable releases, and revert back when needed.

Got it. I figured that’s the way. Although with separate repos for terraform modules and then separate repos for helmfile charts, these would be a ton of repos

https://memegenerator.net/img/instances/25342902/repos-repos-everywhere.jpg

You can try to set

disableValidation: true

in issuers

Is it not possible to make helmfile just run the install of cert-manager first and then install the issuers?

Being a part of helmfile apply helm-diff couldn’t find corresponding cert-manager’s api. If you run helmfile sync your configuration will work

Thanks!

we usually set missingFileHandler in release template, *dbcc in your case. Release templates can reference to top-level templates, thus it’s not a problem to get DRY.

We are doing something like this:

# my_helmfile.yaml

helmDefaults:
  verify: false
  wait: false
  timeout: 600
  recreatePods: false
  force: false

environments:
  bim:
    values:
      - environments/dev/my.yaml

helmfiles:
  - path: git::<https://gitlab>+deploy-token-XX:{{ requiredEnv "REPO_TOKEN" }}@gitlab.mydomain.com/mygroup/myrepo.git@deployment/helmfile.yaml?ref={{ requiredEnv "INFRA_VER" }}
    values:
      - environments/dev/my.yaml

Assuming you know how Helm works you might want to check bitnami’s helm chart as a starting point: https://github.com/bitnami/charts/tree/master/bitnami/postgresql

Thank you Andrey, i will check that out

For DB creation and user mgmt you may be interested in this as well https://github.com/movetokube/postgres-operator

Thanks man, i will take a look at it as well

essentially I am trying to do

{{- $secrets   := exec "split" (list "mybash.sh" "-l" "60" "vars" )}}

@here anyone ?

Had this problem before so what I did was just put the releases that depend on one another in one helmfile. Other releases should be broken up with their own separate helmfiles. Also seems easier to manage this way since we won’t have interdependencies across separate helmfiles.

It uses vals if we are talking about the same functionality). Here it is: https://github.com/variantdev/vals Don’t see any Consul support.

What error are you seeing?

this was surprising just because the chart can be installed with helm install

of course the helm diff upgrade command generated by helmfile does fail when run manually as well. im trying to dbug starting there in helm

also trying to update the diff plugin to see if that is where the issue is

just discovered the disableValidation property in a release. giving that a try

https://github.com/roboll/helmfile/pull/1373 looks to be relevant

that seems to have worked!

Yes, you were on a right track)

The reason was that diff couldn’t find resources because they were not presented in the cluster. That’s why disableValidation is needed

Yup, it’s a bit of a chicken and egg problem when the CRDs haven’t been created by the initial install.

The approach we are taking is deploying infrastructure and platform with terraform. Argo is part of the platform, so we use terraform with helm provider.

Got it thanks for the response! I haven’t explored using terraform to deploy anything inside my cluster. I just use terraform to create the cluster itself so I’m curious of the pros and cons.

Does this also mean you guys haven’t been using the collection of helmfiles directly you have here? Or you meant using the helmfile provider?

Thanks again!

Nono, check the commit times on those

We’re using helmfiles extensively, but starting to use terraform to deploy some of the foundational platform services.

https://registry.terraform.io/modules/cloudposse/tfc-cloud-agent/kubernetes/latest

I’ve actively avoided letting Terraform manage anything to do with helm. It’s hard to explain why, but something feels wrong about it, possibly when helm and TF getting in conflict if a helm fails an upgrade or so, I forget why. Has the world improved since a year or so ago I made the decision to keep TF away from Helm?

It’s just there’s a chicken and the egg problem. If we want to use ArgoCD, we need to provisioned somehow. We could deploy it by hand, but we also have CD for terraform. So by phase-shifting to an alternate platform, we’re able to mitigate the cold-start problem while providing a long-term solution for maintenance.

Many ways to solve it…. Terraform just slots in nicely for us.

If it works for you then thats great, and that also means that maybe the TF helm(and or helmfile) providers have matured to a usable / production ready state now, from when I last attempted it some 12m+ ago.

Ya, both have gone through a lot of iternations. It took a while for the helm provider to support helm3. And similarly the kubernetes provider didn’t support custom resources. That’s all been addressed.

I remember having to a lot of the time go back to using helm CLI to fix upset state between helm and TF so we dropped it in the end, worked fine the initial run, but when we then used it for upgrading etc helm charts …some charts didn’t upgrade needed deleting etc just got in a mess

yeah, TF to manage CRD’s is tempting, we’ve not solved our CRD upgrade issues as of yet, very painful

Interesting discussion. I’ve considered adding ArgoCD via Tterraform as well in our next iteration of clusters. @Erik Osterman (Cloud Posse) If I may ask: When using ArgoCD, do you still use helmfile? Or have you converted any existing helmfile setups to vanilla Helm?

We at first explored using it, and it can work, but you can’t expect to use things like hooks, native ssm integration (which would leak secrets to git), etc.

Yeah, was reading up on this GH discussion and was wondering about hooks as well: https://github.com/argoproj/argo-cd/issues/2143

So for this reason, we’re not adopting helmfile with argocd. But here’s the deal. There are different lifecycles: what deploys argocd? that can still be helmfile.

What deploys the ingress that is required before you can even access argocd? that’s also helmfile.

So I take you cut out helmfile and simply specify values files in the App.

About the lifecycles. Our setup has some resemblance with startt of this thread: Terraform for bootstrapping EKS clusters + some RBAC setup. Then a lo-fi Jenkins pipeline based on helmfile installs all system components.

That pipeline has it’s quirks so replacing that with ArgoCD’s more pull-like mechanism looks interesting. And in our scenario, similar to what you described on top, bootstrapping ArgoCD via Terraform would be ideal.

So the mental model/separation of concerns we’re considering is that argocd is really more for the inhouse applications which change all the time. but for platform-level services (ingress, external-dns, cert-manager, etc) those are not managed by argo.

(just created argocd )

Hmm, that ingress one sure is interesting…. Although I assume it can start headless and eventually will install ingress being one of the system components it manages. But yeah, where does it’s own ingress come from.

On second thought, if it installs the ingress object and the controller actually handling it arrives later (pulled by ArgoCD). Might just work.

yea, it might work…

‘might’ is good enough for now. Thx for your input! Great Slack DevOps community you’ve got here!

but for platform-level services (ingress, external-dns, cert-manager, etc) those are not managed by argo. Nice that I came to the same conclusion! Practicing gitops for 3rd party helm charts (ingress, cert-manager, etc) didn’t make a lot of sense so I still let helmfile manage them.

From this thread, I guess there are still 2 ways to manage 3rd party charts, terraform helm or helmfile. I’ve gotten used to using helmfile but I agree just using the helm provider is worth exploring, especially when you already have a terraform CI pipeline ready. It also makes referencing dependencies easier (since everything is in TF), so you don’t have to hardcode values.

Anyway, thanks Erik for the discussion!

There’s a terraform helmfile provider I believe which I guess would allow you to keep the helmfile wrapper but pass in the dynamics via TF->helmfile in the single run.

right now we 3 step our in a CI pipeline, TF for EKS -> istioctl -> helmsman(will prob go helmfile at some stage). Business apps are fairly easy, as you say the ‘fun’ is the 3rd party platform apps .

There’s the Capabilities built-in object. But I think you’re looking for more specific things to read from the cluster.

My thoughts then would be preceding helmfile command with glue code like this:

export MY_VAR=$(kubectl -n get cm my-config -o jsonpath="{.data.some-key}")

I would like to elimitate all envs from my helmfile, so when I have access to the cluster I only need to run a helmfile apply and it gets all data from the cluster