#helmfile (2021-01)

https://github.com/helmfile/helmfile

Questions and discussion around helmfile https://github.com/roboll/helmfile and https://github.com/cloudposse/helmfiles

Archive: https://archive.sweetops.com/helmfile/

2021-01-01

roth.andy avatar
roth.andy

Quick little utility program I came up with to convert Kubernetes YAML to a format that can be used by the incubator/raw helm chart: https://github.com/RothAndrew/convert-to-helm-raw-values

RothAndrew/convert-to-helm-raw-values

Small utility that will convert K8s YAML files to a values.yaml file capable of being used with the incubator/raw helm chart - RothAndrew/convert-to-helm-raw-values

1

2021-01-04

Nenad Strainovic avatar
Nenad Strainovic

Hi everyone, Does anyone have a problem with helmfile concurrency? I’m not able to install multiple releases in parallel, I see only one helm upgrade/install process at the time. Latest helmfile, tried on Windows (exe) and also tried to run as docker image.

Nenad Strainovic avatar
Nenad Strainovic

It seems that variable tillerless if is set to true has this side effect… but I’m still investigating…

Nenad Strainovic avatar
Nenad Strainovic

I thought it will disable tiller but helm 3 does not have tiller anymore. So it should be false as it is by default. I hope this may help somebody :)

Cameron Boulton avatar
Cameron Boulton

Happy New Year @mumoshu, hope you are well! Just wanted to get your attention on a vals PR when you have time: https://github.com/variantdev/vals/pull/40 Hoping to get this in Helmfile after merge too. Please let me know if there is a different maintainer I should ping instead.

2021-01-05

zadkiel avatar
zadkiel

Hello there! Are we able to import releases through http with helmfile? I’d like to use releases from cloudposse/helmfiles without using submodules/cloning the whole repo

zadkiel avatar
zadkiel

Ok, just found it few lines upper. Thanks

1

2021-01-06

muhaha avatar

Guys? How are You using secrets in helmfile ? Currently I am sourcing bash script, which will via azure cli get proper secrets and then I am using requiredEnv in helmfile… Now I have these problems:

  1. argocd can install only binary plugins, so helm-secret will not help me, unless i customize my own argocd image, which is no go ( sops with azure keyvault )
  2. i am not sure what exactly is variantdev/vals and if it can replace helm-secret plugin
  3. seems that helm file has no integration to azure keyvault Any idea? Thanks
muhaha avatar

Mighty @mumoshu, can You take a look at https://github.com/variantdev/vals/pull/39 ? Thanks

mumoshu avatar
mumoshu

thanks for the reminder! merged / released. i’ll downstream it to helmfile soon

2
Sam Buckingham avatar
Sam Buckingham

Hey guys, thought I’d finally join the Slack channel since I’ve being working with Helmfile for a fair while now. I’ve just come back to work after a short holiday and I’m facing a strange issue after the latest update if anyone has any ideas what could be causing it?

in ./helmfile-3dce096a66df5e8515ae85f40bd83a2bc10620b819f7b5c0019505065acd204d.yaml: in .helmfiles[1]: in helmfile/releases/01-misc-helmfile.yaml: 2 errors:
10:44:31  err 0: failed processing release nfs-server-qa: open .terraform/helmfile/temp-7847bc44b4/qa-nfs-server-qa-values-855cb5cbd4: no such file or directory
10:44:31  err 1: failed processing release redis-tasks-qa: open .terraform/helmfile/temp-7847bc44b4/qa-redis-tasks-qa-values-5c78bf6c78: no such file or directory

This happens on any charts, the only one it does not happen on is rbac chart but that is only because it never loads a values file. Anyone faced this before?

Sam Buckingham avatar
Sam Buckingham

^ Continuing on from above it would seem if I revert back to the previous version of the helmfile provider 0.11.1 I don’t experience this issue so I’m guess it has something to do with the –skip-diff-on-install @mumoshu

Sam Buckingham avatar
Sam Buckingham

This is using 14.4 terraform and latest versions of both helmfile provider and helmfile

Abhishek avatar
Abhishek

Hey everyone ! Does helmfile status showing STATUS: deployed guarantee that the pods are in running state or it shows deployed as soon as they are scheduled ?

roth.andy avatar
roth.andy

Depends on the value of the wait: param

https://github.com/roboll/helmfile

--wait                         if set, will wait until all Pods, PVCs, Services, and minimum number of Pods of a Deployment, StatefulSet, or ReplicaSet are in a ready state before marking the release as successful. It will wait for as long as --timeout
roboll/helmfile

Deploy Kubernetes Helm Charts. Contribute to roboll/helmfile development by creating an account on GitHub.

roth.andy avatar
roth.andy
Helm Install

Helm - The Kubernetes Package Manager.

roth.andy avatar
roth.andy

also present on helm upgrade which I think is the one Helmfile actually uses (helm upgrade --install)

Abhishek avatar
Abhishek

@roth.andy If wait is not enabled, it’ll show deployed as soon as helm install returns ? Thanks for the resources, I completely forgot about this option in helmfile.

Abhishek avatar
Abhishek

Also, if the wait timeouts, will the status be pending-install or failed ?

roth.andy avatar
roth.andy


If wait is not enabled, it’ll show deployed as soon as helm install returns ?
yep
Also, if the wait timeouts, will the status be pending-install or failed ?
not sure what you mean by status. Helm/Helmfile will return with a non-zero exit code

Abhishek avatar
Abhishek

I meant will it update my release status . ohh ! so the sync itself would fail. got it ! Thanks!

2021-01-07

muhaha avatar

Any idea if is possible to include raw kubeconfig as variable, instead of passing kubeconfig file in https://github.com/mumoshu/terraform-provider-helmfile ? For example with this provider https://github.com/vmware-tanzu/terraform-provider-carvel/blob/develop/pkg/provider/kubeconfig.go#L33 is possible to reference kubeconfig as variable without need of existing kubeconfig file …

cole avatar

Hello! I am very thankful for the work on helmfile, which has been a lot of fun to use!

A bit of a strange question here - is it possible to “pass” variables / outputs between releases? I.e. I have a helm release that depends on dynamic values from a previous helm/helmfile release (i.e. in a different directory) - I could get the values by shelling out to kubectl or something like that, but that feels really dirty. Wondering if anyone has ideas for a better way

cole avatar

I shelled out to kubectl with exec and called it a day Still interested to see if there are any other ideas or mechanisms I should be using though!!

voron avatar

well, you may try to introduce some helmfile state variable and use it in all the places you need as long as all the releases are managed via helmfile

cole avatar

Interesting!! Thanks! I have no context for what a helmfile state variable is / how to use them / what things can be put into them - do you have an example perhaps? I have seen them mentioned in the docs, but it wasn’t clear to me how to use them

voron avatar

helmfile.yaml:

environments:
  dev:
    values:
      - domain: example.org

kibana.yaml.gotmpl:

  hosts:
    - "kibana.{{ .Values.domain }}"
cole avatar

Ahhh I see, thanks!! So it still won’t be able to get a dynamically generated value off of the kubernetes objects (like an IP, let’s say) once created. This is a useful tool to keep in mind, though, thanks!

voron avatar

well, this will not work with generated values on the k8s side. It’s better to move generation to terraform/helmfile side to keep it “static” instead of dynamic

cole avatar

I would love to do that and generally agree. I will probably come back to it and see if there is a better solution (or if our app can determine the values at runtime rather than put them in configuration).

Basically, right now I need the nfs-server-provisioner’s service IP (because kubelet does not resolve the NFS service’s domain name), and the nfs-server-provisioner’s export path (which is a GUID and does not seem to be configurable) to do “sub-mounts”. There may be a way to statically define these things, but I am unfortunately not aware of them

voron avatar


because kubelet does not resolve the NFS service’s domain name
I think you need to fix it. k8s depends on working resolver highly.

cole avatar

It’s an outstanding (yet closed) bug in some kubernetes implementations: https://github.com/kubernetes/kubernetes/issues/44528#issuecomment-745438475

volumes/nfs example: service name instead hardcoded IP · Issue #44528 · kubernetes/kubernetes

Kubernetes version (use kubectl version): Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.1", GitCommit:"82450d03cb057bab0950214ef122b67c83fb1…

voron avatar

ah, issue with integrated storage part of k8s

Balazs Varga avatar
Balazs Varga

@cole how did you shelled out that? I would like to get few info from a running cluster and not use the envs

cole avatar

So you can do it either in helmfile.yaml.gotmpl or a values.yaml.gotmpl file. (In this case, we are doing the latter)

 {{ $nfsIP := exec "kubectl" (list "get" "svc" "nfs-nfs-server-provisioner" "--output=jsonpath={.spec.clusterIP}") }}
 {{ $nfsVolume := exec "kubectl" (list "get" "pvc" "ha-storage" "--output=jsonpath={.spec.volumeName}" ) }}

then use the variables in the template like:

{{ $nfsIP }}
{{ $nfsVolume }}

I hope that helps! We abuse this exec pattern a bit more than we probably should, but it works

1
Balazs Varga avatar
Balazs Varga

yeah that helped a lot. thanks

1

2021-01-08

2021-01-11

Balazs Varga avatar
Balazs Varga

hello all, can somebody give me an example about strategicMergePatches?

Balazs Varga avatar
Balazs Varga

got this error : [exec: “kustomize”: executable file not found in $PATH

Balazs Varga avatar
Balazs Varga

problem found missing helm-x plugin

Balazs Varga avatar
Balazs Varga

https://github.com/mumoshu/helm-x/tree/master/examples/helmfile

if I run with

$ helmfile --helm-binary ~/.helm/plugins/helm-x/bin/helm-x --log-level debug apply

I get error:

 panic: exec: "": executable file not found in $PATH

any idea ?

mumoshu/helm-x

Treat any Kustomization or K8s manifests directory as a Helm chart - mumoshu/helm-x

Balazs Varga avatar
Balazs Varga

could go forward , but still not working

Balazs Varga avatar
Balazs Varga

If I use –helm-binary and points to helm-x binary as mentioned in doc I got the following error:

F0112 11:08:07.843810  217384 main.go:59] unknown command "dependency" for "helm-x"]

withouth helm–binary option different error came

[exec: "kustomize": executable file not found in $PATH

any advice ? thanks in advance

Balazs Varga avatar
Balazs Varga

plugin works… remain problem: I see it patches the files,but I don’t see the pod deployed….

Balazs Varga avatar
Balazs Varga

how can I double check that helmfile generated patched yaml with kustomize?

Joaquin Menchaca avatar
Joaquin Menchaca

How do I escape values that have $ in the values? I am getting:

in ./helmfile.yaml: failed processing release my-prometheus: failed to render values files "dgraph-app-alert-rules.yaml.gotmpl": failed to render [dgraph-app-alert-rules.yaml.gotmpl], because of template: stringTemplate:11: undefined variable "$labels"
Joaquin Menchaca avatar
Joaquin Menchaca

Please not have

{{`{{"{{ $labels.target }}"}}`}}

as the solution. (reference: https://archive.sweetops.com/helmfile/2020/12/)

SweetOps #helmfile for December, 2020

SweetOps Slack archive of #helmfile for December, 2020.

cole avatar

Have you tried \$ ? I think I used that for a . once and it worked well!

Joaquin Menchaca avatar
Joaquin Menchaca

That doesn’t work for gotemplate

cole avatar

So the key in your values or a variable has a $ in it? Something like .[Values.my](http://Values.my)$thing? Does quoting work? Something like .Values.'my$thing'? I’ll probably need a reproducible example of a helmfile and what you’re trying to accomplish for me to be any help Maybe someone else will know top-of-mind

Joaquin Menchaca avatar
Joaquin Menchaca

This doesn’t work because the actual values is a gotemplate, so you have gotemplate in a gotemplate. I used

{{ printf "{{ $label.target }}" }}

to shield the value string so that it doesn’t get interpolated. The alternative was to escape it, using:

{{`{{"{{ $labels.target }}"}}`}}
1
Joaquin Menchaca avatar
Joaquin Menchaca

The printf method was more readable to me.

2021-01-12

Ion avatar

Hi, all!

Do you know if it’s possible to have helmfiles in helmfiles? 2 levels

Andrew Nazarov avatar
Andrew Nazarov

Kinda. It depends on what you mean by helmfiles in helmfiles:) See helmfiles: for example

Andrew Nazarov avatar
Andrew Nazarov
roboll/helmfile

Deploy Kubernetes Helm Charts. Contribute to roboll/helmfile development by creating an account on GitHub.

Joaquin Menchaca avatar
Joaquin Menchaca

I am unsure what to do with this, as I don’t use environments for this mini-project:

 0: --
 1: repositories:
 2:   - name: prometheus-community
 3:     url: <https://prometheus-community.github.io/helm-charts>
 4:   - name: stable
 5:     url: <https://charts.helm.sh/stable>
 6: 
 7: releases:
 8:   - name: my-prometheus
 9:     namespace: monitoring
10:     chart: prometheus-community/kube-prometheus-stack
11:     values:
12:       - ./dgraph-prometheus-operator.yaml
13:       - grafana:
14:           adminPassword: admin123
15:      disableValidation: true
16: 

could not deduce `environment:` block, configuring only .Environment.Name. error: failed to read helmfile.yaml.part.0: reading document at index 1: yaml: line 2: mapping values are not allowed in this context

Is there something obviously wrong in yaml that I missed?

Joaquin Menchaca avatar
Joaquin Menchaca
could not deduce environment blcok error when no environment specified · Issue #1649 · roboll/helmfile

Description Helmfile failing due to error with environment block, which there is none, and I am not using environments. Config helmfile: v0.137.0 helm v3.4.2 helm diff plugin 3.1.3 kubectl v1.18.2 …

Joaquin Menchaca avatar
Joaquin Menchaca

OMG. Indentation error, /wrist. I wish there was better output to tell me what the error is… nuts.

Andrew Nazarov avatar
Andrew Nazarov

Have you figured it out?

voron avatar


Indentation error,
seems so

2021-01-13

Balazs Varga avatar
Balazs Varga

when I use mergepatch and try to deploy patched version of chart to my cluster it quits with error it does not find secret,configmap,statefulset etc… every time it created the mentioned resources but not in the time when it checked. it creates the release secret so next time I cannot install it again it sees everything was fine during install. If I remove the release secret file I can run helmfile apply again and it goes 1 step further and exit with different resource is missing error however those resources are in the patched yaml… Any idea ?

Tony Hirsch avatar
Tony Hirsch

how can i limit helmfile to a specific release?

Ievgenii Shepeliuk avatar
Ievgenii Shepeliuk

$ helmfile -l name=$RELEASE_NAME ...

2021-01-14

Christian avatar
Christian

For remote helmfiles (helmfiles referenced by git path), do people keep those helmfiles in separate repositories for easy versioning? Similar to how people generally do it with terraform modules?

Mohammed Yahya avatar
Mohammed Yahya

Yes, more separation more headache, but allow you have CHANGELOG.md and proper tags/versions. You want to have stable releases, and revert back when needed.

Christian avatar
Christian

Got it. I figured that’s the way. Although with separate repos for terraform modules and then separate repos for helmfile charts, these would be a ton of repos

1

2021-01-15

2021-01-16

Jonas Sjödin avatar
Jonas Sjödin

Hi, I’m new with helmfile and having a problem with the needs keyword. I have a very simple file which only applies cert-manager and my cluster issuers. When i run helmfile applyit errors out by saying:

no matches for kind “ClusterIssuer” in version “cert-manager.io/v1alpha2

I therefore use the needs keyword to make sure that helmfile installs cert-manager before my issuers but that is not working. If I comment out the issuers and first install cert-manager and then uncomment issuers and install them everything works. I guess I’m at fault here and that it probably is an obvious error but I can’t see what I’m doing wrong?

repositories:
  - name: prometheus-community
    url: <https://prometheus-community.github.io/helm-charts>
  - name: stable
    url: <https://charts.helm.sh/stable>

helmDefaults:
  createNamespace: true

releases:
  - name: cert-manager
    namespace: cert-manager
    chart: jetstack/cert-manager
    wait: true
    set:
      - name: installCRDs
        value: true
    version: v1.1.0
  
  - name: issuers
    namespace: cert-manager
    chart: issuers/
    needs:
      - cert-manager/cert-manager
    secrets:
      - issuers/secrets.yaml
Andrew Nazarov avatar
Andrew Nazarov

You can try to set

disableValidation: true

in issuers

Jonas Sjödin avatar
Jonas Sjödin

Is it not possible to make helmfile just run the install of cert-manager first and then install the issuers?

Andrew Nazarov avatar
Andrew Nazarov

Being a part of helmfile apply helm-diff couldn’t find corresponding cert-manager’s api. If you run helmfile sync your configuration will work

Jonas Sjödin avatar
Jonas Sjödin

Thanks!

2021-01-17

Denis Boulas avatar
Denis Boulas

Hi there! I’m trying to set missingFileHandler: Warn for values: section of my helmfile:

bases:
- ../_lib/defaults.yaml.gotmpl
---
{{ readFile "../_lib/templates.yaml.gotmpl" }}

values:
- hosts: []
#  - name: "example"
#    ingressClass: "nginx"
#    port: "80"
#    targetPort: "80"
#    upstreamAddress: "192.168.0.1"
#    upstreamVhost: "www.example.com"
#    hostname: "www.example.com"
- ../../config/dbcc-external-services/{{ .Environment.Name }}/values.yaml.gotmpl        # Import env-specific list of hosts

releases:
- name: dbcc-external-services
  <<: *dbcc
  chart: incubator/raw
  namespace: miscellaneous-extra
  version: 0.2.3

I cannot use Environments: here, because it’s complimentary helmfile included in a few entry-level helmfiles with different sets of environments… Any advice is much appreciated!

voron avatar

we usually set missingFileHandler in release template, *dbcc in your case. Release templates can reference to top-level templates, thus it’s not a problem to get DRY.

2021-01-18

Nathan Flynn avatar
Nathan Flynn

Hello, are there any examples of how to consume https://github.com/cloudposse/helmfiles/tree/master/releases ? is the design to consume these in ways such as

helmfiles:
- 
  path: git::<https://github.com/cloudposse/helmfiles.git@releases/echo-server/helmfile.yaml?ref=0.125.0>

and if so, how do I have more environments than specified in the echo-server/environments.yaml ?

cloudposse/helmfiles

Comprehensive Distribution of Helmfiles for Kubernetes - cloudposse/helmfiles

Andrew Nazarov avatar
Andrew Nazarov

We are doing something like this:

# my_helmfile.yaml

helmDefaults:
  verify: false
  wait: false
  timeout: 600
  recreatePods: false
  force: false

environments:
  bim:
    values:
      - environments/dev/my.yaml

helmfiles:
  - path: git::<https://gitlab>+deploy-token-XX:{{ requiredEnv "REPO_TOKEN" }}@gitlab.mydomain.com/mygroup/myrepo.git@deployment/helmfile.yaml?ref={{ requiredEnv "INFRA_VER" }}
    values:
      - environments/dev/my.yaml
cloudposse/helmfiles

Comprehensive Distribution of Helmfiles for Kubernetes - cloudposse/helmfiles

charlesz avatar
charlesz

hi, noob here, do you know a howto i can follow with this task i wanted to do? basically i want to create a postgres pod using helm possibly and point it to a persistent drive.. i am not sure if there is a way to pre create the databases as well, everytime a pod with postgres is created? please help point out a resource that i can read

Andrew Nazarov avatar
Andrew Nazarov

Assuming you know how Helm works you might want to check bitnami’s helm chart as a starting point: https://github.com/bitnami/charts/tree/master/bitnami/postgresql

bitnami/charts

Helm Charts. Contribute to bitnami/charts development by creating an account on GitHub.

charlesz avatar
charlesz

Thank you Andrey, i will check that out

Ievgenii Shepeliuk avatar
Ievgenii Shepeliuk

For DB creation and user mgmt you may be interested in this as well https://github.com/movetokube/postgres-operator

movetokube/postgres-operator

PostgreSQL operator for Kubernetes. Contribute to movetokube/postgres-operator development by creating an account on GitHub.

charlesz avatar
charlesz

Thanks man, i will take a look at it as well

2021-01-19

vgdubrea avatar
vgdubrea

hello all , I have a small question on exec template function. I want to execute split command and pass some vars to this command to be able to grab the values out of this , but I am unsure if this exec is for executing the scripts or I can just use any commands out of this , how does this internally execute this (like if in the shell this already runs in bash shell) ?

vgdubrea avatar
vgdubrea

essentially I am trying to do

{{- $secrets   := exec "split" (list "mybash.sh" "-l" "60" "vars" )}}
vgdubrea avatar
vgdubrea

@here anyone ?

Dahs81 avatar

Does anyone know if the needs: keyword works with separate helmfiles (i.e. in a helmfile.d/ directory)? I’d like to break my helmfiles up, but I don’t really like the idea of using the 00- , 01- , etc prefixes for each file in order to get them to apply in a specific order.

Christian avatar
Christian

Had this problem before so what I did was just put the releases that depend on one another in one helmfile. Other releases should be broken up with their own separate helmfiles. Also seems easier to manage this way since we won’t have interdependencies across separate helmfiles.

2021-01-21

Ievgenii Shepeliuk avatar
Ievgenii Shepeliuk

Hi all helmfile supports reading values from Hashicorp Vault via ref tool, but does it support retrieving values from Consul ?

Andrew Nazarov avatar
Andrew Nazarov

It uses vals if we are talking about the same functionality). Here it is: https://github.com/variantdev/vals Don’t see any Consul support.

variantdev/vals

Helm-like configuration values loader with support for various sources - variantdev/vals

Adam Schepis avatar
Adam Schepis

I’m seeing an error where i’m trying to install kube-prometheus-stack using helmfile but it appears to not be installing the stuff it depends on (CRDs specifically). does helmfile expect me to put these in manually or is it likely that im doing something wrong and helmfile would normally install all the chart’s dependencies? using helm3 and helmfile v0.137.0

Andrew Nazarov avatar
Andrew Nazarov

What error are you seeing?

Adam Schepis avatar
Adam Schepis
COMBINED OUTPUT:
  ********************
  	Release was not present in Helm.  Diff will show entire contents as new.
  ********************
  Error: Failed to render chart: exit status 1: Error: unable to build kubernetes objects from release manifest: [unable to recognize "": no matches for kind "Alertmanager" in version "monitoring.coreos.com/v1", unable to recognize "": no matches for kind "Prometheus" in version "monitoring.coreos.com/v1", unable to recognize "": no matches for kind "PrometheusRule" in version "monitoring.coreos.com/v1", unable to recognize "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"]
  Error: plugin "diff" exited with error
Adam Schepis avatar
Adam Schepis

this was surprising just because the chart can be installed with helm install

Adam Schepis avatar
Adam Schepis

of course the helm diff upgrade command generated by helmfile does fail when run manually as well. im trying to dbug starting there in helm

Adam Schepis avatar
Adam Schepis

also trying to update the diff plugin to see if that is where the issue is

Adam Schepis avatar
Adam Schepis

just discovered the disableValidation property in a release. giving that a try

Adam Schepis avatar
Adam Schepis
Add disableValidation and disableOpenAPIValidation per release by mumoshu · Pull Request #1373 · roboll/helmfile

disableOpenAPIValidation: true might be useful for workaround for broken CRDs that is known to be exist in older OpenShift versions, and disableValidation: true is confirmed to allow installing cha…

Adam Schepis avatar
Adam Schepis

that seems to have worked!

Andrew Nazarov avatar
Andrew Nazarov

Yes, you were on a right track)

Andrew Nazarov avatar
Andrew Nazarov

The reason was that diff couldn’t find resources because they were not presented in the cluster. That’s why disableValidation is needed

bazbremner avatar
bazbremner

Yup, it’s a bit of a chicken and egg problem when the CRDs haven’t been created by the initial install.

2021-01-22

2021-01-23

2021-01-24

Christian avatar
Christian

I’ve been using ArgoCD and helmfile together and as suggested in the README, I do a helmfile template in my CI to achieve gitops for my apps.

So question, do people also manage their base infrastructure charts (cert-manager, nginx/traefik chart, etc) in Argo? Or still let helmfile manage them? A bit weird to have two ways to deploy to the cluster but I guess that’s ok.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

The approach we are taking is deploying infrastructure and platform with terraform. Argo is part of the platform, so we use terraform with helm provider.

Christian avatar
Christian

Got it thanks for the response! I haven’t explored using terraform to deploy anything inside my cluster. I just use terraform to create the cluster itself so I’m curious of the pros and cons.

Does this also mean you guys haven’t been using the collection of helmfiles directly you have here? Or you meant using the helmfile provider?

Thanks again!

cloudposse/helmfiles

Comprehensive Distribution of Helmfiles for Kubernetes - cloudposse/helmfiles

mumoshu/terraform-provider-helmfile

Deploy Helmfile releases from Terraform. Contribute to mumoshu/terraform-provider-helmfile development by creating an account on GitHub.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Nono, check the commit times on those

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We’re using helmfiles extensively, but starting to use terraform to deploy some of the foundational platform services.

Nathan Flynn avatar
Nathan Flynn

I’ve actively avoided letting Terraform manage anything to do with helm. It’s hard to explain why, but something feels wrong about it, possibly when helm and TF getting in conflict if a helm fails an upgrade or so, I forget why. Has the world improved since a year or so ago I made the decision to keep TF away from Helm?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

It’s just there’s a chicken and the egg problem. If we want to use ArgoCD, we need to provisioned somehow. We could deploy it by hand, but we also have CD for terraform. So by phase-shifting to an alternate platform, we’re able to mitigate the cold-start problem while providing a long-term solution for maintenance.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Many ways to solve it…. Terraform just slots in nicely for us.

Nathan Flynn avatar
Nathan Flynn

If it works for you then thats great, and that also means that maybe the TF helm(and or helmfile) providers have matured to a usable / production ready state now, from when I last attempted it some 12m+ ago.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Ya, both have gone through a lot of iternations. It took a while for the helm provider to support helm3. And similarly the kubernetes provider didn’t support custom resources. That’s all been addressed.

Nathan Flynn avatar
Nathan Flynn

I remember having to a lot of the time go back to using helm CLI to fix upset state between helm and TF so we dropped it in the end, worked fine the initial run, but when we then used it for upgrading etc helm charts …some charts didn’t upgrade needed deleting etc just got in a mess

Nathan Flynn avatar
Nathan Flynn

yeah, TF to manage CRD’s is tempting, we’ve not solved our CRD upgrade issues as of yet, very painful

TBeijen avatar
TBeijen

Interesting discussion. I’ve considered adding ArgoCD via Tterraform as well in our next iteration of clusters. @Erik Osterman (Cloud Posse) If I may ask: When using ArgoCD, do you still use helmfile? Or have you converted any existing helmfile setups to vanilla Helm?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We at first explored using it, and it can work, but you can’t expect to use things like hooks, native ssm integration (which would leak secrets to git), etc.

TBeijen avatar
TBeijen

Yeah, was reading up on this GH discussion and was wondering about hooks as well: https://github.com/argoproj/argo-cd/issues/2143

helmfile support · Issue #2143 · argoproj/argo-cd

Is your feature request related to a problem? Please describe. Similar to helm, helmfile support would be great. Describe the solution you’d like Support for helmfile.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

So for this reason, we’re not adopting helmfile with argocd. But here’s the deal. There are different lifecycles: what deploys argocd? that can still be helmfile.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

What deploys the ingress that is required before you can even access argocd? that’s also helmfile.

TBeijen avatar
TBeijen

So I take you cut out helmfile and simply specify values files in the App.

About the lifecycles. Our setup has some resemblance with startt of this thread: Terraform for bootstrapping EKS clusters + some RBAC setup. Then a lo-fi Jenkins pipeline based on helmfile installs all system components.

That pipeline has it’s quirks so replacing that with ArgoCD’s more pull-like mechanism looks interesting. And in our scenario, similar to what you described on top, bootstrapping ArgoCD via Terraform would be ideal.

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

So the mental model/separation of concerns we’re considering is that argocd is really more for the inhouse applications which change all the time. but for platform-level services (ingress, external-dns, cert-manager, etc) those are not managed by argo.

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(just created argocd )

TBeijen avatar
TBeijen

Hmm, that ingress one sure is interesting…. Although I assume it can start headless and eventually will install ingress being one of the system components it manages. But yeah, where does it’s own ingress come from.

On second thought, if it installs the ingress object and the controller actually handling it arrives later (pulled by ArgoCD). Might just work.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, it might work…

1
TBeijen avatar
TBeijen

‘might’ is good enough for now. Thx for your input! Great Slack DevOps community you’ve got here!

1
Christian avatar
Christian


but for platform-level services (ingress, external-dns, cert-manager, etc) those are not managed by argo.
Nice that I came to the same conclusion! Practicing gitops for 3rd party helm charts (ingress, cert-manager, etc) didn’t make a lot of sense so I still let helmfile manage them.

Christian avatar
Christian

From this thread, I guess there are still 2 ways to manage 3rd party charts, terraform helm or helmfile. I’ve gotten used to using helmfile but I agree just using the helm provider is worth exploring, especially when you already have a terraform CI pipeline ready. It also makes referencing dependencies easier (since everything is in TF), so you don’t have to hardcode values.

Anyway, thanks Erik for the discussion!

Nathan Flynn avatar
Nathan Flynn

There’s a terraform helmfile provider I believe which I guess would allow you to keep the helmfile wrapper but pass in the dynamics via TF->helmfile in the single run.

Nathan Flynn avatar
Nathan Flynn

right now we 3 step our in a CI pipeline, TF for EKS -> istioctl -> helmsman(will prob go helmfile at some stage). Business apps are fairly easy, as you say the ‘fun’ is the 3rd party platform apps .

2021-01-25

Balazs Varga avatar
Balazs Varga

hello all, Is there a way to get info for helmfile from cluster before we do an apply ? Currently we provide values with env from OS, but it would be nice if we could grab it from k8s.

TBeijen avatar
TBeijen

There’s the Capabilities built-in object. But I think you’re looking for more specific things to read from the cluster.

My thoughts then would be preceding helmfile command with glue code like this:

export MY_VAR=$(kubectl -n get cm my-config -o jsonpath="{.data.some-key}")
Built-in Objects

Built-in objects available to templates.

Balazs Varga avatar
Balazs Varga

I would like to elimitate all envs from my helmfile, so when I have access to the cluster I only need to run a helmfile apply and it gets all data from the cluster

2021-01-27

2021-01-28

    keyboard_arrow_up