#helmfile (2021-05)

https://github.com/roboll/helmfile

Questions and discussion around helmfile https://github.com/roboll/helmfile and https://github.com/cloudposse/helmfiles Archive: https://archive.sweetops.com/helmfile/

2021-05-31

2021-05-30

dennybaa avatar
dennybaa

Hello @mumoshu. I would like to share my current prototype of a “central remote helmfiles” repository approach. When I was pointing that it would be nice to have remote git sync updates in a GH issue I mentioned that I had an idea on “central repository” like cloudposse…

While without any documentation on cloudposse/helmfiles I haven’t seen any goals of using that code, nonetheless I tried to implement an approach that seems suit my needs: https://github.com/dysnix/helmfiles.

Thinking of the upcoming improvements to helmfile, I feel that sharing the above might be a good source for ideas for the next versions Such as remote values and valueTemplates))

dysnix/helmfiles attachment image

Dysnix Distribution of Helmfiles for Kubernetes. Contribute to dysnix/helmfiles development by creating an account on GitHub.

mumoshu avatar
mumoshu

hey! thanks for sharing

dysnix/helmfiles attachment image

Dysnix Distribution of Helmfiles for Kubernetes. Contribute to dysnix/helmfiles development by creating an account on GitHub.

mumoshu avatar
mumoshu

yeah i think i have more understanding of what you’re trying to achieve now

1
mumoshu avatar
mumoshu
Possibility to update remote git sources of sub-helmfiles · Issue #1817 · roboll/helmfile attachment image

Hello @mumoshu! It would be nice to be able to update remote git sub-helmfiles. I've suddenly found this thing for myself and already envision some useful scenarios. It would be nice to have an…

2

2021-05-26

2021-05-25

Denis Palnitsky avatar
Denis Palnitsky

Hi, I’m using helm file in a CD pipeline to deploy Helm charts from a Gitops repository. It works great and I think you guys did a great job I have to questions:

  1. Is there a way to deploy all environments with one command? I want to be able to add env in helmfile in my gitops, commit it and see helmfile deploys it automatically.
  2. how do you people solve environment cleanup problem when you run helmfile in a pipeline? For example, I removed a chart or an env from a helmfile in gitops repo and I would like to detect those cleaned up resources automatically and remove them
Andrew Nazarov avatar
Andrew Nazarov

Don’t know about the first point, we never needed this. As for the second we have a organisational solution. If one wants to remove a release or something else from the helmfile it should be done in stages: first, installed should be set to false for corresponding releases, and second, only after this configuration is applied a block can be removed from a helmfile. We don’t do gitops, nonetheless I believe the same principle can be applied.

1
1
Denis Palnitsky avatar
Denis Palnitsky

Yep, seems like it’s the only solution for now

mumoshu avatar
mumoshu

Regarding the first point, there’s no built-in functionality that helps it. you need to script it

mumoshu avatar
mumoshu

For the second, right that installed: false is the only solution. That way you are extra sure helmfile would ever remove releases that are explicitly marked so.

mumoshu avatar
mumoshu

We’ve once discussed about storing some helmfile “state” in the k8s cluster so that we can track what are already installed onto the cluster but now missing in the definitions. But it turned out to be defeat one of helmfile goals of being a “thin” wrapper around helm.

1
mumoshu avatar
mumoshu

(It’s arguable if helmfile is really “thin” today but it would still be a great idea to keep it as simple as possible

2

2021-05-24

2021-05-22

2021-05-21

Victoria H avatar
Victoria H

Hey all. I’m brand new to helmfile, testing it out as a way to migrate away from my team’s current strategy of using a bash script to clobber together a base helm chart repository with 1+ other repositories with specific environment configurations. The repositories for the different environments contain both definitions files (simple enough to translate to helmfile), as well as specific configuration files that are either added to the base repo or overwrite existing files in the base repo (for example, in apache-httpd2, there is a deployment-specific.conf in the base repo as well as in the repo(s) for the other environments, so that gets overwritten). Does helmfile offer the functionality to insert these environment-specific configuration files depending on the deployment?

mumoshu avatar
mumoshu
roboll/helmfile attachment image

Deploy Kubernetes Helm Charts. Contribute to roboll/helmfile development by creating an account on GitHub.

roboll/helmfile attachment image

Deploy Kubernetes Helm Charts. Contribute to roboll/helmfile development by creating an account on GitHub.

Victoria H avatar
Victoria H

I think that might be on the right track. A better way of stating it– can helmfile be used to manage inserting cluster-specific FILES in addition to chart values? A good example of this in our repository is the use of .crt and .pem files for specific sub-charts. These live in directories adjacent to the microservice’s chart, and there is a template for a config map that takes everything in these directories as data for the config map ({{ (.Files.Glob "securityconfig/*").AsConfig | indent 2 }}). Say I wanted to deploy to my dev cluster, could I use the helmfile to include the desired .crt and .pem files for that environment?

2021-05-20

Brij S avatar
Brij S

Hi all, I’ve been trying to use https://docs.gitlab.com/ee/user/clusters/applications.html#install-gitlab-runner-using-gitlab-cicd

to install gitlab runners on my cluster thats integrated with Gitlab. It mentions that the registration token can be passed in via environment variable which I’ve set in my CICD variables.

However, when I try to run this, my values.yml is not picked up nor is my registration token. Upon investigating, I found their helmfile for it here https://gitlab.com/gitlab-org/cluster-integration/cluster-applications/-/blob/master/src/default-data/gitlab-runner/helmfile.yaml

I notice its doing

{{- if env "GITLAB_RUNNER_GITLAB_URL" }}
    - gitlabUrl: "{{ env "GITLAB_RUNNER_GITLAB_URL" }}"
{{- end }}
{{- if env "GITLAB_RUNNER_REGISTRATION_TOKEN" }}
    - runnerRegistrationToken: "{{ env "GITLAB_RUNNER_REGISTRATION_TOKEN" }}"
{{- end }}
{{- if env "GITLAB_RUNNER_VALUES_FILE" }}
    - {{ env "GITLAB_RUNNER_VALUES_FILE" | quote }}
{{- else }}
    - {{ env "CI_PROJECT_DIR" }}/.gitlab/managed-apps/gitlab-runner/values.yaml
{{- end }}

but that doesnt seem to be working at all. Any ideas how I could get this to work?

mumoshu avatar
mumoshu

hey! it should just work, as you can indeed use go template expressions like {{ env }} in your helmfile.yaml template.

mumoshu avatar
mumoshu

how did you confirm it doesn’t work? have you tried running helmfile build to see what the rendered helfile.yaml template looks like?

mumoshu avatar
mumoshu

have you tried running it with --debug like helmfile --debug template to see what helmfile sees?

Brij S avatar
Brij S

well the helmfile is in a container that I dont have access to

Brij S avatar
Brij S

and in this case, I have added the GITLAB_RUNNER_REGISTRATION_TOKEN as a CICD variable in gitlab which means its available to the environment. but the token doesnt get picked up and the pods fail to register — this is how I know it doesnt work

mumoshu avatar
mumoshu

ah okay… then the only thing i could suggest would be

mumoshu avatar
mumoshu

try running something like echo $GITLAB_RUNNER_REGISTRATION_TOKEN | shasunm on CI and compare the output with that of the same command run on your local machine

mumoshu avatar
mumoshu

to see if the envvar is really set or not

mumoshu avatar
mumoshu

helmfile should just read the env and that’s all i can say from helmfile’s perspective

2021-05-19

Jake avatar

Just a bit curious as to what precedence applies when specifying muliple values against a release e.g.

releases:
 - name: a
   values:
     - a.yaml
     - b.yaml

Do values in a.yaml override b.yaml (or vice versa)?

Jake avatar

I’m cheekily using mergeOverwrite at the minute, just for a current “workaround”

Rene Hernandez avatar
Rene Hernandez

b.yaml should override a.yaml

Jake avatar

ah cool, cheers.

2021-05-18

muhaha avatar
muhaha

Guys? Is possible to render helmfile including namespaces ? helmfile template does not honor namespaces specified in helmfile.yaml

Lea Zusman avatar
Lea Zusman

Hi :slightly_smiling_face: I’m trying to add multiple hooks that perform few kubectl commands. For example, helmfile.yaml:

.....
releases: 
.....
 - name: XXX
    chart: ./charts/...
    version: 0.0.1
    labels:
      testable: true
    values:
    - ./templates/XXX.values.yaml.gotmpl
    hooks:
      - events: ["prepare"]
          showlogs: true
          command: "kubectl"
          args: ["annotate", "configMap", "XXX",  "-n", "YYY", "meta.helm.sh/release-name=XXX"]
      - events: ["prepare"]
          showlogs: true
          command: "kubectl"
          args: ["annotate", "configMap", "XXX",  "-n", "YYY", "meta.helm.sh/release-namespace=YYY"]
      - events: ["prepare"]
          showlogs: true
          command: "kubectl"
          args: ["label", "configMap", "XXX",  "-n", "YYY", "app.kubernetes.io/managed-by=Helm"]

Now, it looks like there is no way to combine those hooks together (it throws an exception) is there a way to add few hooks \ few commands in one hook - or is there other way to implement this logic ? (using a bash command and run them all together ? )

mumoshu avatar
mumoshu

haven’t tried that myself. what kind of “exception”s d o you actually see?

Andrew Nazarov avatar
Andrew Nazarov

We are successfully using subsequent postsync hooks:

    hooks:
      - events: ["postsync"]
        showlogs: true
        command: "sleep"
        args: ["30s"]
      - events: ["postsync"]
        command: "kubectl"
        args: ["annotate", "--overwrite", "ns", ...]
      - events: ["postsync"]
        command: "kubectl"
        args: ["label", "--overwrite", "ns", ...]
1
1
cool-doge1
Lea Zusman avatar
Lea Zusman

I got an exception about invalid index. apparently I added unnecessary spaces

Lea Zusman avatar
Lea Zusman

@Andrew Nazarov thank you for the example. I fixed it and now it works

1

2021-05-17

anilkumar861 avatar
anilkumar861

Can helmfile hooks can be placed in seperate file and include that file under releases section? I need to manage hooks seperately for each release. Can it be possible?

mumoshu avatar
mumoshu

Maybe no. But would you mind giving me an ideal and imaginary configuration example that you have in your mind? I was thinking that you can just use {{ readFile }} to include some file under releases

Tencho Tenev avatar
Tencho Tenev

Hello, I’ve been trying out helmfile today and want to propose it to my team. However, I stumbled upon an issue with the diff plugin that helmfile uses for helmfile diff . I’d like to find a workaround or contribute a fix for this before I demo the tool to my team. The revision is added as a label in many of our charts, I don’t know if this is common or bad, but it causes helmfile apply to sync every time. Does anyone have suggestions?

`.Release.Revision` not working correctly in diff · Issue #253 · databus23/helm-diff attachment image
Hi, we have a pod label as follows that includes the helm revision number as a value: tags.datadoghq.com/version: {{ .Release.Revision default 0 quote }} This seems to work fine with helm, but …
mumoshu avatar
mumoshu

@ Hey! Thanks for bringing it up. This sounds like a good chance to contribute.

`.Release.Revision` not working correctly in diff · Issue #253 · databus23/helm-diff attachment image
Hi, we have a pod label as follows that includes the helm revision number as a value: tags.datadoghq.com/version: {{ .Release.Revision default 0 quote }} This seems to work fine with helm, but …
mumoshu avatar
mumoshu

https://github.com/databus23/helm-diff/issues/253#issuecomment-842733991

I think the first step would be to try to build your own helm-diff after changing this line, so that helm-diff use helm upgrade --dry-run instead of template

https://github.com/databus23/helm-diff/blob/master/cmd/helm3.go#L141

`.Release.Revision` not working correctly in diff · Issue #253 · databus23/helm-diff attachment image
Hi, we have a pod label as follows that includes the helm revision number as a value: tags.datadoghq.com/version: {{ .Release.Revision default 0 quote }} This seems to work fine with helm, but …
databus23/helm-diff attachment image

A helm plugin that shows a diff explaining what a helm upgrade would change - databus23/helm-diff

Tencho Tenev avatar
Tencho Tenev

Hi, so I did the first step and have helm diff using upgrade dry-run instead of template. This produces diffs showing the next revision which makes more sense.

However, the perfect behaviour for me is to produce no diff if the only reason for change is the revision changing. This is not so trivial, in fact, I think it requires a change in helm itself.

After looking at a few projects, I feel like using release revision in all charts is not a best practice. I will suggest changes to the charts my team owns to remove the use of revisions. I may still raise a PR on helm diff with the change for upgrade instead of template because at least the result is more intuitive.

mumoshu avatar
mumoshu

@ ah, gotcha! just fixing the chart makes sense in that case. good.

mumoshu avatar
mumoshu

In case you can’t fix the chart for whatever reason, I think it’s still a good idea to raise a feature request to helm-diff

mumoshu avatar
mumoshu

to add something like ignoreDifferences in ArgoCD https://argoproj.github.io/argo-cd/user-guide/diffing/

1
mumoshu avatar
mumoshu

It seems to be implemneted by removing the keys at the jsonpaths before computing diff https://github.com/argoproj/argo-cd/blob/b37eee1054e42c873699460dd5e2447c2f9fe5a6/util/argo/normalizers/diff_normalizer.go#L28-L63

argoproj/argo-cd attachment image

Declarative continuous deployment for Kubernetes. Contribute to argoproj/argo-cd development by creating an account on GitHub.

mumoshu avatar
mumoshu

helm-diff does textual diffing here https://github.com/databus23/helm-diff/blob/d42b177462211ce1f15db07f6309359818a005f9/diff/diff.go#L157

so adding some logic for ignoreDiefferences here would work

databus23/helm-diff attachment image

A helm plugin that shows a diff explaining what a helm upgrade would change - databus23/helm-diff

mumoshu avatar
mumoshu

again, if fixing the chart works, that’s okay

mumoshu avatar
mumoshu

just my two cents

2021-05-16

muhaha avatar
muhaha

Guys? Is possible to set kubeconfig context in Helmfile TF helmfile_release_set resource ? Seem its possible in helmfile_release, but not in *_set.

https://github.com/mumoshu/terraform-provider-helmfile/blob/master/pkg/helmfile/resource_release.go#L294

@mumoshu is this intentional ? thanks

mumoshu avatar
mumoshu

Probably it’s not intentional

2021-05-13

2021-05-12

Arthur Jaron avatar
Arthur Jaron

Hi guys, I’ve just inherited a rather large codebase using helmfile, which is new to me. In there is the following construct: helmfile.yaml

releases:
  - name: namespace-dev
    chart: ../../../../charts/namespace
    version: 0.0.1
    createNamespace: false
    values:
      - values.yaml
      - test: {{ .Values.testing }}
      - namespace: dev

values.yaml

testing: yes

I’m simply trying to read “testing” from the yaml file, but I get this error message:

in ./helmfile.yaml: in .helmfiles[0]: in namespaces/helmfile.yaml: error during helmfile.yaml.part.0 parsing: template: stringTemplate:11:24: executing "stringTemplate" at <.Values.testing>: map has no entry for key "testing

What am I doing wrong? This should be trivial from what I’ve read so far…

Rene Hernandez avatar
Rene Hernandez

The testing field won’t be available in that scope if it is defined in a values file. You would need to move it to another yaml file that you would import as an environment (default one or specific one)

Rene Hernandez avatar
Rene Hernandez

e.g: environment.yaml

testing: yes

helmfile.yaml

environments:
  default:
    values:
      - ./environment.yaml

releases:
  - name: namespace-dev
    chart: ../../../../charts/namespace
    version: 0.0.1
    createNamespace: false
    values:
      - values.yaml
      - test: {{ .Values.testing }}
      - namespace: dev
brokencode avatar
brokencode

Hi, I’m wondering, can helmfile is meant to be used like helm-operator/controller? I.e. can you deploy it to kubernetes and have it sync changes from github/gitlab and apply them to the cluster? If so, are there any example deployments available?

Ievgenii Shepeliuk avatar
Ievgenii Shepeliuk

I’ve read somewhere some time ago that ArgoCD has some support or can be hacked to process Helmfiles from git repos :)

1
voron avatar
voron

helmfile apply inside CI/CD pipelines does this for about 1y in one of dev envs

Andrew Nazarov avatar
Andrew Nazarov

You also could find this thread useful: https://github.com/roboll/helmfile/issues/153

Flux, Argo and helmfile-operator are mentioned there

[In the future] Helmfiles as a chart / Helmfile as a K8S CR · Issue #153 · roboll/helmfile attachment image

So that we can leverage everything helm provides to power helmfile :) User-defined Lua functions included in charts (Perhaps in Helm v3) Helm chart repositories to distribute versioned helmfiles fo…

1

2021-05-11

Balazs Varga avatar
Balazs Varga

hello all, let’s say I have dev and prod env-s If I would like to deploy dev I just need to run helmfile apply, for prod I need to mention with -e prod apply… is there a way to check it from a variable that I can get from cluster ?

Michael Seiwald avatar
Michael Seiwald

Hi all, are remote helmfiles via HTTPS supposed to work? E.g. this helmfile.yaml leads to an error:

helmfiles:
  - path: <https://raw.githubusercontent.com/cloudposse/helmfiles/master/releases/external-dns/helmfile.yaml>
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yes, we use it everywhere. Best to show the error message.

Michael Seiwald avatar
Michael Seiwald

This is the error message:

no matches for path: <https://raw.githubusercontent.com/cloudposse/helmfiles/master/releases/external-dns/helmfile.yaml>
There are no repositories defined in your helmfile.yaml.
This means helmfile cannot update your dependencies or create a lock file.
See <https://github.com/roboll/helmfile/issues/878> for more information.
err: no releases found that matches specified selector() and environment(default), in any helmfile
Michael Seiwald avatar
Michael Seiwald

Just to add I don’t mean “Git over HTTPS” but direct “plain” HTTPS.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

aha, good distinction. I see we’re using this:

helmfiles:
- path: "git::<https://github.com/cloudposse/[email protected]/efs-provisioner.yaml?ref=0.112.0>"

Which is “Git over HTTPS”, and I don’t know if we’ve tested the “plain” HTTPS.

2021-05-07

2021-05-05

CH avatar

Is it at all possible to set values for a release from a file located inside a chart? Something like:

releases:
- name: my-app
  chart: my-helm-repo/my-app
  version: 1.0.2
  values:
  - {{ .Release.ChartPath }}/values/{{ .Environment.Name }}.yaml
CH avatar

The goal here is to version our environment values same way our app charts are versioned, and promote them in lockstep.

CH avatar

So that the same commit that adds a new value to a chart can also populate it in the environment values files (and likewise for removal or other changes).

CH avatar

Would be super simple if we could just include the environment values files in the chart itself, instead of trying to download them from Git or something

Vladimir Avdoshka avatar
Vladimir Avdoshka

interesting idea, I wonder to hear if there is some solution

2021-05-04

yuri avatar

Hey guys, could not find an answer to what i was looking for… is there a way to pass ENV VARS to helm-secrets plugin? the plugin supports some flags that im interested in like: HELM_SECRETS_DRIVER_ARGS and HELM_SECRETS_DEC_SUFFIX, i tried to export them before helmfile runs and also with prepare hooks, but it doesn’t seems to affect the plugin in any way, if i export them and then use directly helm secrets …. it works

yuri avatar
roboll/helmfile attachment image

Deploy Kubernetes Helm Charts. Contribute to roboll/helmfile development by creating an account on GitHub.

2021-05-03

Balazs Varga avatar
Balazs Varga

hello all, I have a define :

{{- define "x" -}}
{{ if .value }}
  {{- printf .value -}}
{{ else }}
  {{- exec "kubectl" (list "get" "configmap" "data" "-n" "namespace" "--output=jsonpath={.data.userdata }} }" ) -}}
{{ end }}
{{ end }}

and a template to call it

userdata: {{ template "x" . }}

is there a way to pass params to the define line from template like

(dict "value" .Values.data.user "key" "region")

If I could pass params like namespace, resource type, endpoint in jsonpath… then I could use only few defines and not lots of… thanks in advance

Antoine Taillefer avatar
Antoine Taillefer

I believe so, try:

{{ template "x" (merge (dict "value" .Values.data.user "key" "region") .) }}
1
Balazs Varga avatar
Balazs Varga

I think I found it. my problem was I use kubectl to get data if env not exists and I just passed the value to the jsonpath and it did not work, so I hjad to pass the full –output… part and it works.

Joaquin Menchaca avatar
Joaquin Menchaca

How do you configure helm --name_template in a helmfile.yaml? I didn’t see an option for this, and I was digging through the code, I couldn’t spot if it was or was not supported.

    keyboard_arrow_up