#helmfile (2020-06)

It looks like --values and --set arguments are passed “as is” to helm, and aren’t parsed by helmfile. You may use things like

{{ readFile "myvalues.yaml" }}

to include values into helmfile context

but usually it’s enough to use values/valuesTemplate helmfile directives to use values from file

Apparently --state-values-file works?

I thought that was more like including a helmfile but apparently not.

--state-values-file should affect helmfile context --state-values-set affects it in my tests

similar to adding values on top of helmfile.yaml

values:
  - k0: v0
    k1: v1
---
...
the rest of helmfile here

What I am going for is to build a tool I feed a single yaml file into to configure an environment. So I basically want to pass the “environment” values in almost entirely at runtime. My understanding was that --values would do that, and the other flag seems to.

Ahhh for your example, if the file passed had just:

k0: v0
k1: v1

?

If that’s right, then this IS the flag I want. I thought it did something else, as I use the --state-values-set to sort of create an environment on the fly. (and it seems to work), so I thought it was merging at the root of the helmfile.

Ahhh for your example, if the file passed had just: yes.

Thanks for clarifying.

Also you may look to dynamic configuration like https://github.com/roboll/helmfile/blob/master/docs/writing-helmfile.md#layering-state-template-files

That sounds up my alley… I’ll try to re-read it… a bit confused on the difference between “bases” and “helmfiles”

use bases to build helmfile.yaml use helmfiles to join multiple [autonomous] helmfile’s.yaml imho

Thanks. I may have more annoying questions as I explore this. What I am hoping to achieve is via helmfile, and helm git, being able to place charts, a helmfile fragment, and a values.yaml.gotmpl w/ the code in the git-repo, and then compose things via a bunch of git references in the “top” helmfile.

@mumoshu would you be able to comment on whether this feature fits the overall goal of the project? I’m sorry I pinged you directly but I haven’t gotten any feedback on the PR for a week and the feature would be great for our workflow (having a single repo for all global values across different data centres). Thanks and sorry for the intrusive direct pinging.

Very cool @Zachary Loeber!

@Steve Boardwell could be an interesting strategy

Great job!

releases[].values are for helm. it’s a different concept than helmfile values ({{.Values}}) in your helmfile.yaml

helmfile values can be defined using the top-evel values:

values:
- service:
    port: <ref+awsssm://foo/stage/CONTAINER_PORT?region=us-west-1>


---

releases:
- name: foo
  values: 
  - values-apps/foo.values.yaml.gotmpl

That’s helpful, but also a little confusing.

yeah true! when we added this, everyone seemed to agree that this should better be called values so that it become familiar to any helm users

which turned out not true :cry: maybe we can try renaming it to variables or anything else hoping it makes things a little less confusing

I’ve got several releases in one helmfile that all use the same chart. Additionally, I’m supporting two environments, so with the example you posted I’m not sure how I would set service.port for a different release since I can’t use it in the releases section.

lemme get a more accurate example of what i’m doing to make it more clear

Or you could just go and give me a great example

you can even do this. but i’m not a fan of abstracting things so much / building a yet another configuration language on top of yaml+gotmpl

values:
- foo:
     service:
       port: 1234
  bar:
     service:
       port: 2345

---

releases:
{{ range $name, $values := .Values }}
- name: $name
   values:
  - {{ $values | nindent 4 }}
}}

accept a bit of repetition in releases like my first example and it can be reviewed easily

Yeah, the first example definitely seems more readable.

anyway, if you could share me a concrete example, i can probably review/suggest something on/confirm it

So this is what I’ve currently got:

templates:
  default: &default
    chart: charts/commonApp
    labels:
      app: "{{`{{ .Release.Name }}`}}"
    values:
      - global.yaml.gotmpl
      - values-apps/{{ .Values.repo }}.values.yaml.gotmpl
      - envs/{{ .Environment.Name }}.yaml.gotmpl
    valuesTemplate:
      - appName: "{{`{{ .Release.Name }}`}}"
      - environment: {{ .Environment.Name }}
      - repo: {{ .Values.repo }}
      - service:
          port: secretref+awsssm://{{ .Values.repo }}/{{ .Environment.Name }}/CONTAINER_PORT?region=us-west-1

releases:
  - name: foo-{{ .Values.branchSlug }}
    version: 1.0
    <<: *default

  - name: bar-{{ .Values.branchSlug }}
    version: 1.0
    <<: *default

I’m not entirely clear on values vs valuesTemplate and this is before you suggested using the top level values object.

Based on what you said, I think what I need to do is move some stuff to environments, something like this:

environments:
  stage:
    foo:
      service:
        port: 1234
  prod:
    foo:
      service:
        port: 5678

valuesTemplate is usually used for access to Release.Name and Release.Namespace(I thought it was possible

yeah, your use of environments seems valid to me

in addition to that, you may find the top-level values useful for defining default values for environment values

ahh, that’s helpful

do you need different service.port values for releases foo and bar?

if not, your use of templates seems good

yeah, foo and bar are be different apps, so they’ll have different service.port values. But I use the same name in SSM for the thing following the pattern /appName/envName/varName

That’s why I was hoping I could use release vars - then I don’t have to set the same things for every app separately in the environments section.

then i think it should probably be

secretref+awsssm://{{` {{.Release.Name}} `}}/{{ .Environment.Name }}/CONTAINER_PORT?region=us-west-1

the helmfile template renders

secretref+awsssm://{{` {{.Release.Name}} `}}/{{ .Environment.Name }}/CONTAINER_PORT?region=us-west-1

into

secretref+awsssm://{{.Release.Name}}/prod/CONTAINER_PORT?region=us-west-1

and the release template renders

secretref+awsssm://{{.Release.Name}}/prod/CONTAINER_PORT?region=us-west-1

into e.g.

<secretref+awsssm://foo-BRANCH_SLUG/prod/CONTAINER_PORT?region=us-west-1>

That actually shows why I’m using {{ .Values.repo }} instead of the release name – I don’t want to include the branch slug in the ssm path.

I hear you. But it can’t evaluate {{.Values.repo}} to a different value per release

(Feel free to tell me I’m crazy and need to change my approach :D)

So maybe you’d better build on Release.Name

i think it would work

I’ve lost track, where would you suggest I put the secretref line?

Under valuesTemplate, as it was in your example

Ok, testing that

(fingers crossed)

you can debug it using helmfile build, btw.

ahh, nice. I’ve been using helmfile template - haven’t tried build yet

yeah both helps. it’s just that build is for unit testing your helmfile.yaml, and template can be used for integration testing with helm and helmfile

It did not like that..

failed executing templates in release "apps.yaml"."foo-bar": failed executing template expressions in release "foo-bar".values[3] = "[115 101 114 118 105 99 101 58 10 32 32 112 111 114 116 58 32 115 101 99 114 101 116 114 101 102 43 97 119 115 115 115 109 58 47 47 32 123 123 32 46 82 101 108 101 97 115 101 46 78 97 109 101 32 124 32 115 112 108 105 116 32 34 45 34 32 124 32 105 110 100 101 120 32 48 32 125 125 32 47 115 116 97 103 101 47 67 79 78 84 65 73 78 69 82 95 80 79 82 84 63 114 101 103 105 111 110 61 117 115 45 119 101 115 116 45 49 10]": template: stringTemplate:2:59: executing "stringTemplate" at <index 0>: error calling index: can't index item of type int

seems like i confused argument ordering for index

0 should be the second argument to index

hmm, same thing

oh, same? shouldn’t it emit another error at least

oof, can’t read template: stringTemplate:2: unexpected "(" in operand

ok, that was my bad when I have the space like I should I get template: stringTemplate:2:31: executing "stringTemplate" at <index (.Release.Name | split "-") 0>: error calling index: value has type int; should be string

ahh okay, we should use splitList rather than split

http://masterminds.github.io/sprig/string_slice.html

Yep, that got rid of the error

But.. I’m back where I started

failed to render values files "values-apps/foo.values.yaml.gotmpl": failed to render [values-apps/foo.values.yaml.gotmpl], because of template: stringTemplate:12:21: executing "stringTemplate" at <.Values.service.port>: map has no entry for key "service"

Do you still have reference to Values.service.port in values-apps/{{ .Values.repo }}.values.yaml.gotmpl ?

Yeah, not sure what to set it to?

you shouldn’t need that, as you already set the same service.port in valuesTemplate now

I’m trying to use the value in the .gotmpl file, not set it. I’m guessing that isn’t possible?

ah gotcha! yes, that’s not possible

you should do this in valuesTemplate instead

deployment:
  readinessProbe:
    exec:
      command:
        - "/bin/bash"
        - "/app/health.sh"
        - "localhost"
        - secretref+awsssm://{{` {{ .Release.Name | split "-" | index 0 }}
 `}}/{{ .Environment.Name }}/CONTAINER_PORT?region=us-west-1
        - ".1s"

``

Yeah, that makes sense. I was attempting to avoid that as the the readinessProbe stuff won’t always be the same and that means more repetition, but it seems I should either make them all the same or be ok with more repetition

gotcha

you can use {{ define "somehelper" }} and {{template "somehelper" "someargs" }} template functions to reuse a templated yaml sturucture

or even use a helper defined in a separate fiile using

{{ tpl (readFile "somehelper.tpl") "someargs" }}

hope that helps to alleviate the repetition issue

yeah, definitely. I hadn’t thought of using templates for this, but it makes perfect sense.

Thanks for all your help with this @mumoshu, I really appreciate it.

helmfile is a really great tool, and you’ve helped me wrap my head around a couple things that were missing for me.

glad to help! and thanks for the feedback. that really motivates me

environments:
  preprod:

# Chart repositories used from within this state file
repositories:
- name: stable
  url: <https://kubernetes-charts.storage.googleapis.com>
- name: incubator
  url: <https://kubernetes-charts-incubator.storage.googleapis.com>
- name: bitnami
  url: <https://charts.bitnami.com/bitnami>

helmfiles:
{{ if eq .Environment.Name "preprod" }}
  - path: helmfile-preprod.yaml
{{ end }}

This is my main helmfile.yaml. if I do helmfile repos, I get an error about no selector match. if i do helmfile -e preprod repos , nothing happens at all.

If I stick the repositories down into an actual helmfile containing releases, it works just fine

I’ve always had to include the repositories block in my helmfiles. same with the helmdefaults block. I ended up floating those up to their own files and have something like the following at the top of my helmfiles:

--- bases: - ../config/environments.yaml - ../config/helmdefaults.yaml - ../config/repositories.yaml ---

It works but also means that for any helmfile that gets deployed ALL repos will get synced. Not ideal.

I’ve yet to round back on how to optimize this particular bit

I have a helm chart that we use to manage namespaces, which I hate, but in this case it actually came in handy because its a pre-req that it runs first

so I shoved the repos in that helmfile for that chart, and all the other runs subsequently get it

Funny, that’s why I kept namespace creation as a separate chart as well, it is a handy dependency anchor

Afaik this is the expected behaviour. Nothing is inherited from the main file.

As of managing namespaces with Helm chart - chances to destroy everything by accident scare me a bit). But we are managing namespaces with a chart too.

Did you tried some magic like

{{ if has (env "UPDATE_REPO" | default "no") (list "yes" "True" "true") }}
repositories:
 - name: app1
    url: <git+ssh://[email protected]/examplecom/helm-charts@app1?ref={{> .Values | getOrNil "app1._version" | default "master" }}&sparse=0
 - name: app2
    url: <git+ssh://[email protected]/examplecom/helm-charts@app2?ref=>{{ .Values | getOrNil "app2._version" | default "master" }}&sparse=0
{{ end }}

?

the idea is to use [env] var as a condition to add all repositories to the helmfile.

let me try that magic

I would use get instead of getOrNil wherever possible

I am still trying but still not sure how to update a single app, a list, or all.

it looks like helm doesn’t support specifying repos you wanna update. All or nothing.

I wonder how Cloud Posse might be using this. I believe I read in one of the comments that they might be using a combination of Helm-Helmfile-Helm-git, Is that right @Erik Osterman (Cloud Posse)?

I think he’s at Office Hours on call

@Erik Osterman (Cloud Posse) would you have any recommendation on this?

yes, we use the helm-git plugin all the time

it’s the secret weapon. with helm-git, you can say goodbye to needing helm repos and instead get something more akin to what we have with NPM or go, where you can import basically any chart

Here is an example

Note the bug though:

Cannot use release tags, see https://github.com/aslafy-z/helm-git/issues/9

Thanks. Do you keep multiple apps in the same repo like I mentioned? Helm-git is great. But cloning all repos for a 1 app diff or apply makes the process much longer. I wonder if you have a better way to control it

Looks like you have a helmfile per application.

that’s part of the yaml spec: https://yaml.org/type/merge.html

And check out this

https://github.com/roboll/helmfile/blob/master/docs/writing-helmfile.md#release-template--conventional-directory-structure

Take a look at helmfile release hooks, for example https://github.com/roboll/helmfile#helmfile--kustomize

Interesting, I’ll check that out. I was looking at the hooks, and having it just run a straight “kubectl apply”, but that doesn’t cover situations like destroying a deployment.

I was hoping for a “copy this yaml into the chart/template directory so helm will process it” type command

use hooks to get upstream chart, patch in on the fly and then release patched local chart via helmfile

also possible, yeah. Was trying to keep it simple since helmfile already grabs the upstream chart and caches it temporarily.

tell helmfile to use local chart to deploy, and helmfile will not grab upstream chart then

sure. I’ll fiddle with this. Thanks!

you should be able to use env instead of requiredEnv piped to default , eg:

scheme: {{ env "SCHEME" | default "https" }}

version: {{.Environment.Values | getOrNil "salt_master.version" | default (env "SALT_MASTER_VERSION") | default ""}}

I think you misunderstand selectors a little bit

just set labels in your subhelmfiles for the apps and whack the selectors from your main helmfile.yaml

then it should work as you like I believe

though, I honestly did not know I could set selectors in the main helmfiles block like that, nifty

Anyways, otherwise when you apply your main helmfile it is saying to deploy both app1.yaml with a selector label of name: app1, and app2.yaml with a selector label of name: app2

helmfiles: - ../helmfiles/helmfile.traefik.yaml - ../helmfiles/helmfile.certmanager.yaml - ../helmfiles/helmfile.rbacmanager.yaml - ../helmfiles/helmfile.consul.yaml - path: ../helmfiles/helmfile.vault.yaml selectors: - component=vault

I just did a test with that helmfile.yaml and applied it all

ohhhhhhh k, ty so much

awesome. will try that too

it deployed everything, including vault, but did not install the additional (separate) ingress deployments within the vault.yaml chart

so, I could see using this logic too, just don’t apply it via cli and drop the filters in your main helmfile chart,

your question learned me something new thanks

sorry can i just ask what you put for the label in your sub helmfile?

@Zachary Loeber

no sorries, glad to share:

labels: chart: vault component: vault namespace: {{ .Values | getOrNil "vault.namespace" | default "vault" }}

for better or worse that is what I’ve personally been using as my standard go to labels for all releases

(the namespace one is likely overkill and is sourced from the values.yaml file that is referenced in my environment.yaml

ok ty so much, this is really helpful

https://github.com/zloeber/CICDHelper

that’s all of them, I’m not using tpl templates or anything fancy (yet) so it may not be as dry as it should be

weird, so did the helmfile --environment=dev --selector component=app1 diff but same result it runs through both even though under the release i have the labels set

and you removed the selector in your main helmfile?

ahhhhh nope, missed that ok

so removed path and selectors from main helmfile, it diffs the one app now, tho for some reason still feels the need to add the repo of the other app

That’s something I cannot help with (unfortunately), I’m still trying to figure out the best way to do repo filtering

hahahaha ok, ya, for now i can just run helmfile -f path/to/subhelmfile.yaml i guess

though if you know you have the latest you can use the –skip-deps flag I think

ok, ill look into that ty

lemme know if you find anything

ya will do, thx for all the help!

I’d really like to know the answer to this one too actually, I can find the cached repository data easily enough but cannot seem to find the downloaded charts

Because how cool would it be to have a plugin that stores the downloaded/versioned charts and rewrites the chart repos to the local cached version for offline deployments?

WDYT helmfile downloads chart[s] w/o helm ? Did you tried to download chart directly via helm pull inside hook ?

right, this is more of a helm question than helmfile one

that helm pull in a hook option may be a good way to get the charts downloaded into one location though… I’ll have to look into that option…

I’m curious about caching/go-offline mode in this case

I’d look to pull down all the charts into a dockerfile image and would rewrite the chart urls as part of that process if possible

Idea would be to have an immutable deployment artifact for the cluster state that will always apply the same (regardless if upstream chart sources update beneath our feet)

That’s an interesting idea!

My use case is a little simpler…I just want to dynamically add another template to an upstream chart. I could copy the file with a hook, but I need to know where to copy the file to…

U will, in case of helm pull

If I’m running helmfile, I don’t manually do a “helm pull”, so how would I get the path?

Do you wanna patch the chart on the fly ? Specify local filesystem chart in helmfile and use (prepare ?) hook to pull upstream chart via helm and then modify it. Path is current directory, as you specify current directory via chart : ./my-local-chart

Hmm, ok. I’ll see if I can swing that

use shell script as a hook to debug and get a PoC

I don’t see any similar options in the code https://github.com/roboll/helmfile/blob/master/pkg/state/state.go#L104

Also couldn’t find it.

The thing I’m running into (posted to Kubernetes #helm slack channel):

I’m running into an issue with a chart (linkerd) referencing it’s requirements as for example file://../add-ons/grafana.

What I’m trying to do is: helm pull to have the chart locally in codebase (easy developing, meaningful diffs when upgrading chart). I use helmfile which out of the box executes a helm dependency build for every local chart. This breaks for the linkerd chart because there’s no ‘add-ons’ locally. (Note that this step can be disabled via a flag, which allows me to move forward).

If I compare this to the prometheus-operator chart, that references it’s requiremens using external repos, which works fine in the above setup.

This makes me wonder: Is publishing charts having requirements to relative paths considered a good practice?

Is publishing charts having requirements to relative paths considered a good practice? I don’t think so

@mumoshu ping

I think no by default, but we wanted to implement this. I can check this out soon.

You mean the --suppress-secrets functionality?

oh,

Something more like https://github.com/k14s/kapp/issues/67

Its working well with kapp binary, like masking secrets even in CRDs

We are collecting pipeline logs like plan/apply/destroy phases and suppresing secrets in outputs is something that we like

--suppress-secrets is enabled by default since v0.2.0 thanks to Andrey’s https://github.com/mumoshu/terraform-provider-helmfile/pull/13

oh, i need helm plugin install <https://github.com/aslafy-z/helm-git>

With helm chart maintenance falling back into the realms of maintainers it would be nice if git sources were a first-class citizen in helm

@Tim Birkett agreed!

should I in fact have a separate helmfile for sub-chart-A and sub-chart-B and a top level helmfile and then have those version numbers templated?

where do you specify chart dependencies ? release, templates, sub-helmfile, … ?

I’m very new to helmfile (hours) and we currently have an umbrella chart that has 10+ dependencies comprising our entire product. The pain point has been setting the versions for the subcharts as the Chart.yaml (or requirements.yaml before) cannot be templated.

Probably you’ll find the following useful:

https://github.com/roboll/helmfile/issues/1291

General info about hooks in helmfile: https://github.com/roboll/helmfile#hooks

thanks will check them

I see the cloudposse guys did

{{ exec "curl" (list "-fsSL" (print "<https://raw.githubusercontent.com/cloudposse/grafana-dashboards/>" (env "GRAFANA_DASHBOARDS_VERSION" | default "1.0") "/nginx-ingress/nginx.json")) | indent 12 }}

somewhere. Not sure if that can extend to interpolating helmfile vars though ?

I believe you don’t need to interpolate there; you should be able to:

{{ exec "./mycmd" (list .Values.region "arg2" "--flag1") | indent 2 }}

No, they are not inherited. You should have an environments: block in subhelmfiles as well.

ok, thanks for confirming that. I’m trying to convert an umbrella chart with 10+ into a helmfile setup, but it looks a bit challenging to keep things DRY

I raised a discussion about this some time ago:

https://github.com/roboll/helmfile/issues/1010

right, I’m running into multiple challenges with my conversion from the umbrella chart and I’m starting to doubt if it can be done elegantly. My current setup relies on having a common base values file and another per environment (=namespace) so that updating becomes like:

helm upgrade my-software --namespace env1 --values base.yaml --values env1.yaml

Did you tried any hacks in go templates ?

{{- tpl (readFile "values.yaml" | <REMOVE-ALL-UNNEEDED-STAFF>) . | nindent 0 }}

Imho it may be simpler and clealer to move values to separate files. All these tricks are hard to maintain.

I considered it, but found it too cumbersome. I currently have one common value file for the entire project (umbrella chart) and one specific value file per environment. The specific one overriding common values as needed. This pair controls all of the subcharts and I would find it difficult to maintain if I would have a separate value file for each subchart

you built helmfile using umbrella chart and now don’t wanna migrate to real helmfile

we have had the umbrella chart for quite some time. I got interested in helmfile because I thought it would help me with my main problem with the current setup: As the Chart.yaml (or requirements.yaml in helm 2.x) cannot be templated, I’ve been looking for a way to control the version numbers of my subcharts. For development we use ranges and that is great, but when it’s time to make a prod release I’d like to “lock” the subcharts to specific versions - which would be easy if the Chart.yaml would accept templating. I’m still not sure if this can be done with helmfile, but I definitely haven’t yet got my head around it.

but how can I reference subchart1 values in helmfile without typing them out individually? But how do you do this with helm ? I mean - deploy of single subchart.

I suppose you don’t deploy subcharts separately

exactly - we don’t deploy them individually. They make up one product and are not really usable separately.

Some subcharts are mandatory in the sense that the product cannot work if they are missing, while others are optional “features” and for those I have a condition setup that can be flipped on or off via the values file

but now you wanna split umbrella chart to helmfile charts with dependencies, but don’t wanna split values files

well, like I said, my experience with helmfile is so far just some hours and I’m probably doing a lot of things wrong. I figured out that I would have to split the umbrella chart in order to control the subchart versions, but I thought I could still feed the values from one file.

https://github.com/roboll/helmfile#importing-values-from-any-source did you tried yq to get your subchart values like

{{ readFile "values.yaml"| exec "yq" ( list "r" "-" "subchart2" ) }}

?

“yq” was new to me, thanks for pointing that out! However, I didn’t quite get it to work yet, but perhaps is just about my broken yaml syntax now. With the code

values: |
    {{ readFile "values.yaml" | exec "yq" ( list "r" "-" "subvaluesA" ) }}

I managed to produce values like this:

 0: releases:
 1: - name: "sub-chart-A"
 2:   namespace: "dummy"
 3:   chart: "mycharts/subA"
 4:   version: 1.2
 5:   values: |
 6:     fullnameOverride: "mysubservice1"
 7: image:
 8:   repository: myrepo/myservice
 9:   pullPolicy: Always
10: service:
11:   type: NodePort
12:   annotations:
13:     subannotation: 'value'

But at least it does indeed pick up only the values from under the “subchart2” name

in fact yq might be all I need at the moment. I just tried manipulating Chart.yaml with it and it works like a charm:

yq w -i Chart.yaml 'dependencies.(name==subchartA).version' 2.0.120

Thanks @voron for telling me about it!

kind of unexpected downgrade

I see it more like an unexpected plot twist on my adventure to get into helmfile. Which will continue!

did you play around helmfile deps ?

fyi, no need to use exec yq. helmfile comes with a way to do that natively:

{{ readFile "values.yaml" | fromYaml | get "someKey" "defaultValueIfNotExist" }}