#helmfile (2020-04)

E.g. I might have a helmfile.yaml that contains:

releases:
    - name: app-init
      namespace: app
      chart: ./helm/chart/app-init
      installed: true
      values:
        - helm/vars/app.yaml.gotmpl

    - name: app
      namespace: app
      chart: ./helm/chart/app
      installed: true
      values:
        - helm/vars/app.yaml.gotmpl

And helm/vars/app.yaml.gotmpl may contain something like the following (that doesn’t work):

configFiles:
  environment_file: |
    {{- if eq .Release.Name "app-init" }
    DB_USER={{ requiredEnv "ADMIN_DB_USER" }}
    DB_PASS={{ requiredEnv "ADMIN_DB_PASS" }}
    {{- else }}
    DB_USER={{ requiredEnv "DB_USER" }}
    DB_PASS={{ requiredEnv "DB_PASS" }}
    {{- end }}

The intention above being to populate some values using different environment variables depending on which release is going out. The values file is unable to access .Release.Name and so the above doesn’t work.

I’ve also tried changing the values: file in the release example above to something like:

      values:
        - HelmReleaseName: app-init
        - helm/vars/app.yaml.gotmpl

and:

      values:
        - HelmReleaseName: app
        - helm/vars/app.yaml.gotmpl

And changing the test in the values file to be:

{{- if eq .Values.HelmReleaseName "app-init" }}

But again this doesn’t work.

Is there a way to have a test in the values file be able to see something unique based on the release being installed?

@Jim Release.Name isn’t available within the gotmpl file. It’s only available in helmfile.yaml template. This is to avoid misuses(like “hiding” the release specific values)

And - you don’t need to complicate things with the if-else for your specific example.

I’d suggest this:

releases:
    - name: app-init
      namespace: app
      chart: ./helm/chart/app-init
      installed: true
      values:
        - helm/vars/app-init.yaml.gotmpl

    - name: app
      namespace: app
      chart: ./helm/chart/app
      installed: true
      values:
        - helm/vars/app.yaml.gotmpl

where app-init.yaml.gotmpl is:

configFiles:
  environment_file: |
    DB_USER={{ requiredEnv "ADMIN_DB_USER" }}
    DB_PASS={{ requiredEnv "ADMIN_DB_PASS" }

app.yaml.gotmpl:

configFiles:
  environment_file: |
    DB_USER={{ requiredEnv "DB_USER" }}
    DB_PASS={{ requiredEnv "DB_PASS" }}

Thanks @mumoshu but that isn’t going to work for us too well :disappointed: I’ve simplified the configuration a lot for this example. The values files (of which there are multiple being passed) contain large configuration files that will end up in Kubernetes secrets and directly mounted into the file-system within the containers. I’m trying to avoid having to duplicate those configuration files multiple times(in app-init.yaml.gotmpl and again in app.yaml.gotmpl)

So you have a large duplication within the single string values(configFiles.environment_file) for app and app-init?

Yeah (for this example, again it doesn’t really match reality). We have a number of configuration files below configFiles: and each one is for a different component (PHP, Apache, the Application itself) and some of those configuration files are quite large. We have about 5 variables that are used (sometimes once, sometimes across multiple files) that need to differ for one release vs the other, where both releases use the same configuration files. So I was trying to solve that by using some helmfile conditional logic as part of templating out the config files.

How about aggregating all the common env vars into a single yaml file common.env and reusing it?

app-init.yaml.gotmpl:

configFiles:
  environment_file: |
    DB_USER={{ requiredEnv "ADMIN_DB_USER" }}
    DB_PASS={{ requiredEnv "ADMIN_DB_PASS" }
    {{ readFile "common.env" | nindent 4 }}

app.yaml.gotmpl

configFiles:
  environment_file: |
    DB_USER={{ requiredEnv "DB_USER" }}
    DB_PASS={{ requiredEnv "DB_PASS" }}
    {{ readFile "common.env" | nindent 4 }}

the more control syntax you introduce, the more your config template gets unreadable/buggy.

so i would prefer avoiding using if whenever possible

otherwise - filing a feature request to add support for Release.Name within values.yaml.gotmpl might be needed

Ok. I’m not sure I can easily do what you’ve suggested, but I’ll give it a shot. Thanks for your help.

But the take-away is that currently there is no way me to test anything defined in the release form within the values file right?

This looks like a related ticket: https://github.com/roboll/helmfile/issues/760

yes. i thought i intentionally decided to not implement that because it seemed to become a maintainance/user support nightmare

awesome! I couldn’t recall that

If I have trouble implementing the solution you’ve got above, would you consider allowing us to shoot ourselves in the foot?

do you mean you still need .Release.Name within values.yaml.gotmpl?

if so - yeah i would definitely consider implementing it. i think that feature is requested too many times to be ignored

Yeah, if I can’t break up the configuration files like you’ve shown above, it would be nice if I could test for .Release.Name as an alternative.

undestood. would you mind adding your comment on the issue saying “i need this too”?

Yes, I’ll do that now. Thank you for your help.

thanks for your support too!

@mumoshu , btw, does double render work for values.yaml.gotmpl? Like {{ {{ .Release.Name }} }} . I always get confused with it.

unfortunately no

I attempted that.

and double rendering only render helmfile.yaml twice

it doesn’t happen (and unnecessary) for values.yaml.gotmpl

Ok, good to know. Thanks!

{{` {{ ... }} `}}

is for release templates

np!

see this for release templates

https://github.com/roboll/helmfile/blob/master/docs/writing-helmfile.md#release-template--conventional-directory-structure

We’ve never used gotmpl values files, so I don’t know any details. But we are using this double-render trick in our main helmfile.yaml

yes, that makes sense. you should avoid gotmpl values whenever possible.

Yes, thanks). I’m aware of how it works for releases. But thank you anyway)

(i got to think that the complication from release templates vs values go tmpl vs double rendering all coming from the fact that we’re abusing yaml templates too much…

regarding that i’ve recently talked with @Erik Osterman (Cloud Posse) about a dream: hcl2-based dsl for helmfile

https://sweetops.slack.com/archives/CFFQ9GFB5/p1585873245019000

On the other hand, sometimes you can do some crazy things with helmfile)). But the readability suffers a lot)

FYI. I got this technique to work. Oh but the horrible hacks I had to do in order to get it to work makes it pretty unreadable. Unfortunately the order of the content of the files in question mattered (they’re not just simple env files) and the variables were spread across them, and so the files had to be broken up into multiple parts. Having to change content of the file means finding its multiple parts, plus some bits in multiple helmfile values files. Think of a values file like:

configFiles:
  config_file_1: |
    {{- readFile "start_of_file" | nindent 4 }}
    VAR1={{ requiredEnv "X" }}
    VAR2={{ requiredEnv "Y" }}
    {{- readFile "bit_more_of_file" | nindent 4 }}
    VAR3={{ requiredEnv "Z" }}
    {{- readFile "end_of_file" | nindent 4 }}

  # The same sort of butchery in the next few config files too...
  config_file_2: | 

Then having to have lots of values files because there are different combinations of the VARs that can happen for different releases. So I ended up with lots of file “parts” and 4x values files to drive different releases that all used the same config files, but with the values varying.

If .Release.Name was available, then I would just have 1 file per configuration file, with some statements like the following within it:

{{- if eq .Release.Name "X" }}
VAR1=a
VAR2=b
{{- else if .Release.Name "Y" }}
VAR1=x
VAR2=y
{{- else }}
VAR1=m
VAR2=n
{{- end }}

Although messy… I still think it’s heaps cleaner than what I had to do.

ok, the question is what are the defaults for each section?

for

helmDefaults

and for

releases

ah okay i believe we don’t have a refence for that

because I have the feeling the eg.

releases:
 atomic: true #default is actually false

yeah, it is atomic: false by default

it mostly correspond to helm flags with the similar names

can I make a pull request and start adding defaults? because I was under the impression that the values are all defaults. especially for ( all booleans)

so the default would be no flags passed to helm

ok, so the reference here would be Helm and not Helmfile right?

which means

helmDefaults:
  #tillerNamespace:  
  tillerless: false
  kubeContext: ""
  cleanupOnFail: false
  args:
  verify: false
  wait: false
  timeout: 300
  recreatePods: false
  force: false
  tls: false
  #tlsCACert: "$HELM_HOME/ca.pem"
  #tlsCert: "$HELM_HOME/cert.pem"
  #tlsKey: "$HELM_HOME/key.pem"
  historyMax: 10

ok, so the reference here would be Helm and not Helmfile right? correct. and translating helm defaults would produce something like https://sweetops.slack.com/archives/CE5NGCB9Q/p1586342938045600?thread_ts=1586342576.043500&cid=CE5NGCB9Q

could we update the docs with that values

how would you update it? just replacing the whole helmDefaults in the example?

it would make life easyier fo so many just by looking at your example. i take the example as a reference

okay that makes sense

but would you also leave the information about how you could customize it

yes, or making comments eg-

 historyMax: 1 # dafaults 10

like

# When set to true, enable TLS for request to Tiller
  tls: false

yeah that would work too

for me and my team mates this file is the reference, i usually only look at this one: If I need more I look at the description If I am still clueless I look at helm.sh docs

if you prefer the uncommented line to denote a default, im fine with

```

# Number of hostorical releases to be retained in the cluster. Defaults to 10. Set it to e.g. 1 if you want only 1 release to be retained historyMax: 10

in this order, that why i call it the reference

this is also fine

got it. that makes sense

would you mind giving it a shot?

ok,

i’ll make a PR wait

i mean would you mind submitting a pr for that as the foundation

thanks a lot!

Hello @mumoshu https://github.com/roboll/helmfile/pull/1189

There are some inconsistencies in the docs. Some defaults are written on separate line some other behind the values. I added defaults as a comment behind the values.

However I am open to make adaptions.

An alternative might be.

1. if a comment above KV exist add the default in there.
2. If no comment exist append default in the same line as the value

What do you think @mumoshu? or is it ok as is?

I prefer the option 1 and making it consistent in all places(incl. helmDefaults and releases[]) if possible

thanks anyway! just reviewed it

@mumoshu so only 1. not the combination of 1+2?

yes that’s my intention.

just curious, but do you have any motivation to mix 1 and 2? (i had no intention to do that myself - i just didn’t read that part of readme that much these days

2. If no comment exist append default in the same line as the value I would put the comments and explanation into helmDefaults and keep the release clean as much as possible

also the defaults in releasedepend on helmDefaults if you dont set wait it will fallback to helmDefault

adding a default in there might be not correct as it depends on helmDefaults

agreed. so how about only doing the option 1 for helmDefaults and removing comments for defaults under releases[]?

I mean comments in this part of the example does look like confusing:

https://github.com/roboll/helmfile/pull/1189/files#diff-04c6e90faac2675aa89e2176d2eec7d8R156-R160

https://github.com/roboll/helmfile/pull/1189/files#diff-04c6e90faac2675aa89e2176d2eec7d8R155 this one looks okay

this one seems ok to me. would u also agree to revert it?

https://github.com/roboll/helmfile/pull/1189/files#diff-04c6e90faac2675aa89e2176d2eec7d8L149

https://github.com/roboll/helmfile/pull/1189/files#diff-04c6e90faac2675aa89e2176d2eec7d8L151 should be just wait for k8s resources via --wait..

Defaults to false` in the end of the line looks confusing

ok.

I guess I understand. let me give it another try

thank you so much!

regarding https://github.com/roboll/helmfile/pull/1189/files#diff-04c6e90faac2675aa89e2176d2eec7d8L149 I would prefer to have the comment in helmDefault section, as the values also references to helmDefault. This way there is we can keep it a bit DRY

@mumoshu what do you think now?

this is a gotemplate issue

a string is expected but state.EnvironmentTemplateData is set

ok, and how can I fix this?

the error suggests the input of the tpl function which si set to . is not corrent. somehow it expects a string but don’t know why

I’ve been using syntax like the following if that helps?

{{ tpl (readFile "helpers/add_http_repos.tpl") . }}

@Andrew Nazarov Very odd. I take this as terraform has saved the old INFRA_VERSION value into the tfstate and used it to run terraform plan with it, even though var.infra_version has updated. Am I following you correctly?

Could there be any bug in Terraform around dependency resolution among vars, envvars, etc?

If so, would it work if you avoid envvars as the way for communication between tf and helmfile?

i mean, try using:

helmfiles:
  - path: git::<https://my_user>:{{ requiredEnv "REPO_TOKEN" }}@gitlab.my_domain.com/my_project/my-repo.git@deployment/helmfile.yaml?ref={{ .Values.infraVersion }}

and

resource "helmfile_release_set" "yourapp" {
  values = {
    infra_version = "v3.42"
  }
}

or

resource "helmfile_release_set" "yourapp" {
   values = {
    infra_version = var.infra_version
  }
}

Am I following you correctly? Yes, that how it looks like.
resource “helmfile_release_set” “yourapp” {
values = {
infra_version = “v3.42”
}
} I always get executing "stringTemplate" at <.Values.infraVersion>: map has no entry for key "infraVersion" no matter of how I pass variables to tf. I tried

resource "helmfile_release_set" "yourapp" {
  values = {
    infra_version = "v3.42"
  }
}

and

resource "helmfile_release_set" "yourapp" {
  values = {
    infraVersion = "v3.42"
  }
}

and

values_files = [
		<<EOF
{"infraVersion": "v3.37"}
EOF
  ]

I’m using

terraform:0.12.24
helm:v3.1.2
helmfile:v0.106.3
helmdiff:v3.1.1
terraform-provider-helmfile:built from master

Hm, it might be that my helmfile provider version is not that fresh. I’m rebuilding it.

ah sry! it must be json under values, e.g.

	values = [
		<<EOF
{"infraVersion": "v3.37"}
EOF
	]

Still executing "stringTemplate" at <.Values.infraVersion>: map has no entry for key "infraVersion" . Actually previous errors of that sort were due to the old version of terraform-provider-helmile. Now it’s rebuilt from sratch from the master branch.

I’m wondering if helmfile diff is somehow triggered against resources stored in the tf state files first. This way it’s natural that I have this error, because this value is not in the state yet. And it corresponds well to the initial error where probably the old environment variable was used (taken from the state). And then it does the same for local resources in tf files. I don’t know the logic behind the code (how tf plan works under the hood), but it feels like something strange is happening here.

Oh! And it might be related to this: https://sweetops.slack.com/archives/CE5NGCB9Q/p1583329251056000 and https://sweetops.slack.com/archives/CE5NGCB9Q/p1583333215058300

Should I file an issue on GH?

Regarding our problem, we’ve decided to bypass TF and set environment variable outside of it. It seems it was overengineering initially.

Ahh that makes sense. Terraform internally call underlying provider’s get operation, and seems to compare the results of gets for the old and the new desired state.

terraform-helmfile-provider runs helmfile diff to respond get so yeah, it should require both the old and the new helmfile.yaml to work.

Should I file an issue on GH? Yes! That definitely helps

Finally I’ve created an issue. Sorry for such a delay. Was overwhelmed with other things.

https://github.com/mumoshu/terraform-provider-helmfile/issues/9

@Andrew Nazarov No worry, thanks! I replied there. In short, I think we can use content = file("path/to/helmfile.yaml") syntax instead

And deprecate/remove path attr as it’s so confusing

As I see it cannot deletes clusterrole and customresources sometimes

If they are created via hooks or something it will be like that.

Also I’ve noticed leftovers after failed deletion (permission issues or other troubles). Thus a Helm release was removed, but objects were still in the cluster.

Same here, especially if the release is not fully deployed before you call destroy

thanks, will check them

Only on a case-by-case basis.

e.g. some let you disable ingress then we use the raw chart to define what we need.

but there’s no generic way that works for all situations.

Damn Well thanks for letting me know!

This plugin has been a lifesaver.

environments did not work with bases for us

ended up not using bases

maybe there is a solution for that (did not spend much time investigating)

very strange the docs say something different: https://github.com/roboll/helmfile/blob/master/docs/writing-helmfile.md#layering-state-files

I think the doc is correct. Perhaps you’re missing --- between bases and other parts of helmfile.yaml?

@Andriy Knysh (Cloud Posse) Did you use --- as documented when you tried to make bases + environment values work?

I ran into that with rules for prometheus. You can escape it like this:

{{` {{short_version}} `}}

Yes, we’ve fixed it basically the same way, but escaped the whole json

data:
  javamelody-dashboard.json: |-
    {{`{
    ....
    }`}}

Good point. I’ve used both approaches.

found an easier way:

- kind: ConfigMap
  apiVersion: v1
  metadata:
    name: postgres-grafana-dashboard
    labels:
      grafana_dashboard: '1'
    annotations:
      dashboardSource: "<https://grafana.com/api/dashboards/11376/revisions/2/download>"
  data:
    postgres-grafana-dashboard-11376.json: {{ readFile "postgres-grafana-dashboard-11376.json" | toJson }}

readfile and then toJson. toRawJson would be better, but helmfile doesn’t use an up to date sprig version

@Meerkat Could you try running helmfile template against your example to see it does include the RBAC resources from the dep charts?

It should just work. Helmfile fully delegates the install/update/deletion of the chart and its dependencies to Helm. So if Helm could handle the installation correctly, so should Helmfile.

Running helm template against the chart does produces RBAC-related resources.

$ helm3 template dapr dapr/dapr --namespace dapr-system | grep kind
kind: ServiceAccount
kind: Secret
kind: Secret
kind: ClusterRole
kind: ClusterRoleBinding
- kind: ServiceAccount
  kind: ClusterRole
kind: ClusterRoleBinding
- kind: ServiceAccount
  kind: ClusterRole
kind: Service
kind: Service
kind: Service
kind: Service
kind: Deployment
kind: Deployment
kind: Deployment
kind: Deployment
kind: Configuration
kind: MutatingWebhookConfiguration

helmfile template should also produce the exact result.

When I do a helm template I can see the RBAC objects, but it doesn’t install with the rest. When I do a helm install it shows up just fine.

but it doesn’t install with the rest. What’s actually missing?

I see several services and deployments, clusterroles etc when i run helm template. I can see they do exist in the cluster after helm install

So one of cluster roles managed by the chart includes secret-reader, which does exist in my cluster after installation

$ k get clusterrole | grep secret
secret-reader                                                          57s

Perhaps I need to ask about how you confirmed “it doesn’t install with the rest”

So that I can better understand your issue

I’ll replicate my problem again here in a bit and give you a better description.

I think you want to flip this around.

Just use the raw chart and use Helmfile to provision:

apiVersion: v1
kind: ConfigMap
metadata:
  name: ...
data:
...

Where do these files come from?

Do you want helmfile to stage these files in the directory for you? e.g. using hooks?

As long as the chart uses .Files.Glob to grab the files, it’s impossible.

.Files.Glob by definition has access to files contained in the chart only

Helmfile by its nature has no ability to modify charts on the fly

Perhaps the intended way for the chart author would be to fork the chart.

Then from the helmside you can point to the locally forked chart with chart: ./path/to/forked/chart for your use

gotcha

that makes sense

since it’s impossible, as @mumoshu mentioned, I’m taking @Erik Osterman (Cloud Posse) approach to create the configmap and override

my next battle is using exec with a git.sh script

the cat at the end doesn’t get slurped into helmfile. but I think this is my problem to solve. don’t want to distract you guys with this

Would you mind giving me a context in the values.yaml.gotmpl/helmfile.yaml around the exec line?

Several lines around the exec would be enough

yea

the monochart concept is within calm lol

It shousld be nindent 8

Not sure it’s the source of the whole problem, though

oh, gotcha

lemme mess with it more, it’s surely on my side

Does the script alone work on your machine?

For me, it’s like

$ ./git.sh [email protected]:calm/locust.git//locustfile.py
Cloning into '/tmp/locust'...
fatal: remote error:
   is not a valid repository name
  Email [email protected] for help

yeah, it’s private repo

i found the problem

the repo was already there, so it exit > 0

so just needed

||: 

at the end of it

git clone "${git_full}" "/tmp/${repo}" 1>&2 ||:

i need to harden it

but at least it’s working as expected for the happy path

Even after adding ||: it’s not working for my own repo:

$ ./git.sh [email protected]:mumoshu/variant//README.md
Cloning into '/tmp/variant\'...
fatal: remote error:
   is not a valid repository name
  Email [email protected] for help
cat: /tmp/variant\/README.md: No such file or directory

Maybe an escaping issue? But never mind if it’s working fine for you

oh, it’s because it’s missing .git

yeah, that command worked locally for me

for your repo’s README.md

What does ||: do? First time to see that notation.

I was guessing if it’s || : i.e. || true?

aha

like :>file truncates a file. that : is funky

heh, yea https://stackoverflow.com/a/3224910

Also it’s usually used as the function name in forkbombs

:(){ :|: & };:

https://www.geeksforgeeks.org/fork-bomb/

Looks like a hipster emoji to me.

The decrypted file is specified as follows.

helm/datadog/datadog/helmfile.yaml
environments:
 dev:
  secrets:
  - environment/dev/secrets.encrypted.yaml

I’m encrypting the API key with SOPS.

bash-4.2# head helm/datadog/datadog/environment/dev/secrets.encrypted.yaml
apikey: ENC[AES256GCM,data:hogehogehogehogehoge,type:str]

I use helm3 & helmfile v0.108.0

bash-4.2# helm version
version.BuildInfo{Version:"v3.2.0-rc.1", GitCommit:"7bffac813db894e06d17bac91d14ea819b5c2310", GitTreeState:"clean", GoVersion:"go1.13.10"}
bash-4.2# helmfile --version
helmfile version v0.108.0

I think this part is bad because if you give a raw API key, it will be deployed.

@oza_shu Probably you’re confusing and mixing envrionment secrets with release secrets

What you can refer from the chart template(any files contained in the template directory of your chart) must be release values and secrets.

So give dev.yaml a place under the yaml array at releases[].secrets

not environments.ENV_NAME.secrets.

the latter is for templating the whole helmfile.yaml

the former is for helm

Thank you for your reply! I can pass api_key to template file! Thanks!

nvm, i found this https://github.com/roboll/helmfile/issues/1181

Yes subhelmfiles doesn’t support DAG

But why you need that?

Sub-helmfiles basically means that everything contained in the parent helmfile is dependent on sub-helmfiles

which is basically an another basic form of DAG

thanks, so i assume that the order of sub helmfiles does install them by this order?

yes

cool, thank u for the quick response!

glad to help!

@s_slack I’d apreciate it if you could submit a pr for that

I already provided a brief guidance to implement it. It should be a good-first-issue.

I don’t need this specific feature myself. But I’m eager to review/merge it if someone could contribute!

Not sure if I can . Not familiar with the code.

what do you mean by append here?

are you trying to concat yaml arrays? like concatenating an array from default values.yaml with an array from your values.yaml?

if so - that’s not possible. it’s not even possible with helm. we don’t have any way to let helm use alternative merge strategy