#helmfile (2020-12)

Okay found the answer, gotta update the entrypoint to do the append from the configMap volume before running the main command

You could also probably use an init-container to read the config setting and append it to a file.

TIL about init containers, thanks Erik!

Do init containers share a filesystem with the main container or does that interaction still need to happen via a volume?

(the entrypoint thing worked about to be difficult because of permissions issues with the app user in the main container)

does that interaction still need to happen via a volume? yes, kind of emptyDir

We got used to the current behaviour)). And for us it would have been a disaster once if helmfile had had this functionality by default.

You will find some thoughts on this here (like the absence of state and others): https://github.com/roboll/helmfile/issues/194

Well, you may pass .Release.Version to chart values and then to pods controller’s template as long as you control the chart

Sorry, does .Release.Version exist? Or do I have to set it explicitly (for example, in the CLI)?

I’ve just tried using it without actually setting it, and the value isn’t set. Maybe I’m doing something wrong.

.Chart.Version

But .Chart.Version won’t change if I’m just changing the values passed to the chart

values should be used inside templates or these values are useless

chart values should affect k8s state

Indeed, but if I’m changing a value used in a ConfigMap or a Secret, I want pods using that CM/Secret to be recreated

So that they can pick up the new values

this is exactly helm3 documented case with hash of configmap/secret

https://helm.sh/docs/howto/charts_tips_and_tricks/#automatically-roll-deployments

Yes, but unless I’m doing something wrong, that hash only takes the template as an input – It doesn’t take the actual values. So it doesn’t help if I want to change the values without changing the template or releasing a new version of the chart.

with new chart version helm will recreate pods due to controller change

the template as an input it takes rendered template with values as input

thus, any value change triggers pod restart

Hmmm, I must’ve done something wrong when I tried it then. I’ll have another go later

Thanks voron!

check your configmap/secret templates and be sure you’re using all CM/secrets in controller annotations

Good point. I think that’s probably the problem… I must’ve missed a CM/secret

helmfile diff shows if there are any expected diffs, and it takes 2 files at least - configmap/secret and controller annotation (and thus pod recreate) due to hash change

well, you may separate your release-specific values into different files and include these values via something like

values:
  - values/{{`{{ .Release.Name }}`}}.yaml.gotmpl

Hey, thanks for your reply! My release-specific values are in a separate file. I need to be able to use them to template other values which apply to the base helm chart.

then your release-specific values are not release-specific

do you wanna “calculate/evaluate” some base values based on release-specific values ?

Yes that. My base chart expects a value called environment which I’m trying to compute based on the app-chart’s default values and my release-specific values.

did you tried gotmpl ?

I tried splitting into three files:

• chart-default-values.yaml (contains default values for the environment vars) • release-specific-values.yaml (contains the overrides for that release) • environment-values.yaml.gotmpl (sets the environment variables for the base-chart’s values) If in my release I specify:

releases:
  - name: "myrelease"
    values:
      - ./chart-default-values.yaml
      - ./release-specific-values.yaml
      - ./environment-values.yaml.gotmpl

I find that in environment-values.yaml.gotmpl the Values and environment vars are empty (when I add: {{ .Values }} and ``{{ .Environment }} to the file and run helmfile template it prints []map and {default map[] map[]}` in the generated templates. I cannot access the values from chart-default-values or release-specific-values

helmfile version v0.135.0 btw

what about

{{`{{ .Values }}`}}

?

That renders the string {{ .Values }} into my configmap without evaluating it

I think using it unescaped in the environment-values.yaml.gotmpl is correct as it renders to a string representation of itself map[]. The issue is that it is empty, and I would like it to be populated with the previous two values file’s properties.

We are using embedded vals functionality to get outputs from tf state and pass them to helmfile.

https://github.com/variantdev/vals

https://github.com/roboll/helmfile/blob/master/docs/remote-secrets.md

Yes, I am using exactly that. No problems in setting that up. I just do for example role_arn: <ref+awsssm://path/to/param> then I just made a terraform module for the k8s IAM roles that also takes care of putting everything to SSM.

My question is more of if it’s the right way to go about this? Since it would be essentially throwing everything to SSM, wherein it essentially acts like the glue between helmfile and terraform.

I guess you are doing the same thing then? Sorry idk, this approach just feels. . .hacky. Maybe just me haha

Yeah, it looks pretty the same indeed. We just don’t have this intermediate “publish to SSM” step, but pull directly from the tf state. I will be glad to hear any better alternatives as well, but we are pretty satisfied with our current solution:)

helmfile provider might be an alternative, but to us the current solution seems a bit more reliable.

Haven’t had time to check the most recent version of the provider though.

Ah I actually didn’t notice that vals also supports reading directly from the terraform state. I wonder if that is the better approach, instead of publishing everything to SSM.

Do you also pull sensitive values/secrets directly from the tf state?

Last time I looked, it only supported local state, not remote/S3 backed, but that’s a while ago now. I’ve just gone with Terraform writing to SSM (or humans/other code if it’s something that Terraform can’t handle) and then Helmfile reads from SSM.

At least that way, regardless of how something was provisioned/configured, there’s a consistent contract with Helmfile

Last time I looked, it only supported local state, not remote/S3 backed It still doesn’t support anything. For example it doesn’t support GCS backend so we have to pull the state and operate with it as if it’s local. That’s what looks hack-ish to me right now, speaking of hacks:) Actually it’s an issue of the underlying tfstate-lookup
Do you also pull sensitive values/secrets directly from the tf state? Not yet:) So far we are interested in endpoints and LB IP addresses.

Last time I looked, it only supported local state, not remote/S3 backed Ah got it. I don’t think this is an option for me then.

Thanks for the discussion! I’d just stick with what I have :)

S3 should work ok

Yeah I’ve verified that it works with S3 backed tfstate before. GCS isnt supported yet

Anyways, I think the way @Christian using SSM as the connector between tf and helmfile is nice.

That way you can make tf and helmfile more decoupled than reading tfstate directory

Thanks for the confirmation @mumoshu! This gives me a nice confidence boost to my current setup

Basically I’m trying to run a kubectl to apply a label and it has no idea about the Kube context anymore when it runs and tried to connect to localhost:8080.

I created a new release with this set of dummy hooks, the last one to deliberately make it quit before installing the chart:

    hooks:
      - events:
        - presync
        showlogs: true
        command: sh
        args:
          - -c
          - "env | grep KUBECONFIG | sort;echo [$KUBECONFIG]; test -n \"$KUBECONFIG\" && ls -l $KUBECONFIG; cat $KUBECONFIG"
      - events:
        - presync
        showlogs: true
        command: kubectl
        args:
          - config
          - current-context
      - events:
        - presync
        showlogs: true
        command: sh
        args:
          - -c
          - "exit 1"

When run using helm2, the output of the hooks is:

helmfile.yaml: basePath=.

hook[presync] logs | []
hook[presync] logs | 
helmfile.yaml: basePath=.

hook[presync] logs | test2.he0.io
hook[presync] logs | 
helmfile.yaml: basePath=.

hook[presync] logs | 
in ./helmfile.yaml: failed processing release cert-manager: hook[sh]: command `sh` failed: command "/usr/bin/sh" exited with non-zero status:

PATH:
  /usr/bin/sh

ARGS:
  0: sh (2 bytes)
  1: -c (2 bytes)
  2: exit 1 (6 bytes)

ERROR:
  exit status 1

EXIT STATUS
  1

That’s fine, it shows my context, then deliberately exits with an error in the next hook as I wanted.

When I run it with helm3 however the output is:

helmfile.yaml: basePath=.

hook[presync] logs | KUBECONFIG=/tmp/tmp.018hsO8A7s
hook[presync] logs | [/tmp/tmp.018hsO8A7s]
hook[presync] logs | -rw------- 1 kube-hetest kube-hetest 0 Dec 10 13:18 /tmp/tmp.018hsO8A7s
hook[presync] logs | 
helmfile.yaml: basePath=.

hook[presync] logs | 
in ./helmfile.yaml: failed processing release cert-manager: hook[kubectl]: command `kubectl` failed: command "/usr/local/bin/kubectl" exited with non-zero status:

PATH:
  /usr/local/bin/kubectl

ARGS:
  0: kubectl (7 bytes)
  1: config (6 bytes)
  2: current-context (15 bytes)

ERROR:
  exit status 1

EXIT STATUS
  1

STDERR:
  error: current-context is not set

COMBINED OUTPUT:
  error: current-context is not set

For some reason it has set the KUBECONFIG environment variable and pointed it at an empty temporary file. Then the next command to show the context dies because there is no selected context.

Regular chart hooks have been working fine with Helm3, it’s only this helmfile hook that no longer works when I want to run kubectl command because the kubecontext has been overridden. Note that if I set --kubeconfig to point to my config file and also set --context in the call to kubectl in the hook, then it works with helm3, but that isn’t a viable option.

I don’t know why it is setting the KUBECONFIG environment variable within the hook when helm3 is used, nor how to stop it from doing so. The versions of software I am using are

$ helmfile version
helmfile version v0.135.0

$ helm3 version
version.BuildInfo{Version:"v3.4.2", GitCommit:"23dd3af5e19a02d4f4baa5b2f242645a1a3af629", GitTreeState:"clean", GoVersion:"go1.14.13"}

I still don’t have a solution to this. Does anyone have an idea of why I’m seeing this different behaviour with helmfile hooks depending on if I’m using Helm 2 vs Helm 3?

Oh damn that’s unfortunate, what are we supposed to use for the same purpose? 

I think we need to wait for @mumoshu to host this chart in some separate helm repo It’s still possible to use “deprecated” chart, as the chart is fine itself, the problem is with incubator repo.

Yeah the chart is marked as deprecated due to the incubator repo being abandoned. But it still works as before

I’m using chartcenter’s mirror of the incubator repo these days https://chartcenter.io/incubator/raw

But it’s still with “deprecated” label

Most, but not all, charts have migrated to their own repos/chart locations.

ohhhh

Yeah I am still using it was just worried they might drop it one day, thanks for the info everyone

A lot of charts are being abandoned unfortunately.

I have the same problem as yours, here is my solution. Create a file called prometheusrule.yaml

# prometheusrule.yaml
prometheusRule:
  enabled: true
  rules:
    - alert: UrlDown
      annotations:
        message: The status code of {{ $labels.target }} ...

And just reference it

values:
  - prometheusrule.yaml

Seems that

{{`{{"{{ $labels.target }}"}}`}}

is working

You can choose the better one for your use case

{{ toYaml .Values.foo | indent 8 }} worked for me

We’ve updated all of our helmfiles to use environments: https://github.com/cloudposse/helmfiles/tree/master/releases/

I am also looking about what or how I can use all the features with chartify in helmfile. I am currently using hooks with shell scripts, but if there is a smoother way to do kustomize in helmfile, that would be better.

We are wrapping this in if ne .Environment.Name "default" . I must admit the overall readability suffers:)

1. is it possible to introduce a CLI flag or parameter to helmfile.yaml ?
2. unfortunately i didn’t understand the idea of that if , could you provide an example ? In fact i want to render locally template for different environments, I just don’t want to perform resolution at all ( speed of render + permissions issues )

the issue is that I can’t even override those values via --set since helmfile still tries to perform a resolution, and it’s logical

The thing is that we use “default” environment only to test things, for real environments we have correspondent environment names and settings in helmfile. With that being said it just a matter of the following simple if:

loadBalancerIP: {{- if ne .Environment.Name "default" }} ref+tfstate:///{{ env "PATH_TO_TF_STATE" }}/output.external_ip.value {{- end }}

Got it, thanks

I’m not exactly sure if or how it’s possible to refer to values in the template, but if you want to do it a bit differently, you can set a version under the releases in the helmfile.yaml like so:

- name: "test"                                                                                                                                                              
  version: {{ .Values | default (env "TEST_VERSION") | default "1.0.0"}}                                                                                              
  <<: *default   

@Jonathan the problem is that I have around 100 releases )

So, I need to template it somehow

Have you tried specifying the version value like this?

{{ .Values | get "Release.Name" "" }}

just to try getting the value in a slightly different way?

The reason I’m not sure it worksis that it might try to access the .Release.Name before the values.yaml have been properly rendered, so you might have to do some {{ ` {{ .Values.Release.Name }} `}} magic

@Jonathan Hmmm, good catch! Will try

This worked

templates:
  default: &default
    namespace: test
    chart: repo/{{ .Release.Name }}
    version: 0.0.0-{{ .Values.release | get .Release.Name "develop" }}

where values are

release:
  service: 1.0.0

ah, .gotmpl extension

yep

If the disableValidation: true is specified as above, I don’t think that it’s specified in a way that shows that the field is supposed to be part of the .Values. You’d have to do something like this:

releases:
  - name: prometheus
    namespace: default
    chart: prometheus-community/kube-prometheus-stack
    values:
      - ./alertmanager.yaml
      - disableValidation: true

disableValidation is a key in helmfile, not the prometheus-community helm chart.

The solution was to delete the existing CRDs. An earlier version installed CRDs that have since then been updated. Once CRDs were purged, helm chart would now work, as it installs new CRDs.

you can define the release in the helmfile in the order you want them installed, and then use helmfile --concurrency=1 to run them sequentially

It may be better to use deps in helmfile

Yeah, so, the sync command would work, but the diff if the crd is not installed will not work

is it due to helm diff cannot complete w/o required&installed CRDs ?

@voron Yes

In Helm 3, the release metadata is stored in the same namespace as the release itself. See https://helm.sh/docs/faq/#release-names-are-now-scoped-to-the-namespace

@Christian yea I mean that appears to be demonstrably false

Did you able to get needs to work inside single helmfile.yaml ? needs may need to specify k8s context too just before namespace.

Yup, got it working in a single helmfile. I’d try specifying the k8s context, thanks!

well, use hpa.yaml.gotmpl and templating features

I had to ouse toyaml thanks

Did you specify environment via helmfile -e ?

I think it’s possible if you use it under:

environments:

But would be cool indeed to have it under releases or at least under helmfiles

https://github.com/roboll/helmfile#loading-remote-environment-values-files

helmfiles is already possible but value make thing easier for my user case with different repo for each app , i wanted to just have param for the app not helm config in those repo

each release could get his own file , which can be different from the same environment

also i could do something like `

git::<https://github.com/me/app/config-{{> .Environment.Name }}.yaml?re=master

and in another release have

git::<https://github.com/me/app2/config-{{> .Environment.Name }}.yaml?re=master

similar thing is possible with secrets And you still have exec

in the end what i did was to use multiple helmfile which all have there own environment and url.

i manage to make it work with git without problem but i believe that the http is rather trouble some as you need specific content/header to be pulled