#helmfile (2020-08)

You could use a job with post-install helm hook

That post-install trigger only if the deployment is succesfull?

How would I target the job strictly at the container from the deployment?

I will run in another container inside your pod

it’s always a bad pattern to run a command inside a running container

Agreed but in this case i need to execute a binary inside the container

you can run a job with exact same image I guess https://helm.sh/docs/topics/charts_hooks/

if it runs in same pod, it will have same environment

you use this in pre-install

I remember going this route and having a problem but i will take a look once again and see what comes up

for our staging, it deploys database, kafka schema, etc, and after, helm deploys ours pods

Ah I remember one thing is to see the log output in the ci/cd pipeline and also have it fail if there’s an error

about what?

I think postinstall should be what you want. It’s called only after successful install.

In the hook command, perhaps using

kubectl get po -l release={{` {{.Release.Name}} `}}

and then kubectl exec on the pod would work?

postsync is called after install/upgrade regardless of success/failure, and you should be able to check {{.Error}} for emptiness to determine if it was successful or not. try that instead of postinstall you need testing after upgrade, too.

@mumoshu postinstall are run locally? I thought hook was only for resource creations like jobs

hooks can be any local processes. you’ll usually use preopare and cleanup hooks for creating/deleting charts and presync for creating resources not managed by required by release

for postsync… im not sure. i thought i heard anyone used it for notification

and postinstall is a specialization of postsync

you can put these specific resources in hooks, see https://github.com/kubernetes/kubernetes/issues/45398#issuecomment-478619253 for an example

Are you talking about delete&recreate on every helm upgrade, including cases when immutable resource isn’t changed ?

yes. well of course it may or may not apply to your use-cases, depending on which resources you need to “force update”. In our case for example, we used to do it on PDB, and now we’re doing it on Jobs, so deleting and recreating is not an issue

but I agree that it’s a pain

I hit this with Service and StatefulSet

which part of the spec ?

statefulset example , service example

do helm hooks require to add on-the-fly helmfile json patches to stock charts to apply hooks to these charts?

ah yes, the volumeClaimTemplates in the statefulset, we had this issue too…

hopefully the pvc size won’t change often, you can do a 1-time operation (backup, delete, recreate, restore data)

PVC isn’t affected during statefulset recreation

so backup+restore isn’t required

about json patches, that depends on the charts, if they support adding annotations or not

and it’s still a semi-solution just to emulate helm2 behaviour with drawbacks like recreating the resource every time

I have helmfile apply over 50 releases in a dev CI/CD pipeline, and it works fine, but it’s helm2, not 3.

We are dealing with this manually. I.e. we do helm delete and then trigger pipelines with helmfile and stuff. Not an ideal solution for sure.

yes, donwtime and side effects are everywhere

Do you wanna mix helm gotmpl with helmfile gotmpl in the same line ?

So, you wanna pull values from remote instead of local ?

well, values are already in the chart. I was hoping that I could just reference the file I want to use

I know I can put an URL to retrieve values but its a real shame since everything is already included in the chart

helmfile doesn’t have access to the chart, it just calls helm

Ok, and helm is not able to use values from a remote chart then? the file must be present when we run helm -f xxx ?

you need to get values into helmfile, thus shell script(helm3 show values?) or remote yaml file in helmfile is your option.

ok Thanks for confirming. Shame this is not possible straight away

I don’t think that keeping unused values file in the chart is a common practice.

Ok. So It maybe make sense to move values-XXX.yml along with helmfile instead of keeping it in the chart

this looks a common practice. Keep charts and settings separated. You hit some sensitive data in settings usually and you cannot store it in the chart anyway.

Ok Thanks for the advice. I saw couple of charts storing values-XXX.yaml in their repo, but not that many. I will keep them separated.

storing values-XXX.yaml in their repo I suppose it was a git repo, not a helm repo with charts archives.

Its a git repo, but its in the chart archive. Example: https://github.com/bitnami/charts/tree/master/bitnami/rabbitmq

It looks like additional values file cannot be used by helm directly from the chart archive. Manual specifies values.yaml only.

yes for this case this is clear. I just would have liked a comment on the helm website that we can’t specify custom value file even if its included in the chart

Create a PR

Yes

otherwise, would the following work ? I haven’t tried it yet: https://github.com/zendesk/helm-secrets/issues/61#issuecomment-429053736 Its pulling the chart locally through hooks

yep, this should be fine except it will trigger on every helmfile

did you tried something similar to

string: "{{`{{ .Values.key1.user }}:{{ .Value.key1.password}}`}}"

?

@Jonathan could u try “fetchSecretValue”?

https://github.com/roboll/helmfile/blob/b1190508b2fbbb56b80de916aa7e66dd9f07db2f/docs/remote-secrets.md#fetching-single-key

using fetchSecretValue worked, thanks!

I think recently this became supported? something with go-getter

https://github.com/roboll/helmfile/commit/b5830a30117392650bfd5811f1b0b6c863d3c046

https://github.com/roboll/helmfile/pull/1374

Do you mean I should be able to deploy like that ?

releases:                                                                                                                                                                                                         
  - name: ingress                                                                                                                                                                                                 
    chart: ./raw/ingress-dev.yaml

I have the following error returned:

COMBINED OUTPUT:                                                                                                                                                                                                  
  Error: only unpacked charts can be updated                                                                                                                                                                      

there is an example in the pull request

I’ve never used it personally

he targets `

git::<http://github.com/jetstack/cert-manager.git@deploy/crds?ref=v0.15.2>

which is here https://github.com/jetstack/cert-manager/tree/master/deploy/crds (except this is master and not that tag)

but the tl;dr is that its a repo directory with only a bunch of yaml manifests

Which I believe is what your goal is ?

Oh yes its working if I just pass the folder ./raw instead of ./raw/ingress-dev.yaml

So, I guess in your case if you wanted a single manifest you’d need to do a directory with just that single manifest inside it

Yes I guess so. Thanks a lot for your finding!

helmfile apply with helm2 in dev env

Is it related to https://github.com/roboll/helmfile/issues/932 ?

What about

environments:
  default:
    values:
      - foo:
          common:
            kind: Secret

?

Works! Thanks I already tried:

environments:
  default:
    values:
      - foo:
          common:
            kind: Secret
repositories:
- name: incubator
  url: <https://kubernetes-charts-incubator.storage.googleapis.com>
releases:
- name: test
  namespace: test
  createNamespace: true
  chart: incubator/raw
  version: 0.2.3
  values:
    - resources:
      - apiVersion: v1
        kind: "{{ .Values.foo.common.kind }}"

which is not working…

sry this is very confusing indeed, but the root cause here is that environments.NAME.values needs to be a yaml array, rather than hash

to allow merging multiple values entries

so it should be

environments:
  default:
    values:
    - foo:
        common:
          kind: Secret

the best practice is to separate environments and other parts of your helmfile.yaml with --- like

environments:
  default:
    values:
        foo:
          common:
            kind: Secret

---

repositories:
- name: incubator
  url: <https://kubernetes-charts-incubator.storage.googleapis.com>
releases:
- name: test
  namespace: test
  createNamespace: true
  chart: incubator/raw
  version: 0.2.3
  values:
    - resources:
      - apiVersion: v1
        kind: {{ .Values.foo.common.kind }}

so you get a much better error

in ./helmfile2.yaml: failed to read helmfile2.yaml: reading document at index 1: yaml: unmarshal errors:
  line 4: cannot unmarshal !!map into []interface {}

which tells that you’re mistakenly trying to define values as map, rather than array([])

You can’t use a git repository as source for helm

your repository must be for helm chart, with a specific index.html for listing available charts, etc

gitlab provides a repo : <https://charts.gitlab.io/>

see : https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/doc/index.md

@Issif of course i can https://github.com/aslafy-z/helm-git, point is not to use charts from https://charts.gitlab.io/

I found out that the URL has to be <https://our-nexus/><dir>@<file> but then helmfile tries to fetch just the directory which leads to a HTTP 404..

<http://example.com/dir@file> isn’t a http url

True, but it’s what helmfile expects. Otherwise you get an error like this (after digging into the code and fmt.Printing some errors that are otherwise not exposed to the user)

invalid src format: it must be `[<getter>::]<scheme>://<host>/<path/to/dir>@<path/to/file>?key1=val1&key2=val2: got <https://my.ordinary.http.url/dir/file>

So something like <http://example.com/dir/file> is not even considered by helmfile

It kind of makes sense because helmfile needs the whole directory since the remote helmfile can also reference remote files.

But you cannot “download” a directory over http which makes me think that HTTP ist just not supported at all for remote statefiles?

Just a few hours ago I had a conversation with my colleagues and I mentioned @mumoshu in a way that it’s hard to imagine how he does so many things with such a quality and passion. An example for all of us.

helmfile + variant2 =

I can’t tell you how much I appreciate all your help!

Honestly, I had mixed feelings working on OSS. It’s too tough in regard to how much time and effort I need to put onto it, while I’ve been grateful for having chances of any kinds of collaborations with you all.

Discussing about potential bug, get to distill it into the root cause, finally fixing it together. Receiving feature requests, talking about use-cases, come up with a solution together. Prototyping something and receiving early feedback about it(recent examples would be terraform-provider-helmfile and variant2). I think I love all of those experiences and processes while working on OSS - probably that’s why I’ve been maintaining OSS like Helmfile until now.

Sponsorship - I think this is my first time to receive sponsorships more than one or two at once. Simply I had never realized I would receive such many supports! Combining all my experiences so far, your kind words, and this time, sponsorships, I’m feeling encouraged and motivated more than ever for my work now .

Thank you all for continuing to work with me. Expect me to keep this

// Excuse me for slow response. I had to have some time to organize what I felt

more than 30 likes, but just 8 sponsors on github

Hrm.. I’m not familiar with the condition in helmfile - we haven’t used it, but my guess is that it works like the installed flag?

My guess is that borrows from the upstream convention in the Chart.yaml . See https://helm.sh/docs/topics/charts/#the-chartyaml-file

Anyone know the difference between condition and installed?

condition with false doesn’t delete the release, it’s just ignored, like there is no such release. installed/installedTemplate with false will delete --purge release.

Taken from state.go

    op    |  true   |    false       |
--------------------------------------
installed | install | delete --purge |
condition | install | skip/ignore    |

kind of

hrm… then how is condition different from enabled? sounds like condition is the same as enabled

do we have a keyword enabled in helmfile?

hah, you know - I must be confused. Doesn’t look like it and then it all makes more sense.

Yes. A recently introduced disableValidation flag might help. I’ve just mention this in a different thread:)

You might want to use disableValidation flag to mitigate this

Helmfile has native support for getting values from SSM as it uses vals (https://github.com/variantdev/vals)

https://github.com/roboll/helmfile/blob/master/docs/remote-secrets.md

The remote-secrets.md file has some very basic examples, and the vals readme has a section on using SSM to get more info on how to use it

Nice. That simplifies a bunch. Though how do I handle the fact that I need username/password either in application.properties or as env variable. Though I don’t want it in plain text in the pod or in git.

Create a k8s secret using helmfile, and load that secret as environment variables in the pod.

That sounds much easier than I thought. I will try it out. Thank you @bradym

np

Curious. Is it common practice to put password in secrets file not encrypted (just encoded base64)? Even though it retrieves from SSM one could easily get the password from secrets

I have a SSM parameter with the following values:

param1: value1
param2: value2

I use in values.yaml.gotmpl

...
Secrets:
{{ .Environment.Values.someapp | expandSecretRefs | toYaml | nindent 2 }}

in someapp.yaml

...
- releases:
  - name: some-app  
    values:
      - ../releases/someapp/values.yaml.gotmpl
      - Secrets:
        - <ref+awsssm://someapp/testssm?region=us-west-2>

in secrets.yaml

...
data:
{{- range $key, $value := .Values.Secrets}}
  {{ $value | b64enc | quote | indent 2 }}
{{- end }}

returns:

...
+ data:
+   0: "param1: value1\nparam2: value2"

I can’t seem to be able to get the $key= param1 and $value= value1, etc for those variables. The SSM entry gets populated in $value only, $key is always empty.

Alternatively, you can try this option https://thefirstapril.com/2019/06/29/Secret-Management-in-Helm-Kubernetes/ Use helm secrets plugin to encrypt/decrypt the secret file with AWS KMS and pass the encrypted file to helmfile so that all the secret files which are stored in version control system are encrypted #helm secrets enc secrets.yaml releases:

• name: test namespace: default version: “0.2.0” chart: test set:
  • name: “clusterName” value: test secrets:
  • secret.yaml

https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-helmfiles/

They propose some apps in their managed k8s clusters and for installing them they use helm right. A apps I develop will be added in a next release and that’s why I added it on helm hub last week

I am just positively surprised by their adoption. What app did you work on ?

https://github.com/falcosecurity/falcosidekick

falco has been integrated by gitlab in 13.2

oh that is nice! But what I mean is they are using helmfile to setup their own infrastructure (what runs gitlab.com) as well now..

Oh I get your point now, great

@Pierre Humberdroz Good find , But I am just wondering why are you positively surprised? I mean is this something that you expect they could have rolled their own solution with helm ? we primarily use gitlab.com for everything we do with our customers and are heavy consumers of pretty much all the solutions they offer.

No but a lot of people I talk to are not even aware that helmfile exists. And seeing some bigger company adopting it makes me happy! I also wanted to share how others set their helmfiles up.

helmfile allows to get values from exec, thus you may adapt it with git as a source too

There are some workarounds like: https://github.com/roboll/helmfile/issues/469#issuecomment-613782055

Also this is merged: https://github.com/roboll/helmfile/pull/1296

I was thinking https://github.com/roboll/helmfile/pull/1296 is related to values inside environment solely and doesn’t affect release/template values

Ah, @voron probably you are right. Thanks for a tip

@Sam Buckingham may I know how are you using condition in the helmfile

apparently you tried to write something similar to

values:
- whatever/*.yaml

which is not yet supported

Haha yup. Was nice to get such a thorough error message

glad to hear it worked! btw, pls feel free to open a feature request if you really need it

Just wrote a small bash script to print out all the files in the dir in a yaml format and use an exec

Both do the same. *Template were added to use inside yaml templates to have an ability to merge template and release values. When you use values in template and values in release - release values override template values. Same with valuesTemplate - using it in both template and release leads to release valuesTemplate overrides template’s one. But when you use valuesTemplate in template and values in release(or vise versa) - resulting helm values are merged from values and valuesTemplate values. Similar with set/setTemplate

Huge thanks! Is it documented somewhere and I missed it? Or you know it from inspecting the code/following issues?

maybe here

We use yaml templates, that’s why I’m aware of this.

I’ve seen this, but without trying is hard to get the idea. You description is way better. Thanks again!

np

I want to be able to deploy global-helmfile.yaml with environemnt variable support and also test-helmfile.yaml with static vars and possible release override. Any idea?

Solved with:

helmfiles:
- path: ../../helmfiles.d/global-helmfile.yaml
  values:
   - foo: bar

What does not work me is:

# test-helmfile.yaml

helmfiles:
- path: ../../helmfiles.d/global-helmfile.yaml
  values:
    - foo: bar

# global-helmfile.yaml

environments:
  default:
    values:
      - foo: {{ requiredEnv "FOO" }}

repositories:
releases:

error calling requiredEnv: required env var `FOO` is not set

foo var should be overriden with static foo value, isnt it a bug? @mumoshu

( Its working with {{ env "FOO" }}, but it can render empty value, if FOO is not specified, which is problematic )

Thanks

foo: bar is merged AFTER global-helmfile.yaml is executed as a go template. so, it must be {{ env "FOO" }} if you need to override foo in the callee side from the caller

one possible option is to use installed/installedTemplate in your template or release

installedTemplate: {{ eq .Environment.Name "development" | toYaml }}

another possible option is to use similar condition

environments:
  development:
    values:
    - MySecondApp:
        enabled: true
  staging:
    values:
    - MySecondApp:
        enabled: false
  production:
    values:
    - MySecondApp:
        enabled: false
templates:
  ...
releases:
  - name: MyApp
    <<: *app
  - name: MySecondApp
    condition: MySecondApp.enabled
    <<: *app

Of course, you may use files to load environment values instead of inlining

thank you, that helps

where do you specify kubeContext ?

at the release level

what helmfile version do you use ?

v0.125.5

could you pls reproduce this error on the latest version, and then retry with --debug to get actual helm arguments?

I suppose release: already exists error is from helm, thus your helm may use wrong context. It’s better to test on single release using selector by release name

we use context at helmDefaults and template level, don’t noticed any unfixed issues.

does this release actually exist ?

helm --kube-context dev_eu-frankfurt-1_dataplane --namespace example ls -la 

I suppose this issue is related to helm solely. Are you able to reproduce this issue by executing helm with all these args from debug log ?

well, i can see its issuing the helm release to the same kubecontext twice

that shouldn’t be happening

We are using a nested loop to generate our releases

but it shouldn’t produce this outcome. Here is an example

ah, dynamic releases. I’m able to reproduce this issue - all the releases get the same kubeContext - the last generated one. It may be by design - you’re on the same single helmfile environment, kubeContext isn’t used as some uniq part to distinguish releases, and releases with the same name+namespace are merged in therms of kubeContex.

File a bug or submit a PR.

Interesting results are with

  - name: example-grafana
    chart: "{{ $okeKubeContext }}"
    kubeContext: "{{ $okeKubeContext }}"

chart is uniq, while k8s context is the same in debug output. This leads to expected release: already exists

But as soon as “release name + namespace” are uniq - no issues - k8s context is uniq too.

  - name: "{{ $okeKubeContext }}"
    chart: bitnami/grafana
    kubeContext: "{{ $okeKubeContext }}"
or
  - name: example-grafana
    chart: bitnami/grafana
    namespace: "{{ $okeKubeContext }}"
    kubeContext: "{{ $okeKubeContext }}"

I thought kubecontext WAS used as a unique part to distinguish a release? https://github.com/roboll/helmfile/pull/1390

or at least it was intended to be

and yes, all of your tests above mimic exactly what I’m seeing

@mumoshu just CC’ing you here because I brought this up to you before as well

https://github.com/roboll/helmfile/issues/1440

and now you don’t hit duplicate release error in helmfile, but you get actual duplicate releases.

thank you for all your help

you’re welcome

looks like issue is fixed by @mumoshu.

could you pls provide larger example? I don’t see variable location in your example.

Sure, In addition to the first snippet, suppose

git::<https://git.com/ops/[email protected]?ref=master>

Looks like this:

  ---
  repositories:
~ - name: {{ .Values.a }}
~   url: <https://chartmuseum>.{{ .Values.a }}
~   username: {{ .Values.a }}
~   password: {{ .Values.a }}

  helmDefaults:
    createNamespace: false
    wait: true
    timeout: 600

  templates:
    default: &default
~_    chart: {{ .Values.a }}/{{`{{ .Release.Name }}`}}
      missingFileHandler: Warn

  releases:

~ - name: cluster-metadata
    <<: *default
    namespace: kube-system
~   version: {{ .Environment.Values | get "overrides.cluster-metadata" "0.1.0" }}
    labels:
      default: true
~     app: cluster-metadata
    values:
    - cluster:
~       location: {{ .Values.a }}
~       zone_prefix: {{ .Values.a }}
~       zone_postfix: {{ .Values.a }}

then error will be during helmfile.yaml.part.0 parsing: template: stringTemplate18: executing “stringTemplate” at <.Values.a>: map has no entry for key “a”

so it cannot template

url: <https://chartmuseum>.{{ .Values.a }}

Which i assume it should get from

environments:
    default:
      values:
~_      - a: b

In the first file

helmfiles: allows to specify values per each add helmfile. Did you tried to move your values to file and then specify this file under values? See Advanced Configuration: Nested States on https://github.com/roboll/helmfile

That’s my next option. I was hoping to describe all my app layer in one file per cluster :<

It’s not possible then? I should stick with separate files for values?

Just move values to files from the beginning

 environments:
    default:
      values:
      - /path/to/values.yaml

I’m not sure if this helps unfortunatelly

It’s still a separate file ^^, ladno, spasibo!

You’ll need either a values: block under each sub-helmfile definition or environments: inside each sub-helmfile manifest:)