#helmfile (2020-08)

https://github.com/roboll/helmfile

Questions and discussion around helmfile https://github.com/roboll/helmfile and https://github.com/cloudposse/helmfiles Archive: https://archive.sweetops.com/helmfile/

2020-08-31

jason800 avatar
jason800

Hey All, good morning :slightly_smiling_face: Is it intended that helmfile repos attempts to template all releases? I was attempting to use it as a pre-step to my releases and it was taking a long time to run. Looking at --debug, it appears to be templating all the releases.

jason800 avatar
jason800

ah, looks like the trick for my setup is not to supply an env

jason800 avatar
jason800

so no environments are imported/run into the main helmfile, have to use --allow-no-matching-release

Release notes from helmfile avatar
Release notes from helmfile
02:06:33 AM

v0.126.1: Fix test flake for commonLabels (#1449) 3e6542e (HEAD, tag: v0.126.1, origin/master, origin/HEAD, master) Fix test flake for commonLabels (<a class=”issue-link js-issue-link” data-error-text=”Failed to load title” data-id=”689738084” data-permission-text=”Title is private”…

Release v0.126.1: Fix test flake for `commonLabels` (#1449) · roboll/helmfile

3e6542e (HEAD, tag: v0.126.1, origin/master, origin/HEAD, master) Fix test flake for commonLabels (#1449) d3daea3 Fix panic on hook since v0.126.0 (#1448) 19d7942 Add experimental “forceNamespace” …

2020-08-29

2020-08-28

owlz avatar

Hello! I’m trying to reuse values block for different helmfiles but getting template error: template: stringTemplate:30:26: executing “stringTemplate” at <.Values.location>: map has no entry for key “location” It works if i use the same values in “values block” instead of global environments. Any ideas on how to fix this? Or suggestions how to get what i want other way? Thanks! (I’d like to keep all in one file if possible)

--- 
 environments:
    default:
      values:
~_      - a: b

+ ---
  helmfiles:
~   - path: git::<https://git.com/ops/[email protected]?ref=master>
+   - path: git::<https://git.com/ops/[email protected]?ref=master>
voron avatar
voron

could you pls provide larger example? I don’t see variable location in your example.

owlz avatar

Sure, In addition to the first snippet, suppose

git::<https://git.com/ops/[email protected]?ref=master>

Looks like this:

  ---
  repositories:
~ - name: {{ .Values.a }}
~   url: <https://chartmuseum>.{{ .Values.a }}
~   username: {{ .Values.a }}
~   password: {{ .Values.a }}

  helmDefaults:
    createNamespace: false
    wait: true
    timeout: 600

  templates:
    default: &default
~_    chart: {{ .Values.a }}/{{`{{ .Release.Name }}`}}
      missingFileHandler: Warn

  releases:

~ - name: cluster-metadata
    <<: *default
    namespace: kube-system
~   version: {{ .Environment.Values | get "overrides.cluster-metadata" "0.1.0" }}
    labels:
      default: true
~     app: cluster-metadata
    values:
    - cluster:
~       location: {{ .Values.a }}
~       zone_prefix: {{ .Values.a }}
~       zone_postfix: {{ .Values.a }}
owlz avatar

then error will be during helmfile.yaml.part.0 parsing: template: stringTemplate:3:18: executing “stringTemplate” at <.Values.a>: map has no entry for key “a”

owlz avatar

so it cannot template

url: <https://chartmuseum>.{{ .Values.a }}

Which i assume it should get from

environments:
    default:
      values:
~_      - a: b

In the first file

voron avatar
voron

helmfiles: allows to specify values per each add helmfile. Did you tried to move your values to file and then specify this file under values? See Advanced Configuration: Nested States on https://github.com/roboll/helmfile

roboll/helmfile

Deploy Kubernetes Helm Charts. Contribute to roboll/helmfile development by creating an account on GitHub.

owlz avatar

That’s my next option. I was hoping to describe all my app layer in one file per cluster :<

owlz avatar

It’s not possible then? I should stick with separate files for values?

voron avatar
voron

Just move values to files from the beginning

 environments:
    default:
      values:
      - /path/to/values.yaml
voron avatar
voron

I’m not sure if this helps unfortunatelly

owlz avatar

It’s still a separate file ^^, ladno, spasibo!

1
Andrey Nazarov avatar
Andrey Nazarov

You’ll need either a values: block under each sub-helmfile definition or environments: inside each sub-helmfile manifest:)

Release notes from helmfile avatar
Release notes from helmfile
03:16:23 AM

v0.125.9: Merge pull request #1442 from roboll/fix-duplicate-with-kubectx-per-r… c575587 (HEAD, tag: v0.125.9, origin/master, origin/HEAD, master) Merge pull request <a class=”issue-link js-issue-link” data-error-text=”Failed to load title” data-id=”688441462” data-permission-text=”Title is private”…

Release v0.125.9: Merge pull request #1442 from roboll/fix-duplicate-with-kubectx-per-r… · roboll/helmfile

c575587 (HEAD, tag: v0.125.9, origin/master, origin/HEAD, master) Merge pull request #1442 from roboll/fix-duplicate-with-kubectx-per-release f2dfa7c Fix kubeContext not taken into account for rele…

Merge pull request #1442 from roboll/fix-duplicate-with-kubectx-per-r… · roboll/[email protected]

…elease Fix kubeContext not taken into account for release uniqueness

Release notes from helmfile avatar
Release notes from helmfile
06:36:21 AM

v0.126.0: Fix build error after helmfile build --embed-values addition cd0ecc5 (HEAD, tag: v0.126.0, origin/master, origin/HEAD, master) Fix build error after helmfile build –embed-values addition <a class=”commit-link” data-hovercard-type=”commit”…

Release v0.126.0: Fix build error after `helmfile build --embed-values` addition · roboll/helmfile

cd0ecc5 (HEAD, tag: v0.126.0, origin/master, origin/HEAD, master) Fix build error after helmfile build –embed-values addition dc6c59d Print command output in line (#1354) 0fc0869 feat: helmfile bu…

2020-08-27

jason800 avatar
jason800

Does anyone else still have problems with releases going to different kubeContext are still getting duplicate release errors?

STDERR:
  Error: release: already exists
voron avatar
voron

where do you specify kubeContext ?

jason800 avatar
jason800

at the release level

voron avatar
voron

what helmfile version do you use ?

jason800 avatar
jason800

v0.125.5

voron avatar
voron

could you pls reproduce this error on the latest version, and then retry with --debug to get actual helm arguments?

voron avatar
voron

I suppose release: already exists error is from helm, thus your helm may use wrong context. It’s better to test on single release using selector by release name

voron avatar
voron

we use context at helmDefaults and template level, don’t noticed any unfixed issues.

jason800 avatar
jason800
voron avatar
voron

does this release actually exist ?

helm --kube-context dev_eu-frankfurt-1_dataplane --namespace example ls -la 
voron avatar
voron

I suppose this issue is related to helm solely. Are you able to reproduce this issue by executing helm with all these args from debug log ?

jason800 avatar
jason800

well, i can see its issuing the helm release to the same kubecontext twice

jason800 avatar
jason800

that shouldn’t be happening

jason800 avatar
jason800

We are using a nested loop to generate our releases

jason800 avatar
jason800

but it shouldn’t produce this outcome. Here is an example

jason800 avatar
jason800
releases:
{{- range (list "us-ashburn-1" "us-phoenix-1") }}
  {{- $region := . }}
  {{- range (list "controlplane" "dataplane") }}
    {{- $okeKubeContext := (printf "%s_%s_%s" "dev" $region .) }}
  - name: example-grafana
    chart: bitnami/grafana
    kubeContext: "{{ $okeKubeContext }}"
  {{- end }}
{{- end }}
voron avatar
voron

ah, dynamic releases. I’m able to reproduce this issue - all the releases get the same kubeContext - the last generated one. It may be by design - you’re on the same single helmfile environment, kubeContext isn’t used as some uniq part to distinguish releases, and releases with the same name+namespace are merged in therms of kubeContex.

File a bug or submit a PR.

Interesting results are with

  - name: example-grafana
    chart: "{{ $okeKubeContext }}"
    kubeContext: "{{ $okeKubeContext }}"

chart is uniq, while k8s context is the same in debug output. This leads to expected release: already exists

But as soon as “release name + namespace” are uniq - no issues - k8s context is uniq too.

  - name: "{{ $okeKubeContext }}"
    chart: bitnami/grafana
    kubeContext: "{{ $okeKubeContext }}"
or
  - name: example-grafana
    chart: bitnami/grafana
    namespace: "{{ $okeKubeContext }}"
    kubeContext: "{{ $okeKubeContext }}"
jason800 avatar
jason800

I thought kubecontext WAS used as a unique part to distinguish a release? https://github.com/roboll/helmfile/pull/1390

jason800 avatar
jason800

or at least it was intended to be

jason800 avatar
jason800

and yes, all of your tests above mimic exactly what I’m seeing

jason800 avatar
jason800

@mumoshu just CC’ing you here because I brought this up to you before as well

jason800 avatar
jason800
Helmfile does not take kubeContext into account for unique releases · Issue #1440 · roboll/helmfile

Currently helmfile does not take kubeContext at the release level into the uniqueness of a release. For a use-case where you want to deploy the same chart to many clusters you have to use dynamic r…

voron avatar
voron

and now you don’t hit duplicate release error in helmfile, but you get actual duplicate releases.

:--1:1
jason800 avatar
jason800

thank you for all your help

voron avatar
voron

you’re welcome

voron avatar
voron

looks like issue is fixed by @mumoshu.

:--1:1

2020-08-26

Andrey Nazarov avatar
Andrey Nazarov

When using release templates what’s the difference between

values:
  - config/{{`{{ .Release.Name }}`}}/values.yaml

and

valuesTemplate:
  - config/{{`{{ .Release.Name }}`}}/values.yaml

? There is not much info about this.

Andrey Nazarov avatar
Andrey Nazarov
08:54:19 AM

It’s interesting that for secrets there is no template or it’s a typo

voron avatar
voron

Both do the same. *Template were added to use inside yaml templates to have an ability to merge template and release values. When you use values in template and values in release - release values override template values. Same with valuesTemplate - using it in both template and release leads to release valuesTemplate overrides template’s one. But when you use valuesTemplate in template and values in release(or vise versa) - resulting helm values are merged from values and valuesTemplate values. Similar with set/setTemplate

:--1:1
Andrey Nazarov avatar
Andrey Nazarov

Huge thanks! Is it documented somewhere and I missed it? Or you know it from inspecting the code/following issues?

voron avatar
voron

maybe here

roboll/helmfile

Deploy Kubernetes Helm Charts. Contribute to roboll/helmfile development by creating an account on GitHub.

voron avatar
voron

We use yaml templates, that’s why I’m aware of this.

Andrey Nazarov avatar
Andrey Nazarov

I’ve seen this, but without trying is hard to get the idea. You description is way better. Thanks again!

voron avatar
voron

np

muhaha avatar
muhaha

Guys? How can I override variables if I am using:


\# test-helmfile.yaml
environments:
  default:
    values:
      - foo: bar

helmfiles:
- ../../helmfiles.d/global-helmfile.yaml

\# global-helmfile.yaml
environments:
  default:
    values:
      - foo: {{ env "FOO" }}

repositories:
releases:
helmfile -f test-helmfile.yaml apply

Problem is that foo is empty and is overriden by nested helmfile ..

muhaha avatar
muhaha

I want to be able to deploy global-helmfile.yaml with environemnt variable support and also test-helmfile.yaml with static vars and possible release override. Any idea?

muhaha avatar
muhaha

Solved with:

helmfiles:
- path: ../../helmfiles.d/global-helmfile.yaml
  values:
   - foo: bar
muhaha avatar
muhaha

What does not work me is:


\# test-helmfile.yaml

helmfiles:
- path: ../../helmfiles.d/global-helmfile.yaml
  values:
    - foo: bar

\# global-helmfile.yaml

environments:
  default:
    values:
      - foo: {{ requiredEnv "FOO" }}

repositories:
releases:
error calling requiredEnv: required env var `FOO` is not set

foo var should be overriden with static foo value, isnt it a bug? @mumoshu

( Its working with {{ env "FOO" }}, but it can render empty value, if FOO is not specified, which is problematic )

Thanks

mumoshu avatar
mumoshu

foo: bar is merged AFTER global-helmfile.yaml is executed as a go template. so, it must be {{ env "FOO" }} if you need to override foo in the callee side from the caller

Alex avatar

hello all,

does anyone know if there is a way to specify environments per release? example:

environments:
  development:
  staging:
  production:  

templates:
  app: &app
    namespace: {{ .Environment.Name }}
    chart: charts/{{`{{ .Release.Name }}`}}
    missingFileHandler: Error
    values:
      - apps/{{`{{ .Release.Name }}`}}/values.{{ .Environment.Name }}.yaml
      - apps/{{`{{ .Release.Name }}`}}/version.{{ .Environment.Name }}.yaml
      
releases:
  - name: MyApp
    <<: *app
  - name: MySecondApp
    <<: *app

here, is it possible to specify that MySecondApp is only for development?

(I see ways around it like having 3 separate helmfile.<environment>.yaml or having multiple templates but not ideal)

voron avatar
voron

one possible option is to use installed/installedTemplate in your template or release

installedTemplate: {{ eq .Environment.Name "development" | toYaml }}
voron avatar
voron

another possible option is to use similar condition

environments:
  development:
    values:
    - MySecondApp:
        enabled: true
  staging:
    values:
    - MySecondApp:
        enabled: false
  production:
    values:
    - MySecondApp:
        enabled: false
templates:
  ...
releases:
  - name: MyApp
    <<: *app
  - name: MySecondApp
    condition: MySecondApp.enabled
    <<: *app

Of course, you may use files to load environment values instead of inlining

Alex avatar

thank you, that helps

2020-08-25

jason800 avatar
jason800

Hey @mumoshu, just ran across this funny little message

err 0: glob patterns in release values and secrets is not supported yet. please submit a feature request if necessary
err 1: glob patterns in release values and secrets is not supported yet. please submit a feature request if necessary
mumoshu avatar
mumoshu

apparently you tried to write something similar to

values:
- whatever/*.yaml

which is not yet supported

jason800 avatar
jason800

Haha yup. Was nice to get such a thorough error message

mumoshu avatar
mumoshu

glad to hear it worked! btw, pls feel free to open a feature request if you really need it

jason800 avatar
jason800

Just wrote a small bash script to print out all the files in the dir in a yaml format and use an exec

:--1:1

2020-08-24

4c74356b41 avatar
4c74356b41

hey folks, I’m having a really weird issue, where I’m using the same container for local testing and for CI (container has helmfile, terraform, helm, etc) and after I deploy from my local, CI build redeploys everything with a really weird diff:

+ apiVersion: [networking.istio.io/v1alpha3](http://networking\.istio\.io/v1alpha3)
+ kind: VirtualService
+ metadata:
+   name: raven
+   namespace: ccc-testing
    labels:
      [app.kubernetes.io/name](http://app\.kubernetes\.io/name): raven
      [app.kubernetes.io/instance](http://app\.kubernetes\.io/instance): testing
      [app.kubernetes.io/component](http://app\.kubernetes\.io/component): raven
      [app.kubernetes.io/part-of](http://app\.kubernetes\.io/part\-of): ccc
-     [app.kubernetes.io/managed-by](http://app\.kubernetes\.io/managed\-by): Helm
- spec:
-   hosts:
-   - xxx
-   gateways:
-   - yyy
-   http:
-   - headers:
-       response:
-         set:
-           Strict-Transport-Security: "max-age=31536000; includeSubDomains; preload"
-     route:
-     - destination:
-         host: raven
-         port:
-           number: 8080
-     timeout: 10000s
+     [app.kubernetes.io/managed-by](http://app\.kubernetes\.io/managed\-by): Helm
+ spec:
+   hosts:
+   - xxx
+   gateways:
+   - yyy
+   http:
+   - headers:
+       response:
+         set:
+           Strict-Transport-Security: "max-age=31536000; includeSubDomains; preload"
+     route:
+     - destination:
+         host: raven
+         port:
+           number: 8080
+     timeout: 10000s
4c74356b41 avatar
4c74356b41

so essentially its exactly the same contents of the file. any ideas what to look at? and this happens with every rendered file

2020-08-20

Sam Buckingham avatar
Sam Buckingham

Does anyone know if you can use remote helm chart values? For instance:

releases:
- name: ingress-internal-{{ requiredEnv "PLATFORM_ENV" }}  # name of this release
    namespace: ingress    
    chart: ingress
    values:
    - git::<ssh://[email protected]:ORG/REPO.git//values/ingress/@internal_{{> requiredEnv "PLATFORM_ENV" }}.yaml?ref=master 
    wait: true
    condition: ingress-internal.enabled
    atomic: true
    installed: true
voron avatar
voron

helmfile allows to get values from exec, thus you may adapt it with git as a source too

roboll/helmfile

Deploy Kubernetes Helm Charts. Contribute to roboll/helmfile development by creating an account on GitHub.

Andrey Nazarov avatar
Andrey Nazarov
Proposal: add option for remote values files located in other repos · Issue #469 · roboll/helmfile

Hey, I think it would be awesome to have the option of pulling values files over http/s and git ssh. Something like: values: - https://some-url.com/master/values.yaml.gotmpl - <git://[email protected]>:…

Add the ability to load a remote environment values file by kevinjqiu · Pull Request #1296 · roboll/helmfile

This PR enables the user to specify a remote path for an environment values file, e.g., environments: cluster-azure-us-west: values: - git://git.company.org/helmfiles/global/azur>…

voron avatar
voron

I was thinking https://github.com/roboll/helmfile/pull/1296 is related to values inside environment solely and doesn’t affect release/template values

Add the ability to load a remote environment values file by kevinjqiu · Pull Request #1296 · roboll/helmfile

This PR enables the user to specify a remote path for an environment values file, e.g., environments: cluster-azure-us-west: values: - git://git.company.org/helmfiles/global/azur>…

Andrey Nazarov avatar
Andrey Nazarov

Ah, @ probably you are right. Thanks for a tip

2020-08-19

2020-08-17

vgdubrea avatar
vgdubrea

anybody has a example for helmfile template . I am very confused of how this command is expected to render what I desire it to , at the moment what I figured is if I run this command in a directory it by default reads helmfile.yaml and start to render the releases in this main file. would appreciate if I can see some examples of it.

2020-08-16

2020-08-15

Pierre Humberdroz avatar
Pierre Humberdroz

just saw that even gitlab is using helmfile => https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com

Issif avatar
Issif

They propose some apps in their managed k8s clusters and for installing them they use helm right. A apps I develop will be added in a next release and that’s why I added it on helm hub last week

:--1:1
Pierre Humberdroz avatar
Pierre Humberdroz

I am just positively surprised by their adoption. What app did you work on ?

Issif avatar
Issif
falcosecurity/falcosidekick

A simple daemon to help you with falco’s outputs. Contribute to falcosecurity/falcosidekick development by creating an account on GitHub.

Issif avatar
Issif

falco has been integrated by gitlab in 13.2

Pierre Humberdroz avatar
Pierre Humberdroz

oh that is nice! But what I mean is they are using helmfile to setup their own infrastructure (what runs gitlab.com) as well now..

:--1:1
Issif avatar
Issif

Oh I get your point now, great

vgdubrea avatar
vgdubrea

@Pierre Humberdroz Good find , But I am just wondering why are you positively surprised? I mean is this something that you expect they could have rolled their own solution with helm ? we primarily use gitlab.com for everything we do with our customers and are heavy consumers of pretty much all the solutions they offer.

Pierre Humberdroz avatar
Pierre Humberdroz

No but a lot of people I talk to are not even aware that helmfile exists. And seeing some bigger company adopting it makes me happy! I also wanted to share how others set their helmfiles up.

1

2020-08-14

2020-08-13

s_slack avatar
s_slack

Eks+helm: I have many applications using helm charts but these charts also have a secrets.yaml. This contains an application.properties encrypted. I want to remove the username/password from the property file and store it in AWS SSM. This way I would be able to decrypt the property file, have its values in helmfile. After that create a ConfigMap template that gets populated based on the property values from helmfile. I don’t want to commit the username/password so I would store it in AWS SSM. Though not sure if this is a good idea or how to pass the username/password to the deployment object and not have it in plain text. GoDaddy has this https://ca.godaddy.com/engineering/2019/04/16/kubernetes-external-secrets/ and I wonder if there are any other alternatives. Cheers Ps. I use helm + helmfile + helm-git

Kubernetes External Secrets — GoDaddy Engineering Blog

Engineering teams at GoDaddy use Kubernetes with secret management systems, like AWS Secrets Manager.

bradym avatar
bradym

Helmfile has native support for getting values from SSM as it uses vals (https://github.com/variantdev/vals)

variantdev/vals

Helm-like configuration values loader with support for various sources - variantdev/vals

bradym avatar
bradym
roboll/helmfile

Deploy Kubernetes Helm Charts. Contribute to roboll/helmfile development by creating an account on GitHub.

bradym avatar
bradym

The remote-secrets.md file has some very basic examples, and the vals readme has a section on using SSM to get more info on how to use it

s_slack avatar
s_slack

Nice. That simplifies a bunch. Though how do I handle the fact that I need username/password either in application.properties or as env variable. Though I don’t want it in plain text in the pod or in git.

bradym avatar
bradym

Create a k8s secret using helmfile, and load that secret as environment variables in the pod.

s_slack avatar
s_slack

That sounds much easier than I thought. I will try it out. Thank you @bradym

bradym avatar
bradym

np

s_slack avatar
s_slack

Curious. Is it common practice to put password in secrets file not encrypted (just encoded base64)? Even though it retrieves from SSM one could easily get the password from secrets

s_slack avatar
s_slack

I have a SSM parameter with the following values:

param1: value1
param2: value2

I use in values.yaml.gotmpl

...
Secrets:
{{ .Environment.Values.someapp | expandSecretRefs | toYaml | nindent 2 }}

in someapp.yaml

...
- releases:
  - name: some-app  
    values:
      - ../releases/someapp/values.yaml.gotmpl
      - Secrets:
        - <ref+awsssm://someapp/testssm?region=us-west-2>

in secrets.yaml

...
data:
{{- range $key, $value := .Values.Secrets}}
  {{ $value | b64enc | quote | indent 2 }}
{{- end }}

returns:

...
+ data:
+   0: "param1: value1\nparam2: value2"

I can’t seem to be able to get the $key= param1 and $value= value1, etc for those variables. The SSM entry gets populated in $value only, $key is always empty.

bbhupati avatar
bbhupati

Alternatively, you can try this option https://thefirstapril.com/2019/06/29/Secret-Management-in-Helm-Kubernetes/ Use helm secrets plugin to encrypt/decrypt the secret file with AWS KMS and pass the encrypted file to helmfile so that all the secret files which are stored in version control system are encrypted #helm secrets enc secrets.yaml releases:

  • name: test namespace: default version: “0.2.0” chart: test set:
    • name: “clusterName” value: test secrets:
    • secret.yaml
Secret Management in Helm - Kubernetes

Secret management in HelmHelm is a Kubernetes package manager, Helm helps developer deploy their application to Kubernetes. Helm also provide chart as dependencies for your application at https://hub.

2020-08-12

jason800 avatar
jason800

Does anyone else know if its a bug or feature that you can’t do an apply to a helmfile which contains two releases which have a CRD/CR relationship because of the helm-diff ? The diff will run on both releases before the apply and the second release will fail becaues the CR it tries to diff doesn’t exist as a type yet, since its created in the first chart by a CRD

Andrey Nazarov avatar
Andrey Nazarov

You might want to use disableValidation flag to mitigate this

:--1:1
TJMiller avatar
TJMiller

can i use vals for helm repo credentials?

2020-08-11

jason800 avatar
jason800

Has anyone run into issues before when deploying releases in the same helmfile using needs: but the latter releases are using a custom resource created by a CRD from the earlier releases?

STDERR:
  Error: Failed to render chart: exit status 1: Error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "Vault" in version "[vault.banzaicloud.com/v1alpha1](http://vault\.banzaicloud\.com/v1alpha1)"
  Error: plugin "diff" exited with error
Andrey Nazarov avatar
Andrey Nazarov

Yes. A recently introduced disableValidation flag might help. I’ve just mention this in a different thread:)

1
jason800 avatar
jason800

the Vault custom resource is created in the release previous and they are forming a dependency with needs:

2020-08-10

2020-08-08

Vadim Bauer avatar
Vadim Bauer

Hello could someone explain how condition is working, somehow I don’t get it

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Hrm.. I’m not familiar with the condition in helmfile - we haven’t used it, but my guess is that it works like the installed flag?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

My guess is that borrows from the upstream convention in the Chart.yaml . See https://helm.sh/docs/topics/charts/#the-chartyaml-file

Charts

Explains the chart format, and provides basic guidance for building charts with Helm.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Anyone know the difference between condition and installed?

voron avatar
voron

condition with false doesn’t delete the release, it’s just ignored, like there is no such release. installed/installedTemplate with false will delete --purge release.

voron avatar
voron

Taken from state.go

roboll/helmfile

Deploy Kubernetes Helm Charts. Contribute to roboll/helmfile development by creating an account on GitHub.

voron avatar
voron
    op    |  true   |    false       |
--------------------------------------
installed | install | delete --purge |
condition | install | skip/ignore    |

kind of

:--1:1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

hrm… then how is condition different from enabled? sounds like condition is the same as enabled

voron avatar
voron

do we have a keyword enabled in helmfile?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

hah, you know - I must be confused. Doesn’t look like it and then it all makes more sense.

2020-08-07

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@here So we’ve been a helmfile user for many years now. I think there were only a couple dozen stars on the repo when we started, now there are thousands. The project has transformed dramatically during this time and we’ve all benefited immensely from this. I just want to say how grateful I am for everyones help on this, but especially your’s @mumoshu because it would not have been possible without it. I sincerely don’t understand how you can be so productive on all these projects (helmfile, helm-diff, eksctl, brigade, helm, kube-aws, etc) and squash bugs as fast and tirelessly as you do, but you still manage to pull it off. If there ever was such a thing as a 10x engineer, you’re the definition of it. Anyways, we appreciate all the tremendous effort you put forth not just on coding but also on supporting feature requests, answering questions, and just always being so helpful. I regret not stepping up sooner, but just want to share our support and hope others will too! https://github.com/sponsors/mumoshu

Sponsor @mumoshu on GitHub Sponsors

Support mumoshu’s open source work

8
6
:1000:7
13
Andrey Nazarov avatar
Andrey Nazarov

Just a few hours ago I had a conversation with my colleagues and I mentioned @mumoshu in a way that it’s hard to imagine how he does so many things with such a quality and passion. An example for all of us.

Sponsor @mumoshu on GitHub Sponsors

Support mumoshu’s open source work

:1000:6
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

helmfile + variant2 =

mumoshu avatar
mumoshu

I can’t tell you how much I appreciate all your help!

Honestly, I had mixed feelings working on OSS. It’s too tough in regard to how much time and effort I need to put onto it, while I’ve been grateful for having chances of any kinds of collaborations with you all.

Discussing about potential bug, get to distill it into the root cause, finally fixing it together. Receiving feature requests, talking about use-cases, come up with a solution together. Prototyping something and receiving early feedback about it(recent examples would be terraform-provider-helmfile and variant2). I think I love all of those experiences and processes while working on OSS - probably that’s why I’ve been maintaining OSS like Helmfile until now.

Sponsorship - I think this is my first time to receive sponsorships more than one or two at once. Simply I had never realized I would receive such many supports! Combining all my experiences so far, your kind words, and this time, sponsorships, I’m feeling encouraged and motivated more than ever for my work now .

Thank you all for continuing to work with me. Expect me to keep this

// Excuse me for slow response. I had to have some time to organize what I felt

4
voron avatar
voron

more than 30 likes, but just 8 sponsors on github

2020-08-06

Florent Valdelievre avatar
Florent Valdelievre

Is it possible to deploy a raw kubernetes manifest in helmfile/helm without using kubectl? For example, If i want to deploy a kubernetes ingress only.

jason800 avatar
jason800

I think recently this became supported? something with go-getter

jason800 avatar
jason800
Add ability to use go-getter for fetching remote manifests directory … · roboll/[email protected]

…as chart (#1374) This, in combination with #1172, allows you to use go-getter-supported URL for K8s manifests on chart, so that Helmfile automatically fetches it and then turning it into a te…

jason800 avatar
jason800
Add ability to use go-getter for fetching remote manifests directory as chart by mumoshu · Pull Request #1374 · roboll/helmfile

This, in combination with #1172, allows you to use go-getter-supported URL for K8s manifests on chart, so that Helmfile automatically fetches it and then turn it into a temporary local chart, which…

Florent Valdelievre avatar
Florent Valdelievre

Do you mean I should be able to deploy like that ?

releases:                                                                                                                                                                                                         
  - name: ingress                                                                                                                                                                                                 
    chart: ./raw/ingress-dev.yaml
Florent Valdelievre avatar
Florent Valdelievre

I have the following error returned:

COMBINED OUTPUT:                                                                                                                                                                                                  
  Error: only unpacked charts can be updated                                                                                                                                                                      
jason800 avatar
jason800

there is an example in the pull request

jason800 avatar
jason800

I’ve never used it personally

jason800 avatar
jason800

he targets `

git::<http://github.com/jetstack/[email protected]/crds?ref=v0.15.2>
jason800 avatar
jason800

which is here https://github.com/jetstack/cert-manager/tree/master/deploy/crds (except this is master and not that tag)

jetstack/cert-manager

Automatically provision and manage TLS certificates in Kubernetes - jetstack/cert-manager

jason800 avatar
jason800

but the tl;dr is that its a repo directory with only a bunch of yaml manifests

jason800 avatar
jason800

Which I believe is what your goal is ?

Florent Valdelievre avatar
Florent Valdelievre

Oh yes its working if I just pass the folder ./raw instead of ./raw/ingress-dev.yaml

1
jason800 avatar
jason800

So, I guess in your case if you wanted a single manifest you’d need to do a directory with just that single manifest inside it

Florent Valdelievre avatar
Florent Valdelievre

Yes I guess so. Thanks a lot for your finding!

1
jason800 avatar
jason800

In practice do you guys just apply via sync or apply with helmfile in your ci/cd ?

voron avatar
voron

helmfile apply with helm2 in dev env

Shawn Petersen avatar
Shawn Petersen

No

muhaha avatar
muhaha

Guys? I am getting:

error during gitlab-test2.yaml.part.0 parsing: template: stringTemplate:22:24: executing "stringTemplate" at <.Values.foo.common.kind>: map has no entry for key "foo"

Not sure whats wrong…

environments:
  default:
    values:
        foo:
          common:
            kind: Secret

repositories:
- name: incubator
  url: <https://kubernetes-charts-incubator.storage.googleapis.com>

releases:

- name: test
  namespace: test
  createNamespace: true
  chart: incubator/raw
  version: 0.2.3
  values:
    - resources:
      - apiVersion: v1
        kind: {{ .Values.foo.common.kind }}
muhaha avatar
muhaha
feat: Predictable Helmfile template · Issue #932 · roboll/helmfile

TL;DR; I want to add a new helmfile.yaml field to make templating helmfile configs easier. Problem Helmfile&#39;s double-rendering has opened a wide variety of use-cases that requires you to write …

Andrey Nazarov avatar
Andrey Nazarov

What about

environments:
  default:
    values:
      - foo:
          common:
            kind: Secret

?

:--1:1
muhaha avatar
muhaha

Works! Thanks I already tried:

environments:
  default:
    values:
      - foo:
          common:
            kind: Secret
repositories:
- name: incubator
  url: <https://kubernetes-charts-incubator.storage.googleapis.com>
releases:
- name: test
  namespace: test
  createNamespace: true
  chart: incubator/raw
  version: 0.2.3
  values:
    - resources:
      - apiVersion: v1
        kind: "{{ .Values.foo.common.kind }}"

which is not working…

mumoshu avatar
mumoshu

sry this is very confusing indeed, but the root cause here is that environments.NAME.values needs to be a yaml array, rather than hash

mumoshu avatar
mumoshu

to allow merging multiple values entries

mumoshu avatar
mumoshu

so it should be

environments:
  default:
    values:
    - foo:
        common:
          kind: Secret
mumoshu avatar
mumoshu

the best practice is to separate environments and other parts of your helmfile.yaml with --- like

environments:
  default:
    values:
        foo:
          common:
            kind: Secret

---

repositories:
- name: incubator
  url: <https://kubernetes-charts-incubator.storage.googleapis.com>
releases:
- name: test
  namespace: test
  createNamespace: true
  chart: incubator/raw
  version: 0.2.3
  values:
    - resources:
      - apiVersion: v1
        kind: {{ .Values.foo.common.kind }}

so you get a much better error

mumoshu avatar
mumoshu
in ./helmfile2.yaml: failed to read helmfile2.yaml: reading document at index 1: yaml: unmarshal errors:
  line 4: cannot unmarshal !!map into []interface {}

which tells that you’re mistakenly trying to define values as map, rather than array([])

muhaha avatar
muhaha

Guys? I am trying to add gitlab helm chart repository:

repositories:
- name: custom
  url: git+<https://gitlab.com/gitlab-org/charts/gitlab?ref=v4.2.1&sparse=0>

releases:
- name: gitlab
  namespace: gitlab
  createNamespace: true
  chart: custom/gitlab

, but getting:

COMBINED OUTPUT:
  Error in plugin 'helm-git': Target file name has to be either 'index.yaml' or a tgz release
  Error: looks like "git+<https://gitlab.com/gitlab-org/charts/gitlab?ref=v4.2.1&sparse=0>" is not a valid chart repository or cannot be reached: plugin "helm-git" exited with error

Any idea? Of course that I am aware of official charts ( another thing is that gitlab helm chart is somehow complicated, its using nested subcharts )

Thanks

Issif avatar
Issif

You can’t use a git repository as source for helm

Issif avatar
Issif

your repository must be for helm chart, with a specific index.html for listing available charts, etc

Issif avatar
Issif

gitlab provides a repo : <https://charts.gitlab.io/>

muhaha avatar
muhaha

@Issif of course i can https://github.com/aslafy-z/helm-git, point is not to use charts from https://charts.gitlab.io/

aslafy-z/helm-git

Helm plugin to fetch charts from Git repositories. Contribute to aslafy-z/helm-git development by creating an account on GitHub.

Michael Seiwald avatar
Michael Seiwald

Hi, is it really not possible to just specify a plain HTTPS URL for a sub-helmfile like this?

helmfiles:
  - path: <https://our-nexus/bla/bla/bla>
  - path: git::<ssh://[email protected]>.....

The first URL gives a no matches for path error message. The second (git::ssh ) works..

Michael Seiwald avatar
Michael Seiwald

I found out that the URL has to be <https://our-nexus/><dir>@<file> but then helmfile tries to fetch just the directory which leads to a HTTP 404..

voron avatar
voron

<http://example.com/[email protected]> isn’t a http url

Michael Seiwald avatar
Michael Seiwald

True, but it’s what helmfile expects. Otherwise you get an error like this (after digging into the code and fmt.Printing some errors that are otherwise not exposed to the user)

invalid src format: it must be `[<getter>::]<scheme>://<host>/<path/to/dir>@<path/to/file>?key1=val1&key2=val2: got <https://my.ordinary.http.url/dir/file>
Michael Seiwald avatar
Michael Seiwald

So something like <http://example.com/dir/file> is not even considered by helmfile

Michael Seiwald avatar
Michael Seiwald

It kind of makes sense because helmfile needs the whole directory since the remote helmfile can also reference remote files.

Michael Seiwald avatar
Michael Seiwald

But you cannot “download” a directory over http which makes me think that HTTP ist just not supported at all for remote statefiles?

2020-08-05

Jonathan avatar
Jonathan

I’m trying to get some secrets from gcp secret manager into a string, but I’m having some issues. What I have is a reference to a secret in my common/dev.yaml file like so:

key1:
  user: "<ref+gcpsecrets://projectname/secret-name?version=latest#/path/to/user>"
  password: "<ref+gcpsecrets://projectname/secret-name?version=latest#/path/to/password>" 

The issue is that when I try to use the value of the secrets in a values.gotmpl file like so: string: "{{ .Values.key1.user }}:{{ .Value.key1.password}}" I get the actual string "<ref+gcpsecrets://projectname/secret-name?version=latest#/path/to/secret>" from each value, rather than the value of what the secret reference is. Is there any way to force the reference to be evaluated before being passed along? Or should i structure this some other way? The reason I have it like this is because I need to have the values combined into a string so that the code that is using these values is happy.

voron avatar
voron

did you tried something similar to

string: "{{`{{ .Values.key1.user }}:{{ .Value.key1.password}}`}}"

?

mumoshu avatar
mumoshu
roboll/helmfile

Deploy Kubernetes Helm Charts. Contribute to roboll/helmfile development by creating an account on GitHub.

Jonathan avatar
Jonathan

using fetchSecretValue worked, thanks!

2

2020-08-04

Florent Valdelievre avatar
Florent Valdelievre

Hi, I have a remote chart including somes yaml value files (i.e values-dev.yaml) I thought I would have been able to do that:

releases:
  - name: xxx
    chart: helm-private-repo/foo
    values:
      - values-dev.yaml

But I have the following response:

in ./helmfile.yaml: failed processing release xxx: values file matching "values-dev.yaml" does not exist in "."

It is working when the chart is local but not working when the chart is remote. Is there a way to tell helmfile to use values-dev.yaml from the downloaded chart instead of trying to find values-dev.yaml in ‘.’ ?

voron avatar
voron

So, you wanna pull values from remote instead of local ?

Florent Valdelievre avatar
Florent Valdelievre

well, values are already in the chart. I was hoping that I could just reference the file I want to use

Florent Valdelievre avatar
Florent Valdelievre

I know I can put an URL to retrieve values but its a real shame since everything is already included in the chart

voron avatar
voron

helmfile doesn’t have access to the chart, it just calls helm

Florent Valdelievre avatar
Florent Valdelievre

Ok, and helm is not able to use values from a remote chart then? the file must be present when we run helm -f xxx ?

voron avatar
voron

you need to get values into helmfile, thus shell script(helm3 show values?) or remote yaml file in helmfile is your option.

Florent Valdelievre avatar
Florent Valdelievre

ok Thanks for confirming. Shame this is not possible straight away

voron avatar
voron

I don’t think that keeping unused values file in the chart is a common practice.

Florent Valdelievre avatar
Florent Valdelievre

Ok. So It maybe make sense to move values-XXX.yml along with helmfile instead of keeping it in the chart

voron avatar
voron

this looks a common practice. Keep charts and settings separated. You hit some sensitive data in settings usually and you cannot store it in the chart anyway.

Florent Valdelievre avatar
Florent Valdelievre

Ok Thanks for the advice. I saw couple of charts storing values-XXX.yaml in their repo, but not that many. I will keep them separated.

voron avatar
voron


storing values-XXX.yaml in their repo
I suppose it was a git repo, not a helm repo with charts archives.

Florent Valdelievre avatar
Florent Valdelievre

Its a git repo, but its in the chart archive. Example: https://github.com/bitnami/charts/tree/master/bitnami/rabbitmq

voron avatar
voron

It looks like additional values file cannot be used by helm directly from the chart archive. Manual specifies values.yaml only.

Charts

Explains the chart format, and provides basic guidance for building charts with Helm.

voron avatar
voron
NOTE: The default values file included inside of a chart must be named values.yaml.
Florent Valdelievre avatar
Florent Valdelievre

yes for this case this is clear. I just would have liked a comment on the helm website that we can’t specify custom value file even if its included in the chart

voron avatar
voron

Create a PR

Florent Valdelievre avatar
Florent Valdelievre

Yes

Florent Valdelievre avatar
Florent Valdelievre

otherwise, would the following work ? I haven’t tried it yet: https://github.com/zendesk/helm-secrets/issues/61#issuecomment-429053736 Its pulling the chart locally through hooks

voron avatar
voron

yep, this should be fine except it will trigger on every helmfile

2020-08-03

voron avatar
voron

Does anyone use helm3 in CI/CD ( maybe inside helmfile too)? How do you deal with missing real --force in helm3 compared to helm2? It looks like https://github.com/helm/helm/pull/7431 isn’t going to be merged to add real force to helm3.

feat(helm): add recreate upgrade (rollback) strategy by dastrobu · Pull Request #7431 · helm/helm

An additional optional flag –recreate can be passed on upgrade (or rollback) of a release. In combination with the –force flag the following strategies are employed when updating a resource (whic…

Vincent Behar avatar
Vincent Behar

you can put these specific resources in hooks, see https://github.com/kubernetes/kubernetes/issues/45398#issuecomment-478619253 for an example

PodDisruptionBudget updates are forbidden · Issue #45398 · kubernetes/kubernetes

PodDisruptionBudget is immutable at this moment. It would be a nice improvement to allow changes to it, at the least to the minAvailable field. Not sure whether this counts as a bug or a feature re…

voron avatar
voron

Are you talking about delete&recreate on every helm upgrade, including cases when immutable resource isn’t changed ?

Vincent Behar avatar
Vincent Behar

yes. well of course it may or may not apply to your use-cases, depending on which resources you need to “force update”. In our case for example, we used to do it on PDB, and now we’re doing it on Jobs, so deleting and recreating is not an issue

Vincent Behar avatar
Vincent Behar

but I agree that it’s a pain

voron avatar
voron

I hit this with Service and StatefulSet

Vincent Behar avatar
Vincent Behar

which part of the spec ?

voron avatar
voron
[statefulset example> , <https://github.com/helm/helm/issues/6378 service example](https://gist.github.com/voron/6954a6af3a193ad2ae7ba1d34bbf9b00#file-gistfile1-txt-L25)
Helm 3 - upgrade nginx - spec.clusterIP: Invalid value: "": field is immutable · Issue #6378 · helm/helm

I use the following to install / upgrade a chart: ./helm upgrade –install –set rbac.create=false –set controller.replicaCount=2 –set controller.service.loadBalancerIP=$ip –wait main-ingress st…

voron avatar
voron

do helm hooks require to add on-the-fly helmfile json patches to stock charts to apply hooks to these charts?

Vincent Behar avatar
Vincent Behar

ah yes, the volumeClaimTemplates in the statefulset, we had this issue too…

Vincent Behar avatar
Vincent Behar

hopefully the pvc size won’t change often, you can do a 1-time operation (backup, delete, recreate, restore data)

voron avatar
voron

PVC isn’t affected during statefulset recreation

voron avatar
voron

so backup+restore isn’t required

Vincent Behar avatar
Vincent Behar

about json patches, that depends on the charts, if they support adding annotations or not

voron avatar
voron

and it’s still a semi-solution just to emulate helm2 behaviour with drawbacks like recreating the resource every time

voron avatar
voron

I have helmfile apply over 50 releases in a dev CI/CD pipeline, and it works fine, but it’s helm2, not 3.

Andrey Nazarov avatar
Andrey Nazarov

We are dealing with this manually. I.e. we do helm delete and then trigger pipelines with helmfile and stuff. Not an ideal solution for sure.

voron avatar
voron

yes, donwtime and side effects are everywhere

hari avatar

Hi Everyone,

hari avatar

i have come across a interesting requirement in my helm charts

hari avatar
kind: Deployment
spec:
  template:
    metadata:
      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}

in this config , i need to change the configmap.yaml name dynamically w.r.t environment Like this one checksum/config: {{ include (print $.Template.BasePath “/configmap-{{ .Release.Name }}.yaml”) . | sha256sum }}

hari avatar

OR checksum/config: {{ include (print $.Template.BasePath “/configmap-{{ .Values.env }}.yaml”) . | sha256sum }}

voron avatar
voron

Do you wanna mix helm gotmpl with helmfile gotmpl in the same line ?

hari avatar

Does anyone have come across this situation ?

hari avatar

unfortunately, none of these will work

2020-08-02

2020-08-01

Paul Catinean avatar
Paul Catinean

@mumoshu Not sure if this is a usual/tipical scenario but i started using helmfile in my CI/CD pipeline and one requirement I have is after the deployment is succesful (this can be confirmed with the wait parameter) I need to execute a manual command inside the deployed pod. For this I use the helmfile list with labels in order to identify the helm release name and then get the pod from the release name. Not sure if there’s a better way to do this? I parse the output of helme list with json (which is why I also got an issue opened here https://github.com/roboll/helmfile/issues/1270)

Json output for labels · Issue #1270 · roboll/helmfile

When doing a helmfile list –output=json the labels presented in the output are of csv format. I do a lot of jq parsing to get meaningful information from the releases for other external scripts an…

Issif avatar
Issif

You could use a job with post-install helm hook

Json output for labels · Issue #1270 · roboll/helmfile

When doing a helmfile list –output=json the labels presented in the output are of csv format. I do a lot of jq parsing to get meaningful information from the releases for other external scripts an…

Paul Catinean avatar
Paul Catinean

That post-install trigger only if the deployment is succesfull?

Paul Catinean avatar
Paul Catinean

How would I target the job strictly at the container from the deployment?

Issif avatar
Issif

I will run in another container inside your pod

Issif avatar
Issif

it’s always a bad pattern to run a command inside a running container

Paul Catinean avatar
Paul Catinean

Agreed but in this case i need to execute a binary inside the container

Issif avatar
Issif

you can run a job with exact same image I guess https://helm.sh/docs/topics/charts_hooks/

Chart Hooks

Describes how to work with chart hooks.

Issif avatar
Issif

if it runs in same pod, it will have same environment

Issif avatar
Issif

you use this in pre-install

Paul Catinean avatar
Paul Catinean

I remember going this route and having a problem but i will take a look once again and see what comes up

Issif avatar
Issif

for our staging, it deploys database, kafka schema, etc, and after, helm deploys ours pods

Paul Catinean avatar
Paul Catinean

Ah I remember one thing is to see the log output in the ci/cd pipeline and also have it fail if there’s an error

Issif avatar
Issif

about what?

mumoshu avatar
mumoshu

I think postinstall should be what you want. It’s called only after successful install.

In the hook command, perhaps using

kubectl get po -l release={{` {{.Release.Name}} `}}

and then kubectl exec on the pod would work?

mumoshu avatar
mumoshu

postsync is called after install/upgrade regardless of success/failure, and you should be able to check {{.Error}} for emptiness to determine if it was successful or not. try that instead of postinstall you need testing after upgrade, too.

Issif avatar
Issif

@mumoshu postinstall are run locally? I thought hook was only for resource creations like jobs

mumoshu avatar
mumoshu

hooks can be any local processes. you’ll usually use preopare and cleanup hooks for creating/deleting charts and presync for creating resources not managed by required by release

mumoshu avatar
mumoshu

for postsync… im not sure. i thought i heard anyone used it for notification

mumoshu avatar
mumoshu

and postinstall is a specialization of postsync

Issif avatar
Issif

1
Paul Catinean avatar
Paul Catinean

Maybe I’m going about it the wrong way or something

    keyboard_arrow_up