#helmfile (2020-09)

https://github.com/roboll/helmfile

Questions and discussion around helmfile https://github.com/roboll/helmfile and https://github.com/cloudposse/helmfiles Archive: https://archive.sweetops.com/helmfile/

2020-09-23

vixus0 avatar
vixus0

Are there any best practices for annotating a namespace that gets created by Helm during sync?

Bart M. avatar
Bart M.

anyone here who uses the vals tool standalone?

Bart M. avatar
Bart M.

not sure where I would have to go with questions about this, seems pretty interlinked with helmfile

Andrey Nazarov avatar
Andrey Nazarov

You may try to ask it in this channel since mumoshu is always here.

vixus0 avatar
vixus0

How do I use bases: when I have a directory of helmfiles?

vixus0 avatar
vixus0

I have

bases:
- ../base.yaml

at the top of every helmfile but it seems to get ignored

voron avatar
voron

sometimes we use something like {{- tpl (readFile "../common/templates.yaml") . | nindent 0 }} instead of bases

vixus0 avatar
vixus0

hmm, I guess it’s because bases get combined as independent YAML documents

vixus0 avatar
vixus0

so if I define templates: in a base, it isn’t useable

vixus0 avatar
vixus0

@ is {{- readFile "../common/templates.yaml" | toYaml }} not sufficient?

voron avatar
voron

toYaml shouldn’t be required imho

voron avatar
voron

try and get an answer

vixus0 avatar
vixus0

hmm, I just seem to end up with a blank string (even though the file exists)

vixus0 avatar
vixus0

oh nevermind it only appears on the third pass

vixus0 avatar
vixus0

any ideas what this error means: err: no releases found that matches specified selector() and environment(local), in any helmfile? This only seems to happen when I have a helmfiles section.

vixus0 avatar
vixus0

the YAML that gets generated looks fine to me:

 3: repositories:
 4:   - name: incubator
 5:     url: <https://kubernetes-charts-incubator.storage.googleapis.com>
 6:   - name: uswitch
 7:     url: <https://uswitch.github.io/kiam-helm-charts/charts>
 8:   - name: bitnami
 9:     url: <https://charts.bitnami.com/bitnami>
10:   - name: traefik
11:     url: <https://containous.github.io/traefik-helm-chart>
12:   - name: pomerium
13:     url: <https://helm.pomerium.io>
14:
15: helmfiles:
16:   - path: ./cluster.d
17:     values:
18:       - ../deployments/local/config.yaml.gotmpl
19:   - path: ../deployments/local/helmfile.d
20:     values:
21:       - ../deployments/local/config.yaml.gotmpl
voron avatar
voron

I suppose you need to define environments in your root helmfile

voron avatar
voron

That’s what we have in our root helmfile

bases:
  - "common/repositories.yaml"
  - "common/helm-defaults.yaml"
environments:
  dev:
  prod:
helmfiles:
  - dev/helmfile.yaml
  - prod/helmfile.yaml

and in env-specific prod/helmfile

bases:
  - "../common/repositories.yaml"
  - "../common/helm-defaults.yaml"

environments:
  prod:
{{- tpl (readFile "../common/templates.yaml") . | nindent 0 }}

releases:
...
vixus0 avatar
vixus0

oh I see, so you can include all the environment-specific helmfiles but they only get selected if they have a key in the environments block

vixus0 avatar
vixus0

do you have a way of sharing releases between environments (with different values)?

voron avatar
voron

use the same helmfile with multiple envs inside

voron avatar
voron

use installed(installedTemplate):/condition: to get conditional deploy.

voron avatar
voron

releases are not environment-scoped by default

voron avatar
voron

values/valuesTemplate supports dynamic file names, this feature is super.

voron avatar
voron

combine it with missingFileHandler: Debug to jam missing file warnings, and you may get like 10 values files in template

Andrey Nazarov avatar
Andrey Nazarov


values/valuesTemplate supports dynamic file names, this feature is super.
what do you mean by this?

voron avatar
voron


what do you mean by this?
I mean syntax like

    valuesTemplate:
      - ../common/values-{{`{{ .[Release.Labels.app](http://Release\.Labels\.app) }}`}}.yaml.gotmpl
      - {{ .Environment.Name }}/values-{{`{{ .[Release.Labels.app](http://Release\.Labels\.app) }}`}}.yaml.gotmpl
      - {{ .Environment.Name }}/values-{{`{{ .Release.Name }}`}}.yaml.gotmpl
Andrey Nazarov avatar
Andrey Nazarov

Ah, ok, I see. Yes, this is quite handy indeed

Andrey Nazarov avatar
Andrey Nazarov

I thought there might be some other functionality I’m not aware of:)

Chris Picht avatar
Chris Picht

anyone running containerized Shibboleth and have a helm chart they started with?

muhaha avatar
muhaha

Guys? How can I reference Release.Namespace in values? Like:

releases:
- name: test
  namespace: default
  createNamespace: true
  chart: incubator/raw
  version: 0.2.3
  values:
    - resources:
      - apiVersion: [cert-manager.io/v1](http://cert\-manager\.io/v1)
        kind: Certificate
        metadata:
          name: [test.example.com](http://test\.example\.com)
          namespace: {{ .Release.Namespace }}
        spec:
          secretName: test.example.com-tls
          issuerRef:
            name: {{ .Values.issuer }}
          dnsNames:
          - [test.example.com](http://test\.example\.com)

Thanks

vixus0 avatar
vixus0

{{{{ .Release.Namespace }}}} might work

muhaha avatar
muhaha

nope

 failed to read test.yaml: reading document at index 1: yaml: invalid map key: map[interface {}]interface {}{".Release.Namespace":interface {}(nil)}
Andrey Nazarov avatar
Andrey Nazarov

What about just {{ .Namespace }} ?

muhaha avatar
muhaha

No

muhaha avatar
muhaha

Its rendered and deployed as namespace: null

Andrey Nazarov avatar
Andrey Nazarov

Ah, probably {{ .Namespace }} is rendered when you pass it via --namespace key. Ok, we don’t use this anyway)

muhaha avatar
muhaha

How are You setting namespaces in CRs ? ( Deployed via raw chart.. )

muhaha avatar
muhaha
roboll/helmfile

Deploy Kubernetes Helm Charts. Contribute to roboll/helmfile development by creating an account on GitHub.

vixus0 avatar
vixus0

I’m very confused – I have a simple helmfile with a single release, but when I run helmfile -f simple.yaml diff I get:

err: no releases found that matches specified selector() and environment(default), in any helmfile
jason800 avatar
jason800

do you actually have an environment defined anywhere?

vixus0 avatar
vixus0

no, no environments block and no --environment flag

jason800 avatar
jason800

then its basically just saying it doesn’t see any releases in the default path

jason800 avatar
jason800

is it a single helmfile with a releases: block ?

vixus0 avatar
vixus0

yeah

jason800 avatar
jason800

paste?

vixus0 avatar
vixus0

oh wait, I think it’s because I had a condition: bla.enabled in the release - I passed in --state-values-set bla.enabled=true but that didn’t do anything.

vixus0 avatar
vixus0

removing the condition worked.

:--1:1
jason800 avatar
jason800

Hey All, Is there any way to set --allow-no-matching-release in helmDefaults ?

vixus0 avatar
vixus0

I guess there’s no way to pass through state values from --state-values-file to helmfiles included with helmfiles: []

2020-09-22

vixus0 avatar
vixus0

Hey, I’m getting a bit confused about how state values files are propagated. I’m pointing helmfile at a directory of helmfiles. They contain templating for {{ .Values.bla }} which I provide with --state-values-file …or so I thought. Helmfile complains that it can’t find bla in .Values

voron avatar
voron

Did you tried to debug it with helmfile build --log-level=debug and your --state-values-file or --state-values-set during debug ?

vixus0 avatar
vixus0

I’ll give that a try, thanks

Ievgenii Shepeliuk avatar
Ievgenii Shepeliuk

Hi all - what is the best way to share some values between all releases in helm file ? I just don’t want to copy the same set of values or include values file in each release ?

I tried to declare values via environments, but it seems the environment values are not automatically assigned to releases

is it via release templatting ? https://github.com/roboll/helmfile/blob/master/docs/writing-helmfile.md#release-template--conventional-directory-structure

vixus0 avatar
vixus0

Yeah release templating is the best way I’ve found to share values like that.

Ievgenii Shepeliuk avatar
Ievgenii Shepeliuk

good remainder again, thnx

vixus0 avatar
vixus0

I’m a bit unclear on how the selectors mechanism works – do the selectors only apply within helmfiles: [] or can they be applied to individual releases too?

vixus0 avatar
vixus0

Ok, I was using selectors instead of labels within the releases. But it looks like selectors get applied after templating the state files with {{ .Values.bla }}.

voron avatar
voron

selectors are intended to select some releases instead of all releases, based on labels

Ievgenii Shepeliuk avatar
Ievgenii Shepeliuk

Hi, is it possible to define custom templates inside values template files ? like in Helm charts

{{- define "my.values.template" -}}
...
{{- end -}}
Majan Paul avatar
Majan Paul

as far as i know you cannot use template for the values files.

Bart M. avatar
Bart M.

you can if you name the values files the .yaml.gotmpl extension

Ievgenii Shepeliuk avatar
Ievgenii Shepeliuk

Hello @Bart M. - how can I share those templates between various .yaml.gotmpl files ?

via readFile ?

2020-09-21

2020-09-20

Andrey Nazarov avatar
Andrey Nazarov

I’ve got the following string in my helmfile.yaml:``` … values:

  • … backendBaseUriPattern: “https://$instance$.dev.my-domain.com” …
    ~~And it get’s rendered during `helmfile apply` given the value of instance from the following `range` construction:~~```
    {{- range $instance, $map := .Environment.Values.instances }}
    

    In the end after rendering it looks like backendBaseUriPattern: "<https://my-instance.dev.my-domain.com>" Is this an expected behaviour? Btw, the aforementioned values: block doesn’t belong to this range.

voron avatar
voron

I don’t think that promoting range iterators to global variables is expected behavior. It doesn’t match with gotmpl

A variable's scope extends to the "end" action of the control structure ("if", "with", or "range") in which it is declared, or to the end of the template if there is no such control structure.
template - The Go Programming Language

Go is an open source programming language that makes it easy to build simple, reliable, and efficient software.

Andrey Nazarov avatar
Andrey Nazarov

Ok, that was my fault. I checked the wrong commit of our codebase, my colleagues forgot to put a tag on the fresh code and I didn’t notice this and examined the fresh one.

So, nevermind the question)

Release notes from helmfile avatar
Release notes from helmfile
12:56:17 AM
helmfile test always tests all releases · Issue #1483 · roboll/helmfile

When running helmfile test on a helmfile with conditional releases, releases with condition: false are still tested. Example: repositories: - name: stable url: https://kubernetes-charts.storage.goo

Release notes from helmfile avatar
Release notes from helmfile
01:16:26 AM

v0.129.4: Enable helmfile test testing only enabled and selected releases (#1… b176408 (HEAD, tag: v0.129.4, origin/master, origin/HEAD, master) Enable helmfile test testing only enabled and selected releases (<a class=”issue-link js-issue-link” data-error-text=”Failed to load title” data-id=”703982737”…

mumoshu avatar
mumoshu

contains the fix for the “repository name already exists” error due to the helm update

:--1:2
Release notes from helmfile avatar
Release notes from helmfile
06:56:13 AM

v0.130.0: Add support for ChartCenter (#1492) 942b9a6 (HEAD, tag: v0.130.0, origin/master, origin/HEAD, master) Add support for ChartCenter (<a class=”issue-link js-issue-link” data-error-text=”Failed to load title” data-id=”705302089” data-permission-text=”Title is private”…

2020-09-19

Joaquin Menchaca avatar
Joaquin Menchaca

Is it possible to helmfile apply for only a single release instead of all of them in specified in the helmfile.yaml?

Andrey Nazarov avatar
Andrey Nazarov

You can use selectors for this, but I bet it’s not what you are looking for

2020-09-18

Joaquin Menchaca avatar
Joaquin Menchaca

I just installed helmfile with brew (helmfile version v0.129.3). I’m trying to use repositories I always use, but helmfile gives me this: Error: repository name (jaegertracing) already exists, please specify a different name

repositories:
  - name: jaegertracing
    url: <https://jaegertracing.github.io/helm-charts>
  - name: dgraph
    url: <https://charts.dgraph.io>
Joaquin Menchaca avatar
Joaquin Menchaca

I guess this is underlying problem with helm. I never had it not be idempotent.

Joaquin Menchaca avatar
Joaquin Menchaca

helm has broken idempotence in versions 3.3.2 and 3.3.3. Thus you cannot use helmfile with those versions.

voron avatar
voron
helm repo add now fails in v3.3.2 and v3.3.3 · Issue #8771 · helm/helm

Description The command helm repo add <name> <url> fails as it is no longer idempotent. This worked fine in version v3.3.1 and earlier. It now fails in v3.3.2 and v3.3.3. This is a brea…

change behavior when add repo with same name and url. by yxxhero · Pull Request #8772 · helm/helm

Signed-off-by: yxxhero [email protected] What this PR does / why we need it: resolve: #8771 We could just return exit code zero if the same repo is attempted to be added twice with the same name. …

Joaquin Menchaca avatar
Joaquin Menchaca

The issue is annoying. Before in 2010s, there was a big push to write automation (especially with change configuration) to make idempotent automation. If the state is already in the desired state, do not raise an error.

But now these days, with kubectl and helm, others, it’s like
*idempotent, shemdepotent*

voron avatar
voron

well, I think it’s a good idea to pin versions in automation. on interactive usage it’s possible to use pre-existing helm repos and don’t add it in helmfile

Joaquin Menchaca avatar
Joaquin Menchaca

I’m trying to have documentation for customers that may not be at dev/ops level to manipulate helmfile. helmfile is useful to orchestrate several components that need to work together with no-fuss, no-muss.

roth.andy avatar
roth.andy

I added a note to the PR requesting that --force-update only be added when Helm versions 3.3.2 and 3.3.3 are used. I believe the security issues that Helm addressed with this change are valid, and a PR is being worked that would return the idempotent nature as long as the name and URL are both the same as they were before

https://github.com/roboll/helmfile/pull/1488#discussion_r492055821

roth.andy avatar
roth.andy

Though, I just thought about what would happen if I am using the helm-git plugin (which I do frequently). Changing the ref to another tag or git SHA would be considered a change to the URL, causing the error to be thrown

voron avatar
voron

WDYT about --force-update directive from helmfile side ?

roth.andy avatar
roth.andy

meaning you’d run helmfile apply --force-update?

roth.andy avatar
roth.andy

Not sure, could also do

repositories:
- name: polaris
  url: git+<https://github.com/reactiveops/[email protected]/helm?ref=tags/1.2.3>
  forceUpdate: true
roth.andy avatar
roth.andy

tagging some others for more visibility - @Erik Osterman (Cloud Posse) @mumoshu

Joaquin Menchaca avatar
Joaquin Menchaca

Actually, I didn’t know about the helmfile apply --force-update. I had been using helmfile for less than a week…

roth.andy avatar
roth.andy

Sorry I could have been more clear. That was in response to a suggestion. The --force-update flag does not currently exist for helmfile apply

Joaquin Menchaca avatar
Joaquin Menchaca

I switched to another laptop, installed latest version of helm, and discovered this issue. I think I may have convinced helm folks to change it so that if it will not change the repo name for the same url, not to exit 1.

voron avatar
voron


meaning you’d run helmfile apply --force-update?
No, I was talking about some helmDefaults config option

voron avatar
voron

with per-repository override, like you’ve offered above

Joaquin Menchaca avatar
Joaquin Menchaca

I used the default helmDefaults

Joaquin Menchaca avatar
Joaquin Menchaca

I would prefer that if I have it already to do nothing, that would be nice behavior.

voron avatar
voron

@ we’re talking about feature request of non-existing helmfile functionality

Joaquin Menchaca avatar
Joaquin Menchaca

oh

Joaquin Menchaca avatar
Joaquin Menchaca

yeah

Joaquin Menchaca avatar
Joaquin Menchaca

makes sense.

Joaquin Menchaca avatar
Joaquin Menchaca

Sorry, not enough coffee

Joaquin Menchaca avatar
Joaquin Menchaca

Joaquin Menchaca avatar
Joaquin Menchaca

So that would be fine. But also we get into a situation, that if they have a force-update , and the helm version doesn’t support that option, it craps out on them

roth.andy avatar
roth.andy

They aren’t removing --force-update from Helm, just adding a check for whether the name and url are both the same as they were before

Joaquin Menchaca avatar
Joaquin Menchaca

So it would nice to have it do whatever it needs to do regardless of helm3 version. So if not force-update, then it skips adding it again.

voron avatar
voron

thus, we have 3 helm versions

Fixing issue with idempotent repo add by mattfarina · Pull Request #8777 · helm/helm

A security issue fixed in 3.3.2 caught repos with the same name being added a second time and produced an error. This caused an issue for tools, such as helmfile, that will add the same name with t…

2020-09-17

Cameron Boulton avatar
Cameron Boulton

@mumoshu Hopefully a quick question for you: is there any control/configuration over Helmfile’s exec behavior around errors/non-zero exit codes? Specifically I’d like to do something like: {{ (contains "error: the server doesn't have a resource type" (exec "kubectl" (list "-n" (env "KUBE_NAMESPACE") "get" "<resource>")))) }} Seems like there is no way to proceed with exec if it errors/exits with non-zero?

mumoshu avatar
mumoshu

@Cameron Boulton Hey! Good question - Yes, unfortunately there’s no way for now

mumoshu avatar
mumoshu

It does sound like a valid feature request for me, though

:--1:1
Cameron Boulton avatar
Cameron Boulton

Okay, thank ya. I’ll consider adding a feature request issue to GitHub.

mumoshu avatar
mumoshu

Thanks! Looking forward to it

Cameron Boulton avatar
Cameron Boulton
Improvements to {{exec}} · Issue #1487 · roboll/helmfile

{{exec}} is a great feature of Helmfile and adds a lot of power by enabling the use of any external command/script. However, the current behavior only: Returns stdout but only on success On error (…

1

2020-09-16

Justin Seiser avatar
Justin Seiser

hitting an issue where helmfile apply, appears to run without error

Andrey Nazarov avatar
Andrey Nazarov

Experienced something similar a week or so ago. I had to change some values and run again to actually apply a thing. But I thought in my case it was not related to helmfile. And it was really difficult to reproduce

Justin Seiser avatar
Justin Seiser

but my release is not actually installed.

jason800 avatar
jason800

hey all, is there a way to default to an empty map/list ? {{ .Values | get "my-value" {} }} something like this?

Jonathan avatar
Jonathan

something like this?

version: {{ .Environment.Values | default (env "COMPONENT_VERSION") | default "" }}
jason800 avatar
jason800

well helmfile uses the get syntax, so I’m really just looking for a way to express “empty map” or “empty list”

jason800 avatar
jason800

but yes, something along the lines as what you have

Jonathan avatar
Jonathan

doing

{{ .Values | get "domain" "[dev.example.com](http://dev\.example\.com)" }}

will try to get from .Values.domain, but default to getting from “dev.example.com” if that value does not exists. if you want to assign a default value to the get request, I think this is how you want to do it:

{{ .Values | get "my-domain" "default-domain" | default "{}" }}
jason800 avatar
jason800

man I am banging my head against the wall. I have a helmfile template test that I run locally and it works perfectly and then my gitlab-ci runs it (in the same container) and it fails

jason800 avatar
jason800

in kafka/helmfile-kafka.yaml: "kafka/appmm" depends on nonexistent release "preprod_us-ashburn-1_dataplane/kafka/appkafka"

jason800 avatar
jason800

It makes no sense that the needs: would fail on the exact same code in the CI test

Joaquin Menchaca avatar
Joaquin Menchaca

Can Helmfile work with manifests? I am seeing more and more use operators directly rather than Helm charts.

roth.andy avatar
roth.andy

Yes. Check out the incubator/raw chart

Joaquin Menchaca avatar
Joaquin Menchaca

Or maybe I should look at Kustomize. Any good articles on helmfile driving kustomize?

Joaquin Menchaca avatar
Joaquin Menchaca

I would like to use some Kustomize in non-helmchart releases

bradym avatar
bradym
roboll/helmfile

Deploy Kubernetes Helm Charts. Contribute to roboll/helmfile development by creating an account on GitHub.

Joaquin Menchaca avatar
Joaquin Menchaca

I’m using helmify-kustomize , works great.

roboll/helmfile

Deploy Kubernetes Helm Charts. Contribute to roboll/helmfile development by creating an account on GitHub.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
[Gitlab.com> uses Helmfile <https://about.gitlab.com/blog/2020/09/16/year-of-kubernetes/ https://about.gitlab.com/blog/2020/09/16/year-of-kubernetes/](http://Gitlab.com)
What we learned after a year of GitLab.com on Kubernetes attachment image

It’s been one year since we moved GitLab.com to Kubernetes. We unpack the challenges and learnings from this major migration.

:--1:2
Joaquin Menchaca avatar
Joaquin Menchaca

How do you do dependencies between to releaeses?

Joaquin Menchaca avatar
Joaquin Menchaca

I have a CRD that depends on the operator to be installed, so I am looking for easy way to have that CRD depend on the helm chart to install the operator

Joaquin Menchaca avatar
Joaquin Menchaca

This look like this was added with needs: https://github.com/roboll/helmfile/pull/914

feat: Add `needs: [NS/NAME]` for controlling installation/deletion order declaratively by mumoshu · Pull Request #914 · roboll/helmfile

Introduces DAG-aware installation/deletion ordering to Helmfile. needs controls the order of the installation/deletion of the release: relesaes: - name: somerelease needs: - [TILLER_NAMESPACE/]…

Andrey Nazarov avatar
Andrey Nazarov

What is currently the best practice for installing CRDs before deploying something that needs them? For example, the instructions for using the Helm Operator are to:

  1. Install the HelmRelease CRD using kubectl apply
  2. Deploy the operator using Helm If I want both to happen with one helmfile apply , what’s the best way right now to do that? I’m thinking of the incubator/raw helm chart, but that sounds kinda kludgey
Joaquin Menchaca avatar
Joaquin Menchaca

The needs worked perfectly for me

Joaquin Menchaca avatar
Joaquin Menchaca

I also had to use a manifest to use the operator, and then a helm chart after the manifest

Joaquin Menchaca avatar
Joaquin Menchaca

So for this I used kustomized with a hook to create a helm chart on the fly, then use that. Combined with needs I can mix operators, helm charts, kustomize, all together and use ordering.

2020-09-15

Joaquin Menchaca avatar
Joaquin Menchaca

New to helmfile, I was thinking to use this for testing helm charts after deployment in K8S env.

Joaquin Menchaca avatar
Joaquin Menchaca

What I am fuzzy about it how to inject values into values.yaml per release. I wanted to avoid creating massive directory structure for different types of tests with env.sh, but rather just define values in the releases.

charts:
  dgraph:
    name: dgraph
    enabled: true
    namespace: dg-1
    chart: ./charts/charts/dgraph
    env:
      - path/to/env.sh
    values:
      - charts/dgraph/helmfile/base.yaml.gotmpl
Joaquin Menchaca avatar
Joaquin Menchaca

or maybe:

env:
 DGRAPH_ALPHA_ACL_ENABLED: false
 DGRAPH_ALPHA_ACL_FILE_CONTENT: secret
 DGRAPH_ALPHA_ACL_SECRET_FILE: /dgraph/acl/secret_file
Joaquin Menchaca avatar
Joaquin Menchaca

Is there a way to do something like this?

Joaquin Menchaca avatar
Joaquin Menchaca

I cannot get off the ground with helmfile. Running into this, not sure what to do:

in ./helmfile.yaml: error during helmfile.yaml.part.1 parsing: template: stringTemplate:2:17: executing "stringTemplate" at <.Values.charts>: map has no entry for key "charts"

I setup simple charts.yaml:

charts:
  azuregateway:
    name: minio
    enabled: true
    namespace: minio
    repository:
      name: minio
      url: <https://helm.min.io/>
    chart: minio/minio
    values:
      - envs/aks/tf.minio_config.yaml

and helmfile.yaml with:

helmDefaults:
  timeout: 600
  recreatePods: false
  tillerless: true
  force: true

environments:
  default:
    values:
      - ./charts.yaml
      - ./values.yaml
      {{- if env "ENV_DIR" }}
      - {{ requiredEnv "ENV_DIR" }}/charts.yaml.gotmpl
      - {{ requiredEnv "ENV_DIR" }}/values.yaml.gotmpl
      {{- end }}

repositories:
{{- range .Values.charts }}
{{- if and .enabled (hasKey . "repository") }}
  - name: {{ .repository.name }}
    url: {{ .repository.url }}
{{- end }}
{{- end }}

releases:
{{- range .Values.charts }}
{{- if .enabled }}
  - name: {{ .name }}
    namespace: {{ .namespace }}
    chart: {{ .chart }}
    version: "{{ . | getOrNil "version" }}"
    values:
    {{- range .values }}
      - {{ . }}
    {{- end }}
    secrets:
    {{- range . | getOrNil "secrets" }}
      - {{ . }}
    {{- end }}
  {{- end }}
{{- end }}

Jonathan avatar
Jonathan

How do your values files look like?

Joaquin Menchaca avatar
Joaquin Menchaca

I got it working eventually. There were some issues with starter kit that I was trying to use.

Joaquin Menchaca avatar
Joaquin Menchaca

The values.yaml is

_: {}
Jonathan avatar
Jonathan

Glad you got it working! The reason I was asking is that it seemed you were missing .Values.charts in your values file, but you seem to have solved it nonetheless!

Joaquin Menchaca avatar
Joaquin Menchaca

I’m stuck at trying to render set, having the toughest time.

Joaquin Menchaca avatar
Joaquin Menchaca

I tried adding this snippet in the helmfile:

releases:
{{- range .Values.charts }}
{{- if .enabled }}
    {{- if hasKey . "set" }}
    set:
    {{- range . | getOrNil "set" }}
      - {{ . }}
    {{- end }}
    {{- end }}
    ...
{{- end }}

But when I try to use this:

charts:
  azuredgraph:
    ...
    values:
      - ./charts/dgraph/helmfile/base.yaml.gotmpl
      - ./envs/aks/tf.dgraph_config.yaml
    set:
      - backups.destination: <minio://minio.minio.svc:9000/dgraph-backups>
      - image.tag: v20.03.4
      - backups.incremental.enabled: true
      - backups.full.enabled: true
Joaquin Menchaca avatar
Joaquin Menchaca

But I cannot get set to work in this context.

Joaquin Menchaca avatar
Joaquin Menchaca

It tells me:

second-pass rendering result of "helmfile.yaml.part.0":
 0: 
 1: helmDefaults:
 2:   timeout: 600
 3:   recreatePods: false
 4:   tillerless: true
 5:   force: false
 6: 
 7: environments:
 8:   default:
 9:     values:
10:       - ./charts.yaml.gotmpl
11:       - ./values.yaml
12:       - ./envs/aks/charts.yaml.gotmpl
13:       - ./envs/aks/values.yaml.gotmpl
14: 
15: repositories:
16:   - name: minio
17:     url: <https://helm.min.io/>
18: 
19: releases:
20:   - name: dgraph
21:     namespace: default
22:     chart: /home/joaquin/workarea/charts/charts/dgraph
23:     version: "0.0.11"
24:     values:
25:       - ./charts/dgraph/helmfile/base.yaml.gotmpl
26:       - ./envs/aks/tf.dgraph_config.yaml
27:     set:
28:       - map[backups.destination:<minio://minio.minio.svc:9000/dgraph-backups>]
29:       - map[image.tag:v20.03.4]
30:       - map[backups.incremental.enabled:true]
31:       - map[backups.full.enabled:true]
32:   - name: minio
33:     namespace: minio
34:     chart: minio/minio
35:     version: "6.1.2"
36:     values:
37:       - ./envs/aks/tf.minio_config.yaml
38: 

err: failed to read helmfile.yaml: reading document at index 1: yaml: unmarshal errors:
  line 29: cannot unmarshal !!str `map[bac...` into state.SetValue
  line 30: cannot unmarshal !!str `map[ima...` into state.SetValue
  line 31: cannot unmarshal !!str `map[bac...` into state.SetValue
  line 32: cannot unmarshal !!str `map[bac...` into state.SetValue
in ./helmfile.yaml: failed to read helmfile.yaml: reading document at index 1: yaml: unmarshal errors:
  line 29: cannot unmarshal !!str `map[bac...` into state.SetValue
  line 30: cannot unmarshal !!str `map[ima...` into state.SetValue
  line 31: cannot unmarshal !!str `map[bac...` into state.SetValue
  line 32: cannot unmarshal !!str `map[bac...` into state.SetValue
Joaquin Menchaca avatar
Joaquin Menchaca

Statically if fails too

Joaquin Menchaca avatar
Joaquin Menchaca

I am not sure how to use both set or values (docs: https://github.com/roboll/helmfile#configuration)

roboll/helmfile

Deploy Kubernetes Helm Charts. Contribute to roboll/helmfile development by creating an account on GitHub.

Jonathan avatar
Jonathan

I dont thing helmfile likes having map[...] as values,

you should be able to set them like this:

    set:
    # single value loaded from a local file, translates to --set-file foo.config=path/to/file
    - name: foo.config
      file: path/to/file
    # set a single array value in an array, translates to --set bar[0]={1,2}
    - name: bar[0]
      values:
      - 1
      - 2
Joaquin Menchaca avatar
Joaquin Menchaca

I spotted a little bit after. I did the name/values. Now it is injecting the values.

Joaquin Menchaca avatar
Joaquin Menchaca

I’m using helmfile to run a battery of values easily, as the app I am supporting has 40+ combinations…

Joaquin Menchaca avatar
Joaquin Menchaca

too challenging to keep track of combinations

Joaquin Menchaca avatar
Joaquin Menchaca

I seem to be failing. I changed the values to these

     - name: backups.destination
       value: <minio://minio.minio.svc:9000/dgraph-backups>
     - name: image.tag
       value: v20.03.4

But I am not having them picked up.

Joaquin Menchaca avatar
Joaquin Menchaca

So I went with pure values… that seems to work. I wonder if you have values, then set won’t work

Joaquin Menchaca avatar
Joaquin Menchaca

Thanks for your help.

2020-09-14

Release notes from helmfile avatar
Release notes from helmfile
11:56:27 AM
panic: runtime error: invalid memory address or nil pointer dereference · Issue #34 · variantdev/vals

I&#39;ve downloaded the latest release for both tfstate-lookup and val but running val causes a runtime error when running against state stored in an s3 bucket tfstate-lookup -s .terraform/terrafor…

Release notes from helmfile avatar
Release notes from helmfile
12:06:26 PM

v0.129.2: Bump vals (#1475) 5dd65e8 (HEAD, tag: v0.129.2, origin/master, origin/HEAD, master) Bump vals (<a class=”issue-link js-issue-link” data-error-text=”Failed to load title” data-id=”700691836” data-permission-text=”Title is private”…

mumoshu avatar
mumoshu

Is anyone relying on the undefined behaviour of Helmfile that a multiple negated conditions in a single selector like helmfile -l foo!=foo,bar!=bar is unexpectedly treated as an OR sometimes?

I’m redefining it to be always AND, so that the behavior is consistent:

https://github.com/roboll/helmfile/pull/1478

This might be just a bug but I wanted inform you all for clarity because this seems like a long-standing bug anyway. Thanks!

Treat selector with multiple conditions an AND by mumoshu · Pull Request #1478 · roboll/helmfile

Helmfile had been historically combining multiple -l foo=bar selectors and its equivalent in helmfile.yaml, helmfiles[].selectors[], by OR. I believe this isn&#39;t consistent with kubectl get -l f…

:--1:2
Release notes from helmfile avatar
Release notes from helmfile
01:06:28 AM

v0.129.3: Treat selector with multiple conditions an AND (#1478) 028bcc5 (HEAD, tag: v0.129.3, origin/master, origin/HEAD, master) Treat selector with multiple conditions an AND (<a class=”issue-link js-issue-link” data-error-text=”Failed to load title” data-id=”701512861” data-permission-text=”Title…

Release v0.129.3: Treat selector with multiple conditions an AND (#1478) · roboll/helmfile

028bcc5 (HEAD, tag: v0.129.3, origin/master, origin/HEAD, master) Treat selector with multiple conditions an AND (#1478) Build Info

2020-09-13

Roderik van der Veer avatar
Roderik van der Veer

I’ve got a configmap template that contains othernodes = {{ .Values.orion.otherNodes }} where othernodes is an array of strings. but in this specific config i need each of the strings to be wrapped in quotes but it comes out as [ string, string ] without quotes. Is there a quick way to quote each item in the list?

Roderik van der Veer avatar
Roderik van der Veer

Never mind, othernodes = {{ .Values.orion.otherNodes | toJson }} returns what i need!

Release notes from helmfile avatar
Release notes from helmfile
07:46:29 AM

v0.129.1: Add tailormed to users (#1467) 134d5be (HEAD, tag: v0.129.1, origin/master, origin/HEAD, master) Add tailormed to users (<a class=”issue-link js-issue-link” data-error-text=”Failed to load title” data-id=”698282052” data-permission-text=”Title is private”…

2020-09-12

Release notes from helmfile avatar
Release notes from helmfile
06:36:18 AM

2020-09-11

Gabriel Tiossi avatar
Gabriel Tiossi

Hi there :wave: Is it possible to use remote values from S3 using the go-getter syntax?

1
Gabriel Tiossi avatar
Gabriel Tiossi

I’m getting the following error:


\# helmfile.yaml
environments:
  default:
    values:
      - s3::<https://s3.amazonaws.com/my-s3-bucket/v1/values.yaml>
$ helmfile template
in ./helmfile.yaml: failed to read helmfile.yaml: environment values file matching "s3::<https://s3.amazonaws.com/my-s3-bucket/v1/values.yaml>" does not exist in "."
Gabriel Tiossi avatar
Gabriel Tiossi

Btw, this works with their CLI:

$ go-getter s3::<https://s3.amazonaws.com/my-s3-bucket/v1/values.yaml> /tmp/test/
2020/09/11 08:05:51 success!
Jonathan avatar
Jonathan

What version of helmfile are you using? Looks like it should be supported: https://github.com/roboll/helmfile#loading-remote-environment-values-files

roboll/helmfile

Deploy Kubernetes Helm Charts. Contribute to roboll/helmfile development by creating an account on GitHub.

Gabriel Tiossi avatar
Gabriel Tiossi

The latest one I think

helmfile version v0.128.2
Jonathan avatar
Jonathan

@mumoshu Might have an idea?

1
mumoshu avatar
mumoshu

I have never had chance to try it with s3. This might be a bug or not. I thought it wasn’t trivial to make go-getter as a go library behave the same as the go-getter cli

mumoshu avatar
mumoshu

it may work depending on url format(i don’t have specific idea yet, though..

mumoshu avatar
mumoshu

It did reproduce this locally. Let me see what I can do..

mumoshu avatar
mumoshu

Ah gotcha. @ Could you try s3::<https://s3.amazonaws.com/my-s3-bucket/[email protected]>? Please notice the part before @ is the path to the directory and after is the file name(object key, in s3 term)

mumoshu avatar
mumoshu

I admit this isn’t intuitive, but we had to introduce this dialect of go-getter url to stabilize go-getter’s behaviour for helmfile’s use-case

Gabriel Tiossi avatar
Gabriel Tiossi

Thank you @mumoshu and @Jonathan :grin: I found it intuitive and worked like a charm :+1:

I would like to point that this behavior only works with environments._environmentName_.values , and not with releases[n].values . Don’t know if this is expected or not.

When I use the exact same sucessful string in a releases[n].values, this is the output:

in ./helmfile.yaml: values file matching "s3::<https://s3.amazonaws.com/my-s3-bucket/[email protected]>" does not exist in "."

To work around it, I’m doing something like this:

environments:
  default:
    values:
      - s3::<https://s3.amazonaws.com/my-s3-bucket/[email protected]>

repositories:
  - name: my-repo
    url: <https://my-repo.com>

releases:
  - name: my-release
    chart: my-repo/my-chart
    values:
      - {{ .Environment.Values | toYaml | indent 8 | trim }}

I don’t know if there is a more elegant way to solve it, but it works now

Jonathan avatar
Jonathan

haha don’t thank me, @mumoshu did all the work, I’m just the one who summoned him Glad you solved it!

2
mumoshu avatar
mumoshu

@ Thanks for confirming! Glad to hear it mostly worked.
environments.environmentName.values , and not with releases[n].values
I haven’t tried using it under releases.values so this might be a bug(or just not-implemented-yet)

Let me see what I can do

1
Release notes from helmfile avatar
Release notes from helmfile
01:36:32 PM

v0.129.0: Add experimental write-values command for writing values files only (… 0fad9f0 (HEAD, tag: v0.129.0, origin/master, origin/HEAD, master) Add experimental write-values command for writing values files only (<a class=”issue-link js-issue-link” data-error-text=”Failed to load title” data-id=”699300090”…

Release v0.129.0: Add experimental write-values command for writing values files only (… · roboll/helmfile

0fad9f0 (HEAD, tag: v0.129.0, origin/master, origin/HEAD, master) Add experimental write-values command for writing values files only (#1469) Build Info

2020-09-10

Release notes from helmfile avatar
Release notes from helmfile
12:56:35 PM

v0.128.2: Re-add Release.Namespace in release values.yaml templates (#1466) It was accidentally removed in #1424, and had been unexpectedly unavailable between v0.126.0 and v0.128.1. Fixes <a class=”issue-link js-issue-link”…

Release v0.128.2: Re-add Release.Namespace in release values.yaml templates (#1466) · roboll/helmfile

It was accidentally removed in #1424, and had been unexpectedly unavailable between v0.126.0 and v0.128.1. Fixes #1464

feat: Allow .Release.Name to be used in gotmpl values templates by joshes · Pull Request #1424 · roboll/helmfile

This adds the ability to utilize .Release inside of gotmpl files as discussed here. Resolves: #760 Added a simple test that passed once implemented and all tests are running green.

Release notes from helmfile avatar
Release notes from helmfile
01:06:31 PM

v0.128.2: Re-add Release.Namespace in release values.yaml templates (#1466) 832dcf4 (HEAD, tag: v0.128.2, origin/master, origin/HEAD, master) Re-add Release.Namespace in release values.yaml templates (<a class=”issue-link js-issue-link” data-error-text=”Failed to load title” data-id=”697291695”…

Re-add Release.Namespace in release values.yaml templates (#1466) · roboll/[email protected]

It was accidentally removed in #1424, and had been unexpectedly unavailable between v0.126.0 and v0.128.1. Fixes #1464

2020-09-09

Release notes from helmfile avatar
Release notes from helmfile
11:36:25 AM

v0.128.1: Fix index out of range [1] with length 1 error on env2map (#1463) 0482ba3 (HEAD, tag: v0.128.1, origin/master, origin/HEAD, master) Fix index out of range [1] with length 1 error on env2map (<a class=”issue-link js-issue-link” data-error-text=”Failed to load title” data-id=”695488078”…

Release v0.128.1: Fix `index out of range [1] with length 1` error on env2map (#1463) · roboll/helmfile

0482ba3 (HEAD, tag: v0.128.1, origin/master, origin/HEAD, master) Fix index out of range [1] with length 1 error on env2map (#1463) Build Info

Fix `index out of range [1] with length 1` error on env2map (#1463) · roboll/[email protected]

This happened only when helmfile is run in an environment that any environment variable definition does not include = and the right side.

Sebastian Grateful avatar
Sebastian Grateful
Using nested values defined in environment

I have the following helmfile releases: - name: whoami-mn namespace: whoami-mn-{{ .Environment.Name }} chart: tons/whoami-mn version: {{ .Values.chartVersions.whoami }} installed:…

Sebastian Grateful avatar
Sebastian Grateful

In the readme there’s an example environments.production.values[].vault.enabled: false so I’m surprised my yaml doesn’t work

jason800 avatar
jason800

has anyone figured out a way in helmfile to deploy jobs without hitting the immutable field issue?

jason800 avatar
jason800
Error: UPGRADE FAILED: cannot patch "vault-config-setup" with kind Job: Job.batch "vault-config-setup" is invalid: spec.template:
...
: field is immutable
jason800 avatar
jason800

force: true/false does not get around it

bradym avatar
bradym

I ran into similar issues. I believe the fix was updating to a newer version of helm 3.

jason800 avatar
jason800

oh , i think i am still on 3.2.4

jason800 avatar
jason800

let me check

bradym avatar
bradym

Hmm, 3.2.4 is what I’m using. What version of helmfile are you on?

jason800 avatar
jason800

3.2.4

bradym avatar
bradym

that’s helm, what about helmfile?

jason800 avatar
jason800

0.125.8

bradym avatar
bradym

hmm, that’s pretty recent. Doesn’t seem like a version issue then.

jason800 avatar
jason800

No, i mean this is a very well known and long-discussed issue in helm

jason800 avatar
jason800

I just thought maybe someone had worked around it in helmfile

bradym avatar
bradym

Yeah, I know the issue. I thought the way I got around it was by upgrading helm. But clearly that’s not the case since we’re using the same version of helm.

voron avatar
voron

https://github.com/helm/helm/pull/7431 and related. I cannot understand why can’t we get the same selective replace as we have in helm2.

feat(helm): add recreate upgrade (rollback) strategy by dastrobu · Pull Request #7431 · helm/helm

An additional optional flag –recreate can be passed on upgrade (or rollback) of a release. In combination with the –force flag the following strategies are employed when updating a resource (whic…

voron avatar
voron


worked around it in helmfile
it’s helm-level issue, I don’t thinks it’s possible to provide stable solution at helmfile level.

:--1:2

2020-09-08

Roderik van der Veer avatar
Roderik van der Veer

Did something change recently on how {{ .Namespace }} works?

i just installed v0.128 and i get the following:

helmfile --file helmfile-preip.yaml --selector default=true --selector app=besu --namespace cyan-besu-15 apply

in ./helmfile-preip.yaml: in .helmfiles[0]: in helmfiles/1-before-all.yaml: failed processing release ingress: failed to render values files "values/nginx-ingress-values.yaml.gotmpl": failed to render [values/nginx-ingress-values.yaml.gotmpl], because of template: stringTemplate:6:19: executing "stringTemplate" at <.Namespace>: can't evaluate field Namespace in type state.releaseTemplateData

and the values file contains

controller:
  publishService:
    enabled: true
  scope:
    enabled: true
    namespace: {{ .Namespace }}
  extraArgs:
    default-ssl-certificate: "{{ .Namespace }}/wildcard"
  service:
    enabled: true
  replicaCount: 2
  minAvailable: 1


  resources:
    limits: {}
      #cpu: 500m
      #memory: 512Mi
    requests:
      cpu: 50m
      memory: 128Mi

nameOverride: {{ .Namespace }}-ingress
mumoshu avatar
mumoshu

Seems like a regression introduced recently. I’ll fix it today. Thanks for reporting!

mumoshu avatar
mumoshu

And this is what you’re relying on: https://github.com/roboll/helmfile/commit/7b11ce851a2483f3176079210ce3274271d0237a#diff-114928b5b56bd96452914832f2804038R16-R17

According to the code comment, it’s not intended to be accessible from with values.yaml templates… But we might better fix(repurpose) it if everyone is already using it

mumoshu avatar
mumoshu
Re-add Release.Namespace in release values.yaml templates by mumoshu · Pull Request #1466 · roboll/helmfile

It was accidentally removed in #1424, and had been unexpectedly unavailable between v0.126.0 and v0.128.1.

mumoshu avatar
mumoshu

@Roderik van der Veer I’ve just released v0.128.2 for this

1
Roderik van der Veer avatar
Roderik van der Veer

thx! I modified everything already to make it work so i cannot really test it

:--1:1
Roderik van der Veer avatar
Roderik van der Veer

using {{ .Release.Namespace }} works. If i have to guess, https://github.com/roboll/helmfile/pull/1424 is the culprit

feat: Allow .Release.Name to be used in gotmpl values templates by joshes · Pull Request #1424 · roboll/helmfile

This adds the ability to utilize .Release inside of gotmpl files as discussed here. Resolves: #760 Added a simple test that passed once implemented and all tests are running green.

2020-09-07

muhaha avatar
muhaha

I am thinking loud, but..

Would be nice to have k8s service similar to https://github.com/jetstack/version-checker, of course there it possible to use https://github.com/roboll/helmfile#integrations renovate for gitops version check, but some active checking on actually deployed charts would be a great idea ( with metrics support of course )…

Related: https://github.com/FairwindsOps/nova/

@mumoshu ping

Thanks

EDIT: I have hard time to keep charts updated, or at least have some overview about newer versions available…

mumoshu avatar
mumoshu

Hey! Yeah this makes sense a lot. It should probably be implemented as a K8s controller that runs some code similar to https://github.com/bacongobbler/helm-whatup

bacongobbler/helm-whatup

A Helm plugin to help users determine if there’s an update available for their installed charts. - bacongobbler/helm-whatup

Andrey Nazarov avatar
Andrey Nazarov

Just want to mention that we are very satisfied with Renovate. But I understand your pain;)

muhaha avatar
muhaha

@Andrey Nazarov Are You using regex matching, or direct helmfile support ?

Andrey Nazarov avatar
Andrey Nazarov

We are using regexps

2020-09-06

Andrey Nazarov avatar
Andrey Nazarov

I would like to use this functionality powered by vals: https://github.com/roboll/helmfile/pull/906

As far as I understood it will be rendered only under a release: block. WIn our case we have a common helmfile, and then we reuse this helmfile by referencing form other helmfiles via helmfiles: (sorry for such a cluncky explanation). In the repo with a common helmfile there is no credentials to connect to third-parties to grab the actual value. So, helmfile lint failed.

Does anybody know if any possibility to overwrite this or hide the error exists?

Add integration with Hashicorp Vault, AWS SSM, SecretsManager by klebediev · Pull Request #906 · roboll/helmfile

Fields which are rendered: Release.Values, Release.SetValues.Value, Release.SetValues.Values Example: values: - foo: <ref+vault://mykv/foo?address=http://127.0.0.1:8200#/mykey> set: - name: xyz val…

Andrey Nazarov avatar
Andrey Nazarov

Ok, I think simply wrapping the ref+ link into an if not conditional in a common helmfile might work

Add integration with Hashicorp Vault, AWS SSM, SecretsManager by klebediev · Pull Request #906 · roboll/helmfile

Fields which are rendered: Release.Values, Release.SetValues.Value, Release.SetValues.Values Example: values: - foo: <ref+vault://mykv/foo?address=http://127.0.0.1:8200#/mykey> set: - name: xyz val…

Andrey Nazarov avatar
Andrey Nazarov

Yes it works

mumoshu avatar
mumoshu

I think you’ve already got it! If I can add anything, it works by replacing ref+ urls in yaml hash in values.yaml files, after the go template is rendered to produce the yaml document in the first place.

mumoshu avatar
mumoshu

you can also use fetchSecretValue and expandSecretRefs for more advanced use-cases

mumoshu avatar
mumoshu
roboll/helmfile

Deploy Kubernetes Helm Charts. Contribute to roboll/helmfile development by creating an account on GitHub.

2020-09-04

Craig Dunford avatar
Craig Dunford

anyone know if helmfile supports overriding a chart’s appVersion somehow? (along the lines of what is discussed in this helm issue: https://github.com/helm/helm/issues/8194)

Provide a means of setting .Chart.AppVersion during install or upgrade without editing Chart.yaml · Issue #8194 · helm/helm

A significant number of comments and requests from #3555 were to have the ability to set .Chart.AppVersion at install and upgrade time without having to download and repackage the chart. Those comm…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Carlos R. would probably want this

Provide a means of setting .Chart.AppVersion during install or upgrade without editing Chart.yaml · Issue #8194 · helm/helm

A significant number of comments and requests from #3555 were to have the ability to set .Chart.AppVersion at install and upgrade time without having to download and repackage the chart. Those comm…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

This would be nice as we frequently use our monochart to deploy a wide range of applications and would like to set the .Chart.AppVersion as well

Craig Dunford avatar
Craig Dunford

yeah - like many in that thread, I end up with many deployments out there that show an appVersion that isn’t actually what is deployed

s_slack avatar
s_slack

+1

Carlos R. avatar
Carlos R.
05:30:06 PM

@Carlos R. has joined the channel

roth.andy avatar
roth.andy

What is currently the best practice for installing CRDs before deploying something that needs them? For example, the instructions for using the Helm Operator are to:

  1. Install the HelmRelease CRD using kubectl apply
  2. Deploy the operator using Helm If I want both to happen with one helmfile apply , what’s the best way right now to do that? I’m thinking of the incubator/raw helm chart, but that sounds kinda kludgey
roth.andy avatar
roth.andy

@Erik Osterman (Cloud Posse) I’m pretty sure its been talked about in OO at some point but I don’t remember what was said

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Hah, this topic comes up for us in our engagements all the time. We haven’t found the holy grail yet, but can share the best practices we follow. I am on my phone right now. Will set a reminder

roth.andy avatar
roth.andy

:–1:

Andrey Nazarov avatar
Andrey Nazarov

We’ve already tried different things: helmfile hooks, incubator/raw, home-made charts. Cannot say we are pleased with something specific. Also, we try to use needs: as much as we can to make a dependency tree. When we install third-parties with crds folder we use disableValidation flag that tells helm-diff not to fail if something cannot find a certain object type.

:--1:1
roth.andy avatar
roth.andy

Found the helmfile for cert-manager that uses presync hooks. Looks promising.

https://github.com/cloudposse/helmfiles/blob/master/releases/cert-manager.yaml

cloudposse/helmfiles

Comprehensive Distribution of Helmfiles for Kubernetes - cloudposse/helmfiles

roth.andy avatar
roth.andy
[doc] Recommended way of handling non conventional install such as cert-manager · Issue #538 · roboll/helmfile

HI, When reading the process to install the cert-manager chart (https://hub.helm.sh/charts/jetstack/cert-manager), you can see two steps before installing the chart: installing some CRD prior to th…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Jeremy (Cloud Posse) if he has time can maybe share something quick.

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

You can do everything with helmfile hooks, although I find that unwieldy for all but the simplest commands, so for more complicated stuff, like installing, upgrading, and removing cert-manager CRDs, I prefer to use the hook to run a custom script (which mainly runs kubectl commands, because until last week, jetstack was warning that helm still had issues with CRDs, apparently just resolved in the past few days with the release of helm 3.3.1).

For cert-manager specifically, I have a presync hook that calls the script to install or upgrade the CRDs, a postuninstall script (actually the same script with different arguments) to remove the CRDs, and a postsync hook that does nothing but sleep so that cert-manager has time to set up its CA and webhooks before the next release runs.

The next release, BTW, is in the same helmfile.yaml and “needs” the first one, installs ClusterIssuers. It also has to have

disableValidation: true

because otherwise helmfile apply will fail to install the releases because the CRDs are not yet installed and so the ClusterIssuers are not valid resources yet.

cert-manager/website

Source code for the cert-manager.io website, including project documentation - cert-manager/website

roth.andy avatar
roth.andy

I’m about halfway through @mumoshu’s way to do it with the helmfiles: section. I’ll report back on how that goes and also try the hooks way if I run into any issues.

So far so good though.

Pierre Humberdroz avatar
Pierre Humberdroz
  - name: cert-manager
    chart: jetstack/cert-manager
    namespace: cert-manager
    version: v0.13.1
    verify: false
    values:
      - ../charts/cert-manager/values.yaml
    hooks:
      - events: ["prepare"]
        showlogs: true
        command: "/bin/sh"
        args: ["-c", "kubectl apply --validate=false -f <https://raw.githubusercontent.com/jetstack/cert-manager/release-0.13/deploy/manifests/00-crds.yaml>"]
  - name: prometheus-operator
    chart: stable/prometheus-operator
    namespace: monitoring
    version: 9.3.1
    verify: false
    values:
      - ../charts/prometheus-operator/values.yaml
      - ../charts/prometheus-operator/values-{{ requiredEnv "CI_ENVIRONMENT_SLUG" }}.yaml
    hooks:
      - events: ["prepare"]
        showlogs: true
        command: "/bin/sh"
        args:
          - "-c"
          - "kubectl apply -f <https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.38/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml>"
      - events: ["prepare"]
        showlogs: true
        command: "/bin/sh"
        args:
          - "-c"
          - "kubectl apply -f <https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.38/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml>"
      - events: ["prepare"]
        showlogs: true
        command: "/bin/sh"
        args:
          - "-c"
          - "kubectl apply -f <https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.38/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml>"
      - events: ["prepare"]
        showlogs: true
        command: "/bin/sh"
        args:
          - "-c"
          - "kubectl apply -f <https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.38/example/prometheus-operator-crd/monitoring.coreos.com_prometheusrules.yaml>"
      - events: ["prepare"]
        showlogs: true
        command: "/bin/sh"
        args:
          - "-c"
          - "kubectl apply -f <https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.38/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml>"
      - events: ["prepare"]
        showlogs: true
        command: "/bin/sh"
        args:
          - "-c"
          - "kubectl apply -f <https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.38/example/prometheus-operator-crd/monitoring.coreos.com_thanosrulers.yaml>"
Pierre Humberdroz avatar
Pierre Humberdroz

this is what I am doing so far

Andrey Nazarov avatar
Andrey Nazarov

Prepare hooks will be executed even for helmfile template . Is this intended?

Andrey Nazarov avatar
Andrey Nazarov

As for ClusterIssuer we move it from hooks to a separate Helm release as we had issues when it was not applied. Say, if you fix just a ClusterIssuer manifest helmfile diff won’t see a diff of a cert-manager release and a presync/postsync hook won’t be executed.

2020-09-03

Release notes from helmfile avatar
Release notes from helmfile
01:16:38 AM

v0.128.0: Bump sprig to v3.1.0 and mergo 3.11 (#1456) 9d2c0d4 (HEAD, tag: v0.128.0, origin/master, origin/HEAD, master) Bump sprig to v3.1.0 and mergo 3.11 (<a class=”issue-link js-issue-link” data-error-text=”Failed to load title” data-id=”691909066” data-permission-text=”Title is private”…

Release v0.128.0: Bump sprig to v3.1.0 and mergo 3.11 (#1456) · roboll/helmfile

9d2c0d4 (HEAD, tag: v0.128.0, origin/master, origin/HEAD, master) Bump sprig to v3.1.0 and mergo 3.11 (#1456) Build Info

Bump sprig to v3.1.0 and mergo 3.11 (#1456) · roboll/[email protected]
  • Bump sprig to v3.1.0 test for mergeOverwrite * Let mergo not (accidentally) try to merge unexported fields This is also a good chance separate HelmState with the config loaded from YAML,…

2020-09-02

bradym avatar
bradym

I have an app that I want to deploy to stage only when the branch being deployed matches the branch name specified in an aws ssm parameter. I’m attempting to use installed template on my release, but I haven’t been able to figure out the correct syntax or even if this is possible.

A simple version that doesn’t include ssm (that I’ve confirmed works) would be:

installedTemplate: {{- if eq .Values.branch "master" }} true {{ else }} false {{ end }}

What I’ve tried for the ssm version:

installedTemplate: {{- if eq .Values.branch `{{ <ref+awsssm://DOWNLOAD_WORKER_BRANCH?region=us-west-1> }}` }} true {{ else }} false {{ end }}

Does anyone know if it’s possible to do this? Any pointers in the right direction?

bradym avatar
bradym

When I run helmfile build on it with the ssm version there are no errors, but it always returns false. I assume this is because it’s not rendering the ssm value and just using it as a string.

bradym avatar
bradym

Just discovered fetchSecretValue, now to figure out how to correctly use it

bradym avatar
bradym

And this is what works:

environments:
  stage:
    values:
      - downloadWorkerBranch: <secretref+awsssm://stage/DOWNLOAD_WORKER_BRANCH>
  prod:
    values:
      - downloadWorkerBranch: <secretref+awsssm://prod/DOWNLOAD_WORKER_BRANCH>

Then in the release:

    
installedTemplate: {{- if eq (.Values.branch | quote) (.Values.downloadWorkerBranch | fetchSecretValue | quote) }} true {{ else }} false {{ end }}

If anyone sees that I’m doing something dumb, I’d love your feedback!

:--1:1

2020-09-01

Release notes from helmfile avatar
Release notes from helmfile
08:46:24 AM
Helmfile charts deletion order is not correct if we have more than 6 files in sub-charts folder · Issue #1450 · roboll/helmfile

Hello! Helmfile chart deletion (from the last chart to first) order is not correct if we have more than 6 files in sub-charts folder. For example, for 7 files we have the next deletion order - firs…

Release notes from helmfile avatar
Release notes from helmfile
08:56:26 AM

v0.126.2: Fix wrong deletion order (#1451) 5f1698d (HEAD, tag: v0.126.2, origin/master, origin/HEAD, master) Fix wrong deletion order (<a class=”issue-link js-issue-link” data-error-text=”Failed to load title” data-id=”689932583” data-permission-text=”Title is private”…

Release notes from helmfile avatar
Release notes from helmfile
10:46:20 AM

v0.127.0: Bump sprig to v3 (#1452) Resolves #1294 Resolves <a class=”issue-link js-issue-link” data-error-text=”Failed to load title” data-id=”689490982” data-permission-text=”Title is private”…

Upgrade Sprig library to v3 · Issue #1294 · roboll/helmfile

Hello, I am wondering if upgrading the Sprig library to v3 is on the roadmap? I am a bit too uninvolved to see if it would require a considerable amount of effort to upgrade or not, anyhow I am rea…

Release notes from helmfile avatar
Release notes from helmfile
11:06:24 AM

v0.127.0: Bump sprig to v3 (#1452) efd26f2 (HEAD, tag: v0.127.0, origin/master, origin/HEAD, master) Bump sprig to v3 (<a class=”issue-link js-issue-link” data-error-text=”Failed to load title” data-id=”689956556” data-permission-text=”Title is private”…

1
jason800 avatar
jason800

Hey @mumoshu, I found a bug caused by https://github.com/roboll/helmfile/pull/1442 related to needs:. I have a helmfile that works perfectly fine on v0.125.8 but breaks on v0.125.9. The break is caused by the helmfile thinking a needs: item is invalid when in reality it is completely valid.

jason800 avatar
jason800
err: "preprod_us-phoenix-1_deployment/infra-monitoring/thanos" depends on nonexistent release "infra-monitoring/prometheus-oper-iad-controlplane"

changing working directory back to "/home/jmwitkow/osvc-infra/helm/helmfile-releases"

in infra-monitoring/helmfile-infra-monitoring.yaml: "preprod_us-phoenix-1_deployment/infra-monitoring/thanos" depends on nonexistent release "infra-monitoring/prometheus-oper-iad-controlplane"
jason800 avatar
jason800

If i switch to v0.125.8 it works, v0.125.9 it does not. I am generating the needs: block dynamically, but that probably shouldn’t matter?

jason800 avatar
jason800

the needs block looks like this:

   needs:
    # Generate list of releases that thanos has a dependency on
    {{- range $realmRegions }}
      {{- $regionMap := . }}
      {{- $region := .name }}
      {{- $region_key := .key }}
      {{- range $k8sClusters}}
        {{- $cluster := . }}
        {{- if and ($regionMap | get "deploy_region" false) (eq $cluster "deployment") }}
      - infra-monitoring/prometheus-oper-{{ $region_key }}-{{ $cluster }}
        {{- else if ne $cluster "deployment" }}
      - infra-monitoring/prometheus-oper-{{ $region_key }}-{{ $cluster }}
        {{- end }}
      {{- end }}
    {{- end }}
jason800 avatar
jason800
mumoshu avatar
mumoshu

@ thanks for reporting! could you share me the output of helmfile build for debugging, via a slack direct message if you prefer?

mumoshu avatar
mumoshu

ah i think i’ve spotted the cause. can you prepend preprod_us-ashburn-1_controlplane/ to needs entries?

mumoshu avatar
mumoshu

helmfile since v0.125.9 started to treat kubecontext as a part of release id

mumoshu avatar
mumoshu

that’s because different kubecontext can point to different release in another cluster

mumoshu avatar
mumoshu
0.125.9 release breaks "needs:" parameter. · Issue #1453 · roboll/helmfile

In our releases we build dynamic needs: parameters sometimes. An example of that looks like: needs: # Generate list of releases that thanos has a dependency on {{- range $realmRegions }} {{- $regio…

jason800 avatar
jason800

oh wow thanks

jason800 avatar
jason800

i will try that now

jason800 avatar
jason800

Hey @mumoshu unfortunately this does not appear to fix things

jason800 avatar
jason800
$ helmfile -e dev -f infra-monitoring/helmfile-infra-monitoring.yaml sync
Affected releases are:
  grafana (stable/grafana) UPDATED
  prometheus-oper-fra-controlplane (bitnami/prometheus-operator) UPDATED
  prometheus-oper-fra-dataplane (bitnami/prometheus-operator) UPDATED
  prometheus-oper-fra-deployment (bitnami/prometheus-operator) UPDATED
  prometheus-oper-lhr-controlplane (bitnami/prometheus-operator) UPDATED
  prometheus-oper-lhr-dataplane (bitnami/prometheus-operator) UPDATED
  thanos (bitnami/thanos) UPDATED

panic: invalid state: no nodes have remaining edges: nodes=[dev_eu-frankfurt-1_deployment/infra-monitoring/thanos]
jason800 avatar
jason800

ok, this was related to me changing release names without destroying them first, but I actually am hitting what I believe to be another bug related to dynamic releases and static release names @mumoshu

STDERR:
  Error: failed to untar: a file or directory with the name /tmp/341109564/infra-monitoring/prometheus-operator/bitnami/prometheus-operator/0.31.0/prometheus-operator already exists
jason800 avatar
jason800

it looks like the multiple releases which are unique are using the same temporary directory due to the changes I made for release names to be non-dynamic

jason800 avatar
jason800

and therefore they run into conflicts with files already existing as it loops over the many releases with the same namespace/name

jason800 avatar
jason800

@ you may have some of your typical wisdom here as well?

jason800 avatar
jason800

Looks like this was reported in https://github.com/roboll/helmfile/issues/1384

helmfile lint fails when same release name is used (in multiple namespaces) · Issue #1384 · roboll/helmfile

helmfile lint fails when same release name is used (in multiple namespaces) STDERR: Error: failed to untar: a file or directory with the name /tmp/299510502/keycloak/8.1.0/codecentric/keycloak/keyc…

jason800 avatar
jason800

I’m wondering if the kubeContext uniqueness change broke it again

voron avatar
voron


you may have some of your typical wisdom here as well?
I’m afraid no, we don’t use dynamic releases widely.

jason800 avatar
jason800

I actually believe I have a code fix for it

jason800 avatar
jason800

just trying to figure out how to build helmfile

voron avatar
voron
roboll/helmfile

Deploy Kubernetes Helm Charts. Contribute to roboll/helmfile development by creating an account on GitHub.

jason800 avatar
jason800

Installing Go now

fast_parrot1
voron avatar
voron

don’t forget to pass tests locally before submitting a PR

jason800 avatar
jason800

I’m afraid this may be beyond my experience at this point. I’m getting an error building on my diff but I have no idea why?

jason800 avatar
jason800
diff --git a/pkg/state/state.go b/pkg/state/state.go
index 4eddbd0..9c9422d 100644
--- a/pkg/state/state.go
+++ b/pkg/state/state.go
@@ -972,6 +972,10 @@ func (st *HelmState) PrepareCharts(helm helmexec.Interface, dir string, concurre
                                                pathElems = append(pathElems, release.Namespace)
                                        }

+                                       if release.KubeContext != "" {
+                                               pathElems = append(pathElems, release.KubeContext)
+                                       }
+
                                        chartVersion := "latest"
                                        if release.Version != "" {
                                                chartVersion = release.Version
git ls-files --exclude-standard --modified --deleted --others -x vendor  | grep -v '^go.' | diff /dev/null -
0a1
> pkg/state/state.go
make: *** [pristine] Error 1
jason800 avatar
jason800

almost looks like its upset just because there is a diff

jason800 avatar
jason800

make clean goes fine. make pristine fails, make test succeeds.

PASS
coverage: 65.7% of statements
ok      [github.com/roboll/helmfile/pkg/tmpl](http://github\.com/roboll/helmfile/pkg/tmpl)     0.258s  coverage: 65.7% of statements
?       [github.com/roboll/helmfile/test](http://github\.com/roboll/helmfile/test) [no test files]
(osvc-infra) [[email protected] helmfile]$ echo $?
0
jason800 avatar
jason800

so I’m guessing maybe i shouldn’t be running make pristine ?

jason800 avatar
jason800

i built a binary and confirmed it fixed my issue as well

jason800 avatar
jason800
Append KubeContext to temporary directory path by jwitko · Pull Request #1471 · roboll/helmfile

Fix bug where KubeContext is not taken into account in temporary directories. Fixes #1470

:--1:2
K H avatar

I’m trying to use environments and bases at the same while using variables in helmfile.yaml , I’m not sure if this is a bug or I’m doing something wrong, https://github.com/roboll/helmfile/issues/1454

Defining environments in a an separate layer doesn't work · Issue #1454 · roboll/helmfile

I&#39;m trying to use environments and layers at the same time while using a variable in my helmfile.yaml: bases: - environments.yaml repositories: - name: stable url: https://kubernetes-charts.sto

mumoshu avatar
mumoshu

@ Thanks for reporting! Just replied in the issue.

Defining environments in a an separate layer doesn't work · Issue #1454 · roboll/helmfile

I&#39;m trying to use environments and layers at the same time while using a variable in my helmfile.yaml: bases: - environments.yaml repositories: - name: stable url: https://kubernetes-charts.sto

1
1
    keyboard_arrow_up