#helmfile (2023-10)

do helm support this feature? add custom labels for all resource?

helm does not, but kustomize does for sure

IMHO would the best solution be to include something like a “labels.yaml.gotmpl” file in the environments values section. haven’t managed to get that to work yet, hence my question

Yes, you can use the commonLabels config entry to add a set of labels to everything and can use that in helmfile templates.

There is release attribute installed:, it will install release when true and remove the release when false , while issuing helmfile sync

This looks like a bug in goccy/go-yaml library. You should log a bug there.

Even though, I don’t know all helmfile’s logic around parsing yamls, and in order to pinpoint the issue I need help from someone who knows helmfile codebase

I filled my findings at https://github.com/helmfile/helmfile/issues/1079

Have you tried double escaping the input

So put \ in the tmpl

This is just a guess,

As an example. there is a chart that renders a PrometheusRule CRD; and cutting out the parts that aren’t of interest to the problem, like metadata and other stuff in the spec, the helm template file looks like:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
spec:
  groups:
{{- range .Values.prometheusRules.extraAlertGroups }}
  - {{ tpl (toYaml .) $ | indent 4 | trim }}
{{- end }}

And an unescaped version of the helmfile values file would have something like:

prometheusRules:
  extraAlertGroups:
    - name: custom.rules
      rules:
        - alert: CertificateError
          annotations:
            description: Certificate could not be decoded {{if $labels.secret_name }}in
              Kubernetes secret "{{ $labels.secret_namespace }}/{{ $labels.secret_name
              }}"{{else}}at location "{{ $labels.filepath }}"{{end}}
            summary: Certificate cannot be decoded
          expr: x509_cert_error > 0
          for: 15m
          labels:
            severity: warning

Would result in the error:

in ./helmfile.yaml: failed processing release x509-certificate-exporter: failed to render values files "helm/vars/x509-certificate-exporter.yaml.gotmpl": failed to render [helm/vars/x509-certificate-exporter.yaml.gotmpl], because of template: stringTemplate:21: undefined variable "$labels"

Changing it to use {{ and }} for the description like so

      rules:
        - alert: CertificateError
          annotations:
            description: {{` Certificate could not be decoded {{if $labels.secret_name }}in
              Kubernetes secret "{{ $labels.secret_namespace }}/{{ $labels.secret_name
              }}"{{else}}at location "{{ $labels.filepath }}"{{end}} `}}
            summary: Certificate cannot be decoded
          expr: x509_cert_error > 0
          for: 15m
          labels:
            severity: warning

still results in the exact same error.

I haven’t worked out how to get around it…

Ughhh.. This works.

        - alert: CertificateError
          annotations:
            description: '{{`{{`}}{{printf "\x60"}}{{`Certificate could not be decoded {{if $labels.secret_name }}in
              Kubernetes secret "{{ $labels.secret_namespace }}/{{ $labels.secret_name
              }}"{{else}}at location "{{ $labels.filepath }}"{{end}}`}}{{printf "\x60"}}{{`}}`}}'
            summary: Certificate cannot be decoded
          expr: x509_cert_error > 0 
          for: 15m
          labels:
            severity: warning

You have the innermost {{ and }} which keeps things from expanding the first time.

Then you have

{{`{{`}}{{printf "\x60"}}

which produces a

{{

` and

{{printf "\x60"}}{{`}}`}}

to produce

`}}

I couldn’t just use

{{`{{

` and

`}}`}}

Finally I had to put a single quote at the start and the end to stop a error converting YAML to JSON: yaml: line 20: did not find expected key error. As far as I can tell it didn’t like the raw backtick appearing without it being quoted.

The above is a nightmare… Surely there’s an easier way? I even had trouble trying to render those things how I want to in Slack!

If I try:

            description: '{{`{{`Certificate could not be decoded {{if $labels.secret_name }}in
              Kubernetes secret "{{ $labels.secret_namespace }}/{{ $labels.secret_name
              }}"{{else}}at location "{{ $labels.filepath }}"{{end}}`}}`}}'

then I get the error: failed to render [helm/vars/x509-certificate-exporter.yaml.gotmpl], because of template: stringTemplate:32: unexpected "Certificat"... in operand

The nicest way I could find to format the rule was:

        - alert: CertificateError
          annotations:
            description: >
              {{`{{`}}{{printf "\x60"}}
              {{`
              Certificate could not be decoded
              {{if $labels.secret_name }}
                in Kubernetes secret "{{ $labels.secret_namespace }}/{{ $labels.secret_name }}"
              {{else}}
                at location "{{ $labels.filepath }}"
              {{end}}
              `}}
              {{printf "\x60"}}{{`}}`}}
            summary: Certificate cannot be decoded
          expr: x509_cert_error > 0
          for: 15m
          labels:
            severity: warning

We just use .yaml instead of .yaml.gotmpl and it solves the problem

Thank you. I knew I had to be overlooking something that should be obvious. Now I just have to escape once with {{ }} to protect the rule from the tpl function that’s used in the chart.

We don’t have to use any escaping with kube-prometheus-stack chart

Yeah, that chart doesn’t have tpl used within its templates/prometheus/additionalPrometheusRules.yaml where it applies the additionalPrometheusRulesMap and additionalPrometheusRules values.

hey @Tim, I don’t think this line is valid {{ .Values.{{ .Release.Name }}-external-secrets | expandSecretRefs | toYaml | indent 4 }} when referencing variable dynamically. Also, it might be worth creating a script to dynamically retrieve the secrets.

Thanks, I have resolved it by mapping the entire external secret into files on a volume

@Jim please post a issue in to github issues.

https://github.com/helmfile/helmfile/issues/1093 Note that removing the first --- document start marker solves the warning, however having that is valid YAML (and by default yamllint will warn if you don’t have it.

@Jim fixed

Thank you.

Did you consider to create a PR with documentation updates ?

To be honest, I don’t feel comfortable adding a description of current behavior to the docs, because I’m not sure exactly what happens.

I’m confused by this: https://github.com/helmfile/helmfile/pull/602

A --reset-values CLI flag has been added, but there’s no way to do the same via Helmfile’s YAML?

I guess this is how it works: reset is the default. This can be overridden in Helmfile via reuseValues. This can again be overridden via CLI flag --reset-values.

However, that’s not the whole truth, because they can be used together (see comment).

A --reset-values CLI flag has been added, but there’s no way to do the same via Helmfile’s YAML? reset-values is the default behaviour of helmfile imo. That’s why there is no need to do it via yaml. Introduced CLI flag --reset-values allows to override reuseValues yaml option to get back to default behaviour.

I guess this is how it works: reset is the default. This can be overridden in Helmfile via reuseValues. This can again be overridden via CLI flag --reset-values. Correct.
However, that’s not the whole truth, because they can be used together (see comment). And reset will win in this case, same as with helm.
To be honest, I don’t feel comfortable adding a description of current behavior to the docs, because I’m not sure exactly what happens. It may be a good point to create a PR to improve docs to look clear for yourself, and let reviewers to fix issues if any.

not sure how helmfile implements it, but I think the recommended way to handle all creation and updates of kubernetes is now the equivalent of kubectl apply --server-side=true.

https://github.com/kubernetes/community/blob/master/contributors/devel/sig-api-machinery/strategic-merge-patch.md

This document is old and probably obsolete. Server-side Apply is the new solution and is the recommended way to go. It doesn't require any client-side logic.

We use raw chart to do it helmfile-friendly. Yes, it’s another helm release, not an original one.

It’s up to you, and let it evolve with your helmfile experience. A couple of refactors are ok here. Keep it simple and clear for you. The most used layout from our side is the following:

.
├── .sops.yaml
├── common
│   ├── helm-defaults.yaml
│   ├── repositories.yaml
│   ├── templates.yaml
│   └── values
│       └── ingress-nginx.yaml.gotmpl
├── environments
│   ├── asia
│   │   ├── .envrc
│   │   ├── helmfile.yaml
│   │   ├── secrets
│   │   │   └── ingress-nginx.yaml
│   │   └── values
│   │       └── ingress-nginx.yaml.gotmpl
│   ├── europe
│   │   ├── .envrc
│   │   ├── helmfile.yaml
│   │   ├── secrets
│   │   └── values
│   └── us
│       ├── .envrc
│       ├── helmfile.yaml
│       ├── secrets
│       └── values
└── helmfile.yaml

common are values that are used among all environments?

Yes. To keep things reasonably DRY.