#helmfile (2020-10)

Message you provided is a helm message actually, helmfile gets it from helm and outputs to stdout/stderr.

IMHO helmfile and flux&helm-operator perform the same thing. helm-operator looks a little bit young project, as well as tied to flux. I don’t think it’s a good idea to create a chart ( or use incubator/raw chart) just to generate HelmRelease CRD. I don’t see a great secrets solution as well with HelmRelease, as I wanna stay in deploy-via-helm-only and I don’t wanna spread deploy to multiple moving parts.

Yeah, I think Flux / HO vs Helmfile is a “Pick one or the other” decision.

HelmRelease files are not very complicated and templating them via Helmfile wouldn’t make a lot of sense I don’t think. If you needed to template your HelmRelease files (because you have dozens and dozens of them then I would think Kustomize might be worth looking into.

there are any solution ?? because i also want to keep my gitops tool

Sry for the delay! Good to know that there is strvals package to support this. I’ve reviewed it and it looks generally good.

If you could fix the test failure, that would be great. Otherwise I’ll fix it myself in a few days

I’m looking into it now - interestingly I get a different error on my local machine, but I’m guessing the root cause is similar (helm not being on the PATH seen by tests):

oh no maybe not, this is something to do with HelmState.valsRuntime.Eval

(fwiw, and alternatively speaking, helmfile natively supports retrieving secrets from many different backends like SSM, vault, etc)

Also: https://github.com/roboll/helmfile/issues/1539

Man, the speed at which he creates things like this is insane!

Yes, there are different approaches. I’ll try to find a discussion related to this

found this https://sweetops.slack.com/archives/CE5NGCB9Q/p1599251313090200?thread_ts=1599247141.077800&cid=CE5NGCB9Q

Yep, this is exactly it

maybe related to https://github.com/roboll/helmfile/issues/467

oof, my bad, i mixed app version and chart version

Btw, we’ve just migrated to https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack

Back then we used https://github.com/helm/charts/tree/master/stable/prometheus-operator which is now deprecated

thank you @Andrew Nazarov i used this -> https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack to replicate the same things but i showed Error: Chart requires kubernetesVersion: >=1.16.0-0 which is incompatible with Kubernetes v1.14.10 i will try the deprecated link which you provided

It’s possible to use not the latest version of kube-prometheus-stack. We have one cluster with k8s 1.15.12 and we deployed 9.4.10 there.

I have met the problem, and @mumoshu give me a great solution. https://github.com/roboll/helmfile/issues/724#issuecomment-609039954

My usage is just like your case

{{- range $_, $file := ( exec "bash" (list "-c" "echo -n dashboards/general/*.json") | splitList " " ) }}
{{ trimSuffix (ext $file) (base $file) }}:
json: |
    {{- readFile $file | nindent 10 }}
{{- end }}

hmm, you could possibly use the incubator/raw helm chart to do that, using the templates: [] value.

that runs through the helm templating engine

oh although I don’t know what path it would read the files from, so not sure that would work.

Hmm, I guess that’s not what I need. The problem with Helm is that it doesnt read files outside of the helm root folder. And my value files (chart values and environment values) are outside the chart folder

fixed it by

{{- range $_, $file := ( exec "bash" (list "-c" "echo -n secrets-volume/**") | splitList " " ) }}
{{ $bytes := readFile $file }}
{{- end }}

Oh yeah, I always forget about exec

Maybe you could layer your value files en env files. Maybe this article could help you on how to set it up https://github.com/roboll/helmfile/blob/master/docs/writing-helmfile.md

Hi @Joaquin Menchaca Hm… are you sure it is a good idea to mix helmfile and kustomize? Especially if you don’t find approach with overlays too DRY or too attractive? If you prefer working with templates… would you rather consider raw chart instead?

There’s no other choice. In the community at large, *operators* are all the rage, and often there’s no helm chart around installing or using the *operator*. So if I want to use helmfile with the operator and things that use the CRD, I have to use ./helmify.sh.

Now I could instead write a helm chart for every operator that doesn’t support helm charts, but by the time I finished, I would be fired and unemployed, for spending too much time writing helm charts for other projects, rather than adding features to my company’s project.

The last two I came across are jaeger-operator (which can use kafka, spark, cassandra, elasticsearch operators) and rook ceph operator.

Well… I haven’t used ‘helmify.sh’ myself. I hope someone else could assist you with this! I can only add that in case if you are on fire and running out of time maybe consider using presync and postsync event hooks to execute script with direct call to kustomize… I believe you should be able to pass required variables. Or create a lazy chart with all of the values hardcoded… and template it later when you have more time :-)

That sounds like an option. kustomize edit can inject values, very limited. I am not sure how I could down pass values from helmfile environment, like namespace or values.

For a workaround, I am creating kustomize values with "{{ .Values.nfs.path }}", which are then consumed by values I inject to the final rendered helm chart from helmfile. The downside, is that this is (1) overly complex to people that my use it, (2) brittle.

I would like to inject a set of default values, such as a value.yaml into the final rendered chart from helmify.sh. That way I can continue to use factory pattern method documented for helmfile.

Side note, I noticed that helmfile is not able to inject a namespace into the rendered helmchart. I’m not sure why/how this is. So I added a namespace inside the kustomize.yaml to get the same namespace. helmfile at least creates the namespace if it doesn’t exist.

The kustomize was smart enough to inject the namespace to metadata.namespace where appropriate, and even change the ClusterRoleBinding subjects[0].namespace as well.

@Joaquin Menchaca helmfile delegates setting metadat.aname in the helm-template result to… helm.

helm is only able to set metadata.name in the template output if and only if the chart supports it.

not sure this was what you were struggling. but hope it helps!

For example, cluster-autoscaler doesn’t support setting metadat.anamespace in the template output, cuz it doesn’t have namespace: {{ .Release.Namespace }} in the template https://github.com/helm/charts/blob/b9278fa98cef543f5473eb55160eaf45833bc74e/stable/cluster-autoscaler/templates/deployment.yaml#L8

cert-manager is correct. It has namespace: {{ .Release.Namespace }} in the template https://github.com/helm/charts/blob/10ddb0c607525b323d0b0ceb301ca82bbd31b6f9/stable/cert-manager/templates/deployment.yaml#L5

Fortunately theres’ a workaround. Helmfile has the forceNamespace setting to modify the helm-template output and sets metadata.namespae regardless of the chart supports it or not

https://github.com/roboll/helmfile/pull/1444

:point_up: FYI everyone, helmfile has forceNamespace that allows you to set metadata.namespace in the helm-template output even if the chart doesn’t support it

The helmify.sh script from the gist, didn’t have this in there, as it took the final raw k8s manifests.

One thing it would be nice is to actually inject values into the kustomize scripts on the fly, such as namespace (like a gotemplated kustomize), then run the helmify.sh script.

is your helmify.sh is something similar to https://gist.github.com/mumoshu/f9d0bd98e0eb77f636f79fc2fb130690?

It’s exactly that one

ah ok

i haven’t read throughout all this thread yet, but have you tried the kustomize integration?

https://github.com/roboll/helmfile/blob/master/test/advanced/helmfile.yaml#L7

If I could have the kustomize.yaml.gotemplate rendered, then run kustomize… that would be good workflow

tl;dr; you can point your kustomization directory in chart:

so that helmfile automatically turns the kustomization into a temporary chart

also you can use chart values to set certain parameters on the kustomize project. see

https://github.com/variantdev/chartify/blob/master/testdata/kustomize/input/values.yaml

I didn’t know that there was direct support, if I am reading the example correctly.

I was using the hook method style with the helmify-kustomize script

example:

  - name: jaeger
    namespace: monitoring
    chart: ./jaeger
    needs:
      - monitoring/jaeger-operator
    hooks:
      - events:
          - prepare
          - cleanup
        command: ./helmify.sh
        args:
          - "{{`{{if eq .Event.Name \"prepare\"}}build{{else}}clean{{end}}`}}"
          - "{{`{{.Release.Chart}}`}}"
          - badger
    disableValidation: true

gotcha. i certainly forgot to update the doc about that

BTW. Side note, thank you so much for disableValidation and needs.

Lifesaver

glad to hear

When I had an exact problem, you had an exact solution

operators + crds w/ kustomize becoming popular, so this was quite useful…

So chartify is more robust kustomize-2-chart

yeah.. it was pain for me as well to manage kustomize/helm mix

i believe so

i think you can now rewrite your example with

releases:
  - name: jaeger
    namespace: monitoring
    chart: ./jaeger
    needs:
      - monitoring/jaeger-operator

or with predefined values

releases:
  - name: jaeger
    namespace: monitoring
    chart: ./jaeger
    values:
    - namespace: whatever #=> kustomize edit set namespace=whatever
    needs:
      - monitoring/jaeger-operator

That’s gorgeous

So chartify is apart of helmify now, as a library?

My overlay kustomize is:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

bases:
  - ../../base
patches:
  - storage.yaml
  - ingress.yaml

I didn’t overlay w/ environment, but rather features I wanted to toggle on/off

basically yes. chartify is integrated into helmfile

the jaeger has so many backend features, badger, cassandra, elasticsearch for storage, and spark/kafka integration features, so these would be alternative overlays.

That’s awesome.

Some examples, or unit tests, would be nice, I’ll blog about this stuff, add some docs… this is beyond awesome, needs to be shared.

ah gotcha. that sounds like a valid use-case from jaeger authors perspective :slightly_smiling_face:

from helmfile perspective it might be a bit annoying to manage depending on reqs

i believe someone would want to dynamically generate patches array in the kustomization.yaml from helmfile values. that’s impossible today

a workaround for that pattern would be to prepare a few variation of kustomization.yaml, and switch that depending on helmfile values

releases:
  - name: jaeger
    namespace: monitoring
    chart: ./jaeger-{{ .Values.jaegarVarigationName }}
    needs:
      - monitoring/jaeger-operator

other than that the integrated chartify feature should work perfectly

I am in a quasi devops, solutions architect, developer support role, so I create code snippets, docs, for tools (monitoring, observability, logs) that could integrate to our product, so that’s why I code various options for customers, they setup something like env.sh, chose what features.

that does make sense a lot

BTW, separate topic, what do you think about kustomize. Helm is quite different approach.

for me both are fine. but being locked-in to one of them would never be best..

kustomize folks are opinionated in what they think is best practices is, which seems particular implementation of gitops. From their docs.

yeah. and that “opinion” seems generally ok to me

in the context of gitops, for me, it’s just kustomize build vs helm template, and which format the specific user wants read/write, kustomization or helm chart

imho helm chart template is much powerful as you can work on text, not only yaml. but great power comes with great responsibiliy

Yep

Crazy question (another topic), ever thought of doing something similar with docker-compose? Right now, it is pretty inflexible, gotemplating on it sounds nice. It’s becoming a standard (https://www.compose-spec.io/), and supposedly would be an alternative to kubectl manifests.

In the mean time I was thinking of using gomplate for this.

variant2 and https://github.com/cloudposse/atmos should be a great alternative for that cc/ @Erik Osterman (Cloud Posse)

This is orchestrator orchestrator orchestrator orchestrator…

I just gave a demo of helmfile to sres and qa in another group today, they were so excited and full of imagination on the possibilities now.

Hello, helm will merge values from in-chart values.yaml and my-value.yaml , and values from my-value.yaml will override corresponding existing values from values.yaml . You may check official manual too