#helmfile (2020-09)

If i switch to v0.125.8 it works, v0.125.9 it does not. I am generating the needs: block dynamically, but that probably shouldn’t matter?

the needs block looks like this:

   needs:
    # Generate list of releases that thanos has a dependency on
    {{- range $realmRegions }}
      {{- $regionMap := . }}
      {{- $region := .name }}
      {{- $region_key := .key }}
      {{- range $k8sClusters}}
        {{- $cluster := . }}
        {{- if and ($regionMap | get "deploy_region" false) (eq $cluster "deployment") }}
      - infra-monitoring/prometheus-oper-{{ $region_key }}-{{ $cluster }}
        {{- else if ne $cluster "deployment" }}
      - infra-monitoring/prometheus-oper-{{ $region_key }}-{{ $cluster }}
        {{- end }}
      {{- end }}
    {{- end }}

I’ve created https://github.com/roboll/helmfile/issues/1453 with details

@jason800 thanks for reporting! could you share me the output of helmfile build for debugging, via a slack direct message if you prefer?

ah i think i’ve spotted the cause. can you prepend preprod_us-ashburn-1_controlplane/ to needs entries?

helmfile since v0.125.9 started to treat kubecontext as a part of release id

that’s because different kubecontext can point to different release in another cluster

https://github.com/roboll/helmfile/issues/1453#issuecomment-685192949

oh wow thanks

i will try that now

Hey @mumoshu unfortunately this does not appear to fix things

ok, this was related to me changing release names without destroying them first, but I actually am hitting what I believe to be another bug related to dynamic releases and static release names @mumoshu

STDERR:
  Error: failed to untar: a file or directory with the name /tmp/341109564/infra-monitoring/prometheus-operator/bitnami/prometheus-operator/0.31.0/prometheus-operator already exists

it looks like the multiple releases which are unique are using the same temporary directory due to the changes I made for release names to be non-dynamic

and therefore they run into conflicts with files already existing as it loops over the many releases with the same namespace/name

@voron you may have some of your typical wisdom here as well?

Looks like this was reported in https://github.com/roboll/helmfile/issues/1384

I’m wondering if the kubeContext uniqueness change broke it again

you may have some of your typical wisdom here as well? I’m afraid no, we don’t use dynamic releases widely.

I actually believe I have a code fix for it

just trying to figure out how to build helmfile

https://github.com/roboll/helmfile/blob/master/Makefile, make build ?

Installing Go now

don’t forget to pass tests locally before submitting a PR

I’m afraid this may be beyond my experience at this point. I’m getting an error building on my diff but I have no idea why?

almost looks like its upset just because there is a diff

make clean goes fine. make pristine fails, make test succeeds.

PASS
coverage: 65.7% of statements
ok      github.com/roboll/helmfile/pkg/tmpl     0.258s  coverage: 65.7% of statements
?       github.com/roboll/helmfile/test [no test files]
(osvc-infra) [jmwitkow@9464ba3995f6 helmfile]$ echo $?
0

so I’m guessing maybe i shouldn’t be running make pristine ?

i built a binary and confirmed it fixed my issue as well

https://github.com/roboll/helmfile/pull/1471

@K H Thanks for reporting! Just replied in the issue.

When I run helmfile build on it with the ssm version there are no errors, but it always returns false. I assume this is because it’s not rendering the ssm value and just using it as a string.

Just discovered fetchSecretValue, now to figure out how to correctly use it

And this is what works:

environments:
  stage:
    values:
      - downloadWorkerBranch: <secretref+awsssm://stage/DOWNLOAD_WORKER_BRANCH>
  prod:
    values:
      - downloadWorkerBranch: <secretref+awsssm://prod/DOWNLOAD_WORKER_BRANCH>

Then in the release:

    
installedTemplate: {{- if eq (.Values.branch | quote) (.Values.downloadWorkerBranch | fetchSecretValue | quote) }} true {{ else }} false {{ end }}

If anyone sees that I’m doing something dumb, I’d love your feedback!

@Carlos R. would probably want this

This would be nice as we frequently use our monochart to deploy a wide range of applications and would like to set the .Chart.AppVersion as well

yeah - like many in that thread, I end up with many deployments out there that show an appVersion that isn’t actually what is deployed

+1

@Erik Osterman (Cloud Posse) I’m pretty sure its been talked about in OO at some point but I don’t remember what was said

Hah, this topic comes up for us in our engagements all the time. We haven’t found the holy grail yet, but can share the best practices we follow. I am on my phone right now. Will set a reminder

We’ve already tried different things: helmfile hooks, incubator/raw, home-made charts. Cannot say we are pleased with something specific. Also, we try to use needs: as much as we can to make a dependency tree. When we install third-parties with crds folder we use disableValidation flag that tells helm-diff not to fail if something cannot find a certain object type.

Found the helmfile for cert-manager that uses presync hooks. Looks promising.

https://github.com/cloudposse/helmfiles/blob/master/releases/cert-manager.yaml

Also found: https://github.com/roboll/helmfile/issues/538#issuecomment-479852913

@Jeremy G (Cloud Posse) if he has time can maybe share something quick.

You can do everything with helmfile hooks, although I find that unwieldy for all but the simplest commands, so for more complicated stuff, like installing, upgrading, and removing cert-manager CRDs, I prefer to use the hook to run a custom script (which mainly runs kubectl commands, because until last week, jetstack was warning that helm still had issues with CRDs, apparently just resolved in the past few days with the release of helm 3.3.1).

For cert-manager specifically, I have a presync hook that calls the script to install or upgrade the CRDs, a postuninstall script (actually the same script with different arguments) to remove the CRDs, and a postsync hook that does nothing but sleep so that cert-manager has time to set up its CA and webhooks before the next release runs.

The next release, BTW, is in the same helmfile.yaml and “needs” the first one, installs ClusterIssuers. It also has to have

disableValidation: true

because otherwise helmfile apply will fail to install the releases because the CRDs are not yet installed and so the ClusterIssuers are not valid resources yet.

I’m about halfway through @mumoshu’s way to do it with the helmfiles: section. I’ll report back on how that goes and also try the hooks way if I run into any issues.

So far so good though.

this is what I am doing so far

Prepare hooks will be executed even for helmfile template . Is this intended?

As for ClusterIssuer we move it from hooks to a separate Helm release as we had issues when it was not applied. Say, if you fix just a ClusterIssuer manifest helmfile diff won’t see a diff of a cert-manager release and a presync/postsync hook won’t be executed.

Ok, I think simply wrapping the ref+ link into an if not conditional in a common helmfile might work

Yes it works

I think you’ve already got it! If I can add anything, it works by replacing ref+ urls in yaml hash in values.yaml files, after the go template is rendered to produce the yaml document in the first place.

you can also use fetchSecretValue and expandSecretRefs for more advanced use-cases

https://github.com/roboll/helmfile#templating

Hey! Yeah this makes sense a lot. It should probably be implemented as a K8s controller that runs some code similar to https://github.com/bacongobbler/helm-whatup

Just want to mention that we are very satisfied with Renovate. But I understand your pain;)

@Andrew Nazarov Are You using regex matching, or direct helmfile support ?

We are using regexps

Seems like a regression introduced recently. I’ll fix it today. Thanks for reporting!

This is the commit that broke it

https://github.com/roboll/helmfile/commit/7b11ce851a2483f3176079210ce3274271d0237a

And this is what you’re relying on: https://github.com/roboll/helmfile/commit/7b11ce851a2483f3176079210ce3274271d0237a#diff-114928b5b56bd96452914832f2804038R16-R17

According to the code comment, it’s not intended to be accessible from with values.yaml templates… But we might better fix(repurpose) it if everyone is already using it

https://github.com/roboll/helmfile/pull/1466

@Roderik van der Veer I’ve just released v0.128.2 for this

thx! I modified everything already to make it work so i cannot really test it

force: true/false does not get around it

I ran into similar issues. I believe the fix was updating to a newer version of helm 3.

oh , i think i am still on 3.2.4

let me check

Hmm, 3.2.4 is what I’m using. What version of helmfile are you on?

3.2.4

that’s helm, what about helmfile?

0.125.8

hmm, that’s pretty recent. Doesn’t seem like a version issue then.

No, i mean this is a very well known and long-discussed issue in helm

I just thought maybe someone had worked around it in helmfile

Yeah, I know the issue. I thought the way I got around it was by upgrading helm. But clearly that’s not the case since we’re using the same version of helm.

https://github.com/helm/helm/pull/7431 and related. I cannot understand why can’t we get the same selective replace as we have in helm2.

worked around it in helmfile it’s helm-level issue, I don’t thinks it’s possible to provide stable solution at helmfile level.

I’m getting the following error:

# helmfile.yaml
environments:
  default:
    values:
      - s3::<https://s3.amazonaws.com/my-s3-bucket/v1/values.yaml>

$ helmfile template
in ./helmfile.yaml: failed to read helmfile.yaml: environment values file matching "s3::<https://s3.amazonaws.com/my-s3-bucket/v1/values.yaml>" does not exist in "."

Btw, this works with their CLI:

$ go-getter s3::<https://s3.amazonaws.com/my-s3-bucket/v1/values.yaml> /tmp/test/
2020/09/11 08:05:51 success!

What version of helmfile are you using? Looks like it should be supported: https://github.com/roboll/helmfile#loading-remote-environment-values-files

The latest one I think

helmfile version v0.128.2

@mumoshu Might have an idea?

I have never had chance to try it with s3. This might be a bug or not. I thought it wasn’t trivial to make go-getter as a go library behave the same as the go-getter cli

it may work depending on url format(i don’t have specific idea yet, though..

It did reproduce this locally. Let me see what I can do..

Ah gotcha. @Gabriel Tiossi Could you try s3::<https://s3.amazonaws.com/my-s3-bucket/[email protected]>? Please notice the part before @ is the path to the directory and after is the file name(object key, in s3 term)

I admit this isn’t intuitive, but we had to introduce this dialect of go-getter url to stabilize go-getter’s behaviour for helmfile’s use-case

Thank you @mumoshu and @Jonathan :grin: I found it intuitive and worked like a charm :+1:

I would like to point that this behavior only works with environments._environmentName_.values , and not with releases[n].values . Don’t know if this is expected or not.

When I use the exact same sucessful string in a releases[n].values, this is the output:

in ./helmfile.yaml: values file matching "s3::<https://s3.amazonaws.com/my-s3-bucket/[email protected]>" does not exist in "."

To work around it, I’m doing something like this:

environments:
  default:
    values:
      - s3::<https://s3.amazonaws.com/my-s3-bucket/[email protected]>

repositories:
  - name: my-repo
    url: <https://my-repo.com>

releases:
  - name: my-release
    chart: my-repo/my-chart
    values:
      - {{ .Environment.Values | toYaml | indent 8 | trim }}

I don’t know if there is a more elegant way to solve it, but it works now

haha don’t thank me, @mumoshu did all the work, I’m just the one who summoned him Glad you solved it!

@Gabriel Tiossi Thanks for confirming! Glad to hear it mostly worked.
environments.environmentName.values , and not with releases[n].values I haven’t tried using it under releases.values so this might be a bug(or just not-implemented-yet)

Let me see what I can do

Never mind, othernodes = {{ .Values.orion.otherNodes | toJson }} returns what i need!

How do your values files look like?

I got it working eventually. There were some issues with starter kit that I was trying to use.

The values.yaml is

_: {}

Glad you got it working! The reason I was asking is that it seemed you were missing .Values.charts in your values file, but you seem to have solved it nonetheless!

I’m stuck at trying to render set, having the toughest time.

I tried adding this snippet in the helmfile:

releases:
{{- range .Values.charts }}
{{- if .enabled }}
    {{- if hasKey . "set" }}
    set:
    {{- range . | getOrNil "set" }}
      - {{ . }}
    {{- end }}
    {{- end }}
    ...
{{- end }}

But when I try to use this:

charts:
  azuredgraph:
    ...
    values:
      - ./charts/dgraph/helmfile/base.yaml.gotmpl
      - ./envs/aks/tf.dgraph_config.yaml
    set:
      - backups.destination: <minio://minio.minio.svc:9000/dgraph-backups>
      - image.tag: v20.03.4
      - backups.incremental.enabled: true
      - backups.full.enabled: true

But I cannot get set to work in this context.

It tells me:

second-pass rendering result of "helmfile.yaml.part.0":
 0: 
 1: helmDefaults:
 2:   timeout: 600
 3:   recreatePods: false
 4:   tillerless: true
 5:   force: false
 6: 
 7: environments:
 8:   default:
 9:     values:
10:       - ./charts.yaml.gotmpl
11:       - ./values.yaml
12:       - ./envs/aks/charts.yaml.gotmpl
13:       - ./envs/aks/values.yaml.gotmpl
14: 
15: repositories:
16:   - name: minio
17:     url: <https://helm.min.io/>
18: 
19: releases:
20:   - name: dgraph
21:     namespace: default
22:     chart: /home/joaquin/workarea/charts/charts/dgraph
23:     version: "0.0.11"
24:     values:
25:       - ./charts/dgraph/helmfile/base.yaml.gotmpl
26:       - ./envs/aks/tf.dgraph_config.yaml
27:     set:
28:       - map[backups.destination:<minio://minio.minio.svc:9000/dgraph-backups>]
29:       - map[image.tag:v20.03.4]
30:       - map[backups.incremental.enabled:true]
31:       - map[backups.full.enabled:true]
32:   - name: minio
33:     namespace: minio
34:     chart: minio/minio
35:     version: "6.1.2"
36:     values:
37:       - ./envs/aks/tf.minio_config.yaml
38: 

err: failed to read helmfile.yaml: reading document at index 1: yaml: unmarshal errors:
  line 29: cannot unmarshal !!str `map[bac...` into state.SetValue
  line 30: cannot unmarshal !!str `map[ima...` into state.SetValue
  line 31: cannot unmarshal !!str `map[bac...` into state.SetValue
  line 32: cannot unmarshal !!str `map[bac...` into state.SetValue
in ./helmfile.yaml: failed to read helmfile.yaml: reading document at index 1: yaml: unmarshal errors:
  line 29: cannot unmarshal !!str `map[bac...` into state.SetValue
  line 30: cannot unmarshal !!str `map[ima...` into state.SetValue
  line 31: cannot unmarshal !!str `map[bac...` into state.SetValue
  line 32: cannot unmarshal !!str `map[bac...` into state.SetValue

Statically if fails too

I am not sure how to use both set or values (docs: https://github.com/roboll/helmfile#configuration)

I dont thing helmfile likes having map[...] as values,

you should be able to set them like this:

    set:
    # single value loaded from a local file, translates to --set-file foo.config=path/to/file
    - name: foo.config
      file: path/to/file
    # set a single array value in an array, translates to --set bar[0]={1,2}
    - name: bar[0]
      values:
      - 1
      - 2

I spotted a little bit after. I did the name/values. Now it is injecting the values.

I’m using helmfile to run a battery of values easily, as the app I am supporting has 40+ combinations…

too challenging to keep track of combinations

I seem to be failing. I changed the values to these

     - name: backups.destination
       value: <minio://minio.minio.svc:9000/dgraph-backups>
     - name: image.tag
       value: v20.03.4

But I am not having them picked up.

So I went with pure values… that seems to work. I wonder if you have values, then set won’t work

Thanks for your help.

Experienced something similar a week or so ago. I had to change some values and run again to actually apply a thing. But I thought in my case it was not related to helmfile. And it was really difficult to reproduce

something like this?

version: {{ .Environment.Values | default (env "COMPONENT_VERSION") | default "" }}

well helmfile uses the get syntax, so I’m really just looking for a way to express “empty map” or “empty list”

but yes, something along the lines as what you have

doing

{{ .Values | get "domain" "[dev.example.com](http://dev.example.com)" }}

will try to get from .Values.domain, but default to getting from “dev.example.com” if that value does not exists. if you want to assign a default value to the get request, I think this is how you want to do it:

{{ .Values | get "my-domain" "default-domain" | default "{}" }}

Yes. Check out the incubator/raw chart

I’m using helmify-kustomize , works great.

This look like this was added with needs: https://github.com/roboll/helmfile/pull/914

Maybe you would find this useful:

https://sweetops.slack.com/archives/CE5NGCB9Q/p1599247141077800

The needs worked perfectly for me

I also had to use a manifest to use the operator, and then a helm chart after the manifest

So for this I used kustomized with a hook to create a helm chart on the fly, then use that. Combined with needs I can mix operators, helm charts, kustomize, all together and use ordering.

@Cameron Boulton Hey! Good question - Yes, unfortunately there’s no way for now

It does sound like a valid feature request for me, though

Okay, thank ya. I’ll consider adding a feature request issue to GitHub.

Thanks! Looking forward to it

https://github.com/roboll/helmfile/issues/1487

Oh, I see https://github.com/helm/helm/issues/8771 and the PR https://github.com/helm/helm/pull/8772

The issue is annoying. Before in 2010s, there was a big push to write automation (especially with change configuration) to make idempotent automation. If the state is already in the desired state, do not raise an error.

But now these days, with kubectl and helm, others, it’s like
*idempotent, shemdepotent*

well, I think it’s a good idea to pin versions in automation. on interactive usage it’s possible to use pre-existing helm repos and don’t add it in helmfile

I’m trying to have documentation for customers that may not be at dev/ops level to manipulate helmfile. helmfile is useful to orchestrate several components that need to work together with no-fuss, no-muss.

I added a note to the PR requesting that --force-update only be added when Helm versions 3.3.2 and 3.3.3 are used. I believe the security issues that Helm addressed with this change are valid, and a PR is being worked that would return the idempotent nature as long as the name and URL are both the same as they were before

https://github.com/roboll/helmfile/pull/1488#discussion_r492055821

Though, I just thought about what would happen if I am using the helm-git plugin (which I do frequently). Changing the ref to another tag or git SHA would be considered a change to the URL, causing the error to be thrown

WDYT about --force-update directive from helmfile side ?

meaning you’d run helmfile apply --force-update?

Not sure, could also do

repositories:
- name: polaris
  url: git+<https://github.com/reactiveops/polaris@deploy/helm?ref=tags/1.2.3>
  forceUpdate: true

tagging some others for more visibility - @Erik Osterman (Cloud Posse) @mumoshu

Actually, I didn’t know about the helmfile apply --force-update. I had been using helmfile for less than a week…

Sorry I could have been more clear. That was in response to a suggestion. The --force-update flag does not currently exist for helmfile apply

I switched to another laptop, installed latest version of helm, and discovered this issue. I think I may have convinced helm folks to change it so that if it will not change the repo name for the same url, not to exit 1.

meaning you’d run helmfile apply --force-update? No, I was talking about some helmDefaults config option

with per-repository override, like you’ve offered above

I used the default helmDefaults

I would prefer that if I have it already to do nothing, that would be nice behavior.

@Joaquin Menchaca we’re talking about feature request of non-existing helmfile functionality

oh

yeah

makes sense.

Sorry, not enough coffee

So that would be fine. But also we get into a situation, that if they have a force-update , and the helm version doesn’t support that option, it craps out on them

They aren’t removing --force-update from Helm, just adding a check for whether the name and url are both the same as they were before

So it would nice to have it do whatever it needs to do regardless of helm3 version. So if not force-update, then it skips adding it again.

thus, we have 3 helm versions

• pre 3.3.2
• 3.3.2-3.3.3
• and 3.3.4 with https://github.com/helm/helm/pull/8777

You can use selectors for this, but I bet it’s not what you are looking for

I don’t think that promoting range iterators to global variables is expected behavior. It doesn’t match with gotmpl

A variable's scope extends to the "end" action of the control structure ("if", "with", or "range") in which it is declared, or to the end of the template if there is no such control structure.

Ok, that was my fault. I checked the wrong commit of our codebase, my colleagues forgot to put a tag on the fresh code and I didn’t notice this and examined the fresh one.

So, nevermind the question)

Did you tried to debug it with helmfile build --log-level=debug and your --state-values-file or --state-values-set during debug ?

I’ll give that a try, thanks

Yeah release templating is the best way I’ve found to share values like that.

But you should take this into account: https://github.com/roboll/helmfile/blob/master/docs/writing-helmfile.md#layering-release-values

good remainder again, thnx

Ok, I was using selectors instead of labels within the releases. But it looks like selectors get applied after templating the state files with {{ .Values.bla }}.

selectors are intended to select some releases instead of all releases, based on labels

as far as i know you cannot use template for the values files.

you can if you name the values files the .yaml.gotmpl extension

Hello @Bart M. - how can I share those templates between various .yaml.gotmpl files ?

via readFile ?

not sure, haven’t actually tried that, I just know you can use go templating in values files (which we use for small stuff)

Ok thnx

You may try to ask it in this channel since mumoshu is always here.

oh I see, so you can include all the environment-specific helmfiles but they only get selected if they have a key in the environments block

do you have a way of sharing releases between environments (with different values)?

use the same helmfile with multiple envs inside

use installed(installedTemplate):/condition: to get conditional deploy.

releases are not environment-scoped by default

values/valuesTemplate supports dynamic file names, this feature is super.

combine it with missingFileHandler: Debug to jam missing file warnings, and you may get like 10 values files in template

values/valuesTemplate supports dynamic file names, this feature is super. what do you mean by this?

what do you mean by this? I mean syntax like

    valuesTemplate:
      - ../common/values-{{`{{ .Release.Labels.app }}`}}.yaml.gotmpl
      - {{ .Environment.Name }}/values-{{`{{ .Release.Labels.app }}`}}.yaml.gotmpl
      - {{ .Environment.Name }}/values-{{`{{ .Release.Name }}`}}.yaml.gotmpl

Ah, ok, I see. Yes, this is quite handy indeed

I thought there might be some other functionality I’m not aware of:)

{{{{ .Release.Namespace }}}} might work

nope

 failed to read test.yaml: reading document at index 1: yaml: invalid map key: map[interface {}]interface {}{".Release.Namespace":interface {}(nil)}

What about just {{ .Namespace }} ?

No

Its rendered and deployed as namespace: null

Ah, probably {{ .Namespace }} is rendered when you pass it via --namespace key. Ok, we don’t use this anyway)

How are You setting namespaces in CRs ? ( Deployed via raw chart.. )

Its should be possible according to this https://github.com/roboll/helmfile/blob/master/docs/writing-helmfile.md#release-template--conventional-directory-structure

do you actually have an environment defined anywhere?

no, no environments block and no --environment flag

then its basically just saying it doesn’t see any releases in the default path

is it a single helmfile with a releases: block ?

yeah

paste?

oh wait, I think it’s because I had a condition: bla.enabled in the release - I passed in --state-values-set bla.enabled=true but that didn’t do anything.

removing the condition worked.

helm-operator is kinda tied to fluxcd and all weave stack. It’s more about push VS pull model of changes delivery into k8s. Helm-operator doesn’t support sops natively yet, and it’s hard to get layered templated configuration as it’s possible w/ helmfile.

it looks like helmfile cannot merge arrays in values, as well as helm cannot do this.

I guess list object in go template cannot be merged together

why don’t you use following environments.yaml

 0: environments:
 1:   prod:
 2:     missingFileHandler: Error
 3:     values:
 4:       - ../vars/helmfile/realms/prod.yaml
 5:       - ../vars/helmfile/global/vars.yaml

with vars.yaml like

vault_image: my_vault_image:1.5.3

resources are static YAML - so there is no templating (except for helmfile’s templating with yaml.gotmpl files). templates are strings that will be templated by Helm itself during the install process, so you can use any values that have been passed to the chart alongside the templates.

Hey! Not sure if that will be of any help for your question… But would you consider taking secret deployment out of you chart altogether? Say to separate helmfile release or maybe use event hook (release or global) within the same helmfile release?

You should have your kind: Secret using something like

data:
{{- range $key, $val :=  .Values.secret }}
  {{ $key }}: {{ $val | b64enc | quote }}
{{- end }}

inside your helm chart, while you use sops-encrypted secret file with helm values in your helmfile, f.e:

secret:
    key0: encrypted-value0
    key1: encrypted-value1
sops:
  ...

and tell helmfile about this secret file using secrets: directive in your release or template

    secrets:
      - secrets-{{`{{ .Release.Name }}`}}.yaml

helmfile will decrypt secrets file in runtime and pass it’s values as helm values under secret. , and your chart generates k8s secret data: from it.

@Vugar Indeed, it worked. You are my survivor. Thank yoou. I followed example from https://github.com/roboll/helmfile/issues/1484 Thx @voron I check this out as more of those secrets are waiting to be fixed :p

@tomaszjdul glad to hear this! Personally I don’t deploy charts outside of helmfile context at all… so keeping things within helmfile releases and/or hooks works better for me. But what happens if someone say in your team (especially fresh join) will decide to use this chart in the future outside of helmfile? Depending on your environment and requirements I would give what @voron has suggested earlier a second consideration… The only thing I can add is that within helmfile context I prefer remotely sourced (e.g. from AWS SSM) secrets, these are pretty well documented here and here

Found answer disableValidation: true

what about --values ?

--values functions as expected

my change appends the parsed values from --set to the list of files provided by --values

or did you mean I should be testing --values as well?

--set contains raw values, not files

or did you mean I should be testing --values as well? no

--values functions as expected same from my side, --values affects generated with helmfile write-values file

--set contains raw values, not files yup, so the values need to be parsed first – https://github.com/roboll/helmfile/blob/cbbbb7bdc91e678d1ea50dde4f4555594aced5c0/pkg/state/state.go#L1284