#helmfile (2021-04)

Did you find an example that it should work like this ?

no, just tried based on values. it works. so on the same file I have the Release.Name… few lines after i don’t have it ?

I don’t see that strategicMergePatches and jsonPatches do support files with/without rendered name.

it may be better to use valuesTemplate in templates, btw, but it’s not related to the issue you’re seeing

so I cannot give dynamic address of the patch file? I need to set it under all apps in release ?

try and see. If hardcoded file name works - you may submit a PR to get generated names to function there, or open an issue.

    jsonPatches:
      - ../config/app/patches.yaml
 

then it finds the patch and patches the resources I would like.

and what’s with strategicMergePatches ?

same

if I hardcode the app name it finds… if not it does not replace the {{}} with the value and searches under ../config/{{ .Release.Name }}/patches.yaml

where I don’t have the file

well, sometimes hack with quoting like

{{`{{ .Release.Labels.app }}`}}

may work

skipping missing values file matching "../config/{{ .Release.Name }}/patches.yaml"
skipping missing values file matching "../config/{{ .Release.Name }}/{{ .Environment.Name }}-patches.yaml"
skipping missing values file matching "../config/{{{{ .Release.Name }}}}/merge.yaml"
skipping missing values file matching "../config/{{ .Release.Name }}/merge.yaml.gotmpl"

same

it looks it templating only values and secrets. is that correct?

as I said above, you may submit a PR to get generated names to function there, or open an issue. it may be some trivial fix, or may not, IDK

@voron I can confirm, we’re using the same approach

  dockerized: &dockerized
    namespace: {{ .Environment.Values.namespace }}
    missingFileHandler: Warn
    labels:
      group: docker
    values:
      - envs/dockerized.yaml.gotmpl
      - envs/{{`{{ .Environment.Name }}`}}/dockerized.yaml.gotmpl
      - envs/common/dockerized/{{`{{ .Release.Name }}`}}.yaml.gotmpl
      - envs/{{`{{ .Environment.Name }}`}}/{{`{{ .Release.Name }}`}}.yaml.gotmpl

there is no issue w/ values/valuesTemplate or secrets. It’s specific to strategicMergePatches and jsonPatches

yes. I have issue with patches not with values and secrets.

@Balazs Varga did you tried hack with

{{`{{ ... }}`}}

?

yes I tried.. did not work

file a PR or an issue on GH, that’s all I can advise.

will do. Now I am trying to find where is it templating the secrets and values. I mean the path only because I need the path to be templated. the patch contains hardcoded data.

well, another possible option is to fork chart and fix it to get rid of patches

yeah but this looks more elegant way than modifying charts.

today I will spend to solve this if cannot then will modify charts

Hey! I guess you would like valuesTemplate @Balazs Varga cc @voron

For more info find valuesTemplate in https://github.com/roboll/helmfile/blob/master/docs/writing-helmfile.md#release-template--conventional-directory-structure

yes, but if I would like to use here it does not work.

I thought there’s no reason it doesn’t work under templates.

You might be just merging-in the template afterwards using <<: *default, right?

How did your helmfile.yaml with templates and releases looked like when you found valuesTemplate not working?

templates I mentioned. releases was simple like this:

releases:
- name: init
  namespace: default
  chart: ../chart/init
  <<: *default

so the idea was if I put the values and secrets and patches to the selected folder it will template and use it… it searches under those folders but as you see it does not template the patches lines:

skipping missing values file matching "../config/{{ .Release.Name }}/patches.yaml"
skipping missing values file matching "../config/{{ .Release.Name }}/{{ .Environment.Name }}-patches.yaml"
skipping missing values file matching "../config/{{ .Release.Name }}/merge.yaml"
skipping missing values file matching "../config/{{ .Release.Name }}/merge.yaml.gotmpl"
skipping missing values file matching "../config/{{ .Release.Name }}/{{ .Environment.Name }}-merge.yaml"
skipping missing values file matching "../config/app/values.yaml"

@Balazs Varga Could you try (literally) valuesTemplate instead of values for values, then?

You can’t access release template from within values, which is supposed to be a yaml array of plain strings

On the other hand each item in valuesTemplate is considered a go template with accses to release template

yeah will try… few sec

Also if you’d need to do the same templating on file paths in secrets, unfortunately it isn’t supported today. But it should be a relatively easy addition to Helmfile. Please feel free to open a dedicated feature request for that. It should look like secretsTemplate

thanks. using valuesTemplate it works

that is fine to me if I cannot put merge into secret “folder”.

Oh really? Okay then! I just thought you’d want the same level of reusability for the secrets array, too

no, I just wanted to have jsonpatches and mergepatches under config folder to have a light helmfile.d file

templates:
  default: &default
    missingFileHandler: Debug
    valuesTemplate:
      - ../config/{{ .Release.Name }}/values.yaml
      - ../config/{{ .Release.Name }}/values.yaml.gotmpl
      - ../config/{{ .Release.Name }}/{{ .Environment.Name }}.yaml
      - ../config/{{ .Release.Name }}/{{ .Environment.Name }}.yaml.gotmpl
      - ../config/{{ .Release.Name }}/merge.yaml
      - ../config/{{ .Release.Name }}/merge.yaml.gotmpl
      - ../config/{{ .Release.Name }}/{{ .Environment.Name }}-merge.yaml
      - ../config/{{ .Release.Name }}/patches.yaml
      - ../config/{{ .Release.Name }}/{{ .Environment.Name }}-patches.yaml

    secrets:
      - ../config/{{ .Release.Name }}/secrets.yaml
      - ../config/{{ .Release.Name }}/secrets.yaml.gotmpl
      - ../config/{{ .Release.Name }}/{{ .Environment.Name }}-secrets.yaml

this worked to me… if somebody else will have the same issue.

Successfully generated the value file at ../config/test/merge.yaml.gotmpl. produced:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test
  namespace: test
spec:
  ...

Ah, so what I wanted to say was that I’m afraid that this part isn’t working as you might have expected:

    secrets:
      - ../config/{{ .Release.Name }}/secrets.yaml

Almost certainly this is translated to as-is, without templating, not ../config/test/secrets.yaml as you might have expected, but ../config/{{ .Release.Name }}/secrets.yaml

that still works

skipping missing values file matching "../config/prometheus/default.yaml"
skipping missing values file matching "../config/prometheus/default.yaml.gotmpl"
skipping missing values file matching "../config/prometheus/merge.yaml"
skipping missing values file matching "../config/prometheus/merge.yaml.gotmpl"
skipping missing values file matching "../config/prometheus/default-merge.yaml"
skipping missing values file matching "../config/prometheus/patches.yaml"
skipping missing values file matching "../config/prometheus/default-patches.yaml"
skipping missing secrets file matching "../config/prometheus/secrets.yaml"
skipping missing secrets file matching "../config/prometheus/secrets.yaml.gotmpl"
skipping missing secrets file matching "../config/prometheus/default-secrets.yaml"

I just don’t use it here.

oh really!!

What an inconsistency… but good to hear it works

We use templated names with secrets for some time ago, similar to

    secrets:
      - ../../live/{{ .Environment.Name }}/{{`{{ base .Release.Chart }}`}}/secrets-{{`{{ .Release.Name }}`}}.yaml

Hi, this is a recent addition, see https://github.com/roboll/helmfile/pull/1715 and related commit merged in master: https://github.com/roboll/helmfile/commit/2618cfb38b20d867a977f2b295059893d23e507a but not yet released.

Oh, I did not notice this. I thought 138.7 had it cause the readme mentioned the new config. Thanks for the clarification!

Yep, the README is the one of the master branch

Is there any ETA for the next version?

we didn’t have fresh versions for ~ 1 month, so I expect it to become soon

yes it does. probably https://sweetops.slack.com/archives/CE5NGCB9Q/p1618385513079000?thread_ts=1618381282.078300&cid=CE5NGCB9Q clarifies that a bit?

why?(i’m not yet sure if I fully understand your usecase

files referenced from within a sub-helmfile is relative to the sub-helmfile, to make the sub-helmfile portable(not dependent on the parent-helmfile)

so to me the current behaviour makes sub-helmfiles portable, which is nice

i need to learn something- what would be ideal behaviour and how would you rewrite your yamls with that ideal behaviour?

well, maybe I’m doing something wrong. So lemme articulate what I’m doing first and then I’ll attempt to explain what and why I’m doing it.

first, I’m still trying to read through the docs and learn helmfile. The docs are good, but not very simple to follow referentially.

so, all the stuff I showed here is relevant

and the way I’m approaching creating this is by trying to get helmfile to work in the dex release first. The ultimate goal, however, is to run helmfile from a different directory (one that references the environment I want to deploy).

so, my thinking is, get it working first in dex. Then get helmfile to work one directory up from dex (01-secrets-management)

all the way up to generic.

I’m trying to keep this DRY…

I’m not sure any of this makes sense. It’s difficult to articulate.

Sounds good so far

oh good

ok, so, if I run helmfile from dex which uses the ../../common/environments.yaml file to define environments.

the ../../common/environments.yaml file references a file called config.yaml but I’m in dex so the reference to config.yaml has to be referenced relatively from dex but ../../common/environments.yaml will need to be changed when I run helmfile from a different directory.

that’s what I mean by “not very portable”

it would be better if (at least in the way I’m using this) to relatively reference files from the manifest referencing the file.

So, I’m in /var/tmp/helmfile-work/helmfile/helmfile.d/generic/01-secrets-management/dex and ../../common/environments.yaml contains:

environments:
  default:
    values:
      - ../../common/config.yaml
  production:
    values:
      - ../../common/config.yaml

the ../../common/environments.yaml file references a file called config.yaml but I’m in dex so the reference to config.yaml has to be referenced relatively from dex but ../../common/environments.yaml will need to be changed when I run helmfile from a different directory. (edited) ah gotcha!

fwiw, it doesn’t look like what bases is supposed to help today.

the contents of common is:

at 00:58:51 ❯ ls -ltr ../../common
total 16
-rw-rw-r-- 1 jimconn jimconn 540 Apr 13 00:02 repos.yaml
-rw-r--r-- 1 jimconn vboxsf  486 Apr 13 22:10 config.yaml
-rw-rw-r-- 1 jimconn jimconn 129 Apr 13 23:20 environments.yaml
-rw-rw-r-- 1 jimconn jimconn  37 Apr 13 23:26 helmdefaults.yaml

ah

ok

so I’m not using it properly?

I usually recommend using

{{ readFile "common.yaml.gotmpl" | tpl . $someData }}

---

# releases, repositories, etc

ok

I can give that a try in a moment

one more quick question then…something that’s not really documented very well I think

so I’m gonna switch gears on you for a min

this might be a really easy question for you to answer.

i thought there were opening issue(s) about adding parameters to bases but i cant get the exact link urls for them now

ok

all right, here’s the helmfile.yaml again for dex:

---
bases:
  - ../../common/environments.yaml
  - ../../common/repos.yaml
  - ../../common/helmdefaults.yaml

---
## ************************************
## Start of DEX installation
## ************************************
releases:
  - name: dex
    namespace: {{ .Values.dex.namespace }}
    createNamespace: true
    labels:
      tier: "secrets-management"
      app: dex
    chart: repo/dex
    version: {{ .Values.dex.version }}
    values:
      - values.yaml.gotmpl
    set:
      - name: var.aws_nlb_with_tls_termination_at_lb
        value: {{ env "AWS_NLB" | default false }}
      - name: var.arm64_support
        value: {{ env "ARM64" | default false }}

I’m trying to use a templatized values.yaml

my values.yaml.gotmpl (partial) is:

# DO NOT increase replicas to >1 during the initial install.                                                                                                                                                                                                                                                                                             
# There is a bug which causes the GRPC TLS certs not to be issued because of webhook race conditions.                                                                                                                                                                                                                                                    
# Bug : <https://github.com/helm/charts/issues/24229>                                                                                                                                                                                                                                                                                                      
                                                                                                                                                                                                                                                                                                                                                         
#replicas: 3                                                                                                                                                                                                                                                                                                                                             
{{- if var.arm64_support }}                                                                                                                                                                                                                                                                                                                              
image: ghcr.io/dexidp/dex                                                                                                                                                                                                                                                                                                                                
imageTag: "v2.26.0"                                                                                                                                                                                                                                                                                                                                      
{{- end }}                                                                                                                                                                                                                                                                                                                                               
                                                                                                                                                                                                                                                                                                                                                         
tolerations:                                                                                                                                                                                                                                                                                                                                             
- key: "dedicated"                                                                                                                                                                                                                                                                                                                                       
  operator: "Equal"                                                                                                                                                                                                                                                                                                                                      
  value: "utility"                                                                                                                                                                                                                                                                                                                                       
  effect: "NoSchedule"  

I am not able to reference the set parameters in the values.yaml.gotmpl — not sure what I’m doing wrong here.

ah well I haven’t really tried to use it that way but after rereading https://github.com/roboll/helmfile/issues/688

this might work:

values:
- someValuePassedToBases: "foobar"

---

bases:
- common.yaml

---

#releases, repositories, etc

values.yaml.gotmpl is rendered by helmfile. you have access to helmfile-managed values only there

where set sets values passed to helm

ahhh

ok, so I need to set those in a helmfile values somewhere. Maybe the thing you just linked?

values:
- key: "something"

?

you usually use helmfile (environment) values to produce a series of set entries and values files

set and values entries rendered and merged by helmfile are passed to helm

ok, so I need to set those in a helmfile values somewhere. Maybe the thing you just linked? absolutely!

right, so I’m trying to localize those to just dex in this case. Obviously, I’m not just going to use helmfile to deploy dex but it will be used to deploy a bunch of stuff. So, trying to find a way to make this work both “locally” (for just dex) and then from more of an environment deployment perspective is where I’m getting lost.

ok! Let me think through how to do that.

wishing your good luck!

let me also note that i believe it would worth trying https://sweetops.slack.com/archives/CE5NGCB9Q/p1618387448086400?thread_ts=1618381282.078300&cid=CE5NGCB9Q

helmfile is a rich ecosystem of stuff. Trying to wrap my head around it all for my use-case is complicated.

I appreciate your help!

sounds right- keep posting comments/questions/feedbacks here. i or other people will respond soon!

thx for trying helmfile anyway

Yes it is

inherited in context? Hey! Helmfile doesn’t inherit anything to sub-helmfiles. Inheritance of helmfile environments and values usually happen only between the parent helmfile.yaml and bases. Does that clarify it a bit?

That’s also the foundational thing under my comment on your previous question https://github.com/roboll/helmfile/issues/1045#issuecomment-821910447

Selectors aren’t inherited. It’s just that the user specifies the selector and helmfile uses it to filter releases across all the involved helmfie.yaml.

Environment and environment values aren’t inherited. That’s why this doesn’t work:

environments:
  default:
    values:
      - config.yaml

---
helmfiles:
  - "../../../../helmfile.d/"

The documentation may be outdated or incomplete or simply incorrect, seeing your frustration.

Did you find any specific documentation saying that environments and values are inherited down to sub-helmfiles? If so, we’d definitely need to fix it.

Does the documentation say if anything’s inherited to sub-helfmiles or bases?

I thought I even didn’t explicitly say in the documentation that the bases inherit parent’s environments and values, as trying to depend on the parent helmfile.yaml always sounded like a bad idea to me.

Everything will run exactly the same as if I didn’t use any selector except  ONLY the helmfiles specified by that selector (or a negated selector) would  run. --selector(-l) works that way. To do so, Helmfile requires you to make each sub-helmfile independently consumable. That’s why environments arent inherited down to sub-helmfiles, and the basis of comments starting from https://sweetops.slack.com/archives/CE5NGCB9Q/p1618707461115400?thread_ts=1618551534.114100&cid=CE5NGCB9Q

Would there be anything I’ve not yet answered?

Hey @mumoshu also have a selector question, are there maybe reserved words helmfile doesn’t allow to be label selectors?

Consider a simple example with 3 labels in 2 releases:

repositories:
- name: datawire
  url: <https://www.getambassador.io>
- name: incubator
  url: <https://charts.helm.sh/incubator>

releases:
- name: ambassador
  namespace: ambassador
  labels:
    chart: ambassador
    namespace: ambassador
    foo: bar
  chart: datawire/ambassador
  version: 6.6.2

- name: raw
  namespace: ambassador
  labels:
    chart: ambassador
    namespace: ambassador
    foo: bar
  chart: incubator/raw
  version: 0.2.5

If I run helmfile -l namespace=ambassador diff both releases are selected in the diff

If I run helmfile -l foo=bar diff both releases are selected in the diff

However, if i run helmfile -l chart=ambassador diff only the first release is selected in the diff. This is on v0.138.6.

Yes. As far as I remeber, chart and name are reserved, each has the respective value of the release

hah, thank you! it was confusing me for the longest time (why some releases weren’t being applied)

Ah, that makes sense! Probably we’d better enhance the docs to note about reserved labels and make it a validation error when you’ve tried to override a reserved label

Would you mind opening issues?

Yes! sure! That would be much better than silently half-working

https://github.com/roboll/helmfile/issues/1781

Much appreciated!

Thank you for the response @mumoshu! Today is my Sunday and I have other obligations today but I will answer your questions here tomorrow and also your comment for my GH issue! Very much appreciated!

@mumoshu I updated my comments in the GH issue

I feel daunted compared to Jim’s previous question ;) but I managed to break this down to a simple example and suspect i’m just “doing helmfile wrong” though the environments block in the README implies to me that this should work?

basically, i did helm create my-chart and then wrote this helmfile.yaml for it.

 environments:
   dev:
     values:
     - fullnameOverride: my-chart-dev
   live:
     values:
     - fullnameOverride: my-chart-live
 
 releases:
 - name: my-chart
   chart: ../.
   namespace: chart-test
   version: 0.1.0
   # values:
   # - fullnameOverride: {{ .Values.fullnameOverride }}

so helmfile should deploy my chart where the full name for objects is overridden by the environment specific value.

however, helmfile doesn’t inherit those values unless I uncomment the last two lines and explicitly declare each variable I want to make environmentally conditional? Are all the environment values not inherited? I’m hoping to not have to write out each variable in the releases block otherwise I might as well go for the - ./ values/{{ environment.Name }}.yaml method and I wanted to avoid having two full copies of the values.yaml..?

@jedineeper Hey! Well, honestly speaking I don’t understand how the documentation can make you think that environment values are inherited to release values. Probably the documentation needs to specifically say it doesnt?

Environment values are used to render helmfile.yaml templates and helmfile’s values gotmpl files only.

Helmfile doesn’t automatically pass those values to Helm(as release values).

@Jim Conner Maybe you’re interested in this topic, too?

Uncommenting the last two lines is the way I understand using helmfile with environments.

That the usage of {{ .Values.fullnameOverride }} in the chart files and the usage of {{ .Values.fullnameOverride }} in helmfile refer to separate things.

Inside the chart, .Values refer to the helmchart release’s values:

metadata:
  labels:
    foo: {{ .Values.fullnameOverride }}

refers to this value:

 releases:
 - name: my-chart
   chart: ../.
   namespace: chart-test
   version: 0.1.0
   values:
   - fullnameOverride: bar

So the label foo=bar.

Inside the helmfile, .Values refer to the environment’s values:

 releases:
 - name: my-chart
   chart: ../.
   namespace: chart-test
   version: 0.1.0
   values:
   - fullnameOverride: {{ .Values.fullnameOverride }}

refers to these values:

 environments:
   dev:
     values:
     - fullnameOverride: my-chart-dev
   live:
     values:
     - fullnameOverride: my-chart-live

and you have to select between them with the -e environment argument.

The last two lines bridge the two cases together.

@jedineeper If you want to have different “helm” values.yaml per each helmfile named environment, I’d do this:

 environments:
   dev: {}
   live: {}

---

 releases:
 - name: my-chart
   chart: ../.
   namespace: chart-test
   version: 0.1.0
   values:
    - environments/{{ .Environment.Name }}/values.yaml

@vicken Great explanation!

We can even merge the above examples:

---
 environments:
   dev:
     values:
     - fullnameOverride: my-chart-dev
   live:
     values:
     - fullnameOverride: my-chart-live

---

 releases:
 - name: my-chart
   chart: ../.
   namespace: chart-test
   version: 0.1.0
   values:
    - environments/{{ .Environment.Name }}/values.yaml
   - fullnameOverride: {{ .Values.fullnameOverride }}

A common addition to this is when you’d like to make the environment-specific helm values.yaml optional, add missingFileHandler:

 environments:
   dev:
     values:
     - fullnameOverride: my-chart-dev
   live:
     values:
     - fullnameOverride: my-chart-live

---

 releases:
 - name: my-chart
   chart: ../.
   namespace: chart-test
   version: 0.1.0
   missingFileHandler: Warn
   values:
    - environments/{{ .Environment.Name }}/values.yaml
   - fullnameOverride: {{ .Values.fullnameOverride }}

Super useful, thanks both. Looks like I was on the wrong track with my understanding and that’s cleared up now :)

@mumoshu this might help me. I’d still like to read your thoughts outlined in my gh issue but I might be able to make progress with this information.

@Jim Conner In the scenario 3, are you trying to feed environments defined in bluescape/ops/helmfile-project/helmfile/envs/dev/cluster-n/config.yaml into every helmfile yamls defined under bluescape/ops/helmfile-project/helmfile.d?

yes

@mumoshu

This was where I was getting confused with the concept of inheritance in trying to understand what inheritance was with respect to helmfile though you corrected me in one of your responses that helmfile doesn’t inherit. So, let’s not use that term, but I am trying to propagate the values defined in config.yaml in scenario three across the board of whatever I deploy using helmfile from any respective environment directory

It’s much better to treat there’s no inheritance, I think. (Where did you find helmfile does inheritance? I think that doc should be corrected if it’s confusing

Gotcha so

It’s documentation that I read in other github issues…not in your documentation

Thx I see! Firstly I’d repeat https://github.com/roboll/helmfile/issues/1045#issuecomment-821910447 - envs defined in config.yaml doesn’t get automatically passed to sub-helmfiles under helmfile.d.

I guess this should work:

helmfiles:
  - path: "../../../../helmfile.d/"
    values:
    - {{ .Values | toYaml | nindent 6 }}

Lemme give that a try…

taking a read real fast.

I admit this isn’t elengant, but not sure if there’s any better way than this now..

ok. Lemme take a look.

Lemme ask you this…

is the method of how I want to use helmfile odd or something?

It seems logical to me…and helmfile is a very flexible tool, but am I doing it wrong?

Yours seem totally valid

Ok, that’s good to know!

I guess it’s just that no one has ever (loudly) tried to do it in a such elegant way

Wow. Thank you for that comment. Very nice.

I believe Helmfile should add something like the below to better support your usecase(actually this was my original plan was no one has ever requested it or sent me any question that leads to this

so, it seems like it could potentially be complicated to code depending on the library(ies) you might be using to mux objects…

but if you have a solid top-level then inheritance probably isn’t that tough to enable.

taking a look at this now…gimme a few please.

So from your comment in the thread, it’s not very clear to me which helmfile.yaml that you are suggesting change to. also, do you see merit in making a feature request out of this use-case? I’d LOVE to see this functionality.

also, in that thread, I need to define the config.yaml which has the values defined in the top-level directory but it seems your example only defines a scriptsDir which I suppose is the defined directory where the top-level config.yaml file would reside; but if I point the scriptsDir to the same path as where my top-level is then it seems that would cause a circular dependency unless I can specifically say, “use this file” — not just “use this directory”

@mumoshu is there a way to get helmfile template --selector foo=bar to work?

if I point the scriptsDir to the same path as where my top-level is then it seems that would cause a circular dependency This wasn’t clear to me, circular dependency between what, do you mean?

@Jim Conner Did you literally mean the way? You need to place --selector foo=bar before template.

ah

ok, lemme check that out.

ah perfect! Thank you

it’s not very clear to me which helmfile.yaml that you are suggesting change to. I meant helmfile/envs/dev/cluster-a/helmfile.yaml, assuming that’s where you’re trying to say “inherit envs defined in config.yaml to ../../../../helmfile.d/”

ah! Ok. So scriptsDir points to the top-level directory?

Ah, no. It’s just an arbitrary value for illustration purpose.

It is’nt a reserved value in any way so you can’t assume scriptsDir points to anywhere wihout you specifically set it

k, so right now, the following seems to be working but I’m still attempting to test (this is my top-level):

environments:
  default: {}

---
helmfiles:
  - path: "../../../../helmfile.d/"
    values:
      - config.yaml

I don’t have all of the sub-helmfile.yaml manifests completely correct yet, except for dex but unfortunately, my selector is not working.

so I’m trying to figure that out.

0 release(s) matching tier=secrets-management,app=dex found in helmfile.yaml

have you tried helmfile --debug $SUBCOMMAND? it would print a lot of logs to help debugging

yup

I’m reading through it all now to see if I can figure out where my problem lies

funny enough, the error I just posted is exactly after the repo updates. So, gotta figure it out.

0 release(s) matching tier=secrets-management,app=dex found in helmfile.yaml will be printed on any sub-helmfile that contain other releases. But I thought it doesn’t result in an error?

I mean, you’ll see 0 release(s) matching tier=secrets-management,app=dex found in helmfile.yaml in debug logs on every sub-helmfile that didn’t have releases matching the selector. But Helmfile won’t fail when any sub-helmfile had one or more releases that matched.

just a sec. still reading through the output. I found out that I had --selector specified twice on accident on the command line so that was one problem.

the problem seems to be that helmfile is not traversing recursively into certain paths…

could you push your project in github so that i can reproduce?

sure!

gimme a sec.

sorry for taking so long. Wife was in here talking to me for a min. Now, I’m scrubbing the repo as it’s not open source, but there’s nothing of great value in here yet. So, I’ll create in my personal repo temporarily and provide you with the link.

here you: https://github.com/notjames/jimconn-shell/tree/master/temp

I removed secrets files, so if you run this and you get errors about that, that’s why

Thanks!

just a sec

need to do something

the problem seems to be that helmfile is not traversing recursively into certain paths… what were certain paths and what helmfile command did you run to see it?

do a pull real fast

and then cd <repo>/temp/helmfile/envs/dev/atreus

done

thx

now in that directory I ran: helmfile --debug --selector app=dex template

when you run that, you’ll notice that helmfile doesn’t backup the tree after descending into 01a-… to traverse into 01b-…

ahh…you’re saying that the reason is because the external-dns “chart” is broken….that makes sense.

I’ll fix real fast. I haven’t polished all of these yet so I’m not concerned about that being broken

seems so! helmfile traverses sub-helmfile in some alphabetical order and fail-fast

ah!! OK, that’s good to know

lemme fix that real fsat.

does helmfile descend into dot directories?

nm. I’ll rm -rf external-dns and ingress-nginx

haven’t tried that but as long as the exact path or a glob pattern that matches the dotted directory/file, it should just work (I thought I have not programmed helmfile to explicitly ignore dot dir/file

ok

all right, so the error I’m getting now is:

in ./helmfile.yaml: in .helmfiles[0]: in ../../../../helmfile.d/helmfile.yaml: in .helmfiles[0]: in generic/01a-network-and-proxies/helmfile.yaml: in .helmfiles[1]: in ambassador/helmfile.yaml: [Malformed label: tier="network-and-proxies". Expected label in form k=v or k!=v]

That tier is defined in generic/01a-network-and-proxies/helmfile.yaml

so that’s interesting

the idea was, I can specify a tier at a higher level in the directory structure…but I wasn’t sure that would work.

maybe I did something wrong in the helmfile, though

the error seems to state as much

ah, I see

seems like you tried to use " in the selector which isn’t supported?

well, I tried using proper yaml first. That fails.

just tried it again.

❯ \cat helmfile.yaml 
helmfiles:
  - "*/helmfile.yaml"
  - path: "*/helmfile.yaml"
    selectors:
      - tier: network-and-proxies

gives:

n ./helmfile.yaml: in .helmfiles[0]: in ../../../../helmfile.d/helmfile.yaml: in .helmfiles[0]: in generic/01a-network-and-proxies/helmfile.yaml: failed to read helmfile.yaml: reading document at index 1: yaml: unmarshal errors:
  line 5: cannot unmarshal !!map into string

needs to be tier=network-and-proxies

is selectors: requiring a list perhaps?

definitely!

k, lemme fix and try

it must be a lint of strings

with no "?

in where?

as long as it’s a yaml string, it would be okay

- "foo=bar"

works

ok, lemme try

- foo="bar"

is a valid string but invalid in terms of the selector syntax

so

❯ \cat helmfile.yaml 
helmfiles:
  - "*/helmfile.yaml"
  - path: "*/helmfile.yaml"
    selectors:
      - "tier=network-and-proxies"

and this is temp/helmfile.d/generic/01a-network-and-proxies/helmfile.yaml

seems good

OK cool; that error is gone. But now it still can’t find the selector.

merged environment: &{default map[ambassador:map[namespace:ambassador version:6.6.2] arm64_support:false aws_nlb_with_tls_termination_at_lb:false blscauxclusterissuer:map[namespace:certmanager version:0.1.0] certmanager:map[namespace:certmanager version:1.0.4] cluster_id:<nil> dex:map[namespace:dex version:2.15.2] domain_name:<nil> externaldns:map[namespace:externaldns version:v20210203-v0.7.6-28-g44288212-arm64v8] name:somename nginxingress:map[namespace:ingress-nginx version:3.15.2] oauth2proxy:map[namespace:auth-system version:3.2.5] postgres:map[namespace:grafana version:x.x.x] semver:0.0.1 vaultoperator:map[namespace:vault version:1.8.1] vaultsecretswebhook:map[namespace:secrets-webhook version:1.8.2]] map[]}
0 release(s) matching tier=network-and-proxies found in helmfile.yaml

changing working directory back to "/home/jimconn/projects/src/personal/jimconn-shell/temp/helmfile.d/generic/01a-network-and-proxies"
0 release(s) matching app=dex found in helmfile.yaml

err: no releases found that matches specified selector(app=dex) and environment(default), in any helmfile

hmm…. is helmfile looking at the labels in releases: object too or just helmfile objects?

what’s the command now? I’m pretty lost

is helmfile looking at the labels in releases: object too or just helmfile objects? i don’t get it. helmfile “object” doesn’t have labels. only releases have labels to be matched by selectors

lol. I’m sorry. So the command is being run from temp/helmfile/envs/dev/atreus and the command I’m using is: helmfile --debug --selector app=dex template

well

if you notice that I was able to provide a selector in the helmfiles object of a helmfile.yaml for 01a-… and that at least linted.

yep. that means you’re letting helmfile to run helmfile -l $SELECTOR_HERE template on the sub-helmfile

yup

that way, helmfile uses the selector to filter releases defined in the sub-helmfile

cool

so I was thinking that I could also use a selector for a specific helm release given a selector

so in other words, let’s say I’m in my cluster-a environment

that should work

all I want to do is release one single app

so I name all my apps in releases with a selector (which I know this is a helm selector but helmfile seems to understand those, right)?

the point is that no helmfile.yaml under helmfile.d/generic/01a-network-and-proxies/* has releases that have the label app: dex

oh sure! I know. It’s in 01b-security-management/dex

i see! then you need to provide that to helmfile.yaml. currently it doesn’t refer to it

$ cat helmfile.yaml
helmfiles:
  - "*/helmfile.yaml"
  - path: "*/helmfile.yaml"
    selectors:
      - tier=network-and-proxies

my dex helmfile bundle is done so I’m just testing the “e2e” from cluster-a (or atreus) to see if my top-level config works.

ah!

were you trying to say this?

helmfiles:
  - "../*/helmfile.yaml"
  - path: "*/helmfile.yaml"
    selectors:
      - tier=network-and-proxies

ok

well, no

ok. but you need to add e.g. 01b-security-management/dex to helmfiles section at least

k. just a sec. Lemme see

so in my 01b-security-manangement helmfile.yaml I have:

helmfiles:
  - "*/helmfile.yaml"
  - path: */helmfile.yaml
    selectors:
      - tier=secrets-managment

I was thinking that if I place a selector on the command line the helmfile will read everything and simply use anything found matching the requested selector

yep you seem to be correct

and in this case, I have a bundle named dex which only deploys dex and it’s under 01b-security-management/dex

and the helmfile.yaml in there is:

---
bases:
  - ../../common/environments.yaml
  - ../../common/repos.yaml
  - ../../common/helmdefaults.yaml

---
releases:
  - name: dex
    namespace: {{ .Values.dex.namespace }}
    createNamespace: true
    labels:
      app: dex <<<<<=== THERE
    chart: stable/dex
    version: {{ .Values.dex.version }}
    values:
      - values.yaml.gotmpl

so helmfile should find that right?

yep. but where are you running helmfile from?

the atreus directory

and the content of helmfile.yaml under atreus?

oh, lemme post it

so helmfile should read the helmfile.yaml in helmfile.d

which only matches all *.yaml files in every directory

that directory contains the 01a-… and 01b-… directories which have helmfile.yaml files which define a tier selector for that directory of bundles.

so this is what im getting

isn’t this due to that helmfile failing fast due to that it failed to parse generic/01b-secrets-management/helmfile.yaml?

huh. werid.

seems like it. I don’t get that though. Lemme commit/push what I have here and you can run it again and see if you get the same result.

go ahead and pull

i think you’d need to fix each helmfile.yaml to work alone

oh crap

I am getting that

I did that first.

but I might have changed something during all this. Lemme check that real fast.

at glance this is invalid yaml

$ cat ../../../../helmfile.d/generic/01b-secrets-management/helmfile.yaml 
helmfiles:
  - "*/helmfile.yaml"
  - path: */helmfile.yaml
    selectors:
      - tier=secrets-managment

should be

helmfiles:
  - "*/helmfile.yaml"
  - path: "*/helmfile.yaml"
    selectors:
      - tier=secrets-managment

ok

I’ll add the quotes.

awesome!

that seemed to work

well

ish

great

yeah, so that is an interesting error that I’m still fuzzy on in terms of the cause.

this goes back to part of the conversation we had last week with respect to paths…but I still can’t articulate that very well, so I could just be wrong about its implementation

well lets start reading it

assuming we’re running helmfile-template in atreus

in ./helmfile.yaml: in .helmfiles[0]: in ../../../../helmfile.d/helmfile.yaml: in .helmfiles[2]: in generic/common/environments.yaml: failed to read environments.yaml: environment values file matching "../../common/versions.yaml" does not exist in "."

means that

yes

atreus/helmfile.yaml loaded ../../../../helmfile.d/helmfile.yaml, which in turn loaded generic/common/environments.yaml.

while helmfile’s trying to process the last file, environemnts.yaml, it failed to find ../../common/versions.yaml

let’s see what’s in environments.yaml

yep

yes

and this was where I was super confused

because it should have been values.yaml.gotmpl and versions.yaml

but that didn’t work

this was why I asked you the question the other day…if you will recall.

the paths document says that helmfile will use paths relative to the yaml file requesting the path/file

as you’re including it into some helmfile yaml under generic, those paths should be releative to generic

this my finding is that this isn’t consistent

that applies only to helmfile.yaml

which confused the poo outta me

ahhh

ok

that makes more sense as far as what the documentation means

so bases are base configuration or some form of skeleton of your helmfile.yaml

so what about other manifests? How do paths work with respect to others?

if bases are evaluated in the directory of the base helmfile yaml, it would prevent you from reusing it in any useful way

so every path is relative to the helmfile.yaml being processed

ah. hmm

bases modifies the helmfile.yaml

so that might be problematic for how I’m trying to do this.

probably. so you want to use base to codify some convention of where each helmfile.yaml should locate environment values files, right?

it worked nicely if I could make reference to these files relative to the manifest that references files.

yes

and so…it might be a little easier than that

and you wanted all the environment values files to be shared across all the helmfile.yaml files, right?

basically yes

essentially, the way I’m doing this, everything is using the same files whether it’s running as individual releases, directory releases, or environment releases.

lastly, do you want values.yaml.gtoml and versions.yaml to be under the same directory as environments.yaml(base)?

yes as those are basically global settings for everything but I want to be able to redefine stuff set in those (specifically the stuff in values.yaml.gotmpl) with the config.yaml in cluster-n environments.

and you seem to be using the base environments.yaml from helmfile.yaml files in various levels

generic/01a-network-and-proxies/ambassador is trying to load generic/common/environments.yaml

yes, because that’s a common file — is how I’m using it

ok

well it isn’t how base is supposed to be used but i can see how you’d like to use it

some people want to use bases to be evaluated in the context of the helmfile.yaml loaded it so helmfile is currently designed aroud that

yeah…which I think I wanted to do it this way because it makes sense to me in terms of a structure. Set up a default set and then allow the ability for that default set to change based on overriding configurations

ok

can you just embed all the values directly into environments.yaml then?

what’s an example of what that looks like?

something like

ahhh. I see… this might be doable.

well i used wrong term in the example but you get what i meant..

afk ill post compelte example later

yeah, use the yaml reference capability in conjunction with a template

ok!

so i think this should work

templates:
  versions: &versions
    semver: 0.0.1
      name: somename
      # Versions here reflect the chart version, NOT the app version.
      ambassador:
        version: 6.6.2
        namespace: ambassador
      certmanager:
        version: 1.0.4
        namespace: certmanager
      blscauxclusterissuer:
        version: 0.1.0
        namespace: certmanager
      externaldns:
        version: v20210203-v0.7.6-28-g44288212-arm64v8
        namespace: externaldns
      nginxingress:
        version: 3.15.2
        namespace: ingress-nginx
      postgres:
        version: x.x.x
        namespace: grafana
      oauth2proxy:
        version: 3.2.5
        namespace: auth-system
      dex:
        version: 2.15.2
        namespace: dex
      vaultoperator:
        version: 1.8.1
        namespace: vault
      vaultsecretswebhook:
        version: 1.8.2
        namespace: secrets-webhook
  values: *values
    aws_nlb_with_tls_termination_at_lb: {{ .Values | get "aws_nlb_with_tls_termination_at_lb" false }}
    arm64_support: {{ .Values | get "arm64_support" false }}
    domain_name: {{ .Values | get "domain_name" (env "DOMAIN_NAME") }}
    cluster_id: {{ .Values | get "cluster_id" (env "CLUSTER_ID") }}

---

environments:
  default:
    values:
    - *versions
    - *values
  production:
    values:
    - *versions
    - *values

ugh. I just tried validating the configuration of the template test run I did a while ago (I had to re-establish the values.yaml.gotmpl contents, which I removed from my private repo) and unfortunately, I’m observing that the template run shows that the settings in the config.yaml values in atreus are not getting asserted during the template run meaning it seems that the helmfile.yaml isn’t asserting the config from the top-level. :(

lemme try your suggestion real fast though.

I’m observing that the template run shows that the settings in the config.yaml values in atreus are not getting asserted during the template run meaning it seems that the helmfile.yaml isn’t asserting the config from the top-level. what do you mean by “asserting” here?

are you saying that values defined in config.yaml in the below atreus/helmfile.yaml is not accessible from within sub-helmfiles ../../../.../helmfile.d?

environments:
  default: {}

---
helmfiles:
  - path: "../../../../helmfile.d/"
    values:
      - config.yaml

i have not yet fully understood your whole config, but from what i can guess from our conversation so far

yeah, seems so.

you may be missing to pass values in the middle

oh! I need to pass values?

is there a generic way to do that without having to specify every variable?

Yes. As I have said in elsewhere, you must always explicitly pass values to sub-helmfiles

ahhhh….

hence, no inheritance which you mentioned

yeah

so what works today is https://sweetops.slack.com/archives/CE5NGCB9Q/p1618896775125900?thread_ts=1618580632.114500&cid=CE5NGCB9Q

lemme give that a quick try, too

and what we’ll likely to add to helmfile in (near) future https://sweetops.slack.com/archives/CE5NGCB9Q/p1618896996128700?thread_ts=1618580632.114500&cid=CE5NGCB9Q

got it!

one sec

beware the number you pass to nindent function. it’s dependent on the context.

- {{ .Values | toYaml | indent 2

  - {{ .Values | toYaml | indent 4 }}

make sense, but

  - {{ .Values | toYaml | indent 2 }}

or

- {{ .Values | toYaml | indent 4 }}

doesn’t.

this usage of toYaml with nindent is a common trick seen in wirting helm charts but I thought ir worth being explained

in my case, the indent is the same for all of the files and 6 is accurate I believe.

first-pass produced: &{default map[values:[map[aws_nlb_with_tls_termination_at_lb:false] map[arm64_support:false] map[domain_<name:atreus.dev.domain.io>] map[cluster_id:atreus]]] map[]}
first-pass rendering result of "helmfile.yaml.part.0": {default map[values:[map[aws_nlb_with_tls_termination_at_lb:false] map[arm64_support:false] map[domain_<name:atreus.dev.domain.io>] map[cluster_id:atreus]]] map[]}
second-pass rendering result of "helmfile.yaml.part.0":
 0: ---
 1: helmfiles:
 2:   - "generic/*"
 3:   - path: "generic/*"
 4:     values:
 5:       - 
 6:       values:
 7:       - aws_nlb_with_tls_termination_at_lb: false
 8:       - arm64_support: false
 9:       - domain_name: atreus.dev.domain.io
10:       - cluster_id: atreus
11:       
12: 

err: failed to read helmfile.yaml: reading document at index 1: yaml: line 6: did not find expected node content
changing working directory back to "/project/helmfile-project/helmfile/envs/dev/atreus"
in ./helmfile.yaml: in .helmfiles[0]: in ../../../../helmfile.d/helmfile.yaml: failed to read helmfile.yaml: reading document at index 1: yaml: line 6: did not find expected node content

So the failure seems to be coming from the line I just added (helmfile.d/helmfile.yaml

---
helmfiles:
  - "generic/*"
  - path: "generic/*"
    values:
      - {{ .Values | toYaml | nindent 6 }}

it’s not dereferencing .Values

oh wait

it is

I need to fix the config.yaml

just a sec

i’m now counting the size of the indentation there

oh ok

well shouldn’t it be 8?

---
helmfiles:
  - "generic/*"
  - path: "generic/*"
    values:
      - {{ .Values | toYaml | nindent 8 }}

with 8 it should be rendered to

helmfiles:
  - "generic/*"
  - path: "generic/*"
    values:
      - foo: bar
        baz: 1

It looks like it should be 4

I’ll make it 8 and see what happens

why 4? you pasted the helmfile.yaml with wrong indentation?

just going strictly by the output which was:

 4:     values:
 5:       - 
 6:       values:

but I was looking at the wrong values:

those numbers from 4 to 6 are line numbers

yeah, I wasn’t looking at those

ah ok

I was looking at the indentation of the values on line 6 instead of line 4

cool

I think we got past that

nice

now we’re back on the environments.yaml

the template

err: failed to read ../../common/environments.yaml: reading document at index 1: yaml: line 4: mapping values are not allowed in this context

looks like a syntax thing likely

ah, line 4 does have wrong indentation

templates:
  versions: &versions
    semver: 0.0.1
      name: somename
      # Versions here reflect the chart version, NOT the app version.
      ambassador:
        version: 6.6.2
        namespace: ambassador

i wrote this but this should be

I see it

it’s 2+ and it needs to be 2-

OK, so getting closer. If you are OK with helping me fix this one thing then I’ll stop bugging you for the rest of the day…lol (I need to go to bed) and then I’ll try and fix up the rest on my own and if I get stuck, I’ll let you know and hopefully you can help another time?

yeah sure! but expect i won’t be as responsive as today tomorrow. today was my day off.

doh! maybe I’ll just stay up…lol

well, let’s see

There’s something I need to do to fix the other helmfile.yaml files for each chart which rely on environments.yaml in common

err: failed to read ../../common/environments.yaml: reading document at index 1: yaml: unmarshal errors:
  line 4: field semver not found in type state.TemplateSpec
  line 7: field ambassador not found in type state.TemplateSpec
  line 10: field certmanager not found in type state.TemplateSpec
  line 13: field blscauxclusterissuer not found in type state.TemplateSpec
  line 16: field externaldns not found in type state.TemplateSpec
  line 19: field nginxingress not found in type state.TemplateSpec
  line 22: field postgres not found in type state.TemplateSpec
  line 25: field oauth2proxy not found in type state.TemplateSpec
  line 28: field dex not found in type state.TemplateSpec
  line 31: field vaultoperator not found in type state.TemplateSpec
  line 34: field vaultsecretswebhook not found in type state.TemplateSpec
  line 38: field aws_nlb_with_tls_termination_at_lb not found in type state.TemplateSpec
  line 39: field arm64_support not found in type state.TemplateSpec
  line 40: field domain_name not found in type state.TemplateSpec
  line 41: field cluster_id not found in type state.TemplateSpec
changing working directory back to "/project/helmfile-project/helmfile.d/generic/01a-network-and-proxies"
changing working directory back to "/project/helmfile-project/helmfile.d"
changing working directory back to "/project/helmfile-project/helmfile/envs/dev/atreus"
in ./helmfile.yaml: in .helmfiles[0]: in ../../../../helmfile.d/helmfile.yaml: in .helmfiles[0]: in generic/01a-network-and-proxies/helmfile.yaml: in .helmfiles[0]: in ambassador/helmfile.yaml: failed to read ../../common/environments.yaml: reading document at index 1: yaml: unmarshal errors:
  line 4: field semver not found in type state.TemplateSpec
  line 7: field ambassador not found in type state.TemplateSpec
  line 10: field certmanager not found in type state.TemplateSpec
  line 13: field blscauxclusterissuer not found in type state.TemplateSpec
  line 16: field externaldns not found in type state.TemplateSpec
  line 19: field nginxingress not found in type state.TemplateSpec
  line 22: field postgres not found in type state.TemplateSpec
  line 25: field oauth2proxy not found in type state.TemplateSpec
  line 28: field dex not found in type state.TemplateSpec
  line 31: field vaultoperator not found in type state.TemplateSpec
  line 34: field vaultsecretswebhook not found in type state.TemplateSpec
  line 38: field aws_nlb_with_tls_termination_at_lb not found in type state.TemplateSpec
  line 39: field arm64_support not found in type state.TemplateSpec
  line 40: field domain_name not found in type state.TemplateSpec
  line 41: field cluster_id not found in type state.TemplateSpec

this looks like an indentation issue and I think I probably need to fix the reference to match what we did with the other helmfile.yaml files.

it will be helpful if you could push the latest snapshot of your whole setup before asking questions that would reduce forth-and-back

seems to

sure! do you need that now or just in case I need more help tomorrow?

btw, sincerely appreciate your assistance. Very kind!

just update your git repo immediately before you add another question so that i can try to replicate your issue and think concretely

state.TemplateSpec is the underlying go struct that maps to templates in helmfile.yaml and in your case environments.yaml

cool! will do. Due to the nature of the company policies, I just need to sync between the public repo and my private repo here and then commit/push which will take a few mins.

ah! ok, that’s good context.

ah ok turns out we were only able to define some fields under templates.foo

well then we can’t leverage templates that way..

looks like I’ll need to add helmdefaults to the template

oh, we can’t?

helmDefaults contains default values for releases[] so it might be useful elsewhere, but i think it doesn’t work for this

ok

well trying to come up with a workaround..

k

so I guess I’m not understanding what’s wrong with the template method we’re working on right now.

the helmfile.yaml files I updated a while ago were just the ones in the middle

I didn’t update any of the chart helmfile.yaml files yet

how about this…

environments:
  template:
    values:
    - versions: &versions
        semver: 0.0.1
        name: somename
        # Versions here reflect the chart version, NOT the app version.
        ambassador:
          version: 6.6.2
          namespace: ambassador
        certmanager:
          version: 1.0.4
          namespace: certmanager
        blscauxclusterissuer:
          version: 0.1.0
          namespace: certmanager
        externaldns:
          version: v20210203-v0.7.6-28-g44288212-arm64v8
          namespace: externaldns
        nginxingress:
          version: 3.15.2
          namespace: ingress-nginx
        postgres:
          version: x.x.x
          namespace: grafana
        oauth2proxy:
          version: 3.2.5
          namespace: auth-system
        dex:
          version: 2.15.2
          namespace: dex
        vaultoperator:
          version: 1.8.1
          namespace: vault
        vaultsecretswebhook:
          version: 1.8.2
          namespace: secrets-webhook
      values: *values
        aws_nlb_with_tls_termination_at_lb: {{ .Values | get "aws_nlb_with_tls_termination_at_lb" false }}
        arm64_support: {{ .Values | get "arm64_support" false }}
        domain_name: {{ .Values | get "domain_name" (env "DOMAIN_NAME") }}
        cluster_id: {{ .Values | get "cluster_id" (env "CLUSTER_ID") }}

---

environments:
  default:
    values:
    - *versions
    - *values
  production:
    values:
    - *versions
    - *values

lemme give it a shot

ah probably this doesnt work due to yaml anchors not persisnt across ---

what if you removed ---

removing --- isn’t usually a good idea as this gudes you into the deep sea of helmfile “double-rendering” hack

yeah, it seems that one needs to understand when and when not to use ---

probably you’d better not to think about it and just keep adding --- for now

ok, so quick question… I’m trying to set up the ambassador/helmfile.yaml to use the passed .Values and I’m not 100% sure how to properly do that. The old helmfile.yaml for ambassador was:

---
bases:
  - ../../common/environments.yaml
  - ../../common/repos.yaml
  - ../../common/helmdefaults.yaml

---
releases:
  - name: ambassador
    namespace: {{ .Values.ambassador.namespace }}
    createNamespace: true
    labels:
      app: ambassador
    chart: datawire/ambassador
    version: {{ .Values.ambassador.version }}
    values:
      - values.yaml.gotmpl

the one I just tried, which failed was:

---
bases:
  - ../../common/repos.yaml
  - ../../common/helmdefaults.yaml
---
values:
  - {{ .Values | toYaml | nindent 8 }}

releases:
  - name: ambassador
    namespace: {{ .Values.ambassador.namespace }}
    createNamespace: true
    labels:
      app: ambassador
    chart: datawire/ambassador
    version: {{ .Values.ambassador.version }}
    values:
      - values.yaml.gotmpl

wrong indentation there

the last resort- try this if removing --- didn’t work

environments:
  {{ .Environment.Name }}:
    values:
    - versions:
        semver: 0.0.1
        name: somename
        # Versions here reflect the chart version, NOT the app version.
        ambassador:
          version: 6.6.2
          namespace: ambassador
        certmanager:
          version: 1.0.4
          namespace: certmanager
        blscauxclusterissuer:
          version: 0.1.0
          namespace: certmanager
        externaldns:
          version: v20210203-v0.7.6-28-g44288212-arm64v8
          namespace: externaldns
        nginxingress:
          version: 3.15.2
          namespace: ingress-nginx
        postgres:
          version: x.x.x
          namespace: grafana
        oauth2proxy:
          version: 3.2.5
          namespace: auth-system
        dex:
          version: 2.15.2
          namespace: dex
        vaultoperator:
          version: 1.8.1
          namespace: vault
        vaultsecretswebhook:
          version: 1.8.2
          namespace: secrets-webhook
      values:
        aws_nlb_with_tls_termination_at_lb: {{ .Values | get "aws_nlb_with_tls_termination_at_lb" false }}
        arm64_support: {{ .Values | get "arm64_support" false }}
        domain_name: {{ .Values | get "domain_name" (env "DOMAIN_NAME") }}
        cluster_id: {{ .Values | get "cluster_id" (env "CLUSTER_ID") }}

you may need to rename environments.yaml to environments.yaml.gotmpl. i forgot full details but try renaming the file if it failed to render go template at all

oh! yeah, that might be required

values:
  - {{ .Values | toYaml | nindent 8 }}

seems like a invalid indentation

should be 4

if you removed ` - ../../common/environments.yaml ` from the ambassador helmfile.yaml, it should end up like that

because you’ve tried todefine ambassador.namespace in the versions, which was to be loaded via environments.yaml you just removed

but that should be passed in using the intermediate .Values now, though shouldn’t it?

Intermediate .Values doesn’t contain values defined in versions neither, as you’ve omitted it from bases

hmm.

ok

values:
  - {{ .Values | toYaml | nindent 8 }}

is noop.

you are basically assigning Values to Values

ah, yes. I was originally going to try just:

---
{{ .Values | toYaml }}

in helmfiel.yaml?

yeah

(that wasn’t the whole thing…just the values portion)

that would break helmfile.yaml, as it can’t contain arbitary key values like that

oh

I could put it under bases:?

you can only have artibrary key values under values of some kind. like releases[].values or environments[].values

it’s not a free-form yaml

aahhh. OK, that’s good to know.

so let me do this then: … (one min)

nope. bases has its own schema

---
bases:
  - ../../common/repos.yaml
  - ../../common/helmdefaults.yaml

#config.yaml
---
environments:
  default:
    values:
    - {{ .Values | toYaml | nindent 8 }}

---
releases:
  - name: ambassador
    namespace: {{ .Values.ambassador.namespace }}
    createNamespace: true
    labels:
      app: ambassador
    chart: datawire/ambassador
    version: {{ .Values.ambassador.version }}
    values:
      - values.yaml.gotmpl

?

seems ok, except that nindent should be 6

or I just need to push the indentation out 2 to match everything else.

dang flexibleness of yaml

maybe..?

i thought you usually align yaml dict key values at the level indicated by the end of the selection shown in the below picture

which is 6

yaml allows either or

it can be aligned or pushed out

wow really!

I tend to push it out because I’m old school

yep

the new school way seems to be aligned, though

but make two files using both pragmas and yamllint them

it will work on both

oh well but i still don’t get it

lets say you had tis values

foo: bar
bar: baz

environments:
  default:
    values:
    - {{ .Values | toYaml | nindent 8 }}

renders to

environments:
  default:
    values:
    - foo: bar
        bar: baz

oh, maybe I should qualify what I’m talking about…I’m talking only for lists.

this does really work?

ah gotcha

then you should still say nindent 6 there

environments:
  default:
    values:
    - 
{{ .Values | toYaml | indent 6 }}

if it’s indent rather than nindent this should work

or even this

environments:
  default:
    values:
    - 
{{ .Values | toYaml | indent 8 }}

what’s the difference in go templating between nindent and indent btw? I haven’t looked up the docs on that yet.

fixed wrong indentations in indent examples

yep that’s very important

http://masterminds.github.io/sprig/strings.html

that’s why nindent of 8 ends up with https://sweetops.slack.com/archives/CE5NGCB9Q/p1618909727197500?thread_ts=1618580632.114500&cid=CE5NGCB9Q here

this is very common trick in writing not only helmfile templates but also helm templates so i’d highly recommend getting used to this

the things so far

note that I renamed environments.yaml to environments.yaml.gotmpl to respect helmfile to render go templatized file

but I’m not sure that was enough…I feel like I need to tell helmfile to actually use the gotmpl file.

yes

but where?

you need to add that back to bases

oh

i thought you just removed it from bases earlier?

which helmfile.yaml file ? All of them?

I thought you said it needed to be removed.

I might have misunderstood you though

well no

because we were trying to solve the directory path issue…

the directory path issue would have been resolved when we embedded versions.yaml and values.yaml.gotmpl into environments.yaml.gotmpl

neither environments.yaml nor environments.yaml.gotmpl is going to be included in bases or loaded before rendering helmfile.yaml , except you explicitly add it to bases

you need to be explicit about what values to load, what bases to use, what values to be passed to sub-helmfile

ok. so that last thing is pretty good info. That resonates with me…and I’ll need to think about my project in those terms now methinks.

that’s actually something that should probably be added to the documentation like…very. clearly.

let me make I am understanding what you’re saying, too…

definitely. the point is that i know too much about helmfile already and can’t imagine what parts are missing to what levels of people

for a project, one must be explicit (for each helmfile.yaml) about what values to load, what bases to load and then when layering, what values to pass “down the chain”

so if you could contribute doc addition/fix based on your experience, that would be awesome

absolutely!

yep

ok cool. I need to grok that concept then…

and put it in terms of this project

so let me see if I understand how we’ve done things so far in terms of that concept and this project

I envisioned a project where I could deploy dex via helmfile from within its own directory and all it needs to know is where to find the bases, environments values (if any) and any additional values (for this project in this context, that would be passed in via env vars)…

brb…2 mins

i’ll be afk for dinner but keep posting

oh, and I think it worth a new slack thread now

lol. OK

its getting really long and takes a bit of time to fully load, and im also afraid if every update to this thread is being notified to anyone commented on this thread only in very beginning

It’s been very informative actually :)

@jedineeper is the concept I’m trying to accomplish making sense?

just want to make sure I’m not daft

cool, I created a new thread that we can move this to if you want.

This started with me misunderstanding the use of environments so some of it is beyond the scale of my understanding so I’m not a good judge :)

here goes

@jedineeper if you’re interested

@mumoshu I’ve pushed all my changes; and stuff is broke but it’s like almost 430 AM here and I need to get to bed.

if you don’t mind though, give it a looksee. I’ll see if I can make heads or tails of it tomorrow

er I guess…today is more accurate.

will try to find some time!

I’m not thinking straight though, anymore. So, again, really appreciate all your help.

thank you very much! I’ll chat you up later today/tomorrow your time.

g’night

thanks for trying helmfile! gnight

hey @Alex Genco have you tried using dependencies ? you can define a release with its dependencies so that the child chart will inherit values with <child_chart> prefix - see https://github.com/roboll/helmfile/issues/1762 for reference

I guess my question is, what’s the best way to “inline” environments im not sure if i understand fully, but it seems like you’ve already made it concise and DRY enough.

If i were you i won’t try to make it DRY further, thinking doing so would hurt readability

if really want, you can still do this with standard go template techniques

like

{{ define "t" }}
  - name: app-{{ .env }}
    chart: foo/app
    namespace: app-{{ .env }}
    values:
      - {{ .env }}.yaml
      - values.yaml.gotmpl
{{ end }}

releases:
{{ template "t" (dist "env" "staging" ) }}
{{ template "t" (dist "env" "production" ) }}

i believe you’d better use a more serious configuration language like CUE if you need to dynamically generate helmfile.yamls at this level. FYI, CUE is https://cuelang.org/

but i thought i’ve seen many people happy with https://sweetops.slack.com/archives/CE5NGCB9Q/p1618995133219700?thread_ts=1618934290.213300&cid=CE5NGCB9Q so i can imagine you’ll like it, too

But since the parent repo has no concept of environments, it breaks the deploy process. Well, i guess the reality is that the parent repo does have the concept of environments, but they are hidden. If the parent has no environnment, children shouldn’t have environments

If the parent envs are just hidden and you don’t want to write

environments:
  staging: {}
  production: {}

in parent, you may prefer

environments:
  {{ .Environment.Name }}: {}

---

# releases, helmfiles, etc

@jose.amengual Hey! It sounds like issues in the helm chart you’re using. Helmfile’s just calling helm uprade --install or helm delete or whatever according to the definitions in your helmfile.yaml so that kind of issues can be very likely to be caused by the chart, not helmfile

interesting ok, I will have a look

I used a different namespace and everything works

if I destroy, and delete other resources lingering and use the same namespace that did not work before it does not create the alb

so there must be something still there

now I wonder how I can find it

from the docs on NLBs…

Do not modify the service annotation service.beta.kubernetes.io/aws-load-balancer-type on an existing service object. If you need to modify the underlying AWS LoadBalancer type, for example from classic to NLB, delete the kubernetes service first and create again with the correct annotation. Failure to do so will result in leaked AWS load balancer resources.

so this : https://github.com/kubernetes-sigs/aws-load-balancer-controller/issues/1629

that @Andriy Knysh (Cloud Posse) found is real…

if you modify resources created by the controller int he console, you will have orphan resources

in my case the ingress what still there

so I run kubectl patch ingress ingress-name -n namespace-name -p '{"metadata":{"finalizers":[]}}' --type=merge

and then the ingress was gone and I was able to run helmfile sync and helmfile destroy and everything worked as expected

so….just don’t do ClickOps

yes we have some we use

generally no. but im curious why you want to do that?

also, what’s the exact error you’re trying to ignore?

so there is an issue I created a week ago https://github.com/roboll/helmfile/issues/1778

and I am trying to workaround it

we need to override some default rules from kube prometheus stack, my original idea was to use a patch, but since that didn’t work now I am trying another approach

unfortunately it involves a duplicate resource inside the kube prometheus release: I use additionalPrometheusRules with the name of already existing rule

so helm fails when it sees the duplicate, I was under impression that everything still works fine after that but now I see that this actually happens in the middle of deployment, so some resources are missing after it

so probably ignoring the error won’t be actually helpful in this case

but I have another question now:

if a release fails then further apply attempts do not show any errors but also do not deploy missing resources - is that expected behaviour?

(that’s what led me to believe that everything is find after the error)

one possible workaround is to create a hook in which you run a script that performs a helm template on the chart and then you kubectl apply -f - on the template output patching whatever is necessary on the broken bits on a pipe in between template and kubectl — essentially you’d just fix the manifests as they’re being templatized. It’s ugly and a total hack and might be bug prone…but it’s an idea maybe?

thank you for the idea, Jim

@Eugene Korekin I’ve tried to reproduce your issue but had no luck. please see https://github.com/roboll/helmfile/issues/1778#issuecomment-824441138

perhaps your issue had already been fixed later prs?

if you still have some issue, sharing the reproduction steps on a brand-new cluster would be helpful

@mumoshu the behaviour is definitely different with the latest version from master

https://github.com/roboll/helmfile/issues/1778#issuecomment-824451990

i have managed to let helmfile fail with another error and it does seem like a bug

the one i saw seems happen when you do have strategicmergepatches/jsonpatches + CRDs

right, but there is another issue

whats it?

looks like there is some regression in the master comparing to the stable version

just a moment

so here is a simple helmfile

repositories:
  - name: prometheus-community
    url: <https://prometheus-community.github.io/helm-charts>
releases:
  - name: test
    namespace: test
    chart: prometheus-community/kube-prometheus-stack
    version: ~14.4.0

it works without any issue with the stable version please note that it works without disableValidation

and I ensured that the crds weren’t present before the installation (deploying a new clean cluster right now to be 100% sure)

whoa really? i cant imagine how it can work without disableValidation now…

ok, that’s probably because I used --skip-diff-on-install

ah that makes sense

it’s me who added helm diff --disable-validation to avoid helm-diff failing when trying to diff custom resources for CRD that isn’t installed yet

sure, I saw the PR, that was a great idea

so yeah, --skip-diff-on-install would make disableValidation unnecessary. got it

btw, would it be good idea if helmfile from master will print something different as its version number, not the same string as the stable version?

I am having issues trying to understand which version I use sometimes, cause --version gives the same results for stable and master

ah sounds good. i thought the version number if set via TAG = $(shell git describe --tags --abbrev=0 HEAD) in Makefile

i was innocently believing that it would return any tag only on tagged commit but apparently not

Ah okay i see
git-describe
The command finds the most recent tag that is reachable from a commit

looks like there is some regression in the master comparing to the stable version ok, please disregard this I just checked and they both work in the same way

great. thanks for testing!

so, do you need any other information from my side regarding this issue with jsonpatches and crds?

i believe you’ve provided me enough! thx. i’m now wondering how i could fix that

seems helmfile needs to extract CRDs from the patched YAML and put it under crds dir in the temp chart…. shouldn’t there be any easier way than this

thank you very much

if it won’t be possible to fix, I think specifying that json and strategic merge patches don’t work with CRDs in documentation would be very useful

i cant help wishing there was helm template --crds-only

yep true

Matt Farina is a friend of mine. I can suggest that to him.

i would generally prefer putting crds in a separate chart and then you’re fine

I used to work with him.

cool!

@Eugene Korekin another potential solution would be to enhance helmfile to accept skipCRDs: true in helmfile.yaml.

Then you could have a separate release for CRDs only and another for kube-promehtheus-stack with skipCRDs: true

where in the skipCRDsed kube-promehtue-stack release, you can freely use patching

that separate release for CRDs, should it be created manually in that case?