#helmfile (2021-04)

https://github.com/roboll/helmfile

Questions and discussion around helmfile https://github.com/roboll/helmfile and https://github.com/cloudposse/helmfiles Archive: https://archive.sweetops.com/helmfile/

2021-04-30

2021-04-29

2021-04-28

Jim Conner avatar
Jim Conner


I thought I read in the docs once that there was a way to set a dependency up (like terraforms depends on) but of course, differently implemented. Did I misread that?
For instance, our implementation of dex requires a running vault. I’m sure there are ways to set up the environment that such a thing could be jiggered but is there a native way for helmfile to do this type of functionality?

mumoshu avatar
mumoshu

needs?

mumoshu avatar
mumoshu
roboll/helmfile attachment image

Deploy Kubernetes Helm Charts. Contribute to roboll/helmfile development by creating an account on GitHub.

Jim Conner avatar
Jim Conner

@mumoshu if you’re available: how on earth would one define globally used values for all releases in an environment. This has been my absolute biggest blocker. Even better is that the global scoped values.yaml could be also templatized. I cannot figure out how to achieve this goal. Nothing I’ve tried works.

mumoshu avatar
mumoshu

something like this?

values:
- someGobalValue: foo

---

values:
- {{ .Values | toYaml | nindent 2 }}
- anotherGlobalValue: {{ .Values.someGlobalValue }}bar

---

releases:
- name foo
  chart: somechart
  values:
  - some: {{ .Values.someGlobalValue }}
    another: {{ .Values.anotherGlobalValue }}
- name bar
  chart: somechart
  values:
  - some: {{ .Values.someGlobalValue }}
    another: {{ .Values.anotherGlobalValue }}
mumoshu avatar
mumoshu

@ I’ve just merged it. Thanks a lot for your help

1

2021-04-27

es avatar

hi. is this a bug?

if I run helmfile lint against helmfile.yaml that contains

helmfiles:
  - environments/test1/helmfile.yaml
  - environments/test2/helmfile3.yaml

it fails with Error: repo not found it’s referring to a release located in helmfile3.yaml. if I remove the first line (environments/test1/helmfile.yaml) it works fine. btw, I am using helmBinary option to specify helm version in each of those helmfiles

es avatar

if I run helmfile -f environments/test2/helmfile3.yaml repos it works. shouldn’t it automatically fetch charts?

mumoshu avatar
mumoshu

@ Does it work when you run helmfile -f environments/test1/helmfile.yaml repos, too?

es avatar

The repos in that file automatically fetch when I apply so I haven’t tested. I’m on latest helmfile version btw

mumoshu avatar
mumoshu

Are you using any of strategicMergePatches/forceNamespace/jsonPatches/dependencies/transformers in your helmfile.yaml then?

es avatar

No

mumoshu avatar
mumoshu

I’m not aware of any relevant bug that is fixed in the unreleased version of helmfile, at least

es avatar

https://pastebin.com/Gt4EU67h - thanks in advance and please let me know if you need more information.

$ cat helmfile.yaml helmfiles: - "environments/mgmt/helmfile.yaml" - "en - Pastebin.com

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

1
mumoshu avatar
mumoshu

@ Thanks! Would you mind sharing me the exact version numbers of /usr/local/bin/helm and /usr/local/bin/helm3?

es avatar

it’s there, you probably missed it. sec

# helmfile version;helm version;helm3 version
helmfile version v0.138.7
Client: &version.Version{SemVer:"v2.15.1", GitCommit:"cf1de4f8ba70eded310918a8af3a96bfe8e7683b", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.15.1", GitCommit:"cf1de4f8ba70eded310918a8af3a96bfe8e7683b", GitTreeState:"clean"}
version.BuildInfo{Version:"v3.2.0", GitCommit:"e11b7ce3b12db2941e90399e874513fbd24bcb71", GitTreeState:"clean", GoVersion:"go1.13.10"}
mumoshu avatar
mumoshu

Ah thanks. Let me check..

mumoshu avatar
mumoshu

Whoa, this helm2 is super outdated :slightly_smiling_face: I cant even run helm init --client-only

mumoshu avatar
mumoshu

touch ~/.helm/repository/repositories.yaml did the job…

mumoshu avatar
mumoshu

so i was able to reproduce it with only

helmfiles:
- helmfile.jenkins1.yaml
- helmfile.jenkins2.yaml

helmfile.jenkins1.yaml

repositories:
- name: jenkins
  url: "<https://charts.jenkins.io>"

releases:
- name: jenkins11
  chart: jenkins/jenkins
  version: "2.5.0"

helmfile.jenkins2.yaml

repositories:
- name: jenkins
  url: "<https://charts.jenkins.io>"

helmBinary: helm2151

releases:
- name: jenkins21
  chart: jenkins/jenkins
  version: "3.3.9"
mumoshu avatar
mumoshu

this might be due to unnoticed race between helm2 and helm3 fetch

es avatar

yeah - a similar configuration worked before adding helmBinary

mumoshu avatar
mumoshu

after the failure it doesn’t even show up in helm2 repo list so it maybe be race between helm repo add/up of helm v2 and v3

mumoshu avatar
mumoshu

Almost certainly this is due to a helmfile bug. We usually skipp helm repo add calls on already added repo. But we don’t differentiate calls between helm v2 and v3 there

mumoshu avatar
mumoshu
roboll/helmfile attachment image

Deploy Kubernetes Helm Charts. Contribute to roboll/helmfile development by creating an account on GitHub.

roboll/helmfile attachment image

Deploy Kubernetes Helm Charts. Contribute to roboll/helmfile development by creating an account on GitHub.

mumoshu avatar
mumoshu

that results in helm repo add jenkins ... for helm 3 skips the subsequent helm repo add jenkins ... for helm 2

es avatar

yeah, I see. I just renamed the jenkins repo in helmfile3.yaml to something else and it worked fine

mumoshu avatar
mumoshu

Awesome! That’s a nice workaround

es avatar

thanks for helping

mumoshu avatar
mumoshu

@ Would you mind creating a bug report in helmfile issues? I have a working local branch for fix

es avatar

sure thing

1
mumoshu avatar
mumoshu

@ Thanks for the writeup! Created the PR https://github.com/roboll/helmfile/pull/1816

Fix repo sync to work on repos duplicated between helm v2 and v3 by mumoshu · Pull Request #1816 · roboll/helmfile attachment image

Trying to helm repo add the same Helm chart repository to Helm V2 and V3 in a single Helmfile run had been resulting in an incomplete result, only the latter helm repo add being skipped. This fixes…

es avatar

thank you @mumoshu

mumoshu avatar
mumoshu

Anyway, it would be great if you could provide provide us a more complete example for reproduction. At this point it’s too hard to say if it’s either a bug or not

Jim Conner avatar
Jim Conner

@mumoshu I think I have a fundamental misunderstanding of how to use selectors for sub-helmfiles. You already have a basic skeleton of my repo/condition although I’ve made many changes to it so I’m gonna have to articulate my ask.

I’m using a layered approach to my helmfile project as you know. If you look in the generic directory there is a 01a-… and 01b… directory for a tier of tools in each directory so I want to create a selector that would cause helmfile to 1) run everything in the respective 01a-… and 01b-… directories and 2) I want the flexibility to be able to just run one helmfile within a subdirectory of say, 01b-… (essentially just installing a single chart). It seems to me that the documentation says that the selectors are defined in the helmfiles directive of the parent directories to those sub-directories.

given: <project root>/helmfile.d/generic/01a-tier1 and <project root>/helmfile.d/generic/01b-tier2 if I want to be in helmfile.d to run helmfile and I want to install just charts in 01b-tier2 then the helmfile.yaml in helmfile.d would have to define the selectors and point to the helmfile.yaml in generic defining the selectors with a path: pointing to helmfile.yaml in 01b-tier2 and so forth, correct?

Jim Conner avatar
Jim Conner

My helmfile.yaml in helmfile.d looks like:

# helmfile.yaml in helmfile.d
---
helmfiles:
  - "generic/*"
  - path: "generic/common/*"

My helmfile.yaml in generic is:

# helmfile.yaml in generic
---
helmfiles:
  - "*/*.yaml"
  - path: "common/*"

  - path: "01a-network-and-proxies/helmfile.yaml"
    values:
      - {{ .Values | toYaml | nindent 8 }}
    selectors:
      - "tier=network-and-proxies"

  - path: "01b-secrets-management/helmfile.yaml"
    values:
      - {{ .Values | toYaml | nindent 8 }}
    selectors:
      - "tier=secrets-managment"
Jim Conner avatar
Jim Conner

and finally, my helmfile.yaml in 01b-… for instance is:

# helmfile.yaml in security-management
---
helmfiles:
  - "*/helmfile.yaml"
  - path: "../common/*"

  - path: "certmanager/helmfile.yaml"
    values:
      - {{ .Values | toYaml | nindent 8 }}
    selectors:
      - "tier=secrets-managment"
      - "app=certmanager"

  - path: "vault-operator/helmfile.yaml"
    values:
      - {{ .Values | toYaml | nindent 8 }}
    selectors:
      - "tier=secrets-managment"
      - "app=vault-operator"

  - path: "vault-secrets-webhook/helmfile.yaml"
    values:
      - {{ .Values | toYaml | nindent 8 }}
    selectors:
      - "tier=secrets-managment"
      - "app=vault-secrets-webhook"

  - path: "dex/helmfile.yaml"
    values:
      - {{ .Values | toYaml | nindent 8 }}
    selectors:
      - "tier=secrets-managment"
      - "app=dex"

  - path: "oauth2-proxy/helmfile.yaml"
    values:
      - {{ .Values | toYaml | nindent 8 }}
    selectors:
      - "tier=secrets-managment"
      - "app=oauth2-proxy"
Jim Conner avatar
Jim Conner

I’m inclined to think that that is incorrect, though, because when I run sync --selectors tier=secrets-managment,app=dex (or delete) for instance, it runs everything:

Jim Conner avatar
Jim Conner
mumoshu avatar
mumoshu

Why are you giving selectors: for every sub-helmfile? What if you just omitted that and let it install all when helmfile sync and let it install only part of releases with helmfile -l foo=bar sync?

Jim Conner avatar
Jim Conner

I guess because when I tested that, it didn’t work as I’d hoped…

Jim Conner avatar
Jim Conner

maybe I did something wrong.

Jim Conner avatar
Jim Conner

I was just looking at https://github.com/roboll/helmfile/blob/master/examples/README.md and I saw that helmfile is able to use the release labels…

roboll/helmfile attachment image

Deploy Kubernetes Helm Charts. Contribute to roboll/helmfile development by creating an account on GitHub.

Jim Conner avatar
Jim Conner

so I’ll check that out again and see if I can find out why that doesn’t work.

Jim Conner avatar
Jim Conner

nevertheless, I would like to be able to assign each full directory to a tier

Jim Conner avatar
Jim Conner

sorry for the slow response. I was at the store.

Jim Conner avatar
Jim Conner

very cool. That seems to be working. Not sure what I was doing wrong before…

Jim Conner avatar
Jim Conner

noice!

Jim Conner avatar
Jim Conner

I have to go to the store again to get something for my wife but if you can answer this while I’m gone… I thought I read in the docs once that there was a way to set a dependency up (like terraforms depends on) but of course, differently implemented. Did I misread that? For instance, our implementation of dex requires a running vault. I’m sure there are ways to set up the environment that such a thing could be jiggered but is there a native way for helmfile to do this type of functionality?

I’ll be back in about an hour

Jim Conner avatar
Jim Conner

also, quick question about a thought on a feature request which may already exist. It would be super awesome if I could run an argument that would just show me the charts which would be affected by an operation on requested selector(s).

mumoshu avatar
mumoshu

Perhaps that might be something like a “dry-run” feature?

https://github.com/roboll/helmfile/issues/118

Feature Request: Add `--dry-run` option · Issue #118 · roboll/helmfile

what Add option to process helmfile.yaml, but not execute helm why This would be helmfile for development use-case We&#39;re working on a master helmfile, kind of like a &quot;distribution&quot; of…

mumoshu avatar
mumoshu

Or if you’re willing to just review which releases are eing selected by a specific selector, i think helmfile -l foo=bar list works, as unlike helm list, helmfile list is able to list not-yet-installed releases

mumoshu avatar
mumoshu

Have you tried helmfile --interactive apply?

It stops after printing the list of releases to be deleted and updated, and prompt you for confirmation (y/n).

What if helmfile apply --dry-run worked exactly like that and automatically exist without prompting?

Jim Conner avatar
Jim Conner

list works great!

Jim Conner avatar
Jim Conner

I haven’t tried –interactive yet. I’ll give it a go…

Jim Conner avatar
Jim Conner

as for --dry-run, I am not sure (yet) of how helm asserts –dry-run. I would suppose that it does an actual dry-run against the cluster (like kubectl apply --dry-run=(client|server) does (depending)). I would have to research how kubectl asserts dry-runs first and how that correlates to how helm asserts dry runs before intelligently answering the last question. Basically, when I do tell something to do dry-run, I’m assuming I’m performing several actions which will test the actual application state I’m requesting all the way to the end without actually committing anything to k8s.

Jim Conner avatar
Jim Conner

bbiab

Jim Conner avatar
Jim Conner

cool. nm. don’t have to go now.

2021-04-25

mumoshu avatar
mumoshu

I’ve curated a list of important feature requests and planned changes https://github.com/roboll/helmfile/issues/1806 Please feel free to review and add your voices if you find anything interesting to you

Important feature requests and planned changes to be discussed · Issue #1806 · roboll/helmfile attachment image

This is going to be pinned to the header of our GitHub issues list so that everyone can be aware of and redirected to important planned features for discussion :) Allow opting in for inheriting all…

6
fiesta_parrot1

2021-04-24

Jim Conner avatar
Jim Conner

@mumoshu I hope that helps more (the example I just gave in the issue. I’m heading to bed now… Note that the example manifests from upstream (vault) show no single tics around the property. I added those for testing. With or without, it seems to be valid yaml. You can apply it successfully with kubectl but helmfile doesn’t process it right. Thanks for your help.

mumoshu avatar
mumoshu

I can’t even imagine how i could use that yaml in combination with vault-operator! You’d need to provide a step-by-step guide and your original goal.

mumoshu avatar
mumoshu


You can apply it successfully with kubectl but helmfile doesn’t process it right. Thanks for your help.
Why are you comparsing kubectl and helmfile? Should you compare helm and helmfile…?

mumoshu avatar
mumoshu

Is vaultconfig you’ve shown in https://github.com/roboll/helmfile/issues/1798#issuecomment-826049375 is considered a valid k8s manifest yaml that can be kubectl-applyed?

mumoshu avatar
mumoshu

But it doesn’t look so. I’m super confused

mumoshu avatar
mumoshu

Im not even sure why you’re correlating

 56:         externalConfig:
 57:             policies:
 58:             -   name: allow_secrets
 59:                 rules: 'path "secret/*" { capabilities = ["create", "read", "update", "delete", "list"] }'. <<== fails
 60:             auth:
 61:            -   type: kubernetes
...

with

err: failed to read helmfile.yaml: reading document at index 1: yaml: line 68: did not find expected ',' or ']'
in ./helmfile.yaml: failed to read helmfile.yaml: reading document at index 1: yaml: line 68: did not find expected ',' or ']'

the line number doesn’t match

mumoshu avatar
mumoshu
vaultconfig: |-                                                                                                                                                                                                                                                                                                                                          
    kind: Vault                                                                                                                                                                                                                                                                                                                                          
      name: vault                                                                                                                                                                                                                                                                                                                                        
      namespace: vault     

What’s the extra indentation before name and namespace here…?

Shouldn’t that be

vaultconfig: |-                                                                                                                                                                                                                                                                                                                                          
    kind: Vault                                                                                                                                                                                                                                                                                                                                          
    name: vault                                                                                                                                                                                                                                                                                                                                        
    namespace: vault     
mumoshu avatar
mumoshu

I’m not sure how this vaultconfig is supposed to be used. It isn’t look like a valid values accepted by vault-operator chart nor a valid K8s manfist that can be applied.

Are you using it to dynamically render helmfile.yaml?

But then it should be a yaml dict, not a yaml string.

if it’s supposed to be a yaml dict, you should’nt have ` - ` in:
vaultconfig: |-                                                                                                                                                                                                                                                                                                                                          
    kind: Vault                                                                                                                                                                                                                                                                                                                                          
      name: vault                                                                                                                                                                                                                                                                                                                                        
      namespace: vault                                                                                                                                                                                                                                                                                                                                   
    spec:

it should be

vaultconfig:
    kind: Vault                                                                                                                                                                                                                                                                                                                                          
    name: vault                                                                                                                                                                                                                                                                                                                                        
    namespace: vault                                                                                                                                                                                                                                                                                                                                   
    spec:
mumoshu avatar
mumoshu

You say your example is enough for reproduction. This is so tough…. I tried my best but have no idea what you’re trying to achieve, what you tried, why/how this is supposed to work, etc.

mumoshu avatar
mumoshu

Strange indentation here

      externalConfig:
          policies:
          -   name: allow_secrets
              rules: 'path "secret/*" { capabilities = ["create", "read", "update", "delete", "list"] }'
          auth:
         -   type: kubernetes
              roles:
              -   name: default
                  bound_service_account_names:
          auth:
         -   type: kubernetes
              roles:

should be

          auth:
          -   type: kubernetes
              roles:
mumoshu avatar
mumoshu

same here

          auth:
          -   type: kubernetes
              roles:
              -   name: default
                  bound_service_account_names:
         secrets:
          -   path: secret

secrets: should be indented one more

mumoshu avatar
mumoshu

seriously, you shouldn’t embed a complete YAML in a YAML string. That can obfuscate all kinds of yaml errors, I believe.

Jim Conner avatar
Jim Conner

OK. Gotta a lot here I have to address.

Jim Conner avatar
Jim Conner
  1. I somehow managed to miss metadata in my example in the issue. Fixed.
  2. The CR I had in my repository was wrong, as well. It was missing the apiVersion : Fixed.
Jim Conner avatar
Jim Conner

The way this installation works is that the vault-operator is installed via helm and then this manifest, which is the vault configuration (kind: Vault) is installed. The vault operator picks up the requested config and asserts it for vault.

Jim Conner avatar
Jim Conner

an example of how this works can be seen in this comment

Inject secrets in block notation data within k8s secrets/configmap resources · Issue #1270 · banzaicloud/bank-vaults attachment image

Is your feature request related to a problem? Please describe. current use-case involves attempting to install kube-prometheus-stack/alertmanager, which contains three directives we want to store/r…

Jim Conner avatar
Jim Conner

I’ve never heard of a problem embedding yaml in yaml given placing the yaml in an appropriate property using an appropriate yaml modifier. It is common practice in every place I’ve worked so I’ve never heard of anyone say not to do it and I’ve been doing this a loooong time. So, meh, not sure. Agree to disagree on that? =]

There are a couple of important points to make about the indentation issues you pointed out:

  1. it was from copy and paste so something between the copy, paste, and gh markdown might have borked the actual indentation.
  2. The pasted yaml is coming from sops decrypted data and sops re-arranges yaml from the original input to its own from the time a document is first encrypted. For instance, notice the - key: entries? I never write yaml where there is more than one space after the dash. SOPS does that.
Jim Conner avatar
Jim Conner

I’ll go through and fix yaml indentations issues I find. I was in a hurry to get to bed. It was late and I was keeping my wife up be staying up.

Jim Conner avatar
Jim Conner

one final thing. The yaml config I gave you was not a values.yaml file. I guess I should have just called it a CR manifest. I probably worded that in a weird way where confusion ensued. I provided the values.yaml example in the beginning to point out the property pointed out was sanely written as that was the only example, at the time, that I had and it was from a prominent example; upstream. So, that’s all I was using the values file for. We don’t use that values manifest. Moreover, given the sensitivity of what’s in this repo, I can’t just give you a copy. I have to scrub everything. So, the best example I can give you now on how to reproduce this is that GH comment #1270 I gave you, which is another issue I’m working with upstream on to figure out a problem we’re seeing with vault secrets webhook.

Jim Conner avatar
Jim Conner

I just tried using your fixed manifest from GH and it still fails for me so there might be a different issue. Lemme show you my helmfile.yaml

---
bases:
  - ../../common/repos.yaml
  - ../../common/helmdefaults.yaml
  - ../../common/versions.yaml
{{- if not .Values }}
  - ../../common/values.yaml.gotmpl
{{- end }}

---
environments:
  default:
    secrets:
      - secrets.yaml

---
releases:
  - name: vault-operator
    namespace: {{ .Values.vaultoperator.namespace }}
    createNamespace: true
    labels:
      app: vault-operator
    chart: {{ .Values.vaultoperator.repo }}
    version: {{ .Values.vaultoperator.version }}
    hooks:
      - events: ["postinstall"]
        command: "bash"
        args: ["-c","echo \"{{ .Values.vaultconfig }}\"", "kubectl -n {{ .Values.vaultoperator.namespace }} -f -"]
        showlogs: true
    values:
      - values.yaml

The CR I’m trying to load is the secrets.yaml Now, I’m trying to test the postinstall hook, but yaml isn’t even getting that far yet.

When I run helmfile template I get the following:

Jim Conner avatar
Jim Conner
Jim Conner avatar
Jim Conner

and that’s using the fixed manifest you provided with the fixed missing entries I found.

Jim Conner avatar
Jim Conner

ah geez. I just thought of something. I don’t need to load this into a value at all. Sigh. I’ll just use the postinstall hook and call sops to decrypt pipe it into kubectl directly. It would have been nice just use the internal secrets processing but this way I can get around this bug.

mumoshu avatar
mumoshu
        args: ["-c","echo \"{{ .Values.vaultconfig }}\"", "kubectl -n {{ .Values.vaultoperator.namespace }} -f -"]

This looks bad. You have " in vaultconfig so the second arg in args is delimited there, which seems unexpected to you

Jim Conner avatar
Jim Conner

yeah, I just refactored all of that. I wasn’t sure it would work or not

mumoshu avatar
mumoshu
you seem to be trying to read stdin here "kubectl -n {{ .Values.vaultoperator.namespace }} -f -" but you you’re missing | between echo and kubectl
Jim Conner avatar
Jim Conner

yeah…I don’t expect a pipe would work in the args property anyway, which I was using bash-fu to get around, which didn’t work…plus I neglected to even use apply to kubectl, but like I said, I was trying to get to a point to test that.

Jim Conner avatar
Jim Conner

but I’ve switched gears and not even going to use helmfile/helm/secrets et al

Jim Conner avatar
Jim Conner

since I’m applying a cr, which I should have thought about this before. I’m just gonna use sops in a hook and decrypt into kubectl. I’m writing a small wrapper shell function to handle all the things

mumoshu avatar
mumoshu

fyi a common way to install arbitrary k8s resources using helmfile is to use incubator/raw chart

releases:
 - name: resources
   namespace: vault
   createNamespace: true
  chart: incubator/raw
  values:
  - resources:
    - {{ .Values.vaultconfig | nindent4 }}
1
1
Jim Conner avatar
Jim Conner

btw, I’m not sure how well you know bash stuffs…but the method of input I was going for was: $ < file command which a stdin redirect into command…it’s gooder

mumoshu avatar
mumoshu

to let it not fail on first install you’d definitely want to have disableValidation too

releases:
 - name: resources
   namespace: vault
   createNamespace: true
  chart: incubator/raw
  values:
  - resources:
    - {{ .Values.vaultconfig | nindent4 }}
  disableValidation: true
Jim Conner avatar
Jim Conner

I’ve never heard of the raw chart. Very interesting

mumoshu avatar
mumoshu

to make it installed only after youve installed vault-operator, add needs

releases:
 - name: resources
   namespace: vault
   createNamespace: true
  chart: incubator/raw
  values:
  - resources:
    - {{ .Values.vaultconfig | nindent4 }}
  needs:
  - {{ .Values.vaultoperator.namespace }}/vault-operator
Jim Conner avatar
Jim Conner

I’ll give that a try but first I’d need helmfile to successfully read the cr

mumoshu avatar
mumoshu

yep. but that should be easy as long as you won’t break yaml somehow.

Jim Conner avatar
Jim Conner

so but that’s the problem I was trying to surface. I wasn’t breaking any yaml though things didn’t get into the issue very nicely…so that caused problems for you. I was in a rush so I fixed all that this morning.

mumoshu avatar
mumoshu
        args: ["-c","echo \"{{ .Values.vaultconfig }}\"", "kubectl -n {{ .Values.vaultoperator.namespace }} -f -"]

This looks bad. You have " in vaultconfig so the second arg in args is delimited there, which seems unexpected to you

Jim Conner avatar
Jim Conner

well, I already refactored all of it, but lemme run a test real fast with your suggestion.

Jim Conner avatar
Jim Conner

gimme like 5 mins.

mumoshu avatar
mumoshu


I’ve never heard of a problem embedding yaml in yaml given placing the yaml in an appropriate property using an appropriate yaml modifier. It is common practice in every place I’ve worked so I’ve never heard of anyone say not to do it and I’ve been doing this a loooong time. So, meh, not sure. Agree to disagree on that? =]
well sorry but what i wanted to say is that you should really avoid using go template and yaml in such way if it makes debugging harder for you.

it did made debugging harder for you, right?

mumoshu avatar
mumoshu

cant’ you just create a your own local helm chart dedicated to installing Vault resources, so that you don’t even need templating the whole helmfile.yaml?

Jim Conner avatar
Jim Conner

so, there are lotsa ways to skin this cat in my estimation…I’m trying to employ the easiest method I find that works.

Jim Conner avatar
Jim Conner

mumoshu avatar
mumoshu

it looked like the easiest, but apparently not, right? :)

Jim Conner avatar
Jim Conner

apparently

Jim Conner avatar
Jim Conner

ok

mumoshu avatar
mumoshu


I’ve never heard of a problem embedding yaml in yaml given placing the yaml in an appropriate property using an appropriate yaml modifier
maybe you were thinking about embedding yaml in yaml without go templating here, right?

Jim Conner avatar
Jim Conner

so this is the test I just ran

Jim Conner avatar
Jim Conner
---
bases:
  - ../../common/repos.yaml
  - ../../common/helmdefaults.yaml
  - ../../common/versions.yaml
{{- if not .Values }}
  - ../../common/values.yaml.gotmpl
{{- end }}

---
environments:
  default:
    secrets:
      - secrets.yaml

---
releases:
  - name: vault-operator
    namespace: {{ .Values.vaultoperator.namespace }}
    createNamespace: true
    labels:
      app: vault-operator
    chart: {{ .Values.vaultoperator.repo }}
    version: {{ .Values.vaultoperator.version }}
    hooks:
      - events: ["postinstall"]
        command: "echo "
        #args: ["-c","cat<<E {{ .Values.vaultconfig }}\nE", "|", "kubectl -n {{ .Values.vaultoperator.namespace }} apply -f -"]
        args: "{{ .Values.vaultconfig }}"
        showlogs: true
    values:
      - values.yaml
Jim Conner avatar
Jim Conner

is there anything wrong with that so far?

mumoshu avatar
mumoshu

whats in values.yaml now?

mumoshu avatar
mumoshu

yes, you need to remove

#args: ["-c","cat<<E {{ .Values.vaultconfig }}\nE", "|", "kubectl -n {{ .Values.vaultoperator.namespace }} apply -f -"]

` at all

Jim Conner avatar
Jim Conner

values.yaml or secrets.yaml (secrets.yaml is what’s getting read)

Jim Conner avatar
Jim Conner

ok, I’ll remove it

mumoshu avatar
mumoshu
#args: ["-c","cat<<E {{ .Values.vaultconfig }}\nE", "|", "kubectl -n {{ .Values.vaultoperator.namespace }} apply -f -"]

is clearly breaking your yaml because # will be applied to the first line of your loooong yaml string vaultconfig

Jim Conner avatar
Jim Conner

oh yeah…actually that was another thing I noticed about helmfile…comments are not always handled very nicely…but I might just misunderstand something there

mumoshu avatar
mumoshu

you’re somehow treating go template expression can be commeted out with yaml’s # but that’s not the case

mumoshu avatar
mumoshu

you need to be extra sure that you’re writing a go template to generate a valid yaml yourself.

mumoshu avatar
mumoshu

# is a yaml directive so it won’t disable go template in the line

Jim Conner avatar
Jim Conner

hmm. that’s actually new to me. Interesting. I’ll have to look that up.

Jim Conner avatar
Jim Conner

OK…

Jim Conner avatar
Jim Conner

here’s the test I just ran…

Jim Conner avatar
Jim Conner

in an encrypted secrets.yaml:

Jim Conner avatar
Jim Conner

and here’s the helmfile.yaml again:

---
bases:
  - ../../common/repos.yaml
  - ../../common/helmdefaults.yaml
  - ../../common/versions.yaml
{{- if not .Values }}
  - ../../common/values.yaml.gotmpl
{{- end }}

---
environments:
  default:
    secrets:
      - secrets.yaml

---
releases:
  - name: vault-operator
    namespace: {{ .Values.vaultoperator.namespace }}
    createNamespace: true
    labels:
      app: vault-operator
    chart: {{ .Values.vaultoperator.repo }}
    version: {{ .Values.vaultoperator.version }}
    hooks:
      - events: ["postinstall"]
        command: "echo "
        args: "{{ .Values.vaultconfig }}"
        showlogs: true
    values:
      - values.yaml
Jim Conner avatar
Jim Conner

when I run this, I get the following error:

$ helmfile --debug template
...
...
...
err: failed to read helmfile.yaml: reading document at index 1: yaml: line 11: did not find expected key
in ./helmfile.yaml: failed to read helmfile.yaml: reading document at index 1: yaml: line 11: did not find expected key
mumoshu avatar
mumoshu

what’s in the line 11 of your rendered helmfile.yaml shown in the debug log?

mumoshu avatar
mumoshu

could you provide me the whole log?

Jim Conner avatar
Jim Conner

so what I’m seeing is happening is that the echo command is reading the yaml file and then it seems to be trying to assert it to release in the yaml:

this is confusing:

Jim Conner avatar
Jim Conner
mumoshu avatar
mumoshu

man…!

mumoshu avatar
mumoshu
    hooks:
      - events: ["postinstall"]
        command: "echo "
        args: "{{ .Values.vaultconfig }}"

this seems wrong

Jim Conner avatar
Jim Conner

yeah, lemme get you the whole log…

mumoshu avatar
mumoshu

shouldn’t it be

    hooks:
      - events: ["postinstall"]
        command: "echo "
        args:
        -|
          {{ .Values.vaultconfig | nindent 8 }}
Jim Conner avatar
Jim Conner

uhhh

mumoshu avatar
mumoshu

see this!!!

11:         args: "apiVersion: "someapi/endpoint"                                                                                                                                                                                                                                                                                                        
12: manifest:                             
Jim Conner avatar
Jim Conner

should it be? That goes against any convention of using a shell command with args I’ve ever seen…so this would be great to have documented.

Jim Conner avatar
Jim Conner

I’m just trying to get the example simply echo out the variable

mumoshu avatar
mumoshu

yes. but you’re breaking yaml there

Jim Conner avatar
Jim Conner

sorry. this is starting to get frustrating. I think there’s a fundamental lack of understanding of how this hook is supposed to work.

Jim Conner avatar
Jim Conner

It seems like using a hook with a bash command that “args” is not actually supposed to cause yaml interpretation

Jim Conner avatar
Jim Conner

I’m not trying to assert yaml in this example

mumoshu avatar
mumoshu

hmmm… well yes, you seem to have a fundamental misunderstanding

Jim Conner avatar
Jim Conner

I just want to see the yaml output…and I’m doing that with a bash eecho

Jim Conner avatar
Jim Conner

why is helmfile trying to assert that?

mumoshu avatar
mumoshu
"{{ .Values.vaultconfig }}"

doesn’t automatically escape the yaml embedded in valutconfig

mumoshu avatar
mumoshu

does that clarify it a bit?

Jim Conner avatar
Jim Conner

not really. unfortunately.

Jim Conner avatar
Jim Conner

maybe I can explain how this looks to the enduser

mumoshu avatar
mumoshu

hm. then try

        args: {{ .Values.vaultconfig | toYaml }}
Jim Conner avatar
Jim Conner
   hooks:
      - events: ["postinstall"]
        command: "echo "
        args: "{{ .Values.vaultconfig }}"
        showlogs: true
Jim Conner avatar
Jim Conner

this tells me the following

Jim Conner avatar
Jim Conner

there’s a hook event to be executed during postinstall

mumoshu avatar
mumoshu

well not really

Jim Conner avatar
Jim Conner

use the echo command and use the args {{ .Values.vaultconfig }} and echo that out…

Jim Conner avatar
Jim Conner

do nothing else

mumoshu avatar
mumoshu
   hooks:
      - events: ["postinstall"]
        command: "echo "
        args: "{{ .Values.vaultconfig }}"
        showlogs: true

tells you that you render this snippet of go template to generate a yaml

mumoshu avatar
mumoshu

and only after that helmfile’s able to read the hooks definition

Jim Conner avatar
Jim Conner


Helmfile triggers various events while it is running. Once events are triggered, associated hooks are executed, by running the command with args. The standard output of the command will be displayed if showlogs is set and it’s value is true.

Jim Conner avatar
Jim Conner

the documentation says nothing about rendering

mumoshu avatar
mumoshu

yeah true. but that only happens all the go template expressions are rendered

Jim Conner avatar
Jim Conner

it specifically says executed

mumoshu avatar
mumoshu

yeah true. this go template thing applies to helmfile, not only to hooks

mumoshu avatar
mumoshu

that’s why the doc doesn’t bother repeating all over the places….

mumoshu avatar
mumoshu

if you don’t use go template at all, you can treat it helmfie.yaml as a plain yaml file so there’s no issue at all

mumoshu avatar
mumoshu

and i suppose that might be the fundamental misunderstanding you had

Jim Conner avatar
Jim Conner

so…

Jim Conner avatar
Jim Conner

the use of command: to me indicates some kind of terminal based command can be executed

Jim Conner avatar
Jim Conner

and args: indicates arguments which will get passed to command

mumoshu avatar
mumoshu

if helmfile reads yaml before rendering go template, even this doesn’t work

---
bases:
  - ../../common/repos.yaml
  - ../../common/helmdefaults.yaml
  - ../../common/versions.yaml
{{- if not .Values }}
  - ../../common/values.yaml.gotmpl
{{- end }}
mumoshu avatar
mumoshu

because

{{- if not .Values }}
  - ../../common/values.yaml.gotmpl
{{- end }}

is clearly invalid as yaml

Jim Conner avatar
Jim Conner

yes

mumoshu avatar
mumoshu


the use of command: to me indicates some kind of terminal based command can be execute
and args: indicates arguments which will get passed to command
correct

mumoshu avatar
mumoshu

so all your understanding on hooks, command, args seem correct

mumoshu avatar
mumoshu

but you’re breaking yaml before they are defined at all.. which is what helmfile’s complaining about.

Jim Conner avatar
Jim Conner

well, ok, but lemme finish my thought real fast

Jim Conner avatar
Jim Conner

so, it seems that helmfile should only be looking for an exit code from the command…what ever else the command performs should be interpreted by helmfile. I would think.

Jim Conner avatar
Jim Conner

should reading {{ .Values.vaultconfig }} just be the entire yaml output from the file? In it’s own right, it’s valid yaml

Jim Conner avatar
Jim Conner

but helmfile seems to be concatenating it to everything else

mumoshu avatar
mumoshu

yep, because that’s how go template works…

Jim Conner avatar
Jim Conner

maybe

mumoshu avatar
mumoshu

well internally helmfile does differentiate each arg in args.

mumoshu avatar
mumoshu

so
but helmfile seems to be concatenating it to everything else
if you’re referring to how go template is working here, you’re correct

Jim Conner avatar
Jim Conner

so

mumoshu avatar
mumoshu

i can imagine a valid feature request here would be

mumoshu avatar
mumoshu

somehow enhancing args so that you can source a file content into one arg

mumoshu avatar
mumoshu

or a helmfile value content into one arg

mumoshu avatar
mumoshu

like

   hooks:
      - events: ["postinstall"]
        command: "echo "
        args:
        - fromHelmfileValue: vaultconig
        showlogs: true
Jim Conner avatar
Jim Conner
mumoshu avatar
mumoshu

so you’re failing here…

11:         args: "     <<<< ==== everything in the template here should be passed to bash and ignored by helmfile
12:         apiVersion: "someapi/endpoint"
Jim Conner avatar
Jim Conner

The yaml in the file, which is read into .Values.vaultconfig by itself is valid yaml.

mumoshu avatar
mumoshu

yeah but you arent embedding it correctly

Jim Conner avatar
Jim Conner

I’m not trying to embed it!!! lol. That’s the thing. I don’t understand why helmfile is trying to embed that to args in releases — that’s not making sense to me.

mumoshu avatar
mumoshu

why not use

args :
- |
  {{ .Values.vaultconfig | nident SOME_NUMBER

as i’ve suggested

Jim Conner avatar
Jim Conner

so perhaps you can help me understand that one part.

Jim Conner avatar
Jim Conner

I’m trying to find that magic number…just a sec

mumoshu avatar
mumoshu


I don’t understand why helmfile is trying to embed that to args in releases
Becase you’re telling helmfile to do so here…
args: “{{ .Values.vaultconfig }}”

Jim Conner avatar
Jim Conner

this is with nindent 10

Jim Conner avatar
Jim Conner

but I’m not!! I’m telling helmfile to pass the transliterated template to echo

Jim Conner avatar
Jim Conner

or at least that’s what I’m trying to do

Jim Conner avatar
Jim Conner

I don’t want helmfile to attach anything in args to the releases object.

mumoshu avatar
mumoshu

are you really using:

args:
- |
  {{ .Values.vaultconfig | nindent SOME_NUMBER }}

in the above run?

Jim Conner avatar
Jim Conner

I just want it to pass the “stuff” in args to echo

Jim Conner avatar
Jim Conner

nope.

mumoshu avatar
mumoshu

then it won’t work

Jim Conner avatar
Jim Conner
  1 ---                                                                                                                                                                                                                                                                                                                                                  
  2 bases:                                                                                                                                                                                                                                                                                                                                               
  3   - ../../common/repos.yaml                                                                                                                                                                                                                                                                                                                          
  4   - ../../common/helmdefaults.yaml                                                                                                                                                                                                                                                                                                                   
  5   - ../../common/versions.yaml                                                                                                                                                                                                                                                                                                                       
  6 {{- if not .Values }}                                                                                                                                                                                                                                                                                                                                
  7   - ../../common/values.yaml.gotmpl                                                                                                                                                                                                                                                                                                                  
  8 {{- end }}                                                                                                                                                                                                                                                                                                                                           
  9                                                                                                                                                                                                                                                                                                                                                      
 10 ---                                                                                                                                                                                                                                                                                                                                                  
 11 environments:                                                                                                                                                                                                                                                                                                                                        
 12   default:                                                                                                                                                                                                                                                                                                                                           
 13   | secrets:                                                                                                                                                                                                                                                                                                                                         
 14   | | - secrets.yaml                                                                                                                                                                                                                                                                                                                                 
 15                                                                                                                                                                                                                                                                                                                                                      
 16 ---                                                                                                                                                                                                                                                                                                                                                  
 17 releases:                                                                                                                                                                                                                                                                                                                                            
 18   - name: vault-operator                                                                                                                                                                                                                                                                                                                             
 19   | namespace: {{ .Values.vaultoperator.namespace }}                                                                                                                                                                                                                                                                                                 
 20   | createNamespace: true                                                                                                                                                                                                                                                                                                                            
 21   | labels:                                                                                                                                                                                                                                                                                                                                          
 22   | | app: vault-operator                                                                                                                                                                                                                                                                                                                            
 23   | chart: {{ .Values.vaultoperator.repo }}                                                                                                                                                                                                                                                                                                          
 24   | version: {{ .Values.vaultoperator.version }}                                                                                                                                                                                                                                                                                                     
 25   | hooks:                                                                                                                                                                                                                                                                                                                                           
 26   | | - events: ["postinstall"]                                                                                                                                                                                                                                                                                                                      
 27   | | | command: "echo "                                                                                                                                                                                                                                                                                                                             
 28   | | | args: "{{ .Values.vaultconfig | nindent 10 }}"                                                                                                                                                                                                                                                                                               
 29   | | | showlogs: true                                                                                                                                                                                                                                                                                                                               
 30   | values:                                                                                                                                                                                                                                                                                                                                          
 31   | | - values.yaml 
Jim Conner avatar
Jim Conner

ok, so maybe let me see if understand how helmfile works then in this regard…

mumoshu avatar
mumoshu
hooks:                                                                                                                                                                                                                                                                                                                                           
- events: ["postinstall"]                                                                                                                                                                                                                                                                                                                      
 command: "echo "                                                                                                                                                                                                                                                                                                                             
  args:
  - |
    {{ .Values.vaultconfig | nindent SOME_NUMBER }} 

try this

Jim Conner avatar
Jim Conner
Jim Conner avatar
Jim Conner
    hooks:
      - events: ["postinstall"]
        command: "echo "
        args:
          - {{ .Values.vaultconfig | nindent 12 }}
        showlogs: true
mumoshu avatar
mumoshu

yeah it should end up like that..

mumoshu avatar
mumoshu

why you used

args:
- 

instead of what i suggested above:

args:
- |

?

Jim Conner avatar
Jim Conner

tried both.

Jim Conner avatar
Jim Conner

same error

Jim Conner avatar
Jim Conner
 9:       - events: ["postinstall"]
10:         command: "echo "
11:         args:
12:         - 
13:             apiVersion: "someapi/endpoint"
14:             manifest:
15:               kind: Vault
16:               name: vault
Jim Conner avatar
Jim Conner
err: failed to read helmfile.yaml: reading document at index 1: yaml: unmarshal errors:
  line 14: cannot unmarshal !!map into string
in ./helmfile.yaml: failed to read helmfile.yaml: reading document at index 1: yaml: unmarshal errors:
  line 14: cannot unmarshal !!map into string
Jim Conner avatar
Jim Conner

although, indentation is off in second one. Lemme fix that

Jim Conner avatar
Jim Conner
1:         args:
12:         - 
13:           apiVersion: "someapi/endpoint"
14:           manifest:
15:             kind: Vault
err: failed to read helmfile.yaml: reading document at index 1: yaml: unmarshal errors:
  line 14: cannot unmarshal !!map into string
in ./helmfile.yaml: failed to read helmfile.yaml: reading document at index 1: yaml: unmarshal errors:
  line 14: cannot unmarshal !!map into string
Jim Conner avatar
Jim Conner
 25   | hooks:                                                                                                                                                                                                                                                                                                                                           
 26   | | - events: ["postinstall"]                                                                                                                                                                                                                                                                                                                      
 27   | | | command: "echo "                                                                                                                                                                                                                                                                                                                             
 28   | | | args:                                                                                                                                                                                                                                                                                                                                        
 29   | | | - {{ .Values.vaultconfig | nindent 10 }}                                                                                                                                                                                                                                                                                                     
 30   | | | showlogs: true                                                                                                                                                                                                                                                                                                                               
 31   | values:                                                                                                                                                                                                                                                                                                                                          
 32   | | - values.yaml 
mumoshu avatar
mumoshu

where”s the - | now? mind sharing me the whole helmfile.yaml after you’ve wrote - | ?

Jim Conner avatar
Jim Conner
---
bases:
  - ../../common/repos.yaml
  - ../../common/helmdefaults.yaml
  - ../../common/versions.yaml
{{- if not .Values }}
  - ../../common/values.yaml.gotmpl
{{- end }}

---
environments:
  default:
    secrets:
      - secrets.yaml

---
releases:
  - name: vault-operator
    namespace: {{ .Values.vaultoperator.namespace }}
    createNamespace: true
    labels:
      app: vault-operator
    chart: {{ .Values.vaultoperator.repo }}
    version: {{ .Values.vaultoperator.version }}
    hooks:
      - events: ["postinstall"]
        command: "echo "
        args:
        - {{ .Values.vaultconfig | nindent 10 }}
        showlogs: true
    values:
      - values.yaml
mumoshu avatar
mumoshu

wheres - |?

Jim Conner avatar
Jim Conner
hooks:                                                                                                                                                                                                                                                                                                                                           
- events: ["postinstall"]                                                                                                                                                                                                                                                                                                                      
 command: "echo "                                                                                                                                                                                                                                                                                                                             
  args:
  - {{ .Values.vaultconfig | nindent SOME_NUMBER }} 

try this

Jim Conner avatar
Jim Conner

did I miss a |-?

Jim Conner avatar
Jim Conner

should that be after args:?

mumoshu avatar
mumoshu

yes

mumoshu avatar
mumoshu

ah no

mumoshu avatar
mumoshu

well, do this

mumoshu avatar
mumoshu
  args:
  - |
    {{ .Values.vaultconfig | nindent SOME_NUMBER }} 
Jim Conner avatar
Jim Conner

k, sec

Jim Conner avatar
Jim Conner

ok. so no errors that time…

Jim Conner avatar
Jim Conner

checking through all the output.

mumoshu avatar
mumoshu

btw this might also work

args: ["{{`{{.Values.vaultconfig}}`}}"]

probably you’ve already read https://github.com/roboll/helmfile/#hooks but totally missed the existence of “go template comment” there?

Jim Conner avatar
Jim Conner

yeah, I was considering trying that next actually

Jim Conner avatar
Jim Conner

I haven’t read that actually

mumoshu avatar
mumoshu

really don’t forget making it a go template comment if you try the above

{{`

`}}
mumoshu avatar
mumoshu

ok

Jim Conner avatar
Jim Conner

I found an example helmfile.yaml that used that in a hook so I was going to try it

mumoshu avatar
mumoshu

regardless of you use go template comment or not, embed yaml into args at the helmfile.yaml level or before helmfile running the hook command, all i can say is https://sweetops.slack.com/archives/CE5NGCB9Q/p1619299403271500?thread_ts=1619249294.249600&cid=CE5NGCB9Q

   hooks:
      - events: ["postinstall"]
        command: "echo "
        args: "{{ .Values.vaultconfig }}"
        showlogs: true

tells you that you render this snippet of go template to generate a yaml

mumoshu avatar
mumoshu

helmfile firstly renders the whole helmfile.yaml content as a go template. helmfile reads the result as yaml

1
mumoshu avatar
mumoshu

you can use go template comment like https://sweetops.slack.com/archives/CE5NGCB9Q/p1619301420288500?thread_ts=1619249294.249600&cid=CE5NGCB9Q so that you can “defer” the go template rendering.

BUT it doesn’t mean that helmfile stopped rendering the go template.

really don’t forget making it a go template comment if you try the above

{{`

`}}
Jim Conner avatar
Jim Conner

OK, that’s good info there

mumoshu avatar
mumoshu

even if you had no go template expression in helmfile.yaml, helmfile tries to render it as a go template before parsing as yaml.

a valid yaml can easily become a valid go template too. so, as long as you don’t go template in helmfile.yaml at all, helmfile.yaml looks like a vanilla yaml

1
Jim Conner avatar
Jim Conner

So, I’m running this with helmfile sync --args --dry-run — will the hook work during a helm dry-run?

Jim Conner avatar
Jim Conner

I’m not getting errors now but I’m also not seeing the echo of the stuff in the file.

mumoshu avatar
mumoshu

haven’t tried. helmfile --args is not well supported

Jim Conner avatar
Jim Conner

meh, ok. Lemme just sync and see what happens

Jim Conner avatar
Jim Conner

nope. dang it.

Jim Conner avatar
Jim Conner

not even any logs

Jim Conner avatar
Jim Conner

wait a sec

mumoshu avatar
mumoshu

helmfile --args is a historical artifact that only worked in the very beginning of the helmfile project. at that time helmfile sync only ran a single helm command.

nowadays a helmfile run involves many helm commands so --args doesn’t make sense. Which custom args should be passed to helm template, helm template, helm diff, helm repo up, helm dep build, etc? Impossible to deduce

Jim Conner avatar
Jim Conner

oh boy. That’s not very good because sometimes I want to see what helm debug looks like and what it would template out

Jim Conner avatar
Jim Conner

that’s a valid point though

Jim Conner avatar
Jim Conner

seems like it should be a good idea to allow the user to pass dry-run to upgrade --install though.

mumoshu avatar
mumoshu

i agree. sounds like a good chance for you to write a feature request

Jim Conner avatar
Jim Conner

lol. can do

Jim Conner avatar
Jim Conner

so I’m confused here in what is happening in my testing here…

Jim Conner avatar
Jim Conner

no echo command from postinstall

Jim Conner avatar
Jim Conner

ah geez

mumoshu avatar
mumoshu

oh well, you said postinstall?

Jim Conner avatar
Jim Conner

yeah…don’t say it. I figured it out

Jim Conner avatar
Jim Conner

no hook called postinstall.

1
mumoshu avatar
mumoshu

i remember someone has requested it before… found it. https://github.com/roboll/helmfile/issues/1291#issuecomment-638153828

Quick question about hooks · Issue #1291 · roboll/helmfile attachment image

I am just wondering exactly when presync and postsync hooks are run, specifically: Are they run if a release is being uninstalled? Are they run if a release is already uninstalled and installed: fa…

Jim Conner avatar
Jim Conner

yeah, a postinstall for this use-case would be prime

Jim Conner avatar
Jim Conner

ok, giving this a shot now…

Jim Conner avatar
Jim Conner
27   | | | command: "echo"                                                                                                                                                                                                                                                                                                                              
 28   | | | args: ["{{ `{{ .Values.vaultconfig }}` }}","|","kubectl -n {{ `{{ .Vault.vaultoperator.namespace }}` }} apply -f -"]
mumoshu avatar
mumoshu

created an issue for that at #1805. please feel free to add your voice w/ expected use-case there https://github.com/roboll/helmfile/issues/1805

postinstall hook · Issue #1805 · roboll/helmfile

Just wanted to have a dedicated issue for this. Would anyone use it if Helmfile added a new postinstall hook? It has been originally proposed by @cdunford in #1291. Althought I thought it was great…

1
mumoshu avatar
mumoshu

Does adding space between {{ and the backtick like that work?

Jim Conner avatar
Jim Conner

yup. that’s how we do gotemplate comments for everything (it’s our convention)

mumoshu avatar
mumoshu

I thought go template comment had to start with

{{

`

mumoshu avatar
mumoshu

ok then

Jim Conner avatar
Jim Conner

just gotta fixup a couple things here and then this might work….

mumoshu avatar
mumoshu

hm, i think i’ve found another potential issue in your config

Jim Conner avatar
Jim Conner

oh, cool. I’m listening

mumoshu avatar
mumoshu

maybe yo’ve already noticed, but you should use bash or alike instead of directly calling echo

Jim Conner avatar
Jim Conner

yeah, just fixed that

mumoshu avatar
mumoshu

nice

Jim Conner avatar
Jim Conner
        command: "bash"
        args: ["-c", "echo", "{{ `{{ .Values.vaultconfig }}` }}","|","kubectl -n {{ `{{ .Values.vaultoperator.namespace }}` }} apply -f -"]

but I think the | is illegal unless helmfile knows how to handle that. In my experience, most things don’t properly interpret the pipe in this context

mumoshu avatar
mumoshu

yes, i can see your point

Jim Conner avatar
Jim Conner

output from sync

 Listing releases matching ^vault-operator$
vault-operator  vault           10              2021-04-24 15:21:08.377398548 -0700 PDT deployed        vault-operator-1.8.1    1.8.0      

helmfile.yaml: basePath=.

hook[postsync] logs | 
hook[postsync] logs | 

UPDATED RELEASES:
NAME             CHART                               VERSION
vault-operator   banzaicloud-stable/vault-operator     1.8.1
Jim Conner avatar
Jim Conner

but

Jim Conner avatar
Jim Conner
 at 15:21:36 ❯ kg vault vault
Error from server (NotFound): vaults.vault.banzaicloud.com "vault" not found
Jim Conner avatar
Jim Conner

so no errors, but no resource either

Jim Conner avatar
Jim Conner

it’s quick and dirty, but I can echo that out to a tmpfile and have kubectl read the tmp file, then shred it.

Jim Conner avatar
Jim Conner

that’s kind of what I was working on before actually

Jim Conner avatar
Jim Conner

actually…lemme try one more thing

mumoshu avatar
mumoshu

sounds good

mumoshu avatar
mumoshu

i have also managed to do it with bash here doc

mumoshu avatar
mumoshu
values:
- somePlainYaml: |
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: frompostsync
---

releases:
- name: myapp
  chart: incubator/raw
  values:
  - resources:
    - metadata:
        name: myapp
      kind: ConfigMap
      apiVersion: v1
  hooks:
  - events: ["postsync"]
    command: "bash"
    args:
    - -c
    - |
      cat <<EOS | kubectl apply -f -
      {{ .Values.somePlainYaml | nindent 6 }}
      EOS
Jim Conner avatar
Jim Conner

that’s what I was kinda working on…

Jim Conner avatar
Jim Conner

nicely done

1
Jim Conner avatar
Jim Conner

1
Jim Conner avatar
Jim Conner

sheez…forgot to remove the pipe

Jim Conner avatar
Jim Conner

nice

Jim Conner avatar
Jim Conner

it works

Jim Conner avatar
Jim Conner
     - events: ["postsync"]
        command: "bash"
        args:
          - "-c"
          - |-
            < <(echo -e "{{ `{{ .Values.vaultconfig | nindent 6 }}` }}") \
               kubectl --validate=false -n {{ `{{ .Values.vaultoperator.namespace }}` }} apply -f -
        showlogs: true
mumoshu avatar
mumoshu

awesome!

Jim Conner avatar
Jim Conner

man

Jim Conner avatar
Jim Conner

sorry for the roughness there

1
mumoshu avatar
mumoshu

i’d still recommend the use of bash here doc there

Jim Conner avatar
Jim Conner

can you articulate why?

mumoshu avatar
mumoshu

so that you can avoid a potential issue of echo part breaking due to some character in vaultconfig breaking the bash string

Jim Conner avatar
Jim Conner

Jim Conner avatar
Jim Conner

that’s pretty good reason

mumoshu avatar
mumoshu

it won’t break always. but i would still think it as a good practice to use bash here doc there

mumoshu avatar
mumoshu

that way all you need to care is that you don’t have any bash here doc delimiter in vaultconfig

mumoshu avatar
mumoshu

there might be other ways. its just that bash here doc is the best way i can think of

Jim Conner avatar
Jim Conner

in many ways, fd vivication is better but echo can be squirrely

Jim Conner avatar
Jim Conner

I wouldn’t use echo in fact, but the newlines are important and it seemed that newlines got swallowed up in a straight grok in my quick testing.

Jim Conner avatar
Jim Conner

originally I tested (cli) <($(< $(sops -d secrets.yaml))) kubectl -f -

cool-doge1
Jim Conner avatar
Jim Conner

er

Jim Conner avatar
Jim Conner

something like that. I’m trying to go by memory…I’m also switching contexts a lot here so trying to keep everything straight is iffy

Jim Conner avatar
Jim Conner

hey curious…what time is it there for you?

Jim Conner avatar
Jim Conner

I think you’re in Japan right? Or is that a poor assumption?

mumoshu avatar
mumoshu

Right. It was around 7am sunday

1
Jim Conner avatar
Jim Conner

gnight

1

2021-04-23

muhaha avatar
muhaha

guys, is there any replacement for deprecated inclubator/raw chart ?

roth.andy avatar
roth.andy

A few people have forked it and published their version on artifacthub, but none of them have gained community adoption. Personally I still feel fine using the deprecated chart from helm/charts/incubator since it is stable and doesn’t need many updates. It set out to accomplish this one thing people need, attained it, and now just sits, not really needing much maintenance

muhaha avatar
muhaha

That deprecation warning in latest version of chart breaks yaml..

muhaha avatar
muhaha

Of course, I can use older one, but that breaks other things for me

muhaha avatar
muhaha

Seems that bitnami has one..

muhaha avatar
muhaha

Even cloudposse, monochart..

Leia Renée avatar
Leia Renée

Hi guys , Do you have a sample repository which installs kubernetes cluster auto scaler that works with TF 15.0 properly I was using cookie labs which broken after upgrade. Thanks Leia https://www.linkedin.com/in/leia-renee/

2021-04-22

Andrey Nazarov avatar
Andrey Nazarov

Victor Farcic spread the word about Helmfile on his channel: https://www.youtube.com/watch?v=qIJt8Iq8Zb0

1

2021-04-21

Eugene Korekin avatar
Eugene Korekin

is there any way to ignore helm errors for a specific release and continue?

mumoshu avatar
mumoshu

generally no. but im curious why you want to do that?

mumoshu avatar
mumoshu

also, what’s the exact error you’re trying to ignore?

Eugene Korekin avatar
Eugene Korekin

so there is an issue I created a week ago https://github.com/roboll/helmfile/issues/1778

Failed to render chart with strategicMergePatches/jsonPatches and CRDs · Issue #1778 · roboll/helmfile attachment image

When using helmfile apply with repositories: - name: prometheus-community url: https://prometheus-community.github.io/helm-charts releases: - name: test namespace: test chart: prometheus-community/…

Eugene Korekin avatar
Eugene Korekin

and I am trying to workaround it

Eugene Korekin avatar
Eugene Korekin

we need to override some default rules from kube prometheus stack, my original idea was to use a patch, but since that didn’t work now I am trying another approach

unfortunately it involves a duplicate resource inside the kube prometheus release: I use additionalPrometheusRules with the name of already existing rule

Eugene Korekin avatar
Eugene Korekin

so helm fails when it sees the duplicate, I was under impression that everything still works fine after that but now I see that this actually happens in the middle of deployment, so some resources are missing after it

Eugene Korekin avatar
Eugene Korekin

so probably ignoring the error won’t be actually helpful in this case

Eugene Korekin avatar
Eugene Korekin

but I have another question now:

if a release fails then further apply attempts do not show any errors but also do not deploy missing resources - is that expected behaviour?

(that’s what led me to believe that everything is find after the error)

Jim Conner avatar
Jim Conner

one possible workaround is to create a hook in which you run a script that performs a helm template on the chart and then you kubectl apply -f - on the template output patching whatever is necessary on the broken bits on a pipe in between template and kubectl — essentially you’d just fix the manifests as they’re being templatized. It’s ugly and a total hack and might be bug prone…but it’s an idea maybe?

Eugene Korekin avatar
Eugene Korekin

thank you for the idea, Jim

mumoshu avatar
mumoshu

@Eugene Korekin I’ve tried to reproduce your issue but had no luck. please see https://github.com/roboll/helmfile/issues/1778#issuecomment-824441138

Failed to render chart with strategicMergePatches/jsonPatches and CRDs · Issue #1778 · roboll/helmfile attachment image

When using helmfile apply with repositories: - name: prometheus-community url: https://prometheus-community.github.io/helm-charts releases: - name: test namespace: test chart: prometheus-community/…

mumoshu avatar
mumoshu

perhaps your issue had already been fixed later prs?

mumoshu avatar
mumoshu

if you still have some issue, sharing the reproduction steps on a brand-new cluster would be helpful

Eugene Korekin avatar
Eugene Korekin

@mumoshu the behaviour is definitely different with the latest version from master

mumoshu avatar
mumoshu
Failed to render chart with strategicMergePatches/jsonPatches and CRDs · Issue #1778 · roboll/helmfile attachment image

When using helmfile apply with repositories: - name: prometheus-community url: https://prometheus-community.github.io/helm-charts releases: - name: test namespace: test chart: prometheus-community/…

mumoshu avatar
mumoshu

i have managed to let helmfile fail with another error and it does seem like a bug

mumoshu avatar
mumoshu

the one i saw seems happen when you do have strategicmergepatches/jsonpatches + CRDs

Eugene Korekin avatar
Eugene Korekin

right, but there is another issue

mumoshu avatar
mumoshu

whats it?

Eugene Korekin avatar
Eugene Korekin

looks like there is some regression in the master comparing to the stable version

Eugene Korekin avatar
Eugene Korekin

just a moment

Eugene Korekin avatar
Eugene Korekin

so here is a simple helmfile

repositories:
  - name: prometheus-community
    url: <https://prometheus-community.github.io/helm-charts>
releases:
  - name: test
    namespace: test
    chart: prometheus-community/kube-prometheus-stack
    version: ~14.4.0
Eugene Korekin avatar
Eugene Korekin

it works without any issue with the stable version please note that it works without disableValidation

Eugene Korekin avatar
Eugene Korekin

and I ensured that the crds weren’t present before the installation (deploying a new clean cluster right now to be 100% sure)

mumoshu avatar
mumoshu

whoa really? i cant imagine how it can work without disableValidation now…

Eugene Korekin avatar
Eugene Korekin

ok, that’s probably because I used --skip-diff-on-install

mumoshu avatar
mumoshu

ah that makes sense

mumoshu avatar
mumoshu

it’s me who added helm diff --disable-validation to avoid helm-diff failing when trying to diff custom resources for CRD that isn’t installed yet

Eugene Korekin avatar
Eugene Korekin

sure, I saw the PR, that was a great idea

cool-doge1
mumoshu avatar
mumoshu

so yeah, --skip-diff-on-install would make disableValidation unnecessary. got it

Eugene Korekin avatar
Eugene Korekin

btw, would it be good idea if helmfile from master will print something different as its version number, not the same string as the stable version?

Eugene Korekin avatar
Eugene Korekin

I am having issues trying to understand which version I use sometimes, cause --version gives the same results for stable and master

mumoshu avatar
mumoshu

ah sounds good. i thought the version number if set via TAG = $(shell git describe --tags --abbrev=0 HEAD) in Makefile

mumoshu avatar
mumoshu

i was innocently believing that it would return any tag only on tagged commit but apparently not

mumoshu avatar
mumoshu

Ah okay i see
git-describe
The command finds the most recent tag that is reachable from a commit

Eugene Korekin avatar
Eugene Korekin


looks like there is some regression in the master comparing to the stable version
ok, please disregard this I just checked and they both work in the same way

mumoshu avatar
mumoshu

great. thanks for testing!

Eugene Korekin avatar
Eugene Korekin

so, do you need any other information from my side regarding this issue with jsonpatches and crds?

mumoshu avatar
mumoshu

i believe you’ve provided me enough! thx. i’m now wondering how i could fix that

mumoshu avatar
mumoshu

seems helmfile needs to extract CRDs from the patched YAML and put it under crds dir in the temp chart…. shouldn’t there be any easier way than this

Eugene Korekin avatar
Eugene Korekin

thank you very much

if it won’t be possible to fix, I think specifying that json and strategic merge patches don’t work with CRDs in documentation would be very useful

mumoshu avatar
mumoshu

i cant help wishing there was helm template --crds-only

mumoshu avatar
mumoshu

yep true

Jim Conner avatar
Jim Conner

Matt Farina is a friend of mine. I can suggest that to him.

mumoshu avatar
mumoshu

i would generally prefer putting crds in a separate chart and then you’re fine

Jim Conner avatar
Jim Conner

I used to work with him.

mumoshu avatar
mumoshu

cool!

mumoshu avatar
mumoshu

@Eugene Korekin another potential solution would be to enhance helmfile to accept skipCRDs: true in helmfile.yaml.

Then you could have a separate release for CRDs only and another for kube-promehtheus-stack with skipCRDs: true

mumoshu avatar
mumoshu

where in the skipCRDsed kube-promehtue-stack release, you can freely use patching

Eugene Korekin avatar
Eugene Korekin

that separate release for CRDs, should it be created manually in that case?

Eugene Korekin avatar
Eugene Korekin

or is there some automatic way to just install CRDs (from kube-prometheus-stack for example)

mumoshu avatar
mumoshu

you can defintiely create em manually, but can i ask you why?

mumoshu avatar
mumoshu

nope. so i mean you need to copy CRDs from the kube-prometheus-stack into your own local chart

Eugene Korekin avatar
Eugene Korekin

ok, I see yes, that’s what I meant by doing that manually

mumoshu avatar
mumoshu

ah ok

mumoshu avatar
mumoshu

well just a sec

Eugene Korekin avatar
Eugene Korekin

as opposed to providing some option to helmfile so only CRDs will be installed from a chart

mumoshu avatar
mumoshu

can’t we use our go-getter integration to grab and install CRDs only, assuming kube-promehtue-stack is hosted on GitHub?

Eugene Korekin avatar
Eugene Korekin

that would be great

Eugene Korekin avatar
Eugene Korekin

but couldn’t CRDs also use some templating like other helm chart resources?

mumoshu avatar
mumoshu

maybe no..? i have never heard of CRDs being dynamically generated in practice

Eugene Korekin avatar
Eugene Korekin

I mean, it might be easy to get and install them or manually copy them from a chart if they are just static, but what if not?

mumoshu avatar
mumoshu

maybe. it all depends on what we want? if you’d like to avoid manual steps using go-getter integration to install CRDs only in a separate chart can be a good workaround

mumoshu avatar
mumoshu

otherwise just create a local chart containing CRDs only manually, or even install CRDs before running helmfile make sense

Eugene Korekin avatar
Eugene Korekin

sure, it will work if they are always static and having this integration would be helpful because they could change with a new version of a chart

mumoshu avatar
mumoshu

right

Eugene Korekin avatar
Eugene Korekin

I was just thinking about an edge case when they are dynamic, in that case the go-getter integration will break, right?

Eugene Korekin avatar
Eugene Korekin

but I have little understanding about how they are usually used in charts, so probably I am just missing something, nevermind

Eugene Korekin avatar
Eugene Korekin

I will try to use the workaround you suggested for now, thanks a lot

1
mumoshu avatar
mumoshu


I was just thinking about an edge case when they are dynamic, in that case the go-getter integration will break, right?
absolutely.

Eugene Korekin avatar
Eugene Korekin

just an additional thought: if the root of this issue is in chartify behaviour, maybe the proper way to fix it would be on the chartify level?

Eugene Korekin avatar
Eugene Korekin

I don’t know what chartify is though and how it relates to helm

Eugene Korekin avatar
Eugene Korekin

but if it generates helm charts then this:

the temporary chart generated by chartify is not a correct helm3 chart

looks like a bug in it, right?

mumoshu avatar
mumoshu

yes. i consider it’s a bug in chartify and i’m trying to fix it there. the implementation is going to be tedious so that’s why i wished if there was helm template --only-crds earlier in this thread

Eugene Korekin avatar
Eugene Korekin

ah, got it

mumoshu avatar
mumoshu
Bump chartify to 0.8.2 by mumoshu · Pull Request #1793 · roboll/helmfile attachment image

This version fixes that charitfy not to fail when you used the combination of (1)helm 3 and (2)strategicMergePatches/jsonPatches/transformers etc that triggers chartify on (3)a chart that contains …

2021-04-20

Jim Conner avatar
Jim Conner

@mumoshu we can continue the thread here

1
Jim Conner avatar
Jim Conner

here goes

Jim Conner avatar
Jim Conner

@jedineeper if you’re interested

1
Jim Conner avatar
Jim Conner

@mumoshu I’ve pushed all my changes; and stuff is broke but it’s like almost 430 AM here and I need to get to bed.

1
Jim Conner avatar
Jim Conner

if you don’t mind though, give it a looksee. I’ll see if I can make heads or tails of it tomorrow

Jim Conner avatar
Jim Conner

er I guess…today is more accurate.

1
mumoshu avatar
mumoshu

will try to find some time!

Jim Conner avatar
Jim Conner

I’m not thinking straight though, anymore. So, again, really appreciate all your help.

1
Jim Conner avatar
Jim Conner

thank you very much! I’ll chat you up later today/tomorrow your time.

Jim Conner avatar
Jim Conner

g’night

mumoshu avatar
mumoshu

thanks for trying helmfile! gnight

1
Alex Genco avatar
Alex Genco

Hello! I’m new to helmfile and need some guidance on a use case. I’m adding a nested helmfile inside a monorepo-like helmfile repository. The parent repo so far doesn’t use environments since all services are global, but my child helmfile benefits greatly from them, because it defines production and staging namespaces and deployments. My helmfile looks basically like this:

environments:                                
  staging:                                   
    values:                                  
      - staging.yaml                         
  production:                                
    values:                                  
      - production.yaml                      
releases:                                    
  - name: app
    chart: foo/app
    namespace: app-{{.Environment.Name}}
    values:
    - values.yaml.gotmpl

This works great as long as I specify -e staging or -e production during helmfile apply. But since the parent repo has no concept of environments, it breaks the deploy process.

I guess my question is, what’s the best way to “inline” environments into the releases list? i.e. I would like something like this:

releases:                                    
  - name: app-staging
    chart: foo/app
    namespace: app-staging
    values:
      - staging.yaml
      - values.yaml.gotmpl
  - name: app-production
    chart: foo/app
    namespace: app-production
    values:
      - production.yaml
      - values.yaml.gotmpl

However, it’s attempting to “merge” the values files, rather than use the first to fill in the template of the second. Is there a better way to do this, rather than just duplicating the mostly-identical values files?

gonzalez.mariano.gabr avatar
gonzalez.mariano.gabr

hey @ have you tried using dependencies ? you can define a release with its dependencies so that the child chart will inherit values with <child_chart> prefix - see https://github.com/roboll/helmfile/issues/1762 for reference

Local dependencies not working in release definition · Issue #1762 · roboll/helmfile attachment image

Hi all. I am facing an issue with local dependencies declaration part of a release, as follows: releases: - name: foo_rel namespace: foo_ns chart: foo_example dependencies: - name: dep_chart reposi…

mumoshu avatar
mumoshu


I guess my question is, what’s the best way to “inline” environments
im not sure if i understand fully, but it seems like you’ve already made it concise and DRY enough.

If i were you i won’t try to make it DRY further, thinking doing so would hurt readability

mumoshu avatar
mumoshu

if really want, you can still do this with standard go template techniques

mumoshu avatar
mumoshu

like

{{ define "t" }}
  - name: app-{{ .env }}
    chart: foo/app
    namespace: app-{{ .env }}
    values:
      - {{ .env }}.yaml
      - values.yaml.gotmpl
{{ end }}

releases:
{{ template "t" (dist "env" "staging" ) }}
{{ template "t" (dist "env" "production" ) }}
mumoshu avatar
mumoshu

i believe you’d better use a more serious configuration language like CUE if you need to dynamically generate helmfile.yamls at this level. FYI, CUE is https://cuelang.org/

CUE

Validate and define text-based and dynamic configuration

mumoshu avatar
mumoshu

but i thought i’ve seen many people happy with https://sweetops.slack.com/archives/CE5NGCB9Q/p1618995133219700?thread_ts=1618934290.213300&cid=CE5NGCB9Q so i can imagine you’ll like it, too

like

{{ define "t" }}
  - name: app-{{ .env }}
    chart: foo/app
    namespace: app-{{ .env }}
    values:
      - {{ .env }}.yaml
      - values.yaml.gotmpl
{{ end }}

releases:
{{ template "t" (dist "env" "staging" ) }}
{{ template "t" (dist "env" "production" ) }}
mumoshu avatar
mumoshu


But since the parent repo has no concept of environments, it breaks the deploy process.
Well, i guess the reality is that the parent repo does have the concept of environments, but they are hidden. If the parent has no environnment, children shouldn’t have environments

mumoshu avatar
mumoshu

If the parent envs are just hidden and you don’t want to write

environments:
  staging: {}
  production: {}

in parent, you may prefer

environments:
  {{ .Environment.Name }}: {}

---

# releases, helmfiles, etc
jose.amengual avatar
jose.amengual

Hi, I’m doing hemlfile sync to change from a internet-facing alb to a internal alb and I tried destroy, sync and diff but even though there is changes the alb does not get get created, anyone had and issue like this before?

mumoshu avatar
mumoshu

@jose.amengual Hey! It sounds like issues in the helm chart you’re using. Helmfile’s just calling helm uprade --install or helm delete or whatever according to the definitions in your helmfile.yaml so that kind of issues can be very likely to be caused by the chart, not helmfile

jose.amengual avatar
jose.amengual

interesting ok, I will have a look

jose.amengual avatar
jose.amengual

I used a different namespace and everything works

jose.amengual avatar
jose.amengual

if I destroy, and delete other resources lingering and use the same namespace that did not work before it does not create the alb

jose.amengual avatar
jose.amengual

so there must be something still there

jose.amengual avatar
jose.amengual

now I wonder how I can find it

jose.amengual avatar
jose.amengual

from the docs on NLBs…

Do not modify the service annotation service.beta.kubernetes.io/aws-load-balancer-type on an existing service object. If you need to modify the underlying AWS LoadBalancer type, for example from classic to NLB, delete the kubernetes service first and create again with the correct annotation. Failure to do so will result in leaked AWS load balancer resources.
jose.amengual avatar
jose.amengual
Ingres resources are not getting deleted even though the alb ingress controller deployment is deleted · Issue #1629 · kubernetes-sigs/aws-load-balancer-controller attachment image

There are two issues i faced while trying to use single load balancer feature new ingress resource is failing to add ALB with new ingress group name (works only when the resource group name is chan…

jose.amengual avatar
jose.amengual

that @Andriy Knysh (Cloud Posse) found is real…

jose.amengual avatar
jose.amengual

if you modify resources created by the controller int he console, you will have orphan resources

jose.amengual avatar
jose.amengual

in my case the ingress what still there

jose.amengual avatar
jose.amengual

so I run kubectl patch ingress ingress-name -n namespace-name -p '{"metadata":{"finalizers":[]}}' --type=merge

jose.amengual avatar
jose.amengual

and then the ingress was gone and I was able to run helmfile sync and helmfile destroy and everything worked as expected

jose.amengual avatar
jose.amengual

so….just don’t do ClickOps

1
Jim Conner avatar
Jim Conner

@jose.amengual it seems like it would be a better use-case for terraform ¯_(ツ)_/¯ — Do you have some helmchart you’re using to deploy your ALB?

1
jose.amengual avatar
jose.amengual

yes we have some we use

Jim Conner avatar
Jim Conner

@ I’d try to help you but I’m in the same boat as you man. Still learning.

2021-04-19

2021-04-18

2021-04-17

2021-04-16

jedineeper avatar
jedineeper

Do I have to explicitly define a reference to each of the values I include under a named environment block into a release or is there a tidier way to bulk include the contents of the environment block ontop of the release values?

jedineeper avatar
jedineeper

I feel daunted compared to Jim’s previous question ;) but I managed to break this down to a simple example and suspect i’m just “doing helmfile wrong” though the environments block in the README implies to me that this should work?

basically, i did helm create my-chart and then wrote this helmfile.yaml for it.

 environments:
   dev:
     values:
     - fullnameOverride: my-chart-dev
   live:
     values:
     - fullnameOverride: my-chart-live
 
 releases:
 - name: my-chart
   chart: ../.
   namespace: chart-test
   version: 0.1.0
   # values:
   # - fullnameOverride: {{ .Values.fullnameOverride }}

so helmfile should deploy my chart where the full name for objects is overridden by the environment specific value.

however, helmfile doesn’t inherit those values unless I uncomment the last two lines and explicitly declare each variable I want to make environmentally conditional? Are all the environment values not inherited? I’m hoping to not have to write out each variable in the releases block otherwise I might as well go for the - ./ values/{{ environment.Name }}.yaml method and I wanted to avoid having two full copies of the values.yaml..?

mumoshu avatar
mumoshu

@jedineeper Hey! Well, honestly speaking I don’t understand how the documentation can make you think that environment values are inherited to release values. Probably the documentation needs to specifically say it doesnt?

mumoshu avatar
mumoshu

Environment values are used to render helmfile.yaml templates and helmfile’s values gotmpl files only.

Helmfile doesn’t automatically pass those values to Helm(as release values).

mumoshu avatar
mumoshu

@ Maybe you’re interested in this topic, too?

1
vicken avatar
vicken

Uncommenting the last two lines is the way I understand using helmfile with environments.

That the usage of {{ .Values.fullnameOverride }} in the chart files and the usage of {{ .Values.fullnameOverride }} in helmfile refer to separate things.

Inside the chart, .Values refer to the helmchart release’s values:

metadata:
  labels:
    foo: {{ .Values.fullnameOverride }}

refers to this value:

 releases:
 - name: my-chart
   chart: ../.
   namespace: chart-test
   version: 0.1.0
   values:
   - fullnameOverride: bar

So the label foo=bar.


Inside the helmfile, .Values refer to the environment’s values:

 releases:
 - name: my-chart
   chart: ../.
   namespace: chart-test
   version: 0.1.0
   values:
   - fullnameOverride: {{ .Values.fullnameOverride }}

refers to these values:

 environments:
   dev:
     values:
     - fullnameOverride: my-chart-dev
   live:
     values:
     - fullnameOverride: my-chart-live

and you have to select between them with the -e environment argument.

The last two lines bridge the two cases together.

1
cool-doge1
mumoshu avatar
mumoshu

@jedineeper If you want to have different “helm” values.yaml per each helmfile named environment, I’d do this:

 environments:
   dev: {}
   live: {}

---

 releases:
 - name: my-chart
   chart: ../.
   namespace: chart-test
   version: 0.1.0
   values:
    - environments/{{ .Environment.Name }}/values.yaml
mumoshu avatar
mumoshu

@ Great explanation!

mumoshu avatar
mumoshu

We can even merge the above examples:

---
 environments:
   dev:
     values:
     - fullnameOverride: my-chart-dev
   live:
     values:
     - fullnameOverride: my-chart-live

---

 releases:
 - name: my-chart
   chart: ../.
   namespace: chart-test
   version: 0.1.0
   values:
    - environments/{{ .Environment.Name }}/values.yaml
   - fullnameOverride: {{ .Values.fullnameOverride }}
mumoshu avatar
mumoshu

A common addition to this is when you’d like to make the environment-specific helm values.yaml optional, add missingFileHandler:

 environments:
   dev:
     values:
     - fullnameOverride: my-chart-dev
   live:
     values:
     - fullnameOverride: my-chart-live

---

 releases:
 - name: my-chart
   chart: ../.
   namespace: chart-test
   version: 0.1.0
   missingFileHandler: Warn
   values:
    - environments/{{ .Environment.Name }}/values.yaml
   - fullnameOverride: {{ .Values.fullnameOverride }}
jedineeper avatar
jedineeper

Super useful, thanks both. Looks like I was on the wrong track with my understanding and that’s cleared up now :)

1
Jim Conner avatar
Jim Conner

@mumoshu this might help me. I’d still like to read your thoughts outlined in my gh issue but I might be able to make progress with this information.

mumoshu avatar
mumoshu

@ In the scenario 3, are you trying to feed environments defined in bluescape/ops/helmfile-project/helmfile/envs/dev/cluster-n/config.yaml into every helmfile yamls defined under bluescape/ops/helmfile-project/helmfile.d?

Jim Conner avatar
Jim Conner

yes

Jim Conner avatar
Jim Conner

@mumoshu

Jim Conner avatar
Jim Conner

This was where I was getting confused with the concept of inheritance in trying to understand what inheritance was with respect to helmfile though you corrected me in one of your responses that helmfile doesn’t inherit. So, let’s not use that term, but I am trying to propagate the values defined in config.yaml in scenario three across the board of whatever I deploy using helmfile from any respective environment directory

mumoshu avatar
mumoshu

It’s much better to treat there’s no inheritance, I think. (Where did you find helmfile does inheritance? I think that doc should be corrected if it’s confusing

mumoshu avatar
mumoshu

Gotcha so

Jim Conner avatar
Jim Conner

It’s documentation that I read in other github issues…not in your documentation

mumoshu avatar
mumoshu

Thx I see! Firstly I’d repeat https://github.com/roboll/helmfile/issues/1045#issuecomment-821910447 - envs defined in config.yaml doesn’t get automatically passed to sub-helmfiles under helmfile.d.

I guess this should work:

helmfiles:
  - path: "../../../../helmfile.d/"
    values:
    - {{ .Values | toYaml | nindent 6 }}
Environment values not being used in nested states · Issue #1045 · roboll/helmfile

I&#39;ve been trying to use environments in my main helmfile and have multiple sub helmfiles in my releases folder. I wanted all the sub helmfiles to pick up whatever the value I defined in a speci…

Jim Conner avatar
Jim Conner

Lemme give that a try…

Jim Conner avatar
Jim Conner

taking a read real fast.

mumoshu avatar
mumoshu

I admit this isn’t elengant, but not sure if there’s any better way than this now..

Jim Conner avatar
Jim Conner

ok. Lemme take a look.

Jim Conner avatar
Jim Conner

Lemme ask you this…

Jim Conner avatar
Jim Conner

is the method of how I want to use helmfile odd or something?

Jim Conner avatar
Jim Conner

It seems logical to me…and helmfile is a very flexible tool, but am I doing it wrong?

mumoshu avatar
mumoshu

Yours seem totally valid

Jim Conner avatar
Jim Conner

Ok, that’s good to know!

mumoshu avatar
mumoshu

I guess it’s just that no one has ever (loudly) tried to do it in a such elegant way

Jim Conner avatar
Jim Conner

Wow. Thank you for that comment. Very nice.

mumoshu avatar
mumoshu

I believe Helmfile should add something like the below to better support your usecase(actually this was my original plan was no one has ever requested it or sent me any question that leads to this

mumoshu avatar
mumoshu
helmfiles:
  - path: "../../../../helmfile.d/"
    inheritValues: true
1
Jim Conner avatar
Jim Conner

so, it seems like it could potentially be complicated to code depending on the library(ies) you might be using to mux objects…

Jim Conner avatar
Jim Conner

but if you have a solid top-level then inheritance probably isn’t that tough to enable.

Jim Conner avatar
Jim Conner

taking a look at this now…gimme a few please.

Environment values not being used in nested states · Issue #1045 · roboll/helmfile

I&#39;ve been trying to use environments in my main helmfile and have multiple sub helmfiles in my releases folder. I wanted all the sub helmfiles to pick up whatever the value I defined in a speci…

Jim Conner avatar
Jim Conner

So from your comment in the thread, it’s not very clear to me which helmfile.yaml that you are suggesting change to. also, do you see merit in making a feature request out of this use-case? I’d LOVE to see this functionality.

Environment values not being used in nested states · Issue #1045 · roboll/helmfile

I&#39;ve been trying to use environments in my main helmfile and have multiple sub helmfiles in my releases folder. I wanted all the sub helmfiles to pick up whatever the value I defined in a speci…

Jim Conner avatar
Jim Conner

also, in that thread, I need to define the config.yaml which has the values defined in the top-level directory but it seems your example only defines a scriptsDir which I suppose is the defined directory where the top-level config.yaml file would reside; but if I point the scriptsDir to the same path as where my top-level is then it seems that would cause a circular dependency unless I can specifically say, “use this file” — not just “use this directory”

Jim Conner avatar
Jim Conner

@mumoshu is there a way to get helmfile template --selector foo=bar to work?

mumoshu avatar
mumoshu


if I point the scriptsDir to the same path as where my top-level is then it seems that would cause a circular dependency
This wasn’t clear to me, circular dependency between what, do you mean?

mumoshu avatar
mumoshu

@ Did you literally mean the way? You need to place --selector foo=bar before template.

Jim Conner avatar
Jim Conner

ah

Jim Conner avatar
Jim Conner

ok, lemme check that out.

Jim Conner avatar
Jim Conner

ah perfect! Thank you

mumoshu avatar
mumoshu


it’s not very clear to me which helmfile.yaml that you are suggesting change to.
I meant helmfile/envs/dev/cluster-a/helmfile.yaml, assuming that’s where you’re trying to say “inherit envs defined in config.yaml to ../../../../helmfile.d/”

Jim Conner avatar
Jim Conner

ah! Ok. So scriptsDir points to the top-level directory?

mumoshu avatar
mumoshu

Ah, no. It’s just an arbitrary value for illustration purpose.

mumoshu avatar
mumoshu

It is’nt a reserved value in any way so you can’t assume scriptsDir points to anywhere wihout you specifically set it

Jim Conner avatar
Jim Conner

k, so right now, the following seems to be working but I’m still attempting to test (this is my top-level):

environments:
  default: {}

---
helmfiles:
  - path: "../../../../helmfile.d/"
    values:
      - config.yaml
Jim Conner avatar
Jim Conner

I don’t have all of the sub-helmfile.yaml manifests completely correct yet, except for dex but unfortunately, my selector is not working.

Jim Conner avatar
Jim Conner

so I’m trying to figure that out.

Jim Conner avatar
Jim Conner

0 release(s) matching tier=secrets-management,app=dex found in helmfile.yaml

mumoshu avatar
mumoshu

have you tried helmfile --debug $SUBCOMMAND? it would print a lot of logs to help debugging

Jim Conner avatar
Jim Conner

yup

Jim Conner avatar
Jim Conner

I’m reading through it all now to see if I can figure out where my problem lies

Jim Conner avatar
Jim Conner

funny enough, the error I just posted is exactly after the repo updates. So, gotta figure it out.

mumoshu avatar
mumoshu

0 release(s) matching tier=secrets-management,app=dex found in helmfile.yaml will be printed on any sub-helmfile that contain other releases. But I thought it doesn’t result in an error?

mumoshu avatar
mumoshu

I mean, you’ll see 0 release(s) matching tier=secrets-management,app=dex found in helmfile.yaml in debug logs on every sub-helmfile that didn’t have releases matching the selector. But Helmfile won’t fail when any sub-helmfile had one or more releases that matched.

1
Jim Conner avatar
Jim Conner

just a sec. still reading through the output. I found out that I had --selector specified twice on accident on the command line so that was one problem.

Jim Conner avatar
Jim Conner

the problem seems to be that helmfile is not traversing recursively into certain paths…

1
mumoshu avatar
mumoshu

could you push your project in github so that i can reproduce?

Jim Conner avatar
Jim Conner

sure!

Jim Conner avatar
Jim Conner

gimme a sec.

Jim Conner avatar
Jim Conner

sorry for taking so long. Wife was in here talking to me for a min. Now, I’m scrubbing the repo as it’s not open source, but there’s nothing of great value in here yet. So, I’ll create in my personal repo temporarily and provide you with the link.

Jim Conner avatar
Jim Conner
notjames/jimconn-shell

My shell stuffs. Contribute to notjames/jimconn-shell development by creating an account on GitHub.

Jim Conner avatar
Jim Conner

I removed secrets files, so if you run this and you get errors about that, that’s why

mumoshu avatar
mumoshu

Thanks!

Jim Conner avatar
Jim Conner

just a sec

Jim Conner avatar
Jim Conner

need to do something

mumoshu avatar
mumoshu


the problem seems to be that helmfile is not traversing recursively into certain paths…
what were certain paths and what helmfile command did you run to see it?

Jim Conner avatar
Jim Conner

do a pull real fast

Jim Conner avatar
Jim Conner

and then cd <repo>/temp/helmfile/envs/dev/atreus

mumoshu avatar
mumoshu

done

Jim Conner avatar
Jim Conner

thx

Jim Conner avatar
Jim Conner

now in that directory I ran: helmfile --debug --selector app=dex template

mumoshu avatar
mumoshu
err: error during helmfile.yaml.part.0 parsing: template: stringTemplate:5:30: executing "stringTemplate" at <.Environment.Values.namespace>: map has no entry for key "namespace"
changing working directory back to "/home/mumoshu/p/jimconn-shell/temp/helmfile.d/generic/01a-network-and-proxies"
changing working directory back to "/home/mumoshu/p/jimconn-shell/temp/helmfile.d"
changing working directory back to "/home/mumoshu/p/jimconn-shell/temp/helmfile/envs/dev/atreus"
in ./helmfile.yaml: in .helmfiles[0]: in ../../../../helmfile.d/helmfile.yaml: in .helmfiles[0]: in generic/01a-network-and-proxies/helmfile.yaml: in .helmfiles[1]: in external-dns/helmfile.yaml: error during helmfile.yaml.part.0 parsing: template: stringTemplate:5:30: executing "stringTemplate" at <.Environment.Values.namespace>: map has no entry for key "namespace"
Jim Conner avatar
Jim Conner

when you run that, you’ll notice that helmfile doesn’t backup the tree after descending into 01a-… to traverse into 01b-…

Jim Conner avatar
Jim Conner

ahh…you’re saying that the reason is because the external-dns “chart” is broken….that makes sense.

Jim Conner avatar
Jim Conner

I’ll fix real fast. I haven’t polished all of these yet so I’m not concerned about that being broken

mumoshu avatar
mumoshu

seems so! helmfile traverses sub-helmfile in some alphabetical order and fail-fast

Jim Conner avatar
Jim Conner

ah!! OK, that’s good to know

Jim Conner avatar
Jim Conner

lemme fix that real fsat.

Jim Conner avatar
Jim Conner

does helmfile descend into dot directories?

Jim Conner avatar
Jim Conner

nm. I’ll rm -rf external-dns and ingress-nginx

mumoshu avatar
mumoshu

haven’t tried that but as long as the exact path or a glob pattern that matches the dotted directory/file, it should just work (I thought I have not programmed helmfile to explicitly ignore dot dir/file

Jim Conner avatar
Jim Conner

ok

Jim Conner avatar
Jim Conner

all right, so the error I’m getting now is:

in ./helmfile.yaml: in .helmfiles[0]: in ../../../../helmfile.d/helmfile.yaml: in .helmfiles[0]: in generic/01a-network-and-proxies/helmfile.yaml: in .helmfiles[1]: in ambassador/helmfile.yaml: [Malformed label: tier="network-and-proxies". Expected label in form k=v or k!=v]

That tier is defined in generic/01a-network-and-proxies/helmfile.yaml

Jim Conner avatar
Jim Conner

so that’s interesting

Jim Conner avatar
Jim Conner

the idea was, I can specify a tier at a higher level in the directory structure…but I wasn’t sure that would work.

Jim Conner avatar
Jim Conner

maybe I did something wrong in the helmfile, though

Jim Conner avatar
Jim Conner

the error seems to state as much

Jim Conner avatar
Jim Conner

ah, I see

mumoshu avatar
mumoshu

seems like you tried to use " in the selector which isn’t supported?

Jim Conner avatar
Jim Conner

well, I tried using proper yaml first. That fails.

Jim Conner avatar
Jim Conner

just tried it again.

Jim Conner avatar
Jim Conner
first-pass produced: &{default map[] map[]}
first-pass rendering result of "helmfile.yaml.part.0": {default map[] map[]}
second-pass rendering result of "helmfile.yaml.part.0":
 0: helmfiles:
 1:   - "*/helmfile.yaml"
 2:   - path: */helmfile.yaml
 3:     selectors:
 4:       - tier=secrets-managment. <<== apparently not OK
 5
1
Jim Conner avatar
Jim Conner
❯ \cat helmfile.yaml 
helmfiles:
  - "*/helmfile.yaml"
  - path: "*/helmfile.yaml"
    selectors:
      - tier: network-and-proxies

gives:

n ./helmfile.yaml: in .helmfiles[0]: in ../../../../helmfile.d/helmfile.yaml: in .helmfiles[0]: in generic/01a-network-and-proxies/helmfile.yaml: failed to read helmfile.yaml: reading document at index 1: yaml: unmarshal errors:
  line 5: cannot unmarshal !!map into string
mumoshu avatar
mumoshu

needs to be tier=network-and-proxies

Jim Conner avatar
Jim Conner

is selectors: requiring a list perhaps?

mumoshu avatar
mumoshu

definitely!

Jim Conner avatar
Jim Conner

k, lemme fix and try

mumoshu avatar
mumoshu

it must be a lint of strings

Jim Conner avatar
Jim Conner

with no "?

mumoshu avatar
mumoshu

in where?

mumoshu avatar
mumoshu

as long as it’s a yaml string, it would be okay

mumoshu avatar
mumoshu
- "foo=bar"

works

Jim Conner avatar
Jim Conner

ok, lemme try

mumoshu avatar
mumoshu
- foo="bar"

is a valid string but invalid in terms of the selector syntax

Jim Conner avatar
Jim Conner

so

❯ \cat helmfile.yaml 
helmfiles:
  - "*/helmfile.yaml"
  - path: "*/helmfile.yaml"
    selectors:
      - "tier=network-and-proxies"

and this is temp/helmfile.d/generic/01a-network-and-proxies/helmfile.yaml

mumoshu avatar
mumoshu

seems good

Jim Conner avatar
Jim Conner

OK cool; that error is gone. But now it still can’t find the selector.

merged environment: &{default map[ambassador:map[namespace:ambassador version:6.6.2] arm64_support:false aws_nlb_with_tls_termination_at_lb:false blscauxclusterissuer:map[namespace:certmanager version:0.1.0] certmanager:map[namespace:certmanager version:1.0.4] cluster_id:<nil> dex:map[namespace:dex version:2.15.2] domain_name:<nil> externaldns:map[namespace:externaldns version:v20210203-v0.7.6-28-g44288212-arm64v8] name:somename nginxingress:map[namespace:ingress-nginx version:3.15.2] oauth2proxy:map[namespace:auth-system version:3.2.5] postgres:map[namespace:grafana version:x.x.x] semver:0.0.1 vaultoperator:map[namespace:vault version:1.8.1] vaultsecretswebhook:map[namespace:secrets-webhook version:1.8.2]] map[]}
0 release(s) matching tier=network-and-proxies found in helmfile.yaml

changing working directory back to "/home/jimconn/projects/src/personal/jimconn-shell/temp/helmfile.d/generic/01a-network-and-proxies"
0 release(s) matching app=dex found in helmfile.yaml

err: no releases found that matches specified selector(app=dex) and environment(default), in any helmfile

hmm…. is helmfile looking at the labels in releases: object too or just helmfile objects?

mumoshu avatar
mumoshu

what’s the command now? I’m pretty lost

mumoshu avatar
mumoshu
[email protected]:~/p/jimconn-shell/temp/helmfile.d/generic/01a-network-and-proxies$ $HOME/p/helmfile/helmfile template
Adding repo datawire <https://www.getambassador.io>
"datawire" has been added to your repositories

in ./helmfile.yaml: in .helmfiles[0]: in ambassador/helmfile.yaml: failed to render values files "values.yaml.gotmpl": failed to render [values.yaml.gotmpl], because of template: stringTemplate:63:20: executing "stringTemplate" at <.Values.resources.limits.cpu>: map has no entry for key "resources"
mumoshu avatar
mumoshu


is helmfile looking at the labels in releases: object too or just helmfile objects?
i don’t get it. helmfile “object” doesn’t have labels. only releases have labels to be matched by selectors

Jim Conner avatar
Jim Conner

lol. I’m sorry. So the command is being run from temp/helmfile/envs/dev/atreus and the command I’m using is: helmfile --debug --selector app=dex template

Jim Conner avatar
Jim Conner

well

Jim Conner avatar
Jim Conner

if you notice that I was able to provide a selector in the helmfiles object of a helmfile.yaml for 01a-… and that at least linted.

mumoshu avatar
mumoshu

yep. that means you’re letting helmfile to run helmfile -l $SELECTOR_HERE template on the sub-helmfile

Jim Conner avatar
Jim Conner

yup

mumoshu avatar
mumoshu

that way, helmfile uses the selector to filter releases defined in the sub-helmfile

Jim Conner avatar
Jim Conner

cool

Jim Conner avatar
Jim Conner

so I was thinking that I could also use a selector for a specific helm release given a selector

Jim Conner avatar
Jim Conner

so in other words, let’s say I’m in my cluster-a environment

mumoshu avatar
mumoshu

that should work

Jim Conner avatar
Jim Conner

all I want to do is release one single app

Jim Conner avatar
Jim Conner

so I name all my apps in releases with a selector (which I know this is a helm selector but helmfile seems to understand those, right)?

mumoshu avatar
mumoshu

the point is that no helmfile.yaml under helmfile.d/generic/01a-network-and-proxies/* has releases that have the label app: dex

Jim Conner avatar
Jim Conner

oh sure! I know. It’s in 01b-security-management/dex

mumoshu avatar
mumoshu

i see! then you need to provide that to helmfile.yaml. currently it doesn’t refer to it

$ cat helmfile.yaml
helmfiles:
  - "*/helmfile.yaml"
  - path: "*/helmfile.yaml"
    selectors:
      - tier=network-and-proxies
Jim Conner avatar
Jim Conner

my dex helmfile bundle is done so I’m just testing the “e2e” from cluster-a (or atreus) to see if my top-level config works.

Jim Conner avatar
Jim Conner

ah!

mumoshu avatar
mumoshu

were you trying to say this?

helmfiles:
  - "../*/helmfile.yaml"
  - path: "*/helmfile.yaml"
    selectors:
      - tier=network-and-proxies
Jim Conner avatar
Jim Conner

ok

Jim Conner avatar
Jim Conner

well, no

mumoshu avatar
mumoshu

ok. but you need to add e.g. 01b-security-management/dex to helmfiles section at least

Jim Conner avatar
Jim Conner

k. just a sec. Lemme see

Jim Conner avatar
Jim Conner

so in my 01b-security-manangement helmfile.yaml I have:

helmfiles:
  - "*/helmfile.yaml"
  - path: */helmfile.yaml
    selectors:
      - tier=secrets-managment
Jim Conner avatar
Jim Conner

I was thinking that if I place a selector on the command line the helmfile will read everything and simply use anything found matching the requested selector

mumoshu avatar
mumoshu

yep you seem to be correct

Jim Conner avatar
Jim Conner

and in this case, I have a bundle named dex which only deploys dex and it’s under 01b-security-management/dex

Jim Conner avatar
Jim Conner

and the helmfile.yaml in there is:

---
bases:
  - ../../common/environments.yaml
  - ../../common/repos.yaml
  - ../../common/helmdefaults.yaml

---
releases:
  - name: dex
    namespace: {{ .Values.dex.namespace }}
    createNamespace: true
    labels:
      app: dex <<<<<=== THERE
    chart: stable/dex
    version: {{ .Values.dex.version }}
    values:
      - values.yaml.gotmpl
Jim Conner avatar
Jim Conner

so helmfile should find that right?

mumoshu avatar
mumoshu

yep. but where are you running helmfile from?

Jim Conner avatar
Jim Conner

the atreus directory

mumoshu avatar
mumoshu

and the content of helmfile.yaml under atreus?

Jim Conner avatar
Jim Conner

Jim Conner avatar
Jim Conner

oh, lemme post it

Jim Conner avatar
Jim Conner
environments:
  default: {}

---
helmfiles:
  - path: "../../../../helmfile.d/"
    values:
      - config.yaml
Jim Conner avatar
Jim Conner

so helmfile should read the helmfile.yaml in helmfile.d

Jim Conner avatar
Jim Conner

which only matches all *.yaml files in every directory

Jim Conner avatar
Jim Conner

that directory contains the 01a-… and 01b-… directories which have helmfile.yaml files which define a tier selector for that directory of bundles.

mumoshu avatar
mumoshu

so this is what im getting

mumoshu avatar
mumoshu
[email protected]:~/p/jimconn-shell/temp/helmfile/envs/dev/atreus$ $HOME/p/helmfile/helmfile -l app=dex template
Adding repo stable <https://charts.helm.sh/stable>
"stable" has been added to your repositories

Adding repo jetstack <https://charts.jetstack.io>
"jetstack" has been added to your repositories

Adding repo bitnami <https://charts.bitnami.com/bitnami>
"bitnami" has been added to your repositories

Adding repo prometheus <https://prometheus-community.github.io/helm-charts>
"prometheus" has been added to your repositories

Adding repo banzaicloud-stable <https://kubernetes-charts.banzaicloud.com>
"banzaicloud-stable" has been added to your repositories

Adding repo cloudposse <https://charts.cloudposse.com/incubator/>
"cloudposse" has been added to your repositories

Adding repo datawire <https://www.getambassador.io>
"datawire" has been added to your repositories

in ./helmfile.yaml: in .helmfiles[0]: in ../../../../helmfile.d/helmfile.yaml: in .helmfiles[1]: in generic/01b-secrets-management/helmfile.yaml: failed to read helmfile.yaml: reading document at index 1: yaml: line 3: did not find expected alphabetic or numeric character
mumoshu avatar
mumoshu

isn’t this due to that helmfile failing fast due to that it failed to parse generic/01b-secrets-management/helmfile.yaml?

Jim Conner avatar
Jim Conner

huh. werid.

Jim Conner avatar
Jim Conner

seems like it. I don’t get that though. Lemme commit/push what I have here and you can run it again and see if you get the same result.

Jim Conner avatar
Jim Conner

go ahead and pull

mumoshu avatar
mumoshu

i think you’d need to fix each helmfile.yaml to work alone

Jim Conner avatar
Jim Conner

oh crap

Jim Conner avatar
Jim Conner

I am getting that

Jim Conner avatar
Jim Conner

I did that first.

Jim Conner avatar
Jim Conner

but I might have changed something during all this. Lemme check that real fast.

mumoshu avatar
mumoshu

at glance this is invalid yaml

$ cat ../../../../helmfile.d/generic/01b-secrets-management/helmfile.yaml 
helmfiles:
  - "*/helmfile.yaml"
  - path: */helmfile.yaml
    selectors:
      - tier=secrets-managment
mumoshu avatar
mumoshu

should be

helmfiles:
  - "*/helmfile.yaml"
  - path: "*/helmfile.yaml"
    selectors:
      - tier=secrets-managment
Jim Conner avatar
Jim Conner

ok

Jim Conner avatar
Jim Conner

I’ll add the quotes.

Jim Conner avatar
Jim Conner

awesome!

Jim Conner avatar
Jim Conner

that seemed to work

Jim Conner avatar
Jim Conner

well

Jim Conner avatar
Jim Conner

ish

mumoshu avatar
mumoshu

great

Jim Conner avatar
Jim Conner
err: failed to read environments.yaml: environment values file matching "../../common/versions.yaml" does not exist in "."
changing working directory back to "/home/jimconn/projects/src/personal/jimconn-shell/temp/helmfile.d"
changing working directory back to "/home/jimconn/projects/src/personal/jimconn-shell/temp/helmfile/envs/dev/atreus"
in ./helmfile.yaml: in .helmfiles[0]: in ../../../../helmfile.d/helmfile.yaml: in .helmfiles[2]: in generic/common/environments.yaml: failed to read environments.yaml: environment values file matching "../../common/versions.yaml" does not exist in "."
Jim Conner avatar
Jim Conner

yeah, so that is an interesting error that I’m still fuzzy on in terms of the cause.

Jim Conner avatar
Jim Conner

this goes back to part of the conversation we had last week with respect to paths…but I still can’t articulate that very well, so I could just be wrong about its implementation

mumoshu avatar
mumoshu

well lets start reading it

mumoshu avatar
mumoshu

assuming we’re running helmfile-template in atreus

mumoshu avatar
mumoshu
in ./helmfile.yaml: in .helmfiles[0]: in ../../../../helmfile.d/helmfile.yaml: in .helmfiles[2]: in generic/common/environments.yaml: failed to read environments.yaml: environment values file matching "../../common/versions.yaml" does not exist in "."

means that

Jim Conner avatar
Jim Conner

yes

mumoshu avatar
mumoshu

atreus/helmfile.yaml loaded ../../../../helmfile.d/helmfile.yaml, which in turn loaded generic/common/environments.yaml.

mumoshu avatar
mumoshu

while helmfile’s trying to process the last file, environemnts.yaml, it failed to find ../../common/versions.yaml

mumoshu avatar
mumoshu

let’s see what’s in environments.yaml

Jim Conner avatar
Jim Conner
---
environments:
  default:
    values:
      - ../../common/versions.yaml
      - ../../common/values.yaml.gotmpl
  production:
    values:
      - ../../common/versions.yaml
      - ../../common/values.yaml.gotmpl
mumoshu avatar
mumoshu
$ cat ../../../../helmfile.d/generic/common/environments.yaml 
---
environments:
  default:
    values:
      - ../../common/versions.yaml
      - ../../common/values.yaml.gotmpl
  production:
    values:
      - ../../common/versions.yaml
      - ../../common/values.yaml.gotmpl
mumoshu avatar
mumoshu

yep

Jim Conner avatar
Jim Conner

yes

Jim Conner avatar
Jim Conner

and this was where I was super confused

Jim Conner avatar
Jim Conner

because it should have been values.yaml.gotmpl and versions.yaml

Jim Conner avatar
Jim Conner

but that didn’t work

Jim Conner avatar
Jim Conner

this was why I asked you the question the other day…if you will recall.

Jim Conner avatar
Jim Conner

the paths document says that helmfile will use paths relative to the yaml file requesting the path/file

mumoshu avatar
mumoshu

as you’re including it into some helmfile yaml under generic, those paths should be releative to generic

Jim Conner avatar
Jim Conner

this my finding is that this isn’t consistent

mumoshu avatar
mumoshu

that applies only to helmfile.yaml

Jim Conner avatar
Jim Conner

which confused the poo outta me

Jim Conner avatar
Jim Conner

ahhh

Jim Conner avatar
Jim Conner

ok

Jim Conner avatar
Jim Conner

that makes more sense as far as what the documentation means

mumoshu avatar
mumoshu

so bases are base configuration or some form of skeleton of your helmfile.yaml

Jim Conner avatar
Jim Conner

so what about other manifests? How do paths work with respect to others?

mumoshu avatar
mumoshu

if bases are evaluated in the directory of the base helmfile yaml, it would prevent you from reusing it in any useful way

mumoshu avatar
mumoshu

so every path is relative to the helmfile.yaml being processed

Jim Conner avatar
Jim Conner

ah. hmm

mumoshu avatar
mumoshu

bases modifies the helmfile.yaml

Jim Conner avatar
Jim Conner

so that might be problematic for how I’m trying to do this.

mumoshu avatar
mumoshu

probably. so you want to use base to codify some convention of where each helmfile.yaml should locate environment values files, right?

Jim Conner avatar
Jim Conner

it worked nicely if I could make reference to these files relative to the manifest that references files.

Jim Conner avatar
Jim Conner

yes

Jim Conner avatar
Jim Conner

and so…it might be a little easier than that

mumoshu avatar
mumoshu

and you wanted all the environment values files to be shared across all the helmfile.yaml files, right?

Jim Conner avatar
Jim Conner

basically yes

Jim Conner avatar
Jim Conner

essentially, the way I’m doing this, everything is using the same files whether it’s running as individual releases, directory releases, or environment releases.

mumoshu avatar
mumoshu

lastly, do you want values.yaml.gtoml and versions.yaml to be under the same directory as environments.yaml(base)?

Jim Conner avatar
Jim Conner

yes as those are basically global settings for everything but I want to be able to redefine stuff set in those (specifically the stuff in values.yaml.gotmpl) with the config.yaml in cluster-n environments.

mumoshu avatar
mumoshu

and you seem to be using the base environments.yaml from helmfile.yaml files in various levels

mumoshu avatar
mumoshu

generic/01a-network-and-proxies/ambassador is trying to load generic/common/environments.yaml

Jim Conner avatar
Jim Conner

yes, because that’s a common file — is how I’m using it

mumoshu avatar
mumoshu

ok

mumoshu avatar
mumoshu

well it isn’t how base is supposed to be used but i can see how you’d like to use it

mumoshu avatar
mumoshu

some people want to use bases to be evaluated in the context of the helmfile.yaml loaded it so helmfile is currently designed aroud that

Jim Conner avatar
Jim Conner

yeah…which I think I wanted to do it this way because it makes sense to me in terms of a structure. Set up a default set and then allow the ability for that default set to change based on overriding configurations

mumoshu avatar
mumoshu

ok

mumoshu avatar
mumoshu

can you just embed all the values directly into environments.yaml then?

Jim Conner avatar
Jim Conner

what’s an example of what that looks like?

mumoshu avatar
mumoshu

something like

mumoshu avatar
mumoshu
templates:
  values: &values
    semver: 0.0.1
      name: somename
      # Versions here reflect the chart version, NOT the app version.
      ambassador:
        version: 6.6.2
        namespace: ambassador
      certmanager:
        version: 1.0.4
        namespace: certmanager
      blscauxclusterissuer:
        version: 0.1.0
        namespace: certmanager
      externaldns:
        version: v20210203-v0.7.6-28-g44288212-arm64v8
        namespace: externaldns
      nginxingress:
        version: 3.15.2
        namespace: ingress-nginx
      postgres:
        version: x.x.x
        namespace: grafana
      oauth2proxy:
        version: 3.2.5
        namespace: auth-system
      dex:
        version: 2.15.2
        namespace: dex
      vaultoperator:
        version: 1.8.1
        namespace: vault
      vaultsecretswebhook:
        version: 1.8.2
        namespace: secrets-webhook


environments:
  default:
    values:
    - *values
  production:
    values:
    - *values
Jim Conner avatar
Jim Conner

ahhh. I see… this might be doable.

mumoshu avatar
mumoshu

well i used wrong term in the example but you get what i meant..

mumoshu avatar
mumoshu

afk ill post compelte example later

Jim Conner avatar
Jim Conner

yeah, use the yaml reference capability in conjunction with a template

Jim Conner avatar
Jim Conner

ok!

mumoshu avatar
mumoshu

so i think this should work

templates:
  versions: &versions
    semver: 0.0.1
      name: somename
      # Versions here reflect the chart version, NOT the app version.
      ambassador:
        version: 6.6.2
        namespace: ambassador
      certmanager:
        version: 1.0.4
        namespace: certmanager
      blscauxclusterissuer:
        version: 0.1.0
        namespace: certmanager
      externaldns:
        version: v20210203-v0.7.6-28-g44288212-arm64v8
        namespace: externaldns
      nginxingress:
        version: 3.15.2
        namespace: ingress-nginx
      postgres:
        version: x.x.x
        namespace: grafana
      oauth2proxy:
        version: 3.2.5
        namespace: auth-system
      dex:
        version: 2.15.2
        namespace: dex
      vaultoperator:
        version: 1.8.1
        namespace: vault
      vaultsecretswebhook:
        version: 1.8.2
        namespace: secrets-webhook
  values: *values
    aws_nlb_with_tls_termination_at_lb: {{ .Values | get "aws_nlb_with_tls_termination_at_lb" false }}
    arm64_support: {{ .Values | get "arm64_support" false }}
    domain_name: {{ .Values | get "domain_name" (env "DOMAIN_NAME") }}
    cluster_id: {{ .Values | get "cluster_id" (env "CLUSTER_ID") }}

---

environments:
  default:
    values:
    - *versions
    - *values
  production:
    values:
    - *versions
    - *values
Jim Conner avatar
Jim Conner

ugh. I just tried validating the configuration of the template test run I did a while ago (I had to re-establish the values.yaml.gotmpl contents, which I removed from my private repo) and unfortunately, I’m observing that the template run shows that the settings in the config.yaml values in atreus are not getting asserted during the template run meaning it seems that the helmfile.yaml isn’t asserting the config from the top-level. :(

Jim Conner avatar
Jim Conner

lemme try your suggestion real fast though.

mumoshu avatar
mumoshu


I’m observing that the template run shows that the settings in the config.yaml values in atreus are not getting asserted during the template run meaning it seems that the helmfile.yaml isn’t asserting the config from the top-level.
what do you mean by “asserting” here?

mumoshu avatar
mumoshu

are you saying that values defined in config.yaml in the below atreus/helmfile.yaml is not accessible from within sub-helmfiles ../../../.../helmfile.d?

environments:
  default: {}

---
helmfiles:
  - path: "../../../../helmfile.d/"
    values:
      - config.yaml
mumoshu avatar
mumoshu

i have not yet fully understood your whole config, but from what i can guess from our conversation so far

Jim Conner avatar
Jim Conner

yeah, seems so.

mumoshu avatar
mumoshu

you may be missing to pass values in the middle

Jim Conner avatar
Jim Conner

oh! I need to pass values?

Jim Conner avatar
Jim Conner

is there a generic way to do that without having to specify every variable?

mumoshu avatar
mumoshu

Yes. As I have said in elsewhere, you must always explicitly pass values to sub-helmfiles

Jim Conner avatar
Jim Conner

ahhhh….

Jim Conner avatar
Jim Conner

hence, no inheritance which you mentioned

mumoshu avatar
mumoshu

yeah

mumoshu avatar
mumoshu

Thx I see! Firstly I’d repeat https://github.com/roboll/helmfile/issues/1045#issuecomment-821910447 - envs defined in config.yaml doesn’t get automatically passed to sub-helmfiles under helmfile.d.

I guess this should work:

helmfiles:
  - path: "../../../../helmfile.d/"
    values:
    - {{ .Values | toYaml | nindent 6 }}
Jim Conner avatar
Jim Conner

lemme give that a quick try, too

mumoshu avatar
mumoshu
helmfiles:
  - path: "../../../../helmfile.d/"
    inheritValues: true
1
Jim Conner avatar
Jim Conner

got it!

Jim Conner avatar
Jim Conner

one sec

mumoshu avatar
mumoshu

beware the number you pass to nindent function. it’s dependent on the context.

- {{ .Values | toYaml | indent 2
  - {{ .Values | toYaml | indent 4 }}

make sense, but

  - {{ .Values | toYaml | indent 2 }}

or

- {{ .Values | toYaml | indent 4 }}

doesn’t.

this usage of toYaml with nindent is a common trick seen in wirting helm charts but I thought ir worth being explained

1
Jim Conner avatar
Jim Conner

in my case, the indent is the same for all of the files and 6 is accurate I believe.

1
Jim Conner avatar
Jim Conner
first-pass produced: &{default map[values:[map[aws_nlb_with_tls_termination_at_lb:false] map[arm64_support:false] map[domain_<name:atreus.dev.domain.io>] map[cluster_id:atreus]]] map[]}
first-pass rendering result of "helmfile.yaml.part.0": {default map[values:[map[aws_nlb_with_tls_termination_at_lb:false] map[arm64_support:false] map[domain_<name:atreus.dev.domain.io>] map[cluster_id:atreus]]] map[]}
second-pass rendering result of "helmfile.yaml.part.0":
 0: ---
 1: helmfiles:
 2:   - "generic/*"
 3:   - path: "generic/*"
 4:     values:
 5:       - 
 6:       values:
 7:       - aws_nlb_with_tls_termination_at_lb: false
 8:       - arm64_support: false
 9:       - domain_name: atreus.dev.domain.io
10:       - cluster_id: atreus
11:       
12: 

err: failed to read helmfile.yaml: reading document at index 1: yaml: line 6: did not find expected node content
changing working directory back to "/project/helmfile-project/helmfile/envs/dev/atreus"
in ./helmfile.yaml: in .helmfiles[0]: in ../../../../helmfile.d/helmfile.yaml: failed to read helmfile.yaml: reading document at index 1: yaml: line 6: did not find expected node content

So the failure seems to be coming from the line I just added (helmfile.d/helmfile.yaml

---
helmfiles:
  - "generic/*"
  - path: "generic/*"
    values:
      - {{ .Values | toYaml | nindent 6 }}
Jim Conner avatar
Jim Conner

it’s not dereferencing .Values

Jim Conner avatar
Jim Conner

oh wait

Jim Conner avatar
Jim Conner

it is

Jim Conner avatar
Jim Conner

I need to fix the config.yaml

Jim Conner avatar
Jim Conner

just a sec

mumoshu avatar
mumoshu

i’m now counting the size of the indentation there

mumoshu avatar
mumoshu

oh ok

mumoshu avatar
mumoshu

well shouldn’t it be 8?

---
helmfiles:
  - "generic/*"
  - path: "generic/*"
    values:
      - {{ .Values | toYaml | nindent 8 }}

with 8 it should be rendered to

helmfiles:
  - "generic/*"
  - path: "generic/*"
    values:
      - foo: bar
        baz: 1
Jim Conner avatar
Jim Conner

It looks like it should be 4

Jim Conner avatar
Jim Conner

I’ll make it 8 and see what happens

mumoshu avatar
mumoshu

why 4? you pasted the helmfile.yaml with wrong indentation?

Jim Conner avatar
Jim Conner

just going strictly by the output which was:

 4:     values:
 5:       - 
 6:       values:
Jim Conner avatar
Jim Conner

but I was looking at the wrong values:

mumoshu avatar
mumoshu

those numbers from 4 to 6 are line numbers

Jim Conner avatar
Jim Conner

yeah, I wasn’t looking at those

mumoshu avatar
mumoshu

ah ok

Jim Conner avatar
Jim Conner

I was looking at the indentation of the values on line 6 instead of line 4

Jim Conner avatar
Jim Conner

cool

Jim Conner avatar
Jim Conner

I think we got past that

mumoshu avatar
mumoshu

nice

Jim Conner avatar
Jim Conner

now we’re back on the environments.yaml

Jim Conner avatar
Jim Conner

the template

Jim Conner avatar
Jim Conner

err: failed to read ../../common/environments.yaml: reading document at index 1: yaml: line 4: mapping values are not allowed in this context

Jim Conner avatar
Jim Conner

looks like a syntax thing likely

mumoshu avatar
mumoshu

ah, line 4 does have wrong indentation

mumoshu avatar
mumoshu
templates:
  versions: &versions
    semver: 0.0.1
      name: somename
      # Versions here reflect the chart version, NOT the app version.
      ambassador:
        version: 6.6.2
        namespace: ambassador

i wrote this but this should be

Jim Conner avatar
Jim Conner

I see it

Jim Conner avatar
Jim Conner

it’s 2+ and it needs to be 2-

mumoshu avatar
mumoshu
templates:
  versions: &versions
    semver: 0.0.1
    name: somename
    # Versions here reflect the chart version, NOT the app version.
    ambassador:
      version: 6.6.2
      namespace: ambassador
1
Jim Conner avatar
Jim Conner

OK, so getting closer. If you are OK with helping me fix this one thing then I’ll stop bugging you for the rest of the day…lol (I need to go to bed) and then I’ll try and fix up the rest on my own and if I get stuck, I’ll let you know and hopefully you can help another time?

mumoshu avatar
mumoshu

yeah sure! but expect i won’t be as responsive as today tomorrow. today was my day off.

Jim Conner avatar
Jim Conner

doh! maybe I’ll just stay up…lol

Jim Conner avatar
Jim Conner

well, let’s see

Jim Conner avatar
Jim Conner

There’s something I need to do to fix the other helmfile.yaml files for each chart which rely on environments.yaml in common

err: failed to read ../../common/environments.yaml: reading document at index 1: yaml: unmarshal errors:
  line 4: field semver not found in type state.TemplateSpec
  line 7: field ambassador not found in type state.TemplateSpec
  line 10: field certmanager not found in type state.TemplateSpec
  line 13: field blscauxclusterissuer not found in type state.TemplateSpec
  line 16: field externaldns not found in type state.TemplateSpec
  line 19: field nginxingress not found in type state.TemplateSpec
  line 22: field postgres not found in type state.TemplateSpec
  line 25: field oauth2proxy not found in type state.TemplateSpec
  line 28: field dex not found in type state.TemplateSpec
  line 31: field vaultoperator not found in type state.TemplateSpec
  line 34: field vaultsecretswebhook not found in type state.TemplateSpec
  line 38: field aws_nlb_with_tls_termination_at_lb not found in type state.TemplateSpec
  line 39: field arm64_support not found in type state.TemplateSpec
  line 40: field domain_name not found in type state.TemplateSpec
  line 41: field cluster_id not found in type state.TemplateSpec
changing working directory back to "/project/helmfile-project/helmfile.d/generic/01a-network-and-proxies"
changing working directory back to "/project/helmfile-project/helmfile.d"
changing working directory back to "/project/helmfile-project/helmfile/envs/dev/atreus"
in ./helmfile.yaml: in .helmfiles[0]: in ../../../../helmfile.d/helmfile.yaml: in .helmfiles[0]: in generic/01a-network-and-proxies/helmfile.yaml: in .helmfiles[0]: in ambassador/helmfile.yaml: failed to read ../../common/environments.yaml: reading document at index 1: yaml: unmarshal errors:
  line 4: field semver not found in type state.TemplateSpec
  line 7: field ambassador not found in type state.TemplateSpec
  line 10: field certmanager not found in type state.TemplateSpec
  line 13: field blscauxclusterissuer not found in type state.TemplateSpec
  line 16: field externaldns not found in type state.TemplateSpec
  line 19: field nginxingress not found in type state.TemplateSpec
  line 22: field postgres not found in type state.TemplateSpec
  line 25: field oauth2proxy not found in type state.TemplateSpec
  line 28: field dex not found in type state.TemplateSpec
  line 31: field vaultoperator not found in type state.TemplateSpec
  line 34: field vaultsecretswebhook not found in type state.TemplateSpec
  line 38: field aws_nlb_with_tls_termination_at_lb not found in type state.TemplateSpec
  line 39: field arm64_support not found in type state.TemplateSpec
  line 40: field domain_name not found in type state.TemplateSpec
  line 41: field cluster_id not found in type state.TemplateSpec

this looks like an indentation issue and I think I probably need to fix the reference to match what we did with the other helmfile.yaml files.

mumoshu avatar
mumoshu

it will be helpful if you could push the latest snapshot of your whole setup before asking questions that would reduce forth-and-back

mumoshu avatar
mumoshu

seems to

Jim Conner avatar
Jim Conner

sure! do you need that now or just in case I need more help tomorrow?

Jim Conner avatar
Jim Conner

btw, sincerely appreciate your assistance. Very kind!

mumoshu avatar
mumoshu

just update your git repo immediately before you add another question so that i can try to replicate your issue and think concretely

mumoshu avatar
mumoshu

state.TemplateSpec is the underlying go struct that maps to templates in helmfile.yaml and in your case environments.yaml

Jim Conner avatar
Jim Conner

cool! will do. Due to the nature of the company policies, I just need to sync between the public repo and my private repo here and then commit/push which will take a few mins.

Jim Conner avatar
Jim Conner

ah! ok, that’s good context.

mumoshu avatar
mumoshu

ah ok turns out we were only able to define some fields under templates.foo

mumoshu avatar
mumoshu

well then we can’t leverage templates that way..

Jim Conner avatar
Jim Conner

looks like I’ll need to add helmdefaults to the template

Jim Conner avatar
Jim Conner

oh, we can’t?

mumoshu avatar
mumoshu

helmDefaults contains default values for releases[] so it might be useful elsewhere, but i think it doesn’t work for this

Jim Conner avatar
Jim Conner

ok

mumoshu avatar
mumoshu

well trying to come up with a workaround..

Jim Conner avatar
Jim Conner

k

Jim Conner avatar
Jim Conner

so I guess I’m not understanding what’s wrong with the template method we’re working on right now.

Jim Conner avatar
Jim Conner

the helmfile.yaml files I updated a while ago were just the ones in the middle

Jim Conner avatar
Jim Conner

I didn’t update any of the chart helmfile.yaml files yet

mumoshu avatar
mumoshu

how about this…

environments:
  template:
    values:
    - versions: &versions
        semver: 0.0.1
        name: somename
        # Versions here reflect the chart version, NOT the app version.
        ambassador:
          version: 6.6.2
          namespace: ambassador
        certmanager:
          version: 1.0.4
          namespace: certmanager
        blscauxclusterissuer:
          version: 0.1.0
          namespace: certmanager
        externaldns:
          version: v20210203-v0.7.6-28-g44288212-arm64v8
          namespace: externaldns
        nginxingress:
          version: 3.15.2
          namespace: ingress-nginx
        postgres:
          version: x.x.x
          namespace: grafana
        oauth2proxy:
          version: 3.2.5
          namespace: auth-system
        dex:
          version: 2.15.2
          namespace: dex
        vaultoperator:
          version: 1.8.1
          namespace: vault
        vaultsecretswebhook:
          version: 1.8.2
          namespace: secrets-webhook
      values: *values
        aws_nlb_with_tls_termination_at_lb: {{ .Values | get "aws_nlb_with_tls_termination_at_lb" false }}
        arm64_support: {{ .Values | get "arm64_support" false }}
        domain_name: {{ .Values | get "domain_name" (env "DOMAIN_NAME") }}
        cluster_id: {{ .Values | get "cluster_id" (env "CLUSTER_ID") }}

---

environments:
  default:
    values:
    - *versions
    - *values
  production:
    values:
    - *versions
    - *values
Jim Conner avatar
Jim Conner

lemme give it a shot

mumoshu avatar
mumoshu

ah probably this doesnt work due to yaml anchors not persisnt across ---

mumoshu avatar
mumoshu

what if you removed ---

mumoshu avatar
mumoshu
environments:
  template:
    values:
    - versions: &versions
        semver: 0.0.1
        name: somename
        # Versions here reflect the chart version, NOT the app version.
        ambassador:
          version: 6.6.2
          namespace: ambassador
        certmanager:
          version: 1.0.4
          namespace: certmanager
        blscauxclusterissuer:
          version: 0.1.0
          namespace: certmanager
        externaldns:
          version: v20210203-v0.7.6-28-g44288212-arm64v8
          namespace: externaldns
        nginxingress:
          version: 3.15.2
          namespace: ingress-nginx
        postgres:
          version: x.x.x
          namespace: grafana
        oauth2proxy:
          version: 3.2.5
          namespace: auth-system
        dex:
          version: 2.15.2
          namespace: dex
        vaultoperator:
          version: 1.8.1
          namespace: vault
        vaultsecretswebhook:
          version: 1.8.2
          namespace: secrets-webhook
      values: *values
        aws_nlb_with_tls_termination_at_lb: {{ .Values | get "aws_nlb_with_tls_termination_at_lb" false }}
        arm64_support: {{ .Values | get "arm64_support" false }}
        domain_name: {{ .Values | get "domain_name" (env "DOMAIN_NAME") }}
        cluster_id: {{ .Values | get "cluster_id" (env "CLUSTER_ID") }}

environments:
  default:
    values:
    - *versions
    - *values
  production:
    values:
    - *versions
    - *values
mumoshu avatar
mumoshu

removing --- isn’t usually a good idea as this gudes you into the deep sea of helmfile “double-rendering” hack

Jim Conner avatar
Jim Conner

yeah, it seems that one needs to understand when and when not to use ---

mumoshu avatar
mumoshu

probably you’d better not to think about it and just keep adding --- for now

1
Jim Conner avatar
Jim Conner

ok, so quick question… I’m trying to set up the ambassador/helmfile.yaml to use the passed .Values and I’m not 100% sure how to properly do that. The old helmfile.yaml for ambassador was:

---
bases:
  - ../../common/environments.yaml
  - ../../common/repos.yaml
  - ../../common/helmdefaults.yaml

---
releases:
  - name: ambassador
    namespace: {{ .Values.ambassador.namespace }}
    createNamespace: true
    labels:
      app: ambassador
    chart: datawire/ambassador
    version: {{ .Values.ambassador.version }}
    values:
      - values.yaml.gotmpl
Jim Conner avatar
Jim Conner

the one I just tried, which failed was:

---
bases:
  - ../../common/repos.yaml
  - ../../common/helmdefaults.yaml
---
values:
  - {{ .Values | toYaml | nindent 8 }}

releases:
  - name: ambassador
    namespace: {{ .Values.ambassador.namespace }}
    createNamespace: true
    labels:
      app: ambassador
    chart: datawire/ambassador
    version: {{ .Values.ambassador.version }}
    values:
      - values.yaml.gotmpl
Jim Conner avatar
Jim Conner

wrong indentation there

mumoshu avatar
mumoshu

the last resort- try this if removing --- didn’t work

environments:
  {{ .Environment.Name }}:
    values:
    - versions:
        semver: 0.0.1
        name: somename
        # Versions here reflect the chart version, NOT the app version.
        ambassador:
          version: 6.6.2
          namespace: ambassador
        certmanager:
          version: 1.0.4
          namespace: certmanager
        blscauxclusterissuer:
          version: 0.1.0
          namespace: certmanager
        externaldns:
          version: v20210203-v0.7.6-28-g44288212-arm64v8
          namespace: externaldns
        nginxingress:
          version: 3.15.2
          namespace: ingress-nginx
        postgres:
          version: x.x.x
          namespace: grafana
        oauth2proxy:
          version: 3.2.5
          namespace: auth-system
        dex:
          version: 2.15.2
          namespace: dex
        vaultoperator:
          version: 1.8.1
          namespace: vault
        vaultsecretswebhook:
          version: 1.8.2
          namespace: secrets-webhook
      values:
        aws_nlb_with_tls_termination_at_lb: {{ .Values | get "aws_nlb_with_tls_termination_at_lb" false }}
        arm64_support: {{ .Values | get "arm64_support" false }}
        domain_name: {{ .Values | get "domain_name" (env "DOMAIN_NAME") }}
        cluster_id: {{ .Values | get "cluster_id" (env "CLUSTER_ID") }}

you may need to rename environments.yaml to environments.yaml.gotmpl. i forgot full details but try renaming the file if it failed to render go template at all

Jim Conner avatar
Jim Conner

oh! yeah, that might be required

mumoshu avatar
mumoshu
values:
  - {{ .Values | toYaml | nindent 8 }}

seems like a invalid indentation

mumoshu avatar
mumoshu

should be 4

mumoshu avatar
mumoshu

if you removed ` - ../../common/environments.yaml ` from the ambassador helmfile.yaml, it should end up like that

mumoshu avatar
mumoshu

because you’ve tried todefine ambassador.namespace in the versions, which was to be loaded via environments.yaml you just removed

Jim Conner avatar
Jim Conner

but that should be passed in using the intermediate .Values now, though shouldn’t it?

mumoshu avatar
mumoshu

Intermediate .Values doesn’t contain values defined in versions neither, as you’ve omitted it from bases

Jim Conner avatar
Jim Conner

hmm.

Jim Conner avatar
Jim Conner

ok

mumoshu avatar
mumoshu
values:
  - {{ .Values | toYaml | nindent 8 }}

is noop.

mumoshu avatar
mumoshu

you are basically assigning Values to Values

Jim Conner avatar
Jim Conner

ah, yes. I was originally going to try just:

---
{{ .Values | toYaml }}
mumoshu avatar
mumoshu

in helmfiel.yaml?

Jim Conner avatar
Jim Conner

yeah

Jim Conner avatar
Jim Conner

(that wasn’t the whole thing…just the values portion)

mumoshu avatar
mumoshu

that would break helmfile.yaml, as it can’t contain arbitary key values like that

Jim Conner avatar
Jim Conner

oh

Jim Conner avatar
Jim Conner

I could put it under bases:?

mumoshu avatar
mumoshu

you can only have artibrary key values under values of some kind. like releases[].values or environments[].values

mumoshu avatar
mumoshu

it’s not a free-form yaml

Jim Conner avatar
Jim Conner

aahhh. OK, that’s good to know.

Jim Conner avatar
Jim Conner

so let me do this then: … (one min)

mumoshu avatar
mumoshu

nope. bases has its own schema

Jim Conner avatar
Jim Conner
---
bases:
  - ../../common/repos.yaml
  - ../../common/helmdefaults.yaml

#config.yaml
---
environments:
  default:
    values:
    - {{ .Values | toYaml | nindent 8 }}

---
releases:
  - name: ambassador
    namespace: {{ .Values.ambassador.namespace }}
    createNamespace: true
    labels:
      app: ambassador
    chart: datawire/ambassador
    version: {{ .Values.ambassador.version }}
    values:
      - values.yaml.gotmpl

?

mumoshu avatar
mumoshu

seems ok, except that nindent should be 6

Jim Conner avatar
Jim Conner

or I just need to push the indentation out 2 to match everything else.

Jim Conner avatar
Jim Conner

dang flexibleness of yaml

mumoshu avatar
mumoshu

maybe..?

mumoshu avatar
mumoshu
09:06:40 AM

i thought you usually align yaml dict key values at the level indicated by the end of the selection shown in the below picture

mumoshu avatar
mumoshu

which is 6

Jim Conner avatar
Jim Conner

yaml allows either or

Jim Conner avatar
Jim Conner

it can be aligned or pushed out

mumoshu avatar
mumoshu

wow really!

Jim Conner avatar
Jim Conner

I tend to push it out because I’m old school

Jim Conner avatar
Jim Conner

yep

Jim Conner avatar
Jim Conner

the new school way seems to be aligned, though

Jim Conner avatar
Jim Conner

but make two files using both pragmas and yamllint them

Jim Conner avatar
Jim Conner

it will work on both

mumoshu avatar
mumoshu

oh well but i still don’t get it

mumoshu avatar
mumoshu

lets say you had tis values

foo: bar
bar: baz
mumoshu avatar
mumoshu
environments:
  default:
    values:
    - {{ .Values | toYaml | nindent 8 }}

renders to

environments:
  default:
    values:
    - foo: bar
        bar: baz
Jim Conner avatar
Jim Conner

oh, maybe I should qualify what I’m talking about…I’m talking only for lists.

mumoshu avatar
mumoshu

this does really work?

mumoshu avatar
mumoshu

ah gotcha

mumoshu avatar
mumoshu

then you should still say nindent 6 there

mumoshu avatar
mumoshu
environments:
  default:
    values:
    - 
{{ .Values | toYaml | indent 6 }}

if it’s indent rather than nindent this should work

mumoshu avatar
mumoshu

or even this

environments:
  default:
    values:
    - 
{{ .Values | toYaml | indent 8 }}
Jim Conner avatar
Jim Conner

what’s the difference in go templating between nindent and indent btw? I haven’t looked up the docs on that yet.

mumoshu avatar
mumoshu

fixed wrong indentations in indent examples

mumoshu avatar
mumoshu

yep that’s very important

mumoshu avatar
mumoshu
environments:
  default:
    values:
    - {{ .Values | toYaml | nindent 8 }}

renders to

environments:
  default:
    values:
    - foo: bar
        bar: baz
mumoshu avatar
mumoshu

this is very common trick in writing not only helmfile templates but also helm templates so i’d highly recommend getting used to this

1
Jim Conner avatar
Jim Conner

the things so far

Jim Conner avatar
Jim Conner

note that I renamed environments.yaml to environments.yaml.gotmpl to respect helmfile to render go templatized file

Jim Conner avatar
Jim Conner

but I’m not sure that was enough…I feel like I need to tell helmfile to actually use the gotmpl file.

mumoshu avatar
mumoshu

yes

Jim Conner avatar
Jim Conner

but where?

mumoshu avatar
mumoshu

you need to add that back to bases

Jim Conner avatar
Jim Conner

oh

mumoshu avatar
mumoshu

i thought you just removed it from bases earlier?

Jim Conner avatar
Jim Conner

which helmfile.yaml file ? All of them?

Jim Conner avatar
Jim Conner

I thought you said it needed to be removed.

Jim Conner avatar
Jim Conner

I might have misunderstood you though

mumoshu avatar
mumoshu

well no

Jim Conner avatar
Jim Conner

because we were trying to solve the directory path issue…

mumoshu avatar
mumoshu

the directory path issue would have been resolved when we embedded versions.yaml and values.yaml.gotmpl into environments.yaml.gotmpl

mumoshu avatar
mumoshu

neither environments.yaml nor environments.yaml.gotmpl is going to be included in bases or loaded before rendering helmfile.yaml , except you explicitly add it to bases

mumoshu avatar
mumoshu

you need to be explicit about what values to load, what bases to use, what values to be passed to sub-helmfile

Jim Conner avatar
Jim Conner

ok. so that last thing is pretty good info. That resonates with me…and I’ll need to think about my project in those terms now methinks.

1
Jim Conner avatar
Jim Conner

that’s actually something that should probably be added to the documentation like…very. clearly.

Jim Conner avatar
Jim Conner

let me make I am understanding what you’re saying, too…

mumoshu avatar
mumoshu

definitely. the point is that i know too much about helmfile already and can’t imagine what parts are missing to what levels of people

Jim Conner avatar
Jim Conner

for a project, one must be explicit (for each helmfile.yaml) about what values to load, what bases to load and then when layering, what values to pass “down the chain”

mumoshu avatar
mumoshu

so if you could contribute doc addition/fix based on your experience, that would be awesome

Jim Conner avatar
Jim Conner

absolutely!

mumoshu avatar
mumoshu

yep

Jim Conner avatar
Jim Conner

ok cool. I need to grok that concept then…

1
Jim Conner avatar
Jim Conner

and put it in terms of this project

Jim Conner avatar
Jim Conner

so let me see if I understand how we’ve done things so far in terms of that concept and this project

Jim Conner avatar
Jim Conner

I envisioned a project where I could deploy dex via helmfile from within its own directory and all it needs to know is where to find the bases, environments values (if any) and any additional values (for this project in this context, that would be passed in via env vars)…

Jim Conner avatar
Jim Conner

brb…2 mins

mumoshu avatar
mumoshu

i’ll be afk for dinner but keep posting

mumoshu avatar
mumoshu

oh, and I think it worth a new slack thread now

Jim Conner avatar
Jim Conner

lol. OK

mumoshu avatar
mumoshu

its getting really long and takes a bit of time to fully load, and im also afraid if every update to this thread is being notified to anyone commented on this thread only in very beginning

jedineeper avatar
jedineeper

It’s been very informative actually :)

2
2
Jim Conner avatar
Jim Conner

@jedineeper is the concept I’m trying to accomplish making sense?

Jim Conner avatar
Jim Conner

just want to make sure I’m not daft

Jim Conner avatar
Jim Conner

cool, I created a new thread that we can move this to if you want.

jedineeper avatar
jedineeper

This started with me misunderstanding the use of environments so some of it is beyond the scale of my understanding so I’m not a good judge :)

2021-04-15

gonzalez.mariano.gabr avatar
gonzalez.mariano.gabr

hey guys, im trying to add helmfile to a docker image (dockerfile) and not sure if https://github.com/roboll/helmfile/releases/download/v0.138.7/helmfile_linux_amd64 is the actual binary?

Antoine Taillefer avatar
Antoine Taillefer

Yes it is

1
Balazs Varga avatar
Balazs Varga

can we set a “security” to avoid misdeploys? I mean if I accidentally deploy from a wrong branch to a cluster. ? a warning message or interactive promt… and not -i because we can forget it. in default config maybe?

1
Jim Conner avatar
Jim Conner

I have the following directory structure where helmfile.d has the helmfile manifests. The helmfile directory tree has my environments and this is from where I want to invoke environmental helmfile runs:

├── helmfile
│   ├── envs
│   │   ├── dev
│   │   │   └── cluster-a
│   │   ├── preprod
│   │   └── production
│   └── shlib
└── helmfile.d
    └── generic
        ├── 01a-network-and-proxies
        │   ├── ambassador
        │   ├── external-dns
        │   └── ingress-nginx
        ├── 01b-secrets-management
        │   ├── certmanager
        │   ├── dex
        │   ├── oauth2-proxy
        │   │   └── secrets
        │   ├── vault-operator
        │   │   └── secrets
        │   └── vault-secrets-webhook
        └── common

In cluster-a there’s a helmfile.yaml and config.yaml:

❯ \cat helmfile.yaml 
environments:
  default:
    values:
      - config.yaml

helmfiles:
  - "../../../../helmfiles.d/*"

The concept is that any specific configuration values for cluster-a are specified in the config.yaml inside the cluster-a directory. However, when I test invoking helmfile from this path, I get the error:

envvals_loader: loaded config.yaml:map[values:map[keyname:foobar]]
no matches for path: ../../../../helmfiles.d/*
merged environment: &{default map[values:map[keyname:foobar]] map[]}
helm:XVlBz> v3.4.1+gc4e7485
0 release(s) found in helmfile.yaml

err: no releases found that matches specified selector() and environment(default), in any helmfile

I don’t understand why.

Jim Conner avatar
Jim Conner

one thing to note is there’s a helmfile.yaml in each directory which specifies helmfiles property:

.(helmfile.d)
├── generic
│   ├── 01a-network-and-proxies
│   │   └── helmfile.yaml
│   ├── 01b-secrets-management
│   │   └── helmfile.yaml
│   └── helmfile.yaml
└── helmfile.yaml
Jim Conner avatar
Jim Conner

in helmfile.d/helmfile.yaml

---
helmfiles:
  - "generic/*"
Jim Conner avatar
Jim Conner

in generic/helmfile.yaml

helmfiles:
  - "*/*.yaml"
Jim Conner avatar
Jim Conner

won’t the attribute in helmfiles: cause helmfile to “walk up the chain” so-to-speak?

Jim Conner avatar
Jim Conner

sigh, figured it out.

Jim Conner avatar
Jim Conner

for posterity sake, the helmfile.yaml in the cluster-a directory had a misspelled directory name: - "../../../../helmfiles.d/*" (note helmfiles.d vs helmfile.d)

1
Jim Conner avatar
Jim Conner
Environment values not being used in nested states · Issue #1045 · roboll/helmfile

I&#39;ve been trying to use environments in my main helmfile and have multiple sub helmfiles in my releases folder. I wanted all the sub helmfiles to pick up whatever the value I defined in a speci…

Jim Conner avatar
Jim Conner

sigh. I’m really struggling to understand certain points in this documentation. I keep getting blocked on aspects of how to properly use helmfile and I either can’t find documentation to meet my needs or the documentation I think will help me either doesn’t make sense or is not as verbose as necessary to understand the full aspect of what the documentation is trying to point out.

For instance, I’m blocked on certain aspects of this project (tickets and questions are submitted) and I’m moving on to other aspects of this project where I’m not blocked. One of those aspects is the ability to logically select charts. It doesn’t work the way I’m trying to implement so I’m reading the documentation, which doesn’t make sense to me. I don’t understand the context of inheritance with respect to sub-helmfiles and the inherited properties. That point is not well spelled out. What do you mean by inherited in context? Everything? Certain properties? The selectors? I don’t understand.

The use-case I’m trying to solve is to simply use a selector to run helmfiles only identified by that selector. Everything will run exactly the same as if I didn’t use any selector except ONLY the helmfiles specified by that selector (or a negated selector) would run. The details of how that works should not be something the end-user is concerned about if it’s clear-cut as what I was hoping would be the case. I’m simply not understanding, which I think is because the documentation is a little too sparse, but there is a lot of documentation. I can’t put my finger on the problem. It might be me, which I can concede. It shouldn’t be this complicated to understand what software can do and I keep running into roadblocks here. I want to use helmfile because I believe it does what we need it to do. Some assistance would be greatly appreciated.

1
mumoshu avatar
mumoshu


inherited in context?
Hey! Helmfile doesn’t inherit anything to sub-helmfiles. Inheritance of helmfile environments and values usually happen only between the parent helmfile.yaml and bases. Does that clarify it a bit?

mumoshu avatar
mumoshu

That’s also the foundational thing under my comment on your previous question https://github.com/roboll/helmfile/issues/1045#issuecomment-821910447

Environment values not being used in nested states · Issue #1045 · roboll/helmfile

I&#39;ve been trying to use environments in my main helmfile and have multiple sub helmfiles in my releases folder. I wanted all the sub helmfiles to pick up whatever the value I defined in a speci…

mumoshu avatar
mumoshu

Selectors aren’t inherited. It’s just that the user specifies the selector and helmfile uses it to filter releases across all the involved helmfie.yaml.

mumoshu avatar
mumoshu

Environment and environment values aren’t inherited. That’s why this doesn’t work:

environments:
  default:
    values:
      - config.yaml

---
helmfiles:
  - "../../../../helmfile.d/"
mumoshu avatar
mumoshu

The documentation may be outdated or incomplete or simply incorrect, seeing your frustration.

Did you find any specific documentation saying that environments and values are inherited down to sub-helmfiles? If so, we’d definitely need to fix it.

mumoshu avatar
mumoshu

Does the documentation say if anything’s inherited to sub-helfmiles or bases?

I thought I even didn’t explicitly say in the documentation that the bases inherit parent’s environments and values, as trying to depend on the parent helmfile.yaml always sounded like a bad idea to me.

mumoshu avatar
mumoshu


Everything will run exactly the same as if I didn’t use any selector except  ONLY the helmfiles specified by that selector (or a negated selector) would  run.
--selector(-l) works that way. To do so, Helmfile requires you to make each sub-helmfile independently consumable. That’s why environments arent inherited down to sub-helmfiles, and the basis of comments starting from https://sweetops.slack.com/archives/CE5NGCB9Q/p1618707461115400?thread_ts=1618551534.114100&cid=CE5NGCB9Q


inherited in context?
Hey! Helmfile doesn’t inherit anything to sub-helmfiles. Inheritance of helmfile environments and values usually happen only between the parent helmfile.yaml and bases. Does that clarify it a bit?

mumoshu avatar
mumoshu

Would there be anything I’ve not yet answered?

vicken avatar
vicken

Hey @mumoshu also have a selector question, are there maybe reserved words helmfile doesn’t allow to be label selectors?

Consider a simple example with 3 labels in 2 releases:

repositories:
- name: datawire
  url: <https://www.getambassador.io>
- name: incubator
  url: <https://charts.helm.sh/incubator>

releases:
- name: ambassador
  namespace: ambassador
  labels:
    chart: ambassador
    namespace: ambassador
    foo: bar
  chart: datawire/ambassador
  version: 6.6.2

- name: raw
  namespace: ambassador
  labels:
    chart: ambassador
    namespace: ambassador
    foo: bar
  chart: incubator/raw
  version: 0.2.5

If I run helmfile -l namespace=ambassador diff both releases are selected in the diff

If I run helmfile -l foo=bar diff both releases are selected in the diff

However, if i run helmfile -l chart=ambassador diff only the first release is selected in the diff. This is on v0.138.6.

mumoshu avatar
mumoshu

Yes. As far as I remeber, chart and name are reserved, each has the respective value of the release

1
vicken avatar
vicken

hah, thank you! it was confusing me for the longest time (why some releases weren’t being applied)

1
mumoshu avatar
mumoshu

Ah, that makes sense! Probably we’d better enhance the docs to note about reserved labels and make it a validation error when you’ve tried to override a reserved label

mumoshu avatar
mumoshu

Would you mind opening issues?

vicken avatar
vicken

Yes! sure! That would be much better than silently half-working

1
1
mumoshu avatar
mumoshu

Much appreciated!

1
Jim Conner avatar
Jim Conner

Thank you for the response @mumoshu! Today is my Sunday and I have other obligations today but I will answer your questions here tomorrow and also your comment for my GH issue! Very much appreciated!

cool-doge1
Jim Conner avatar
Jim Conner

@mumoshu I updated my comments in the GH issue

Environment values not being used in nested states · Issue #1045 · roboll/helmfile

I&#39;ve been trying to use environments in my main helmfile and have multiple sub helmfiles in my releases folder. I wanted all the sub helmfiles to pick up whatever the value I defined in a speci…

2021-04-14

2021-04-13

Jim Conner avatar
Jim Conner

Good evening everyone. I’m trying to learn helmfile and am struggling to figure out why I’m getting the following errors. My directory structure is:

.(helmfile.d)                                                                                                                                                                                                                                                                                                                                                        
├── generic 
|   ├── helmfile.yaml
│   ├── 01-secrets-management
│   │   ├── dex
│   │   │   ├── helmfile.yaml                                                                                                                                                                                                                                                                                                                            
│   │   │   └── values.yaml                                                                                                                                                                                                                                                                                                                              
│   │   ├── helmfile.yaml                                                                                                                                                                                                                                                                                                                                
│   │   ├── oauth2-proxy                                                                                                                                                                                                                                                                                                                                 
│   │   │   ├── arm64-values.yaml                                                                                                                                                                                                                                                                                                                        
│   │   │   ├── secrets                                                                                                                                                                                                                                                                                                                                  
│   │   │   ├── values.yaml                                                                                                                                                                                                                                                                                                                              
│   │   │   └── wait_for_endpoint.sh                                                                                                                                                                                                                                                                                                                     
│   │   ├── vault-operator                                                                                                                                                                                                                                                                                                                               
│   │   │   ├── helmfile.yaml                                                                                                                                                                                                                                                                                                                            
│   │   │   ├── secrets                                                                                                                                                                                                                                                                                                                                  
│   │   │   │   └── vault-cr-secret-dec.yaml                                                                                                                                                                                                                                                                                                             
│   │   │   └── values.yaml                                                                                                                                                                                                                                                                                                                              
│   │   └── vault-secrets-webhook                                                                                                                                                                                                                                                                                                                        
│   │       └── values.yaml
|   └── common
│       ├── config.yaml
│       ├── environments.yaml
│       ├── helmdefaults.yaml
│       └── repos.yaml
└── helmfile.yaml

The helmfile.yaml in . is:

---
helmfiles:
  - "*/*"

The helmfile.yaml in generic is:

helmfiles:
  - "*"

The helmfile.yaml in 01-secrets-management is:

bases:
  - ../common/environments.yaml
  - ../common/repos.yaml
  - ../common/helmdefaults.yaml
Jim Conner avatar
Jim Conner

finally:

Jim Conner avatar
Jim Conner

The helmfile.yaml in dex is:

bases:
  - ../../common/environments.yaml
  - ../../common/repos.yaml
  - ../../common/helmdefaults.yaml

## ************************************
## Start of DEX installation
## ************************************
releases:
  - name: dex
    namespace: {{ .Values.dex.namespace }}
    createNamespace: true
    labels:
      tier: "secrets-management"
      app: dex
    chart: repo/helm-charts
    version: {{ .Values.dex.version }}
    values:
      - values.yaml

I’m running helmfile -e default --log-level debug lint from the dex directory. I get the following output/error:

Jim Conner avatar
Jim Conner
Jim Conner avatar
Jim Conner

The first error means that it’s not finding/using the common bases. I’m not understanding why. The last error is seemingly cascading from the first.

Jim Conner avatar
Jim Conner

perhaps I’ve misread/misunderstood the documentation on bases but re-reading it it seems that helmfile is trying to render helmfile.yaml BEFORE rendering the other layers. Is that correct?

Jim Conner avatar
Jim Conner

so if that’s the case then that means I can’t use anything defined in environments because it’s not yet read/rendered. Correct?

Jim Conner avatar
Jim Conner

seems kinda useless to me if that’s the case.

Jim Conner avatar
Jim Conner

FTR, if this is how bases works, it seems that it’s not very useful in this case and would be more useful in top-level helmfile.yaml files.

mumoshu avatar
mumoshu

We can’t resolve that chicken-and-egg problem automagically. I may be still missing something but at glance what you wanted seems like

mumoshu avatar
mumoshu
bases:
  - ../../common/environments.yaml
  - ../../common/repos.yaml
  - ../../common/helmdefaults.yaml

---

## ************************************
## Start of DEX installation
## ************************************
releases:
  - name: dex
    namespace: {{ .Values.dex.namespace }}
    createNamespace: true
    labels:
      tier: "secrets-management"
      app: dex
    chart: repo/helm-charts
    version: {{ .Values.dex.version }}
    values:
      - values.yaml
1
mumoshu avatar
mumoshu

Notice --- so that the first part is rendered as a template to produce a YAML structure that includes bases. Bases should be loaded and the env values are loaded before rendering the latter part as a template

Jim Conner avatar
Jim Conner

ahh! Lemme give that a try real fast.

Jim Conner avatar
Jim Conner

nice!

Jim Conner avatar
Jim Conner

not outta the woods yet…but that got me past that roadblock

Jim Conner avatar
Jim Conner

how much time you got @mumoshu? Is it late where you are? I have questions, thoughts, and possibly ideas.

Jim Conner avatar
Jim Conner

and I’m still very new to helmfile so I’m struggling to get a good working set of helmfiles here.

Jim Conner avatar
Jim Conner

The error has changed…but I think this is good as it seems that the bases files are getting read now. These are the new errors.

Jim Conner avatar
Jim Conner

This looks like possibly over-lapping values between my config.yaml or environments.yaml files.

Jim Conner avatar
Jim Conner

actually, this looks like it’s reading environments.yaml which in turn references config.yaml within the directory where environments.yaml resides. This might be a referential problem with respect to … something.

mumoshu avatar
mumoshu

files referenced from within a sub-helmfile is relative to the sub-helmfile, to make the sub-helmfile portable(not dependent on the parent-helmfile)

Jim Conner avatar
Jim Conner
roboll/helmfile

Deploy Kubernetes Helm Charts. Contribute to roboll/helmfile development by creating an account on GitHub.

Jim Conner avatar
Jim Conner

so
Relative paths referenced in the Helmfile manifest itself are relative to that manifest
does this mean in sub-helmfile.yaml files that the path references in those files are relative to them or the relative to the top-level helmfile.yaml?

Jim Conner avatar
Jim Conner

I’m trying to use paths in sub-files as relative to the sub-files.

Jim Conner avatar
Jim Conner

wow.

Jim Conner avatar
Jim Conner

that’s the case.

Jim Conner avatar
Jim Conner

@mumoshu that’s a super-confusing thing methinks.

Jim Conner avatar
Jim Conner

it’s also not very portable imo.

mumoshu avatar
mumoshu

why?(i’m not yet sure if I fully understand your usecase

mumoshu avatar
mumoshu

files referenced from within a sub-helmfile is relative to the sub-helmfile, to make the sub-helmfile portable(not dependent on the parent-helmfile)

mumoshu avatar
mumoshu

so to me the current behaviour makes sub-helmfiles portable, which is nice

mumoshu avatar
mumoshu

i need to learn something- what would be ideal behaviour and how would you rewrite your yamls with that ideal behaviour?

Jim Conner avatar
Jim Conner

well, maybe I’m doing something wrong. So lemme articulate what I’m doing first and then I’ll attempt to explain what and why I’m doing it.

Jim Conner avatar
Jim Conner

first, I’m still trying to read through the docs and learn helmfile. The docs are good, but not very simple to follow referentially.

Jim Conner avatar
Jim Conner

so, all the stuff I showed here is relevant

Good evening everyone. I’m trying to learn helmfile and am struggling to figure out why I’m getting the following errors. My directory structure is:

.(helmfile.d)                                                                                                                                                                                                                                                                                                                                                        
├── generic 
|   ├── helmfile.yaml
│   ├── 01-secrets-management
│   │   ├── dex
│   │   │   ├── helmfile.yaml                                                                                                                                                                                                                                                                                                                            
│   │   │   └── values.yaml                                                                                                                                                                                                                                                                                                                              
│   │   ├── helmfile.yaml                                                                                                                                                                                                                                                                                                                                
│   │   ├── oauth2-proxy                                                                                                                                                                                                                                                                                                                                 
│   │   │   ├── arm64-values.yaml                                                                                                                                                                                                                                                                                                                        
│   │   │   ├── secrets                                                                                                                                                                                                                                                                                                                                  
│   │   │   ├── values.yaml                                                                                                                                                                                                                                                                                                                              
│   │   │   └── wait_for_endpoint.sh                                                                                                                                                                                                                                                                                                                     
│   │   ├── vault-operator                                                                                                                                                                                                                                                                                                                               
│   │   │   ├── helmfile.yaml                                                                                                                                                                                                                                                                                                                            
│   │   │   ├── secrets                                                                                                                                                                                                                                                                                                                                  
│   │   │   │   └── vault-cr-secret-dec.yaml                                                                                                                                                                                                                                                                                                             
│   │   │   └── values.yaml                                                                                                                                                                                                                                                                                                                              
│   │   └── vault-secrets-webhook                                                                                                                                                                                                                                                                                                                        
│   │       └── values.yaml
|   └── common
│       ├── config.yaml
│       ├── environments.yaml
│       ├── helmdefaults.yaml
│       └── repos.yaml
└── helmfile.yaml

The helmfile.yaml in . is:

---
helmfiles:
  - "*/*"

The helmfile.yaml in generic is:

helmfiles:
  - "*"

The helmfile.yaml in 01-secrets-management is:

bases:
  - ../common/environments.yaml
  - ../common/repos.yaml
  - ../common/helmdefaults.yaml
Jim Conner avatar
Jim Conner

and the way I’m approaching creating this is by trying to get helmfile to work in the dex release first. The ultimate goal, however, is to run helmfile from a different directory (one that references the environment I want to deploy).

Jim Conner avatar
Jim Conner

so, my thinking is, get it working first in dex. Then get helmfile to work one directory up from dex (01-secrets-management)

Jim Conner avatar
Jim Conner

all the way up to generic.

Jim Conner avatar
Jim Conner

I’m trying to keep this DRY…

Jim Conner avatar
Jim Conner

I’m not sure any of this makes sense. It’s difficult to articulate.

mumoshu avatar
mumoshu

Sounds good so far

Jim Conner avatar
Jim Conner

oh good

Jim Conner avatar
Jim Conner

ok, so, if I run helmfile from dex which uses the ../../common/environments.yaml file to define environments.

Jim Conner avatar
Jim Conner

the ../../common/environments.yaml file references a file called config.yaml but I’m in dex so the reference to config.yaml has to be referenced relatively from dex but ../../common/environments.yaml will need to be changed when I run helmfile from a different directory.

Jim Conner avatar
Jim Conner

that’s what I mean by “not very portable”

Jim Conner avatar
Jim Conner

it would be better if (at least in the way I’m using this) to relatively reference files from the manifest referencing the file.

Jim Conner avatar
Jim Conner

So, I’m in /var/tmp/helmfile-work/helmfile/helmfile.d/generic/01-secrets-management/dex and ../../common/environments.yaml contains:

environments:
  default:
    values:
      - ../../common/config.yaml
  production:
    values:
      - ../../common/config.yaml
mumoshu avatar
mumoshu


the ../../common/environments.yaml file references a file called config.yaml but I’m in dex so the reference to config.yaml has to be referenced relatively from dex but ../../common/environments.yaml will need to be changed when I run helmfile from a different directory. (edited)
ah gotcha!

fwiw, it doesn’t look like what bases is supposed to help today.

Jim Conner avatar
Jim Conner

the contents of common is:

at 00:58:51 ❯ ls -ltr ../../common
total 16
-rw-rw-r-- 1 jimconn jimconn 540 Apr 13 00:02 repos.yaml
-rw-r--r-- 1 jimconn vboxsf  486 Apr 13 22:10 config.yaml
-rw-rw-r-- 1 jimconn jimconn 129 Apr 13 23:20 environments.yaml
-rw-rw-r-- 1 jimconn jimconn  37 Apr 13 23:26 helmdefaults.yaml
Jim Conner avatar
Jim Conner

ah

Jim Conner avatar
Jim Conner

ok

Jim Conner avatar
Jim Conner

so I’m not using it properly?

mumoshu avatar
mumoshu

I usually recommend using

{{ readFile "common.yaml.gotmpl" | tpl . $someData }}

---

# releases, repositories, etc
Jim Conner avatar
Jim Conner

ok

Jim Conner avatar
Jim Conner

I can give that a try in a moment

Jim Conner avatar
Jim Conner

one more quick question then…something that’s not really documented very well I think

Jim Conner avatar
Jim Conner

so I’m gonna switch gears on you for a min

Jim Conner avatar
Jim Conner

this might be a really easy question for you to answer.

mumoshu avatar
mumoshu

i thought there were opening issue(s) about adding parameters to bases but i cant get the exact link urls for them now

Jim Conner avatar
Jim Conner

ok

Jim Conner avatar
Jim Conner

all right, here’s the helmfile.yaml again for dex:

---
bases:
  - ../../common/environments.yaml
  - ../../common/repos.yaml
  - ../../common/helmdefaults.yaml

---
## ************************************
## Start of DEX installation
## ************************************
releases:
  - name: dex
    namespace: {{ .Values.dex.namespace }}
    createNamespace: true
    labels:
      tier: "secrets-management"
      app: dex
    chart: repo/dex
    version: {{ .Values.dex.version }}
    values:
      - values.yaml.gotmpl
    set:
      - name: var.aws_nlb_with_tls_termination_at_lb
        value: {{ env "AWS_NLB" | default false }}
      - name: var.arm64_support
        value: {{ env "ARM64" | default false }}

I’m trying to use a templatized values.yaml

Jim Conner avatar
Jim Conner

my values.yaml.gotmpl (partial) is:

# DO NOT increase replicas to >1 during the initial install.                                                                                                                                                                                                                                                                                             
# There is a bug which causes the GRPC TLS certs not to be issued because of webhook race conditions.                                                                                                                                                                                                                                                    
# Bug : <https://github.com/helm/charts/issues/24229>                                                                                                                                                                                                                                                                                                      
                                                                                                                                                                                                                                                                                                                                                         
#replicas: 3                                                                                                                                                                                                                                                                                                                                             
{{- if var.arm64_support }}                                                                                                                                                                                                                                                                                                                              
image: ghcr.io/dexidp/dex                                                                                                                                                                                                                                                                                                                                
imageTag: "v2.26.0"                                                                                                                                                                                                                                                                                                                                      
{{- end }}                                                                                                                                                                                                                                                                                                                                               
                                                                                                                                                                                                                                                                                                                                                         
tolerations:                                                                                                                                                                                                                                                                                                                                             
- key: "dedicated"                                                                                                                                                                                                                                                                                                                                       
  operator: "Equal"                                                                                                                                                                                                                                                                                                                                      
  value: "utility"                                                                                                                                                                                                                                                                                                                                       
  effect: "NoSchedule"  
Jim Conner avatar
Jim Conner

I am not able to reference the set parameters in the values.yaml.gotmpl — not sure what I’m doing wrong here.

mumoshu avatar
mumoshu

ah well I haven’t really tried to use it that way but after rereading https://github.com/roboll/helmfile/issues/688

this might work:

values:
- someValuePassedToBases: "foobar"

---

bases:
- common.yaml

---

#releases, repositories, etc
breaking: stop `bases` inheriting parents' values by default · Issue #688 · roboll/helmfile

Extracted from #347 (comment) We&#39;ve introduced bases a month ago via #587. I&#39;d like to make this breaking change(perhaps the first, intended breaking change in helmfile) before too many peo…

mumoshu avatar
mumoshu

values.yaml.gotmpl is rendered by helmfile. you have access to helmfile-managed values only there

mumoshu avatar
mumoshu

where set sets values passed to helm

Jim Conner avatar
Jim Conner

ahhh

Jim Conner avatar
Jim Conner

ok, so I need to set those in a helmfile values somewhere. Maybe the thing you just linked?

values:
- key: "something"

?

mumoshu avatar
mumoshu

you usually use helmfile (environment) values to produce a series of set entries and values files

mumoshu avatar
mumoshu

set and values entries rendered and merged by helmfile are passed to helm

mumoshu avatar
mumoshu


ok, so I need to set those in a helmfile values somewhere. Maybe the thing you just linked?
absolutely!

Jim Conner avatar
Jim Conner

right, so I’m trying to localize those to just dex in this case. Obviously, I’m not just going to use helmfile to deploy dex but it will be used to deploy a bunch of stuff. So, trying to find a way to make this work both “locally” (for just dex) and then from more of an environment deployment perspective is where I’m getting lost.

Jim Conner avatar
Jim Conner

ok! Let me think through how to do that.

mumoshu avatar
mumoshu

wishing your good luck!

let me also note that i believe it would worth trying https://sweetops.slack.com/archives/CE5NGCB9Q/p1618387448086400?thread_ts=1618381282.078300&cid=CE5NGCB9Q

ah well I haven’t really tried to use it that way but after rereading https://github.com/roboll/helmfile/issues/688

this might work:

values:
- someValuePassedToBases: "foobar"

---

bases:
- common.yaml

---

#releases, repositories, etc
Jim Conner avatar
Jim Conner

helmfile is a rich ecosystem of stuff. Trying to wrap my head around it all for my use-case is complicated.

Jim Conner avatar
Jim Conner

I appreciate your help!

mumoshu avatar
mumoshu

sounds right- keep posting comments/questions/feedbacks here. i or other people will respond soon!

mumoshu avatar
mumoshu

thx for trying helmfile anyway

1

2021-04-11

mumoshu avatar
mumoshu

Does anyone rely on helmfile’s current behavior that helmfile -l name=foo apply NOT failing when foo had some needs to other releases?

https://github.com/roboll/helmfile/pull/1772 can be a breaking change to you so please chime-in and leave your comments if you have opinions

Add --{include,skip}-needs to helmfile sync and apply by mumoshu · Pull Request #1772 · roboll/helmfile

Currently, this is going to be a breaking change for whoever relied on the helmfile&#39;s existing behavior of helmfile -l foo=bar apply silently ignoring the unfulfilled needs. Since this change, …

2021-04-08

2021-04-07

abhishrant.m avatar
abhishrant.m

Hi everyone, this is probably me missing something simple but for some reason I am not able to use the new “waitForJobs” config.

I added it to the release section as follows

releases:
  - name: test
	...
    wait: true
    waitForJobs: true
    timeout: 60
helmfiles:
	...
	{{ if eq .Environment.Name "cluster" }}
	- path: environments/cluster/test.yaml
	{{ end }}

environments:
	cluster:

The helmfile apply command fails with

...
[1] in /home/helmfile/helm-installer/resources/helmfile.yaml: in .helmfiles[1]: in environments/cluster/test.yaml: failed to read test.yaml: reading document at index 1: yaml: unmarshal errors:
[1]   line 7: field waitForJobs not found in type state.ReleaseSpec

Helm and helmfile version

bash-5.0# helm version
version.BuildInfo{Version:"v3.5.0", GitCommit:"32c22239423b3b4ba6706d450bd044baffdcf9e6", GitTreeState:"clean", GoVersion:"go1.15.6"}
bash-5.0# helmfile version
helmfile version v0.138.7
Antoine Taillefer avatar
Antoine Taillefer

Hi, this is a recent addition, see https://github.com/roboll/helmfile/pull/1715 and related commit merged in master: https://github.com/roboll/helmfile/commit/2618cfb38b20d867a977f2b295059893d23e507a but not yet released.

abhishrant.m avatar
abhishrant.m

Oh, I did not notice this. I thought 138.7 had it cause the readme mentioned the new config. Thanks for the clarification!

Antoine Taillefer avatar
Antoine Taillefer

Yep, the README is the one of the master branch

abhishrant.m avatar
abhishrant.m

Is there any ETA for the next version?

voron avatar
voron

we didn’t have fresh versions for ~ 1 month, so I expect it to become soon

1

2021-04-06

Balazs Varga avatar
Balazs Varga

any idea why it does not work like values ?

skipping missing values file matching "../config/{{ .Release.Name }}/patches.yaml"
skipping missing values file matching "../config/{{ .Release.Name }}/{{ .Environment.Name }}-patches.yaml"
skipping missing values file matching "../config/{{ .Release.Name }}/merge.yaml"
skipping missing values file matching "../config/{{ .Release.Name }}/merge.yaml.gotmpl"
skipping missing values file matching "../config/{{ .Release.Name }}/{{ .Environment.Name }}-merge.yaml"
skipping missing values file matching "../config/app/values.yaml"
Successfully generated the value file at ../config/app/values.yaml.gotmpl. produced:

My templates settings are :

templates:
  default: &default
    missingFileHandler: Debug
    values:
      - ../config/{{ .Release.Name }}/values.yaml
      - ../config/{{ .Release.Name }}/values.yaml.gotmpl
      - ../config/{{ .Release.Name }}/{{ .Environment.Name }}.yaml
      - ../config/{{ .Release.Name }}/{{ .Environment.Name }}.yaml.gotmpl
    secrets:
      - ../config/{{ .Release.Name }}/secrets.yaml
      - ../config/{{ .Release.Name }}/secrets.yaml.gotmpl
      - ../config/{{ .Release.Name }}/{{ .Environment.Name }}-secrets.yaml
    strategicMergePatches:
      - ../config/{{ .Release.Name }}/merge.yaml
      - ../config/{{ .Release.Name }}/merge.yaml.gotmpl
      - ../config/{{ .Release.Name }}/{{ .Environment.Name }}-merge.yaml
    jsonPatches:
      - ../config/{{ .Release.Name }}/patches.yaml
      - ../config/{{ .Release.Name }}/{{ .Environment.Name }}-patches.yaml
voron avatar
voron

Did you find an example that it should work like this ?

Balazs Varga avatar
Balazs Varga

no, just tried based on values. it works. so on the same file I have the Release.Name… few lines after i don’t have it ?

voron avatar
voron

I don’t see that strategicMergePatches and jsonPatches do support files with/without rendered name.

voron avatar
voron

it may be better to use valuesTemplate in templates, btw, but it’s not related to the issue you’re seeing

Balazs Varga avatar
Balazs Varga

so I cannot give dynamic address of the patch file? I need to set it under all apps in release ?

voron avatar
voron

try and see. If hardcoded file name works - you may submit a PR to get generated names to function there, or open an issue.

Balazs Varga avatar
Balazs Varga
    jsonPatches:
      - ../config/app/patches.yaml
 

then it finds the patch and patches the resources I would like.

voron avatar
voron

and what’s with strategicMergePatches ?

Balazs Varga avatar
Balazs Varga

same

Balazs Varga avatar
Balazs Varga

if I hardcode the app name it finds… if not it does not replace the {{}} with the value and searches under ../config/{{ .Release.Name }}/patches.yaml

Balazs Varga avatar
Balazs Varga

where I don’t have the file

voron avatar
voron

well, sometimes hack with quoting like

{{`{{ .Release.Labels.app }}`}}

may work

Balazs Varga avatar
Balazs Varga
skipping missing values file matching "../config/{{ .Release.Name }}/patches.yaml"
skipping missing values file matching "../config/{{ .Release.Name }}/{{ .Environment.Name }}-patches.yaml"
skipping missing values file matching "../config/{{{{ .Release.Name }}}}/merge.yaml"
skipping missing values file matching "../config/{{ .Release.Name }}/merge.yaml.gotmpl"

same

Balazs Varga avatar
Balazs Varga

it looks it templating only values and secrets. is that correct?

voron avatar
voron

as I said above, you may submit a PR to get generated names to function there, or open an issue. it may be some trivial fix, or may not, IDK

Andrey Bronin avatar
Andrey Bronin

@ I can confirm, we’re using the same approach

  dockerized: &dockerized
    namespace: {{ .Environment.Values.namespace }}
    missingFileHandler: Warn
    labels:
      group: docker
    values:
      - envs/dockerized.yaml.gotmpl
      - envs/{{`{{ .Environment.Name }}`}}/dockerized.yaml.gotmpl
      - envs/common/dockerized/{{`{{ .Release.Name }}`}}.yaml.gotmpl
      - envs/{{`{{ .Environment.Name }}`}}/{{`{{ .Release.Name }}`}}.yaml.gotmpl
voron avatar
voron

there is no issue w/ values/valuesTemplate or secrets. It’s specific to strategicMergePatches and jsonPatches

Balazs Varga avatar
Balazs Varga

yes. I have issue with patches not with values and secrets.

voron avatar
voron

@Balazs Varga did you tried hack with

{{`{{ ... }}`}}

?

Balazs Varga avatar
Balazs Varga

yes I tried.. did not work

voron avatar
voron

file a PR or an issue on GH, that’s all I can advise.

Balazs Varga avatar
Balazs Varga

will do. Now I am trying to find where is it templating the secrets and values. I mean the path only because I need the path to be templated. the patch contains hardcoded data.

voron avatar
voron

well, another possible option is to fork chart and fix it to get rid of patches

Balazs Varga avatar
Balazs Varga

yeah but this looks more elegant way than modifying charts.

Balazs Varga avatar
Balazs Varga

today I will spend to solve this if cannot then will modify charts

mumoshu avatar
mumoshu

Hey! I guess you would like valuesTemplate @Balazs Varga cc @

For more info find valuesTemplate in https://github.com/roboll/helmfile/blob/master/docs/writing-helmfile.md#release-template--conventional-directory-structure

roboll/helmfile

Deploy Kubernetes Helm Charts. Contribute to roboll/helmfile development by creating an account on GitHub.

Balazs Varga avatar
Balazs Varga
06:31:39 AM

yes, but if I would like to use here it does not work.

mumoshu avatar
mumoshu

I thought there’s no reason it doesn’t work under templates.

You might be just merging-in the template afterwards using <<: *default, right?

mumoshu avatar
mumoshu

How did your helmfile.yaml with templates and releases looked like when you found valuesTemplate not working?

Balazs Varga avatar
Balazs Varga

templates I mentioned. releases was simple like this:

releases:
- name: init
  namespace: default
  chart: ../chart/init
  <<: *default

so the idea was if I put the values and secrets and patches to the selected folder it will template and use it… it searches under those folders but as you see it does not template the patches lines:

skipping missing values file matching "../config/{{ .Release.Name }}/patches.yaml"
skipping missing values file matching "../config/{{ .Release.Name }}/{{ .Environment.Name }}-patches.yaml"
skipping missing values file matching "../config/{{ .Release.Name }}/merge.yaml"
skipping missing values file matching "../config/{{ .Release.Name }}/merge.yaml.gotmpl"
skipping missing values file matching "../config/{{ .Release.Name }}/{{ .Environment.Name }}-merge.yaml"
skipping missing values file matching "../config/app/values.yaml"
mumoshu avatar
mumoshu

@Balazs Varga Could you try (literally) valuesTemplate instead of values for values, then?

1
mumoshu avatar
mumoshu

You can’t access release template from within values, which is supposed to be a yaml array of plain strings

mumoshu avatar
mumoshu

On the other hand each item in valuesTemplate is considered a go template with accses to release template

Balazs Varga avatar
Balazs Varga

yeah will try… few sec

mumoshu avatar
mumoshu

Also if you’d need to do the same templating on file paths in secrets, unfortunately it isn’t supported today. But it should be a relatively easy addition to Helmfile. Please feel free to open a dedicated feature request for that. It should look like secretsTemplate

Balazs Varga avatar
Balazs Varga

thanks. using valuesTemplate it works

Balazs Varga avatar
Balazs Varga

that is fine to me if I cannot put merge into secret “folder”.

mumoshu avatar
mumoshu

Oh really? Okay then! I just thought you’d want the same level of reusability for the secrets array, too

Balazs Varga avatar
Balazs Varga

no, I just wanted to have jsonpatches and mergepatches under config folder to have a light helmfile.d file

1
Balazs Varga avatar
Balazs Varga
templates:
  default: &default
    missingFileHandler: Debug
    valuesTemplate:
      - ../config/{{ .Release.Name }}/values.yaml
      - ../config/{{ .Release.Name }}/values.yaml.gotmpl
      - ../config/{{ .Release.Name }}/{{ .Environment.Name }}.yaml
      - ../config/{{ .Release.Name }}/{{ .Environment.Name }}.yaml.gotmpl
      - ../config/{{ .Release.Name }}/merge.yaml
      - ../config/{{ .Release.Name }}/merge.yaml.gotmpl
      - ../config/{{ .Release.Name }}/{{ .Environment.Name }}-merge.yaml
      - ../config/{{ .Release.Name }}/patches.yaml
      - ../config/{{ .Release.Name }}/{{ .Environment.Name }}-patches.yaml

    secrets:
      - ../config/{{ .Release.Name }}/secrets.yaml
      - ../config/{{ .Release.Name }}/secrets.yaml.gotmpl
      - ../config/{{ .Release.Name }}/{{ .Environment.Name }}-secrets.yaml

this worked to me… if somebody else will have the same issue.

Successfully generated the value file at ../config/test/merge.yaml.gotmpl. produced:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test
  namespace: test
spec:
  ...
mumoshu avatar
mumoshu

Ah, so what I wanted to say was that I’m afraid that this part isn’t working as you might have expected:

    secrets:
      - ../config/{{ .Release.Name }}/secrets.yaml

Almost certainly this is translated to as-is, without templating, not ../config/test/secrets.yaml as you might have expected, but ../config/{{ .Release.Name }}/secrets.yaml

Balazs Varga avatar
Balazs Varga

that still works

skipping missing values file matching "../config/prometheus/default.yaml"
skipping missing values file matching "../config/prometheus/default.yaml.gotmpl"
skipping missing values file matching "../config/prometheus/merge.yaml"
skipping missing values file matching "../config/prometheus/merge.yaml.gotmpl"
skipping missing values file matching "../config/prometheus/default-merge.yaml"
skipping missing values file matching "../config/prometheus/patches.yaml"
skipping missing values file matching "../config/prometheus/default-patches.yaml"
skipping missing secrets file matching "../config/prometheus/secrets.yaml"
skipping missing secrets file matching "../config/prometheus/secrets.yaml.gotmpl"
skipping missing secrets file matching "../config/prometheus/default-secrets.yaml"
Balazs Varga avatar
Balazs Varga

I just don’t use it here.

mumoshu avatar
mumoshu

oh really!!

mumoshu avatar
mumoshu

What an inconsistency… but good to hear it works

2
voron avatar
voron

We use templated names with secrets for some time ago, similar to

    secrets:
      - ../../live/{{ .Environment.Name }}/{{`{{ base .Release.Chart }}`}}/secrets-{{`{{ .Release.Name }}`}}.yaml
2
cool-doge2
    keyboard_arrow_up