#helmfile (2022-01)

https://github.com/roboll/helmfile

Questions and discussion around helmfile https://github.com/roboll/helmfile and https://github.com/cloudposse/helmfiles Archive: https://archive.sweetops.com/helmfile/

2022-01-25

2022-01-24

Andrew Nazarov avatar
Andrew Nazarov

With the most recent helmfile and helm-diff I started seeing errors during helmfile template for tfstate references like reading value for myproject-tf-state-dev/terraform/default.tfstate/output.filestore_instance_ip.value[0]: expected an object but got: array (["x.x.x.x"]). What component can cause this? helmfile, vals or tfstate-lookup? Can’t find changes that might affect this.

mumoshu avatar
mumoshu

probably tfstate-lookup. from the error message im not entirely sure at which level it failed though :thinking_face: maybe filestore_instance_ip was formerly a object that contains an array named value but now it’s not?

does rewiting

output.filestore_instance_ip.value[0]

to

output.filestore_instance_ip[0]

work?

1
Andrew Nazarov avatar
Andrew Nazarov

It seems you are totally right. I’m trying to get some values using the most recent vals (0.15.0) and it gets the right value without .value at the end of the reference.

1
mumoshu avatar
mumoshu

thanks for your confirmation! apparently we’re affected by breaking change in tfstate-lookup (but we can’t blame tfstate-lookup as they’re at 0.x and it doesn’t have any feature flag regarding the feature

2022-01-19

2022-01-17

Jim Conner avatar
Jim Conner

‘nother helmfile question by yours truly… so, I’m setting up my environment such that depending on settings in an environment.yaml, charts should be able to be enabled or disabled whereby when a chart is disabled, it’s simply passed during helmfile processing and vice-versa. An example of what I’m doing is: the env-toggle file:

# This file is only allowed to be a single dimension key value map
---
##################################
### globals                    ###
#
## use to set global cluster vars
cluster_domain_name: jimtest.dev.company
cluster_id: jimconn

enable_arm64_support: true
enable_aws_nlb_with_tls_termination_at_lb: false

##################################
### substrate                  ###
#
# use to enable substrate components
enable_argocd: true
enable_dex: true
enable_ambassador: true
enable_ingress_nginx: true
enable_opa_policy_manager: false

eventually, the ambassador release helmfile gets reached. I attempted using (along with a few other permutations):

{{ if .Values.enable_ambassador }}
releases:
  - name: ambassador
    namespace: {{ .Values.ambassador.namespace }}
    labels:
      app: ambassador
      tier: secrets-management
    values:
      - chart_repo: {{ .Values.ambassador.chart_repo }}
      - chart_version: {{ .Values.ambassador.chart_version }}
      - values.yaml.gotmpl
{{ else }}
releases:
  - {{ .Values | toYaml | nindent 4 }}
{{ end }}

which yields an error (along with the other permutations):

defaultVals:[]
second-pass rendering result of "helmfile.yaml.part.0":
 0: 
 1: releases:
 2:   - name: ambassador
 3:     namespace: ambassador
 4:     labels:
 5:       app: ambassador
 6:       tier: secrets-management
 7:     values:
 8:       - chart_repo: datawire/ambassador
 9:       - chart_version: 6.6.2
10:       - values.yaml.gotmpl
11: 
12: 

err: error during helmfile.yaml.part.0 parsing: encountered empty chart while reading release "ambassador" at index 0
changing working directory back to "repo/projects/helmfile-project/charts/10-cluster-mgmt"
changing working directory back to "repo/projects/helmfile-project/charts"
changing working directory back to "repo/projects/helmfile-project"
changing working directory back to "repo/projects/helmfile-project/environments"
changing working directory back to "repo/projects/helmfile-project/environments/devel"
in ./helmfile.yaml: in .helmfiles[0]: in ../helmfile.yaml: in .helmfiles[0]: in ../helmfile.yaml: in .helmfiles[0]: in charts/helmfile.yaml: in .helmfiles[1]: in 10-cluster-mgmt/helmfile.yaml: in .helmfiles[0]: in ambassador/helmfile.yaml: error during helmfile.yaml.part.0 parsing: encountered empty chart while reading release "ambassador" at index 0

I’m trying to do this in a way that is scalable. I mean, I could probably put an if-then within the helmfile.yaml one directory up from the release helmfile.yaml which is:

---
helmfiles:
- path: "*/helmfile.yaml"
  values:
  - {{ .Values | toYaml | nindent 4 }}

but if I use that method, this is not scalable in my setup as I would have to create a conditional for every single release I hold in this directory…vs using a wildcard in the path there. That doesn’t scale….

Any suggestions?

Jim Conner avatar
Jim Conner

I prefer this method (ambassador/helmfile.yaml ):

{{ if .Values.enable_ambassador }}
---
releases:
  - name: ambassador
    namespace: {{ .Values.ambassador.namespace }}
    labels:
      app: ambassador
      tier: secrets-management
    values:
      - chart_repo: {{ .Values.ambassador.chart_repo }}
      - chart_version: {{ .Values.ambassador.chart_version }}
      - values.yaml.gotmpl
{{ end }}

However, that yields:

...
...
...
second-pass rendering failed, input of "helmfile.yaml.part.0":
 0: {{ if .Values.enable_ambassador }}

err: error during helmfile.yaml.part.0 parsing: template: stringTemplate:1: unexpected EOF
changing working directory back to "repo/projects/helmfile-project/charts/10-cluster-mgmt"
changing working directory back to "repo/projects/helmfile-project/charts"
changing working directory back to "repo/projects/helmfile-project"
changing working directory back to "repo/projects/helmfile-project/environments"
changing working directory back to "repo/projects/helmfile-project/environments/devel"
in ./helmfile.yaml: in .helmfiles[0]: in ../helmfile.yaml: in .helmfiles[0]: in ../helmfile.yaml: in .helmfiles[0]: in charts/helmfile.yaml: in .helmfiles[1]: in 10-cluster-mgmt/helmfile.yaml: in .helmfiles[0]: in ambassador/helmfile.yaml: error during helmfile.yaml.part.0 parsing: template: stringTemplate:1: unexpected EOF
Jim Conner avatar
Jim Conner

Dang! I feel dumb. I figured it out. I renamed helmfile.yaml to helmfile.yaml.gotmpl and it renders correctly now.

mumoshu avatar
mumoshu

interesting. helmfile.yaml is rendered as a go template first so giving it a .gotmpl or not shouldn’t make differences

mumoshu avatar
mumoshu
 0: 
 1: releases:
 2:   - name: ambassador
 3:     namespace: ambassador
 4:     labels:
 5:       app: ambassador
 6:       tier: secrets-management
 7:     values:
 8:       - chart_repo: datawire/ambassador
 9:       - chart_version: 6.6.2
10:       - values.yaml.gotmpl
11: 

your release lacks chart: so i thought it’s the cause of this error
encountered empty chart while reading release “ambassador” at index 0

Jim Conner avatar
Jim Conner

Yeah, so I thought that the naming convention of the helmfile.yaml was the problem but in turns out the issue that seemed resolved was in fact only masked by something else. The issue persists.

second-pass rendering failed, input of "helmfile.yaml.gotmpl.part.0":
 0: {{ if .Values.enable_ambassador }}

err: error during helmfile.yaml.gotmpl.part.0 parsing: template: stringTemplate:1: unexpected EOF
changing working directory back to "repo/projects/helmfile-project/charts/10-cluster-mgmt"
changing working directory back to "repo/projects/helmfile-project/charts"
changing working directory back to "repo/projects/helmfile-project"
changing working directory back to "repo/projects/helmfile-project/environments"
changing working directory back to "repo/projects/helmfile-project/environments/devel"
in ./helmfile.yaml: in .helmfiles[0]: in ../helmfile.yaml: in .helmfiles[0]: in ../helmfile.yaml: in .helmfiles[0]: in charts/helmfile.yaml: in .helmfiles[0]: in 10-cluster-mgmt/helmfile.yaml: in .helmfiles[0]: in ambassador/helmfile.yaml.gotmpl: error during helmfile.yaml.gotmpl.part.0 parsing: template: stringTemplate:1: unexpected EOF
Jim Conner avatar
Jim Conner

Is there something wrong with that I’m trying to do here? My helmfile.yaml.gotmpl:

✦ at 17:19:19 ❯ \cat ambassador/helmfile*
{{ if .Values.enable_ambassador }}
---
releases:
  - name: ambassador
    namespace: {{ .Values.ambassador.namespace }}
    chart: {{ .Values.ambassador.chart_repo }}
    labels:
      app: ambassador
      tier: cluster-mgmt
    values:
      - chart_version: {{ .Values.ambassador.chart_version }}
      - values.yaml.gotmpl
{{ end }}
Jim Conner avatar
Jim Conner

@mumoshu

mumoshu avatar
mumoshu

As I’ve written in the issue, -- is used to separate a file into a multiple yaml templates

mumoshu avatar
mumoshu
{{ if .Values.enable_ambassador }}
---
releases:
  - name: ambassador
    namespace: {{ .Values.ambassador.namespace }}
    labels:
      app: ambassador
      tier: secrets-management
    values:
      - chart_repo: {{ .Values.ambassador.chart_repo }}
      - chart_version: {{ .Values.ambassador.chart_version }}
      - values.yaml.gotmpl
{{ end }}

so this is wrong.

Jim Conner avatar
Jim Conner

sure, I understand that.

Jim Conner avatar
Jim Conner

what I’m trying to do is get helmfile to bypass a release if it is not enabled.

mumoshu avatar
mumoshu

??

mumoshu avatar
mumoshu

can you jsut do this instead?

{{ if .Values.enable_ambassador }}
releases:
  - name: ambassador
    namespace: {{ .Values.ambassador.namespace }}
    labels:
      app: ambassador
      tier: secrets-management
    values:
      - chart_repo: {{ .Values.ambassador.chart_repo }}
      - chart_version: {{ .Values.ambassador.chart_version }}
      - values.yaml.gotmpl
{{ end }}
Jim Conner avatar
Jim Conner

sure

Jim Conner avatar
Jim Conner

lemme try

mumoshu avatar
mumoshu

i don’t understand why you need --- to do that

Jim Conner avatar
Jim Conner

I have --- in the top of all of my helmfile.yamls

Jim Conner avatar
Jim Conner

is that…not OK?

mumoshu avatar
mumoshu

sure it’s okay if you want it (for some reason that i don’t know of), if you’re really putting into in the top

mumoshu avatar
mumoshu
{{ if .Values.enable_ambassador }}
---

obvisouly --- isn’t in the top for the above case..? that’s where i confused

Jim Conner avatar
Jim Conner

I might be abusing the template in this example but the thought was that --- starts a new valid release…and if I want the release disabled then effectively it doesn’t exist. In other words, the start of the release, I thought, started with --- but if the release is disabled then --- will never even get parsed….was my thinking

mumoshu avatar
mumoshu

ah well perhaps helmfile fill complain about missing releases and sub-helmfiles if you had an empty document in the beginning of every helmfile.yaml

mumoshu avatar
mumoshu


— starts a new valid release
i couldn’t understand this part

mumoshu avatar
mumoshu
Documentation for helmfile DRYness lacks...and maybe not even DRY at all... · Issue #2036 · roboll/helmfile attachment image

I had to vent because I'm so frustrated with helmfile. I'm really struggling to wrap my head around how to implement this project of mine for helmfile. I'm going to try really hard to a…

mumoshu avatar
mumoshu

--- splits the file into documents BEFORE rendering so that’s why --- within the if block breakks it

mumoshu avatar
mumoshu

--- doesn’t start a new set of releases. it just splits the config into layers where each layer is rendered and merged in that order.

Jim Conner avatar
Jim Conner

I get a little confused about how YAML streams work with respect to --- vs how helmfile might be using streams. If you follow the YAML spec, I thought, then each new stream is supposed to start with a --- so that’s why each of my helmfile.yaml files start with a --- out of just good practice, but that might be something I need to change…

OK so, it’s like this… Let’s assume that this is my helmfile.yaml and when helmfile parses it, it will perform a release.

---
some valid stuff here

Now, let’s say I want to enable some helmfile releases in some directories and disable others…. if I enable a helmfile (stream) then something like:

{{ if .Values.enabled }}
---
helmfile stuff
{{ end }}
mumoshu avatar
mumoshu

our use of --- doesn’t follow the yaml spec.

Jim Conner avatar
Jim Conner

ah hah!!!

Jim Conner avatar
Jim Conner

ok, that’s good to know!!

Jim Conner avatar
Jim Conner

so it’s actually even MORE important than I thought…ok, that’s huge.

mumoshu avatar
mumoshu

yeah glad you understand it now

Jim Conner avatar
Jim Conner

OK, so I’ve removed all of the ---s in my helmfiles where I was using it inside the conditionals

Jim Conner avatar
Jim Conner

I’m getting a different error now so I’ll try and work through that. Thank you @mumoshu

1
mumoshu avatar
mumoshu

well.. im really confused what you understand and not. i think i’ve already explained that --- isn’t that of yaml spec defines in https://sweetops.slack.com/archives/CE5NGCB9Q/p1642641966099100?thread_ts=1642467523.092800&cid=CE5NGCB9Q, but perhaps i wasn’t clear enough…??

mumoshu avatar
mumoshu

wishing you manage it!

Jim Conner avatar
Jim Conner

the concept about what you were trying to explain made sense to a degree but it didn’t make sense with respect to the difference between YAML and helmfile. That difference was never explicitly mentioned so I just assumed that I had to work around helmfile spec and YAML spec….

mumoshu avatar
mumoshu

ah okay, i see where you come from now! i thought it’s obvious that it can’t be --- from the yaml spec, as it can only be either (1) load helmfile.yaml as as a stream of yaml documents separated by --- and render each yaml doc as a go template, merge rendered documents as layers OR (2) load helmfile.yaml as a steram of go templates separated by ---, render each template, parse the render output as a yaml, merge resulting yaml documents

mumoshu avatar
mumoshu

and (1) isn’t possible as a go template file can’t be a valid yaml

mumoshu avatar
mumoshu

the more we talk about this yaml+go template gotchas the more i get to think we must have outgrew it and need a more serious programming or configuration language

mumoshu avatar
mumoshu

thanks anyway for clarifying!

Jim Conner avatar
Jim Conner

no problem!!

Jim Conner avatar
Jim Conner

I appreciate your time. Out of curiosity…do you have thoughts about what other candidates for programming config language?

mumoshu avatar
mumoshu

i’m looking one that fits helmfile for almost 2 years now and have no single answer yet

Jim Conner avatar
Jim Conner

helmfile v2?

Jim Conner avatar
Jim Conner

have you heard of helmwave?

mumoshu avatar
mumoshu

sort of > v2 but i’d keep the current yaml+go template api for a while for maybe it’s more like v1 with additional api with the new language

mumoshu avatar
mumoshu

yes, but i havent tried it yet > helmwave

Jim Conner avatar
Jim Conner

yeah, that’s a good idea

mumoshu avatar
mumoshu

i know it because i also maintain helm-diff which is apparently used by helmwave too and helmwave’s author sometimes come and submit a feature request that helps helmwave’s use case

Jim Conner avatar
Jim Conner

I looked into it. helmfile has already had a steep learning curve. I am not about to go learning something new. Even their documentation is not good enough to easily craft something that would work for our environment. I’m finally starting to get a better grasps of how to use helmfile for how I want to do my project…and even that’s not working exactly perfectly yet.

mumoshu avatar
mumoshu

i was once convinced to adopt jsonnet and cue, but it turned out it’s really not meant as a extensible language. custom functions aren’t well supported in neither jsonnet or cue

Jim Conner avatar
Jim Conner

we use jsonnet in our environment and while it’s powerful, it’s more cognitive load and extraneous work for helm projects. It would be like using an asteroid to kill an ant.

mumoshu avatar
mumoshu

yeah i can see that… and jsonnet and cue can be used to generate a complex and regular (go template free) helmfile.yaml so i tend to recommend it in cases where you’re just using helmfile’s template only for doing somthing that can be possible in jsonnet/cue.

Jim Conner avatar
Jim Conner

out of curiosity… did I describe my project environment well enough for you to understand in this issue https://github.com/roboll/helmfile/issues/2036#issuecomment-1010503017?

mumoshu avatar
mumoshu

yes. and i see that the structure of your project is not that uncommon

mumoshu avatar
mumoshu

so it would work in the end

Jim Conner avatar
Jim Conner

I was hoping to make it so that I could run helmfile from any of the directories minimally specifying requirements to helmfile and that the context of the local helmfile.yaml would be enough to work…but that’s not turning out to be the case yet… that will be something I tackle later. For now, being able to run it from a specific environment should be enough.

Jim Conner avatar
Jim Conner

at any rate, I need to get dinner for my family. I might pop on later and work on this project of mine…Thanks @mumoshu

1
Jim Conner avatar
Jim Conner

ttyl

1

2022-01-12

Alex Bowers avatar
Alex Bowers

Hi, I’m trying to use helmfile to install dependencies, and then some resources for those dependencies (eg, ConfigMap for crossplane), so I have a templates directory alongside my helmfile.yaml. The dependency (crossplane) gets installed absolutely fine, but none of the templates (ConfigMap), are getting installed. Am i doing something wrong?

Alex Bowers avatar
Alex Bowers

This is the structure i’m using

Andrew Nazarov avatar
Andrew Nazarov

What is inside of these templates? helmfile definitions or k8s manifests? How do you install things?

With helmfile you can define helm releases, i.e it operates with charts. It also can make releases out of kustomize. And it’s possible to apply plain k8s manifests via hooks.

Alex Bowers avatar
Alex Bowers

Just plain k8s manifests, but with some {{.Values...}} things in them

Alex Bowers avatar
Alex Bowers

As an example, this is my crossplane-secret.yaml file:

apiVersion: v1
kind: Secret
metadata:
  namespace: crossplane-system
  name: crossplane-db-conn
stringData:
  endpoint: {{ .Values.mysql.endpoint | quote }}
  port: {{ .Values.mysql.port | quote }}
  username: {{ .Values.mysql.username | quote }}
  password: {{ .Values.mysql.password | quote }}
Alex Bowers avatar
Alex Bowers

And this is my values.yaml

mysql:
  endpoint: {{ requiredEnv "MYSQL_ENDPOINT" }}
  port: 3306
  username: {{ requiredEnv "MYSQL_USERNAME" }}
  password: {{ requiredEnv "MYSQL_PASSWORD" }}
Alex Bowers avatar
Alex Bowers

In terms of my install, I have this in my helmfile.yaml

repositories:
- name: crossplane-stable
  url: <https://charts.crossplane.io/stable>

releases:
- name: crossplane
  chart: crossplane-stable/crossplane
  namespace: crossplane-system
  version: v1.6.1
  installed: true
  values:
    - values.yaml.gotmpl

and it is all ran by doing:

helmfile --file $CI_PROJECT_DIR/environments/helm/helmfile.yaml sync
Alex Bowers avatar
Alex Bowers

Whereby $CI_PROJECT_DIR/environments/helm/helmfile.yaml is just a master helmfile.yaml that refers to other helmfile.yaml within subdirectories like so:

helmfiles:
  - path: applications/*/helmfile.yaml
Alex Bowers avatar
Alex Bowers

I think this would work if i had an individual Chart.yaml file in each of these directories too, but I’d like to avoid that, since I don’t actually want these to be charts, just to be installed in addition to the rest of the dependency

Andrew Nazarov avatar
Andrew Nazarov

And how do you you want these configmaps to be installed?) I’m not aware of the most recent features of helmfile , therefore given your example I would either make a separate chart with these configMaps and then define a release in helmfile.yaml or utilise incubator/raw chart and put all these configMaps there as values:)

Alex Bowers avatar
Alex Bowers

Ideally i’d like it to just behave as though i had a separate chart for these dependencies, without me having to define one

Alex Bowers avatar
Alex Bowers

I figure that doesn’t exist though

Ievgenii Shepeliuk avatar
Ievgenii Shepeliuk

incubator/raw is deprecated, this one can be used instead https://github.com/deliveryhero/helm-charts/tree/master/stable/k8s-resources

helm-charts/stable/k8s-resources at master · deliveryhero/helm-charts attachment image

Helm Charts @ Delivery Hero . Contribute to deliveryhero/helm-charts development by creating an account on GitHub.

2022-01-11

Jim Conner avatar
Jim Conner

@mumoshu if you jump in here in a little while, I’d love to have a quick conversation about my last question in #2036 – specifically, wondering how helmfile internally uses --- . It sounds like there’s an order of operation helmfile adheres to but I’m not very clear on that. For instance, after changing/fixing the things suggested by you, it looks like bases loses the .Value read from environments in my helmfile approach. Now I might figure this out before you become available and if that happens, I’ll let you know. For now, I’m workin on this. TIA

1
mumoshu avatar
mumoshu
Documentation for helmfile DRYness lacks...and maybe not even DRY at all... · Issue #2036 · roboll/helmfile attachment image

I had to vent because I&#39;m so frustrated with helmfile. I&#39;m really struggling to wrap my head around how to implement this project of mine for helmfile. I&#39;m going to try really hard to a…

Jim Conner avatar
Jim Conner

and I appreciate it…sorry to keep bothering you.

Jim Conner avatar
Jim Conner

lemme take a look

Jim Conner avatar
Jim Conner

awesome! It’s clear as mud troll but better than nothing! Thank you @mumoshu

Jim Conner avatar
Jim Conner

So, I’m not sure how to grok this:

err: failed to read common/global-default.env.yaml.gotmpl: reading document at index 1: yaml: unmarshal errors:
  line 2: cannot unmarshal !!map into []interface {}
changing working directory back to "/home/jimconn/projects/src/bluescape/ops/projects/helmfile-project/environments/devel"
changing working directory back to "/home/jimconn/projects/src/bluescape/ops/projects/helmfile-project/environments/devel/test-cluster"
in ./helmfile.yaml: in .helmfiles[0]: in ../helmfile.yaml: in .helmfiles[0]: in ../helmfile.yaml: failed to read common/global-default.env.yaml.gotmpl: reading document at index 1: yaml: unmarshal errors:
  line 2: cannot unmarshal !!map into []interface {}

the whole debug output is (see snippet in #helmfile)

it looks like everything mutates fine so far…but the error then says it can’t read the file it’s already read. Confusing.

Jim Conner avatar
Jim Conner

I’ll brb. I gotta pick up my daughter from school

mumoshu avatar
mumoshu

It’s failing while parsing the render output as YAML.

The render output is emitted immediately before the parsing part, so reading it would help debugging.

I took a glance and it seems like the second line in the render output doesn’t conform to the structure of helmfile config

 0: values:
 1:   domain_name: "jimtest.dev.company"

values must be an array, rather than a dict

1
1
Jim Conner avatar
Jim Conner

gotta eat dinner…when I’m done I’ll dig into it.

Jim Conner avatar
Jim Conner

making progress….

Jim Conner avatar
Jim Conner

good stuff.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I heard today that Jenkins X (jx3) is moving to adopt helmfile! That’s pretty cool.

bradym avatar
bradym

I still hope to never touch jenkins again, but that is pretty cool.

1
Jim Conner avatar
Jim Conner

now if Jenkins just wasn’t a piece of junk…that would be neat!

Jim Conner avatar
Jim Conner

troll

Jim Conner avatar
Jim Conner

2022-01-10

2022-01-08

Joaquin Menchaca avatar
Joaquin Menchaca

Helmfile quesiton/discussion; The releases[*].namespace will not populate the {{ .Namespace }} value used within releases[**].values[**].

But I can get this with -n mynamespace arg. I noticed when I do this, I don’t see namespace injected with helm template, but I do see when finally deployed.

This issue comes up when I do a ClusterRoleBinding, and I was the subjects[0].namespace set appropriately.

Thoughts, is the expected behavior? desired behavior?

mumoshu avatar
mumoshu

I thought it’s .Release.Namespace which is populated by releases[*].namespace. Could you confirm?

mumoshu avatar
mumoshu


I noticed when I do this, I don’t see namespace injected with helm template
This is a different issue. It comes from the nature of helm-template. helm-template doesn’t set metadata.name in the resulting resources when the chart templates don’t include the below snippet for every resource:

metadata:
  namespace: {{ .Release.Namespace }}
mumoshu avatar
mumoshu


I do see when finally deployed.
I think this is because helm-upgrade set the namespace when calling the k8s client for updating k8s resources in the cluster.

Even if Helm sees no metadata.namespace in each rendered manifest, K8s give each a corresponding namespace according to the client config.

mumoshu avatar
mumoshu

I’d say it’s an expected behavior. It isn’t really easy to understand, though.

2022-01-07

Joaquin Menchaca avatar
Joaquin Menchaca

I was wondering why SweetOps moved away from using Helmfile. I never understood the use cases, and not sure if I got a reply from the last time I asked this.

Terraform templating is clumsy (not sure if improved, e.g. indenting) where go-template with helmfile was richer, so felt helmfile was far easier to run multiple complex charts.

One advantage with Terraform was that you could use fetch data and secrets from the cloud for use as values in the helm charts without exposing it, where going between different tools, it could get complex.

Segue, I’m not sure if this exist, bu something could be done only in k8s land if there is cloud operator with CRD that could be used (but unfamiliar with this or if it is even viable). Google started something like this with their own CRD in k8s manifests, not sure how popular it is.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

It’s not that we stopped using helmfile, we stopped using helmfile the way we were using it, which reduced the number of places we have it. like all tools, they have a time and place, and sometimes the tools we start with evolve over time.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Inevitably, we find we’re just swapping out one tool with one set of problems for another tool with another set of problems.

2
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

So the problem we ran into is we need to provision secrets, IAM roles and policies, backing services (E.g. redis, rds, etc), and we need to do it after the cluster is online.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Let’s take a prime example: argocd

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We leverage argocd more and more for application CD, but less so for backing services CD, for which we’re big proponents of Spacelift.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Something needs to provision argoCD, which could be helmfile, or terraform. We chose terraform so we can have one component terraform the github repos and provision argocd in one fell swoop.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

But like I said about exchanging problems, this led to a new problem, that we don’t get the pretty diffs of helmfile with helm+helm diff. This seems to be fixed by enabling the experimental feature of the helm provider to set manifest=true , but it’s not perfect.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Here are some related issues we are tracking:

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Show diff for actual state vs desired state · Issue #176 · databus23/helm-diff attachment image

Hi, At the moment, if you make any manual changes to resources (not via helm) helm diff will not reflect these changes. I suggest that the output should reflect the desired vs actual state of the r…

Add support for storing the rendered manifest (i.e Helm diff support) by jrhouston · Pull Request #702 · hashicorp/terraform-provider-helm attachment image

Description This PR adds a feature to the provider that enables storing the rendered manifest of a release so a full diff of changes to the chart&#39;s contents can be shown at plan time. We do thi…

Show diff for actual state vs desired state · Issue #176 · databus23/helm-diff attachment image

Hi, At the moment, if you make any manual changes to resources (not via helm) helm diff will not reflect these changes. I suggest that the output should reflect the desired vs actual state of the r…

upgrade command add three-way-merge option by luxurine · Pull Request #304 · databus23/helm-diff attachment image

upgrade command add three-way-merge option to show diff for actual state vs desired state #176 prepared resource for testing 1、last release: — # Source: base-app/templates/service.yaml apiVersion…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

And it turns out that with helm3 some of the same problelms we encountered with the helm provider exist now in helmfile too (but didn’t exist in helm2).

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

So you mentioned that terraform templating is pretty bad, but in 0.13+, I think it got a lot better. That said, we don’t template YAML when we use terraform because we don’t need to.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Instead, we use native HCL and call jsonencode once we’ve constructed the object.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

The other challenge was problem more related to our type of customer. Customers seek us out precisely because of our terraform chops, so introducing helmfile for this one use-case meant learning another tool with advanced templating. We exploited the out of the advanced templating, and while it made sense to us, it seems like users new to k8s struggled with all the layers. Reducing it to terraform seems to improve our deliverables.

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We have a vast testing infrastructure based on terraform, and an efficient distribution model based on terraform modules and the terraform registry. Using terraform we’re able to deliver an end-to-end implementation in a module which is not really possible in pure helmfile without using hooks and shell scripts to create things like IAM roles.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

For customer apps, we recommend argocd, which still has the same problems that we cannot create infrastructure in the same phase as we deploy apps in argocd. It requires a coordinated rollout.

Andrew Nazarov avatar
Andrew Nazarov

What a great explanation @Erik Osterman (Cloud Posse) ! Actually, we are tracking the same issues and have pretty much the same concerns diffs-wise. Nevertheless we still rely on helmfile dealing with app deployments. More on that, about a year or so ago we started using Google Config Connector which allows us to create cloud resources via k8s manifests. So, nowadays usually we either create/reuse a Helm chart to create a clouds resource or add a manifest of a cloud resource to existed charts enhancing their functionality. And we continue using Terraform for, say, the foundation layer (up to creating a K8s cluster).

We more or less like what we’ve got, the flexibility, the workflow, etc. However, one of the major things here is that there are some cloud services that can be managed only with Terraform/Pulumi: Elastic Cloud, MongoDB Atlas, etc. Adopting these pieces brought up some arguments to reconsider or refine our approach. That’s where we are right now)

mumoshu avatar
mumoshu

Thanks for sharing! I learned a lot :)

The world definitely looks much more complex than a year or two ago. At that time I thought terraform + helmfile was the only option. But now, I see many variants of workflows build from arbitrary combinations of the below:

Infra:

• Terraform + k8s/helm tf provider

• Plumi

• aws-cdk (+cdk-k8s)

• Custom script (e.g. run tf then helmfile) Apps:

• Run kubectl/kustomize/helm/helmfile from CircleCI, GitHub Actions, Atlantis, etc

• ArgoCD (+ Helmfile as a ArgoCD custom config management plugin sometimes)

DaniC (he/him) avatar
DaniC (he/him)

@Erik Osterman (Cloud Posse) kudos for all this great notes,

Zachary Loeber avatar
Zachary Loeber

Thanks @Erik Osterman (Cloud Posse), What a great rundown of the struggles of orchestrating cloud delivery from bootstrap to app deployment. Are you only using cicd pipelines for these stitched together orchestrations or have you been exploring other tools to fill this particular gap?

Joaquin Menchaca avatar
Joaquin Menchaca

Somethings I found in my experience:

• jsonencode converted everything to strings, so if non-strings, like actual ints/booleans, this was a problem.

• Issues w/ K8S provider where HCL changed, but manifests did not. This is with maps or blocks becoming hash/map. Docs are not really good for this.

• Issues w/ K8S provider deploying things differs in behavior from kubectl

• Some niche scenarios where what would be a patch (update) in helm/kubectl, becomes delete and recreate in Terraform. If I stay with basic helm charts, I can get away with mostly using Terraform, but the moment I need to do more advanced templating or mix this with kubernetes manifests, such as for CRDs, then things get messy with pure terraform. This happens in more of platform layer (prometheus, cert-manager, etc) but in the app layer, it makes sense to be pure TF for this.

At my current organization (google shop), they are the opposite, starting with google deployments (ugh) and anchor helm charts (no helmfile), and are now moving at least infra to Terraform, last I checked. They were considering going toward using k8s operator to manage cloud deployments (which is something that seems scary to me).

Jim Conner avatar
Jim Conner

In most typical cluster setups, there are generally and pretty consistently, in my experience, at least three components for environments….

  1. the cluster/infrastructure a. when using a cloud provider b. if using bare-metal then this could be split into two completely disparate components
  2. the substrate e.g. components running on infrastructure to support the main application/micro-service(s)
  3. the application Practices which could be discussed:
  4. always separate infrastructure from software deployments to eliminate accidental actions from occurring. a. terraform was meant for infrastructure. Some people believe it’s a good idea to use terraform for infra and software but I personally think this is a really really bad idea (we’ve been bit in the collective butts by this). b. day-1 deployments typically work fine…but c. extra precautions, code, software must be taken or used to keep unintended issues from occurring on day-2 deployments. which could just add to tech debt…moreover, d. when using terraform to deploy substrate, day-2 operations require terraform to run which means terraform must check the state and possibly apply to your infrastructure before getting to substrate/software deployment, which is usually not desired and unless expected is definitely not suggested. e. in our experience, if something in the substrate needs attention, terraform helm provider doesn’t always catch the correct state of helm charts deployed and extra work needs to be done to “fix” that which for us usually means side-deleting a chart and then re-applying terraform; see #a
  5. substrate software should be deployed using helm and better if deployed using some kind of package release software like helmfile, helmwave (maybe, I don’t know much about this) or home-grown wrapper…to ease release – a. It’s up to engineers how best to perform this task. We’re migrating out of terraform and trying out helmfile.
  6. for deploying your software stack, something like argocd is likely requisite but this is also up to engineering teams to decide. The idea behind tooling is to:
  7. ease toil and burden on operations teams and devs
  8. effectively perform the tasks necessary for proper operations and uptime of application(s)
2
Michael Attard avatar
Michael Attard

Hey @mumoshu!

We are looking for something like https://www.runatlantis.io/ (Terraform Github automation) but for Helmfile.

Are you aware of any tools that we can leverage to show diffs when running Helmfile in a CI env?

Terraform Pull Request Automation | Atlantis

Atlantis: Terraform Pull Request Automation

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

You can definitely run helmfile in Atlantis

Terraform Pull Request Automation | Atlantis

Atlantis: Terraform Pull Request Automation

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Create a custom workflow

Michael Attard avatar
Michael Attard

Thanks for the suggestion Erik

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

If you search the archives you will find example configurations

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I am on my phone so it’s a bit tedious to search

Michael Attard avatar
Michael Attard

Thanks again Erik, it’s good to see that this was done by someone else

Andrew Nazarov avatar
Andrew Nazarov

There are also custom script possibilities in Env0)

tomv avatar

We are running helmfile w/ a custom atlantis workflow

tomv avatar

Works great as long as all your secret material is in k8s Secret resources

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Here’s an example:

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
helmfile:
    plan:
      steps:
        - run: "chamber exec kops -- kops export kubecfg"
        - run: 'chamber exec kops -- direnv exec "$DIR" helmfile diff --suppress-secrets --args="--allow-unreleased --context 5 --no-color"
    apply:
      steps:
        - run: "chamber exec kops -- kops export kubecfg"
        - run: 'test -x "$DIR"/namespace-annotations.sh && chamber exec kops -- direnv exec "$DIR" "$DIR"/namespace-annotations.sh || echo "No script found to annotate namespaces"'
        - run: 'chamber exec kops -- direnv exec "$DIR" helmfile apply --suppress-secrets'

2022-01-06

Jim Conner avatar
Jim Conner

Would someone who knows helmfile really good be able to take a look at https://github.com/roboll/helmfile/issues/2036 and perhaps give me some pointers? The issue is attempting to point out poor documentation but I really do believe that the method I’m attempting to use for helmfile is sound and helmfile just needs better documentation methinks. – TIA

Documentation for helmfile DRYness lacks...and maybe not even DRY at all... · Issue #2036 · roboll/helmfile attachment image

I had to vent because I&#39;m so frustrated with helmfile. I&#39;m really struggling to wrap my head around how to implement this project of mine for helmfile. I&#39;m going to try really hard to a…

bradym avatar
bradym

Honestly the biggest problem with helmfile is the fact that Momushu seems to be the only maintainer. Dude is amazing, and has got to be burned out by all the work he’s put into this project by now. I can’t fathom how he couldn’t be. Documentation the number two issue, which I’m sure comes from a lack of help maintaining the project.

If helmfile is going to continue and evolve I think it’ll have to be by people who use it stepping up to help improve both of these situations.

2
1
mumoshu avatar
mumoshu

The biggest problem for me is that I don’t even have an admin privilege on the repo that is required to add more maintainers to the project myself. I’ve asked the original author(roboll) regarding that about a year ago but we lack a progress

https://github.com/roboll/helmfile/issues/1824

Transfer this repository to a dedicated GitHub organization · Issue #1824 · roboll/helmfile attachment image

I&#39;d like to have one step further to maintain Helmfile better, by moving this repository to a GitHub organization https://github.com/helmfile. This project has originally been created by awesom…

mumoshu avatar
mumoshu


I think it’ll have to be by people who use it stepping up to help improve both of these situations.
Fully agree and I don’t worry much about that. I remember several people have stepped up saying they are eager to become maintainers.

I was also thinking itscaro would be a great candidate as a potential maintainer seeing his continued efforts on helmfile https://github.com/roboll/helmfile/pulls?q=is%3Apr+author%3Aitscaro+

bradym avatar
bradym

Ouch. I was about to ask if forking would be a good idea, but you don’t get issues and PRs that way so that’s clearly a no go.

mumoshu avatar
mumoshu

Yeah, that’s one of issues.

If we’re forking I’d also love it if we could “archive” the current repo so that everyone can notice that the project has been moved. Indeed, it’s impossible without admin privilege either…

bradym avatar
bradym

yeah, true

mumoshu avatar
mumoshu

That said, I’m more inclined to fork the repo these days just because… there seems to be no other options Maybe we’d better set a due date for moving the repo and do fork if it passed? WDYT?

Jim Conner avatar
Jim Conner

Yes. Typically the way to handle that is to fork the repo…

Jim Conner avatar
Jim Conner

what would be best, though, would be for roboll to pass the project on to you (I’m not using GH terms there but it’s something like that).

Jim Conner avatar
Jim Conner

obviously, you’re the main guy now… @mumoshu

mumoshu avatar
mumoshu

yeah… but I’ve already tried that for more than a year in places like https://github.com/roboll/helmfile/issues/1824 and slack dm

Jim Conner avatar
Jim Conner

sucks.

Jim Conner avatar
Jim Conner

well, not to be a bother, but @mumoshu have I got questions for you (#2036 and #2039) – however, right now, I need to get dinner for my family and spend a little time with them tonight, so if you have time in the next few days, I would love to chat.

1
Jim Conner avatar
Jim Conner

bradym avatar
bradym

I think it makes sense to fork the repo at this point.

Jim Conner avatar
Jim Conner

welp, it’s 11PM now. I need to go to bed. @mumoshu perhaps we can talk about those two issues [my] tomorrow ([your tonight]) or something.

Jim Conner avatar
Jim Conner

lemme know. I did my best articulating my thoughts, problems, concerns in the first ticket and in the second ticket, I’m pretty sure something I’m doing is causing the panic, but nevertheless, panics are bad (tm).

Igor Urazov avatar
Igor Urazov

I’m afraid roboll can be considered as MIA now. His personal site and twitter are gone, he doesn’t communicate in gitlab, last activity was https://github.com/roboll/dotfiles was 10 months ago.

Fork would be a best option. Also for existing users awareness, current README can be updated to point to fork, so at least this would be visible enough.

GitHub - roboll/dotfiles: productivity, ootb attachment image

productivity, ootb. Contribute to roboll/dotfiles development by creating an account on GitHub.

Igor Urazov avatar
Igor Urazov

Also forking in general sometimes is the last working resort to get attention of current owner. Maybe roboll will notice it and you’ll be able to sort things out.

Igor Urazov avatar
Igor Urazov

One last thing, forking is totally okay, we’re all working in rapidly changing industry, so breaking changes are expected and we know how to deal with them.

1
Jim Conner avatar
Jim Conner

I couldn’t agree more! The question is whether contributors would be appreciated. Some maintainers really don’t like others touching the code.

Jim Conner avatar
Jim Conner

If I understood the ins and outs of helmfile better, I’d be willing to help with documentation. I did that with vault-secrets-webhook (bank-vaults) for banzaicloud.

Jim Conner avatar
Jim Conner

at any rate, if anyone is able to give any kind of assistance on that question I have there…it would be greatly appreciated it

Jim Conner avatar
Jim Conner

I might just have to wait for @mumoshu to take a look. He doesn’t seem to have been around lately, though, so….maybe he’s on a break.

1
bradym avatar
bradym

Wish I could help, but my use of k8s and helmfiles seems to be so much simpler then everyone else around here.

bradym avatar
bradym

That might sound wrong… I just mean that we only have two clusters and a handful of things that we deploy.

bradym avatar
bradym

Very small shop compared to what I hear most people around here doing.

Jim Conner avatar
Jim Conner

yeah…we have like…15 clusters and ~40 charts per….

Jim Conner avatar
Jim Conner

our infra guys decided to use terraform to deploy helm charts using the terraform helm provider (which I think Mumoshu maintains/wrote actually) but imo, deploying software with terraform is a really really really bad idea blending infrastructure with software, which is poor practice. You really should keep those two things separate. The infra guys and we have been recognizing a great number of problems with this make up so I’m separating all of that out into helmfile. That’s proving to be a chore because of what I’ve identified problematic given the size of our environment and what helmfile documentation speaks to.

mumoshu avatar
mumoshu


terraform helm provider (which I think Mumoshu maintains/wrote actually) but imo,
maybe terraform-provider-helmfile? then yes, I maintain it :slightly_smiling_face:

deploying software with terraform is a really really really bad idea blending infrastructure with software, which is poor practice. You really should keep those two things separate. I generally agree with you!

my motivation for building terraform-provider-helmfile was to use it only for deploying system components whose lifecycles are tied to that of the cluster. that said, I consider deploying “apps” with terraform-provider-helmfile as no-go.

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

This is a good #office-hours topic. Let’s discuss next week

1
bradym avatar
bradym

Unfortunately I can’t make it to office hours due to a recurring work meeting at the same time, but I’ll be interested to hear what comes out of the discussion.

2022-01-04

Jim Conner avatar
Jim Conner

anyone able to explain what this means?

could not deduce `environment:` block, configuring only .Environment.Name. error: failed to read helmfile.yaml.part.0: reading document at index 1: yaml: unmarshal errors:
  line 3: cannot unmarshal !!map into []state.SubHelmfileSpec
error in first-pass rendering: result of "helmfile.yaml.part.0":

A couple of people have cut issues for this error notably one who said he fixed it but didn’t post how he fixed it

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
GitHub - roboll/helmfile: Deploy Kubernetes Helm Charts attachment image

Deploy Kubernetes Helm Charts. Contribute to roboll/helmfile development by creating an account on GitHub.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

looks like if you use .Environment.Name you need to specify environment block(s)

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

or don’t use .Environment.Name since it’s deprecated

Jim Conner avatar
Jim Conner

yup. though I’m not using an environments object so I’m assuming that it will just work…but it doesn’t.

Jim Conner avatar
Jim Conner

at least according to the documentation it seems that it would work.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

this worked for us

environments:
  default:
    values:
      - defaults.yaml
  ## Add additional environments with value overrides like this:
  production_east:
    values:
    - defaults.yaml
    - east.yaml
    - production.yaml
Jim Conner avatar
Jim Conner

so environments is required?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

if your code uses .Environment.xxxxx , yes

Jim Conner avatar
Jim Conner

I don’t use .Environments at all.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

(or could be just bad YAML formatting, happens all the time)

Jim Conner avatar
Jim Conner

really strange. At any rate, I even tried though and it’s still throwing the same error.

Jim Conner avatar
Jim Conner

all my stuff (so far) passes yamllint…though it might just be bad yaml for helmfile’s perspective.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

try to remove parts of helmfile.yaml step-by-step until the error disappears - you would know in what part of the file it is (if it’s an invalid yaml+template)

Jim Conner avatar
Jim Conner

sorry…gimme a sec

Jim Conner avatar
Jim Conner

having a talk with the kid…

Jim Conner avatar
Jim Conner

yeah, that’s the technique I’ve been using. Nothing working so far. I’ll figure it out eventually though. Thanks for your help…

Jim Conner avatar
Jim Conner

My helmfile.yaml is super simple:

---
environments:
  default:
    values:
    - env-toggles.yaml

helmfiles:
  path: ../helmfile.yaml
Jim Conner avatar
Jim Conner
processing file "helmfile.yaml" in directory "."
first-pass rendering starting for "helmfile.yaml.part.0": inherited=&{default map[] map[]}, overrode=<nil>
first-pass uses: &{default map[] map[]}
first-pass rendering output of "helmfile.yaml.part.0":
 0: ---
 1: environments:
 2:   default:
 3:     values:
 4:     - env-toggles.yaml
 5: 
 6: helmfiles:
 7:   path: ../helmfile.yaml
 8: 

could not deduce `environment:` block, configuring only .Environment.Name. error: failed to read helmfile.yaml.part.0: reading document at index 1: yaml: unmarshal errors:
  line 8: cannot unmarshal !!map into []state.SubHelmfileSpec
error in first-pass rendering: result of "helmfile.yaml.part.0":
 0: ---
 1: environments:
 2:   default:
 3:     values:
 4:     - env-toggles.yaml
 5: 
 6: helmfiles:
 7:   path: ../helmfile.yaml
 8: 

first-pass produced: &{default map[] map[]}
first-pass rendering result of "helmfile.yaml.part.0": {default map[] map[]}
second-pass rendering result of "helmfile.yaml.part.0":
 0: ---
 1: environments:
 2:   default:
 3:     values:
 4:     - env-toggles.yaml
 5: 
 6: helmfiles:
 7:   path: ../helmfile.yaml
 8: 

err: failed to read helmfile.yaml: reading document at index 1: yaml: unmarshal errors:
  line 8: cannot unmarshal !!map into []state.SubHelmfileSpec
in ./helmfile.yaml: failed to read helmfile.yaml: reading document at index 1: yaml: unmarshal errors:
  line 8: cannot unmarshal !!map into []state.SubHelmfileSpec
Jim Conner avatar
Jim Conner

ahhh, I think I’m misunderstanding the error. The environment: error is a red-herring of sorts, I think. The main error is at the bottom I think.

Jim Conner avatar
Jim Conner

meh, still haven’t figured it out…but I’ll whittle it down.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

try this:

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

file structure

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

environments.yaml :

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
environments:
  default:
    values:
      - defaults.yaml
## Add additional environments with value overrides like this:
#  production_east:
#    values:
#    - defaults.yaml
#    - east.yaml
#    - production.yaml
#
# See <https://github.com/roboll/helmfile#environment> for details
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

helmfile.yaml:

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
bases:
- environments.yaml
---
repositories:
# Official aws-load-balancer-controller chart repo
- name: "eks"
  url: "<https://aws.github.io/eks-charts>"

releases:
# <https://github.com/aws/eks-charts/tree/master/stable/aws-load-balancer-controller>
# <https://github.com/aws/eks-charts/blob/master/stable/aws-load-balancer-controller/values.yaml>

- name: "alb-controller"
  namespace: "kube-system"
  chart: "eks/aws-load-balancer-controller"
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

use

bases:
- environments.yaml
Jim Conner avatar
Jim Conner

do you work with helmfile like as a maintainer? It might be that what I’m trying to do is flawed in the way I’m setting up directory structure…. I posted an issue for helmfile here

Documentation for helmfile DRYness lacks...and maybe not even DRY at all... · Issue #2036 · roboll/helmfile attachment image

I had to vent because I&#39;m so frustrated with helmfile. I&#39;m really struggling to wrap my head around how to implement this project of mine for helmfile. I&#39;m going to try really hard to a…

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

no, we just used it a lot

Jim Conner avatar
Jim Conner

If you read that issue, the directory structure I’m attempting to use is in there.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

what I posted above (the file structure and the sections in the files) work

Jim Conner avatar
Jim Conner

I need to update the issue with the part I’m working on right now though….

Jim Conner avatar
Jim Conner

well, I’m gonna be a little stubborn for now as the structure I’m trying to use would work very nicely with our infrastructure/cluster setup.

Jim Conner avatar
Jim Conner

otherwise, helmfile assets would just be a big glob of nasty yaml and too complex to maintain in my estimation

Jim Conner avatar
Jim Conner

and it will be more complex for us to maintain.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

you can have any file structure you want

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

just some sections are required, e.g.

bases:
- environments.yaml
Jim Conner avatar
Jim Conner

ftr, I updated https://github.com/roboll/helmfile/issues/2036 so my directory structure is there… I’m running helmfile from the helmfiles/environments/devel/test-cluster directory. That is where I’m getting the error from above.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

the folder structure is not important, the sections in the helmfile.yaml are

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

did you try

bases:
- environments.yaml
Jim Conner avatar
Jim Conner

well so using environments.yaml the way it’s documented may not be something I want to do yet. I’m still figuring out how that might work.

    keyboard_arrow_up