#helmfile (2022-01)

did you check https://github.com/roboll/helmfile#environment @Jim Conner ?

looks like if you use .Environment.Name you need to specify environment block(s)

or don’t use .Environment.Name since it’s deprecated

yup. though I’m not using an environments object so I’m assuming that it will just work…but it doesn’t.

at least according to the documentation it seems that it would work.

this worked for us

environments:
  default:
    values:
      - defaults.yaml
  ## Add additional environments with value overrides like this:
  production_east:
    values:
    - defaults.yaml
    - east.yaml
    - production.yaml

so environments is required?

if your code uses .Environment.xxxxx , yes

I don’t use .Environments at all.

(or could be just bad YAML formatting, happens all the time)

really strange. At any rate, I even tried though and it’s still throwing the same error.

all my stuff (so far) passes yamllint…though it might just be bad yaml for helmfile’s perspective.

try to remove parts of helmfile.yaml step-by-step until the error disappears - you would know in what part of the file it is (if it’s an invalid yaml+template)

sorry…gimme a sec

having a talk with the kid…

yeah, that’s the technique I’ve been using. Nothing working so far. I’ll figure it out eventually though. Thanks for your help…

My helmfile.yaml is super simple:

---
environments:
  default:
    values:
    - env-toggles.yaml

helmfiles:
  path: ../helmfile.yaml

ahhh, I think I’m misunderstanding the error. The environment: error is a red-herring of sorts, I think. The main error is at the bottom I think.

meh, still haven’t figured it out…but I’ll whittle it down.

try this:

file structure

environments.yaml :

helmfile.yaml:

use

bases:
- environments.yaml

do you work with helmfile like as a maintainer? It might be that what I’m trying to do is flawed in the way I’m setting up directory structure…. I posted an issue for helmfile here

no, we just used it a lot

If you read that issue, the directory structure I’m attempting to use is in there.

what I posted above (the file structure and the sections in the files) work

I need to update the issue with the part I’m working on right now though….

well, I’m gonna be a little stubborn for now as the structure I’m trying to use would work very nicely with our infrastructure/cluster setup.

otherwise, helmfile assets would just be a big glob of nasty yaml and too complex to maintain in my estimation

and it will be more complex for us to maintain.

you can have any file structure you want

just some sections are required, e.g.

bases:
- environments.yaml

ftr, I updated https://github.com/roboll/helmfile/issues/2036 so my directory structure is there… I’m running helmfile from the helmfiles/environments/devel/test-cluster directory. That is where I’m getting the error from above.

the folder structure is not important, the sections in the helmfile.yaml are

did you try

bases:
- environments.yaml

well so using environments.yaml the way it’s documented may not be something I want to do yet. I’m still figuring out how that might work.

The biggest problem for me is that I don’t even have an admin privilege on the repo that is required to add more maintainers to the project myself. I’ve asked the original author(roboll) regarding that about a year ago but we lack a progress

https://github.com/roboll/helmfile/issues/1824

I think it’ll have to be by people who use it stepping up to help improve both of these situations. Fully agree and I don’t worry much about that. I remember several people have stepped up saying they are eager to become maintainers.

I was also thinking itscaro would be a great candidate as a potential maintainer seeing his continued efforts on helmfile https://github.com/roboll/helmfile/pulls?q=is%3Apr+author%3Aitscaro+

Ouch. I was about to ask if forking would be a good idea, but you don’t get issues and PRs that way so that’s clearly a no go.

Yeah, that’s one of issues.

If we’re forking I’d also love it if we could “archive” the current repo so that everyone can notice that the project has been moved. Indeed, it’s impossible without admin privilege either…

yeah, true

That said, I’m more inclined to fork the repo these days just because… there seems to be no other options Maybe we’d better set a due date for moving the repo and do fork if it passed? WDYT?

Yes. Typically the way to handle that is to fork the repo…

what would be best, though, would be for roboll to pass the project on to you (I’m not using GH terms there but it’s something like that).

obviously, you’re the main guy now… @mumoshu

yeah… but I’ve already tried that for more than a year in places like https://github.com/roboll/helmfile/issues/1824 and slack dm

sucks.

well, not to be a bother, but @mumoshu have I got questions for you (#2036 and #2039) – however, right now, I need to get dinner for my family and spend a little time with them tonight, so if you have time in the next few days, I would love to chat.

I think it makes sense to fork the repo at this point.

welp, it’s 11PM now. I need to go to bed. @mumoshu perhaps we can talk about those two issues [my] tomorrow ([your tonight]) or something.

lemme know. I did my best articulating my thoughts, problems, concerns in the first ticket and in the second ticket, I’m pretty sure something I’m doing is causing the panic, but nevertheless, panics are bad (tm).

I’m afraid roboll can be considered as MIA now. His personal site and twitter are gone, he doesn’t communicate in gitlab, last activity was https://github.com/roboll/dotfiles was 10 months ago.

Fork would be a best option. Also for existing users awareness, current README can be updated to point to fork, so at least this would be visible enough.

Also forking in general sometimes is the last working resort to get attention of current owner. Maybe roboll will notice it and you’ll be able to sort things out.

One last thing, forking is totally okay, we’re all working in rapidly changing industry, so breaking changes are expected and we know how to deal with them.

terraform helm provider (which I think Mumoshu maintains/wrote actually) but imo, maybe terraform-provider-helmfile? then yes, I maintain it :slightly_smiling_face:

deploying software with terraform is a really really really bad idea blending infrastructure with software, which is poor practice. You really should keep those two things separate. I generally agree with you!

my motivation for building terraform-provider-helmfile was to use it only for deploying system components whose lifecycles are tied to that of the cluster. that said, I consider deploying “apps” with terraform-provider-helmfile as no-go.

It’s not that we stopped using helmfile, we stopped using helmfile the way we were using it, which reduced the number of places we have it. like all tools, they have a time and place, and sometimes the tools we start with evolve over time.

Inevitably, we find we’re just swapping out one tool with one set of problems for another tool with another set of problems.

So the problem we ran into is we need to provision secrets, IAM roles and policies, backing services (E.g. redis, rds, etc), and we need to do it after the cluster is online.

Let’s take a prime example: argocd

We leverage argocd more and more for application CD, but less so for backing services CD, for which we’re big proponents of Spacelift.

Something needs to provision argoCD, which could be helmfile, or terraform. We chose terraform so we can have one component terraform the github repos and provision argocd in one fell swoop.

But like I said about exchanging problems, this led to a new problem, that we don’t get the pretty diffs of helmfile with helm+helm diff. This seems to be fixed by enabling the experimental feature of the helm provider to set manifest=true , but it’s not perfect.

Here are some related issues we are tracking:

• https://github.com/databus23/helm-diff/issues/176

• https://github.com/hashicorp/terraform-provider-helm/pull/702

• https://github.com/databus23/helm-diff/issues/176#issuecomment-572952711

• https://github.com/databus23/helm-diff/pull/304

And it turns out that with helm3 some of the same problelms we encountered with the helm provider exist now in helmfile too (but didn’t exist in helm2).

So you mentioned that terraform templating is pretty bad, but in 0.13+, I think it got a lot better. That said, we don’t template YAML when we use terraform because we don’t need to.

Instead, we use native HCL and call jsonencode once we’ve constructed the object.

The other challenge was problem more related to our type of customer. Customers seek us out precisely because of our terraform chops, so introducing helmfile for this one use-case meant learning another tool with advanced templating. We exploited the out of the advanced templating, and while it made sense to us, it seems like users new to k8s struggled with all the layers. Reducing it to terraform seems to improve our deliverables.

We have a vast testing infrastructure based on terraform, and an efficient distribution model based on terraform modules and the terraform registry. Using terraform we’re able to deliver an end-to-end implementation in a module which is not really possible in pure helmfile without using hooks and shell scripts to create things like IAM roles.

For customer apps, we recommend argocd, which still has the same problems that we cannot create infrastructure in the same phase as we deploy apps in argocd. It requires a coordinated rollout.

What a great explanation @Erik Osterman (Cloud Posse) ! Actually, we are tracking the same issues and have pretty much the same concerns diffs-wise. Nevertheless we still rely on helmfile dealing with app deployments. More on that, about a year or so ago we started using Google Config Connector which allows us to create cloud resources via k8s manifests. So, nowadays usually we either create/reuse a Helm chart to create a clouds resource or add a manifest of a cloud resource to existed charts enhancing their functionality. And we continue using Terraform for, say, the foundation layer (up to creating a K8s cluster).

We more or less like what we’ve got, the flexibility, the workflow, etc. However, one of the major things here is that there are some cloud services that can be managed only with Terraform/Pulumi: Elastic Cloud, MongoDB Atlas, etc. Adopting these pieces brought up some arguments to reconsider or refine our approach. That’s where we are right now)

Thanks for sharing! I learned a lot :)

The world definitely looks much more complex than a year or two ago. At that time I thought terraform + helmfile was the only option. But now, I see many variants of workflows build from arbitrary combinations of the below:

Infra:

• Terraform + k8s/helm tf provider

• Plumi

• aws-cdk (+cdk-k8s)

• Custom script (e.g. run tf then helmfile) Apps:

• Run kubectl/kustomize/helm/helmfile from CircleCI, GitHub Actions, Atlantis, etc

• ArgoCD (+ Helmfile as a ArgoCD custom config management plugin sometimes)

@Erik Osterman (Cloud Posse) kudos for all this great notes,

Thanks @Erik Osterman (Cloud Posse), What a great rundown of the struggles of orchestrating cloud delivery from bootstrap to app deployment. Are you only using cicd pipelines for these stitched together orchestrations or have you been exploring other tools to fill this particular gap?

Somethings I found in my experience:

• jsonencode converted everything to strings, so if non-strings, like actual ints/booleans, this was a problem.

• Issues w/ K8S provider where HCL changed, but manifests did not. This is with maps or blocks becoming hash/map. Docs are not really good for this.

• Issues w/ K8S provider deploying things differs in behavior from kubectl

• Some niche scenarios where what would be a patch (update) in helm/kubectl, becomes delete and recreate in Terraform. If I stay with basic helm charts, I can get away with mostly using Terraform, but the moment I need to do more advanced templating or mix this with kubernetes manifests, such as for CRDs, then things get messy with pure terraform. This happens in more of platform layer (prometheus, cert-manager, etc) but in the app layer, it makes sense to be pure TF for this.

At my current organization (google shop), they are the opposite, starting with google deployments (ugh) and anchor helm charts (no helmfile), and are now moving at least infra to Terraform, last I checked. They were considering going toward using k8s operator to manage cloud deployments (which is something that seems scary to me).

In most typical cluster setups, there are generally and pretty consistently, in my experience, at least three components for environments….

1. the cluster/infrastructure a. when using a cloud provider b. if using bare-metal then this could be split into two completely disparate components
2. the substrate e.g. components running on infrastructure to support the main application/micro-service(s)
3. the application Practices which could be discussed:
4. always separate infrastructure from software deployments to eliminate accidental actions from occurring. a. terraform was meant for infrastructure. Some people believe it’s a good idea to use terraform for infra and software but I personally think this is a really really bad idea (we’ve been bit in the collective butts by this). b. day-1 deployments typically work fine…but c. extra precautions, code, software must be taken or used to keep unintended issues from occurring on day-2 deployments. which could just add to tech debt…moreover, d. when using terraform to deploy substrate, day-2 operations require terraform to run which means terraform must check the state and possibly apply to your infrastructure before getting to substrate/software deployment, which is usually not desired and unless expected is definitely not suggested. e. in our experience, if something in the substrate needs attention, terraform helm provider doesn’t always catch the correct state of helm charts deployed and extra work needs to be done to “fix” that which for us usually means side-deleting a chart and then re-applying terraform; see #a
5. substrate software should be deployed using helm and better if deployed using some kind of package release software like helmfile, helmwave (maybe, I don’t know much about this) or home-grown wrapper…to ease release – a. It’s up to engineers how best to perform this task. We’re migrating out of terraform and trying out helmfile.
6. for deploying your software stack, something like argocd is likely requisite but this is also up to engineering teams to decide. The idea behind tooling is to:
7. ease toil and burden on operations teams and devs
8. effectively perform the tasks necessary for proper operations and uptime of application(s)

You can definitely run helmfile in Atlantis

Create a custom workflow

Thanks for the suggestion Erik

https://archive.sweetops.com/atlantis/2020/09/#ef545741-75ba-4c43-9201-734a5db7c1bb

If you search the archives you will find example configurations

I am on my phone so it’s a bit tedious to search

Thanks again Erik, it’s good to see that this was done by someone else

There are also custom script possibilities in Env0)

We are running helmfile w/ a custom atlantis workflow

Works great as long as all your secret material is in k8s Secret resources

Here’s an example:

https://archive.sweetops.com/atlantis/2019/09/#d7f87ce2-312f-4cc8-a744-c8620b3f3f11

I thought it’s .Release.Namespace which is populated by releases[*].namespace. Could you confirm?

I noticed when I do this, I don’t see namespace injected with helm template This is a different issue. It comes from the nature of helm-template. helm-template doesn’t set metadata.name in the resulting resources when the chart templates don’t include the below snippet for every resource:

metadata:
  namespace: {{ .Release.Namespace }}

I do see when finally deployed. I think this is because helm-upgrade set the namespace when calling the k8s client for updating k8s resources in the cluster.

Even if Helm sees no metadata.namespace in each rendered manifest, K8s give each a corresponding namespace according to the client config.

I’d say it’s an expected behavior. It isn’t really easy to understand, though.

I tried my best explaining how it works https://github.com/roboll/helmfile/issues/2036#issuecomment-1010503017

and I appreciate it…sorry to keep bothering you.

lemme take a look

awesome! It’s clear as mud but better than nothing! Thank you @mumoshu

So, I’m not sure how to grok this:

err: failed to read common/global-default.env.yaml.gotmpl: reading document at index 1: yaml: unmarshal errors:
  line 2: cannot unmarshal !!map into []interface {}
changing working directory back to "/home/jimconn/projects/src/bluescape/ops/projects/helmfile-project/environments/devel"
changing working directory back to "/home/jimconn/projects/src/bluescape/ops/projects/helmfile-project/environments/devel/test-cluster"
in ./helmfile.yaml: in .helmfiles[0]: in ../helmfile.yaml: in .helmfiles[0]: in ../helmfile.yaml: failed to read common/global-default.env.yaml.gotmpl: reading document at index 1: yaml: unmarshal errors:
  line 2: cannot unmarshal !!map into []interface {}

the whole debug output is (see snippet in #helmfile)

it looks like everything mutates fine so far…but the error then says it can’t read the file it’s already read. Confusing.

https://sweetops.slack.com/files/U01U2JYA7NE/F02TMB4HKR8/untitled.txt

I’ll brb. I gotta pick up my daughter from school

It’s failing while parsing the render output as YAML.

The render output is emitted immediately before the parsing part, so reading it would help debugging.

I took a glance and it seems like the second line in the render output doesn’t conform to the structure of helmfile config

 0: values:
 1:   domain_name: "jimtest.dev.company"

values must be an array, rather than a dict

gotta eat dinner…when I’m done I’ll dig into it.

making progress….

good stuff.

https://jenkins-x.io/community/labs/enhancements/proposals/4/readme/

I still hope to never touch jenkins again, but that is pretty cool.

This is the structure i’m using

What is inside of these templates? helmfile definitions or k8s manifests? How do you install things?

With helmfile you can define helm releases, i.e it operates with charts. It also can make releases out of kustomize. And it’s possible to apply plain k8s manifests via hooks.

Just plain k8s manifests, but with some {{.Values...}} things in them

As an example, this is my crossplane-secret.yaml file:

apiVersion: v1
kind: Secret
metadata:
  namespace: crossplane-system
  name: crossplane-db-conn
stringData:
  endpoint: {{ .Values.mysql.endpoint | quote }}
  port: {{ .Values.mysql.port | quote }}
  username: {{ .Values.mysql.username | quote }}
  password: {{ .Values.mysql.password | quote }}

And this is my values.yaml

mysql:
  endpoint: {{ requiredEnv "MYSQL_ENDPOINT" }}
  port: 3306
  username: {{ requiredEnv "MYSQL_USERNAME" }}
  password: {{ requiredEnv "MYSQL_PASSWORD" }}

In terms of my install, I have this in my helmfile.yaml

repositories:
- name: crossplane-stable
  url: <https://charts.crossplane.io/stable>

releases:
- name: crossplane
  chart: crossplane-stable/crossplane
  namespace: crossplane-system
  version: v1.6.1
  installed: true
  values:
    - values.yaml.gotmpl

and it is all ran by doing:

helmfile --file $CI_PROJECT_DIR/environments/helm/helmfile.yaml sync

Whereby $CI_PROJECT_DIR/environments/helm/helmfile.yaml is just a master helmfile.yaml that refers to other helmfile.yaml within subdirectories like so:

helmfiles:
  - path: applications/*/helmfile.yaml

I think this would work if i had an individual Chart.yaml file in each of these directories too, but I’d like to avoid that, since I don’t actually want these to be charts, just to be installed in addition to the rest of the dependency

And how do you you want these configmaps to be installed?) I’m not aware of the most recent features of helmfile , therefore given your example I would either make a separate chart with these configMaps and then define a release in helmfile.yaml or utilise incubator/raw chart and put all these configMaps there as values:)

Ideally i’d like it to just behave as though i had a separate chart for these dependencies, without me having to define one

I figure that doesn’t exist though

incubator/raw is deprecated, this one can be used instead https://github.com/deliveryhero/helm-charts/tree/master/stable/k8s-resources

I prefer this method (ambassador/helmfile.yaml ):

{{ if .Values.enable_ambassador }}
---
releases:
  - name: ambassador
    namespace: {{ .Values.ambassador.namespace }}
    labels:
      app: ambassador
      tier: secrets-management
    values:
      - chart_repo: {{ .Values.ambassador.chart_repo }}
      - chart_version: {{ .Values.ambassador.chart_version }}
      - values.yaml.gotmpl
{{ end }}

However, that yields:

...
...
...
second-pass rendering failed, input of "helmfile.yaml.part.0":
 0: {{ if .Values.enable_ambassador }}

err: error during helmfile.yaml.part.0 parsing: template: stringTemplate:1: unexpected EOF
changing working directory back to "repo/projects/helmfile-project/charts/10-cluster-mgmt"
changing working directory back to "repo/projects/helmfile-project/charts"
changing working directory back to "repo/projects/helmfile-project"
changing working directory back to "repo/projects/helmfile-project/environments"
changing working directory back to "repo/projects/helmfile-project/environments/devel"
in ./helmfile.yaml: in .helmfiles[0]: in ../helmfile.yaml: in .helmfiles[0]: in ../helmfile.yaml: in .helmfiles[0]: in charts/helmfile.yaml: in .helmfiles[1]: in 10-cluster-mgmt/helmfile.yaml: in .helmfiles[0]: in ambassador/helmfile.yaml: error during helmfile.yaml.part.0 parsing: template: stringTemplate:1: unexpected EOF

Dang! I feel dumb. I figured it out. I renamed helmfile.yaml to helmfile.yaml.gotmpl and it renders correctly now.

interesting. helmfile.yaml is rendered as a go template first so giving it a .gotmpl or not shouldn’t make differences

 0: 
 1: releases:
 2:   - name: ambassador
 3:     namespace: ambassador
 4:     labels:
 5:       app: ambassador
 6:       tier: secrets-management
 7:     values:
 8:       - chart_repo: datawire/ambassador
 9:       - chart_version: 6.6.2
10:       - values.yaml.gotmpl
11: 

your release lacks chart: so i thought it’s the cause of this error
encountered empty chart while reading release “ambassador” at index 0

Yeah, so I thought that the naming convention of the helmfile.yaml was the problem but in turns out the issue that seemed resolved was in fact only masked by something else. The issue persists.

second-pass rendering failed, input of "helmfile.yaml.gotmpl.part.0":
 0: {{ if .Values.enable_ambassador }}

err: error during helmfile.yaml.gotmpl.part.0 parsing: template: stringTemplate:1: unexpected EOF
changing working directory back to "repo/projects/helmfile-project/charts/10-cluster-mgmt"
changing working directory back to "repo/projects/helmfile-project/charts"
changing working directory back to "repo/projects/helmfile-project"
changing working directory back to "repo/projects/helmfile-project/environments"
changing working directory back to "repo/projects/helmfile-project/environments/devel"
in ./helmfile.yaml: in .helmfiles[0]: in ../helmfile.yaml: in .helmfiles[0]: in ../helmfile.yaml: in .helmfiles[0]: in charts/helmfile.yaml: in .helmfiles[0]: in 10-cluster-mgmt/helmfile.yaml: in .helmfiles[0]: in ambassador/helmfile.yaml.gotmpl: error during helmfile.yaml.gotmpl.part.0 parsing: template: stringTemplate:1: unexpected EOF

Is there something wrong with that I’m trying to do here? My helmfile.yaml.gotmpl:

✦ at 17:19:19 ❯ \cat ambassador/helmfile*
{{ if .Values.enable_ambassador }}
---
releases:
  - name: ambassador
    namespace: {{ .Values.ambassador.namespace }}
    chart: {{ .Values.ambassador.chart_repo }}
    labels:
      app: ambassador
      tier: cluster-mgmt
    values:
      - chart_version: {{ .Values.ambassador.chart_version }}
      - values.yaml.gotmpl
{{ end }}

@mumoshu

As I’ve written in the issue, -- is used to separate a file into a multiple yaml templates

{{ if .Values.enable_ambassador }}
---
releases:
  - name: ambassador
    namespace: {{ .Values.ambassador.namespace }}
    labels:
      app: ambassador
      tier: secrets-management
    values:
      - chart_repo: {{ .Values.ambassador.chart_repo }}
      - chart_version: {{ .Values.ambassador.chart_version }}
      - values.yaml.gotmpl
{{ end }}

so this is wrong.

sure, I understand that.

what I’m trying to do is get helmfile to bypass a release if it is not enabled.

??

can you jsut do this instead?

{{ if .Values.enable_ambassador }}
releases:
  - name: ambassador
    namespace: {{ .Values.ambassador.namespace }}
    labels:
      app: ambassador
      tier: secrets-management
    values:
      - chart_repo: {{ .Values.ambassador.chart_repo }}
      - chart_version: {{ .Values.ambassador.chart_version }}
      - values.yaml.gotmpl
{{ end }}

sure

lemme try

i don’t understand why you need --- to do that

I have --- in the top of all of my helmfile.yamls

is that…not OK?

sure it’s okay if you want it (for some reason that i don’t know of), if you’re really putting into in the top

{{ if .Values.enable_ambassador }}
---

obvisouly --- isn’t in the top for the above case..? that’s where i confused

I might be abusing the template in this example but the thought was that --- starts a new valid release…and if I want the release disabled then effectively it doesn’t exist. In other words, the start of the release, I thought, started with --- but if the release is disabled then --- will never even get parsed….was my thinking

ah well perhaps helmfile fill complain about missing releases and sub-helmfiles if you had an empty document in the beginning of every helmfile.yaml

— starts a new valid release i couldn’t understand this part

it should align with what i’ve written in https://github.com/roboll/helmfile/issues/2036#issuecomment-1010503017

--- splits the file into documents BEFORE rendering so that’s why --- within the if block breakks it

--- doesn’t start a new set of releases. it just splits the config into layers where each layer is rendered and merged in that order.

I get a little confused about how YAML streams work with respect to --- vs how helmfile might be using streams. If you follow the YAML spec, I thought, then each new stream is supposed to start with a --- so that’s why each of my helmfile.yaml files start with a --- out of just good practice, but that might be something I need to change…

OK so, it’s like this… Let’s assume that this is my helmfile.yaml and when helmfile parses it, it will perform a release.

---
some valid stuff here

Now, let’s say I want to enable some helmfile releases in some directories and disable others…. if I enable a helmfile (stream) then something like:

{{ if .Values.enabled }}
---
helmfile stuff
{{ end }}

our use of --- doesn’t follow the yaml spec.

ah hah!!!

ok, that’s good to know!!

so it’s actually even MORE important than I thought…ok, that’s huge.

yeah glad you understand it now

OK, so I’ve removed all of the ---s in my helmfiles where I was using it inside the conditionals

I’m getting a different error now so I’ll try and work through that. Thank you @mumoshu

well.. im really confused what you understand and not. i think i’ve already explained that --- isn’t that of yaml spec defines in https://sweetops.slack.com/archives/CE5NGCB9Q/p1642641966099100?thread_ts=1642467523.092800&cid=CE5NGCB9Q, but perhaps i wasn’t clear enough…??

wishing you manage it!

the concept about what you were trying to explain made sense to a degree but it didn’t make sense with respect to the difference between YAML and helmfile. That difference was never explicitly mentioned so I just assumed that I had to work around helmfile spec and YAML spec….

ah okay, i see where you come from now! i thought it’s obvious that it can’t be --- from the yaml spec, as it can only be either (1) load helmfile.yaml as as a stream of yaml documents separated by --- and render each yaml doc as a go template, merge rendered documents as layers OR (2) load helmfile.yaml as a steram of go templates separated by ---, render each template, parse the render output as a yaml, merge resulting yaml documents

and (1) isn’t possible as a go template file can’t be a valid yaml

the more we talk about this yaml+go template gotchas the more i get to think we must have outgrew it and need a more serious programming or configuration language

thanks anyway for clarifying!

no problem!!

I appreciate your time. Out of curiosity…do you have thoughts about what other candidates for programming config language?

i’m looking one that fits helmfile for almost 2 years now and have no single answer yet

helmfile v2?

have you heard of helmwave?

sort of > v2 but i’d keep the current yaml+go template api for a while for maybe it’s more like v1 with additional api with the new language

yes, but i havent tried it yet > helmwave

yeah, that’s a good idea

i know it because i also maintain helm-diff which is apparently used by helmwave too and helmwave’s author sometimes come and submit a feature request that helps helmwave’s use case

I looked into it. helmfile has already had a steep learning curve. I am not about to go learning something new. Even their documentation is not good enough to easily craft something that would work for our environment. I’m finally starting to get a better grasps of how to use helmfile for how I want to do my project…and even that’s not working exactly perfectly yet.

i was once convinced to adopt jsonnet and cue, but it turned out it’s really not meant as a extensible language. custom functions aren’t well supported in neither jsonnet or cue

we use jsonnet in our environment and while it’s powerful, it’s more cognitive load and extraneous work for helm projects. It would be like using an asteroid to kill an ant.

yeah i can see that… and jsonnet and cue can be used to generate a complex and regular (go template free) helmfile.yaml so i tend to recommend it in cases where you’re just using helmfile’s template only for doing somthing that can be possible in jsonnet/cue.

out of curiosity… did I describe my project environment well enough for you to understand in this issue https://github.com/roboll/helmfile/issues/2036#issuecomment-1010503017?

yes. and i see that the structure of your project is not that uncommon

so it would work in the end

I was hoping to make it so that I could run helmfile from any of the directories minimally specifying requirements to helmfile and that the context of the local helmfile.yaml would be enough to work…but that’s not turning out to be the case yet… that will be something I tackle later. For now, being able to run it from a specific environment should be enough.

at any rate, I need to get dinner for my family. I might pop on later and work on this project of mine…Thanks @mumoshu

ttyl

probably tfstate-lookup. from the error message im not entirely sure at which level it failed though :thinking_face: maybe filestore_instance_ip was formerly a object that contains an array named value but now it’s not?

does rewiting

output.filestore_instance_ip.value[0]

to

output.filestore_instance_ip[0]

work?

It seems you are totally right. I’m trying to get some values using the most recent vals (0.15.0) and it gets the right value without .value at the end of the reference.

thanks for your confirmation! apparently we’re affected by breaking change in tfstate-lookup (but we can’t blame tfstate-lookup as they’re at 0.x and it doesn’t have any feature flag regarding the feature

fwiw, i was just reading recent changes to tfstate-lookup and found this: https://github.com/fujiwara/tfstate-lookup/pull/23 apparently this was the change broke it for us

Cool! Does it make sense to fix vals README like this: https://github.com/variantdev/vals/pull/70 ?

But this section https://github.com/roboll/helmfile/blob/master/docs/writing-helmfile.md#user-content-layering-release-values says that it is not for attaching additional values

This works if I include values1.yaml.gotmpl file from environments.default.values section, but the problem is that environment variables are global, but I need to fill release-specific values.

I guess it will work if values1.yaml is included in your environment

Yes, it works when included in environment!

But I’m searching ways to have per-release “values” files, instead of one large “values” file in environment, that contains config for all releases ‘cuz it so inconvenient…

Also environment values are global and available for all releases, so instead of simply using something like mariadb.auth.username I must manually add prefix to them like my-chart.mariadb.auth.username (with catching this issue).