#helmfile (2019-12)

https://github.com/roboll/helmfile

Questions and discussion around helmfile https://github.com/roboll/helmfile and https://github.com/cloudposse/helmfiles Archive: https://archive.sweetops.com/helmfile/

2019-12-31

Yingjun Hu avatar
Yingjun Hu

It seems like the releases defined in helmfile are executed async?

Erik Osterman avatar
Erik Osterman

I think there’s a --concurrency flag you can use to set parallelism to 1 and then it would be synchronous

Yingjun Hu avatar
Yingjun Hu

I see, thanks!

2019-12-30

DanB avatar

We are looking into adopting helmfile for our existing helm use cases. We currently have 15-20 active clusters at any given time, but only about 10 are permanent (we spin up clusters several times throughout the day). These clusters are spread accross aws accounts, on prem and across regions. We don’t have the concept of a linear dev -> stage -> prod. To get to the point, can anyone point me to an example helmfile that might fit this use case?

Erik Osterman avatar
Erik Osterman

we use helmfile much the same way. there isn’t necessarily a linear dev->stage->prod SDLC. We also use it to manage releases in multiple clusters, across multiple aws accounts, across multiple customers

Erik Osterman avatar
Erik Osterman

Perhaps this is why we rely more on “environment variables” than the construct of “environments”

Erik Osterman avatar
Erik Osterman
cloudposse/helmfiles

Comprehensive Distribution of Helmfiles for Kubernetes - cloudposse/helmfiles

Erik Osterman avatar
Erik Osterman

here are all of our helmfiles

Erik Osterman avatar
Erik Osterman

We store a lot of settings in SSM and use chamber to call helmfile and pass those settings

Erik Osterman avatar
Erik Osterman

we write a lot of settings to SSM using terraform

Erik Osterman avatar
Erik Osterman

(there’s also now a terraform-helmfile-provider )

DanB avatar

thanks for the links. Where do you keep your environment variables? Big fans of IaC, ie if we multiple charts that reference a specific vpc depending on account and region, where would that vpc id go?

Yingjun Hu avatar
Yingjun Hu

You can probably refer to my earlier discussion https://github.com/roboll/helmfile/issues/1045. But I guess the only way (and the way I’m using) is to define environments in helmfile and inject values to your releases.

Environment values not being used in nested states · Issue #1045 · roboll/helmfile

I've been trying to use environments in my main helmfile and have multiple sub helmfiles in my releases folder. I wanted all the sub helmfiles to pick up whatever the value I defined in a speci…

Erik Osterman avatar
Erik Osterman

In our case, the VPC ID would be an output from a Terraform module, that would be written to SSM

Erik Osterman avatar
Erik Osterman

As much as possible, we try to keep dynamically generated values (e.g. vpc_id out of code because and into the parameter store so that it can be dynamic and easily automated).

DanB avatar

vpc id was maybe a poor example, maybe chart version? ie a different chart version depending on aws account/region.

2019-12-26

2019-12-25

2019-12-24

2019-12-23

Andrey Nazarov avatar
Andrey Nazarov

Do sub-helmfiles inherit helmDefaults?

Yingjun Hu avatar
Yingjun Hu

wondering the same thing. Still want to find a way to let sub-helmfiles inherit values from upstream

mumoshu avatar
mumoshu

no they don’t. the general rule is that nothing other than Environment.Name should be inherited

Roderik van der Veer avatar
Roderik van der Veer

I’m trying to run helmfile with helm 3, (3.0.2) and i installed helm diff 3.0.0-rc.7 but when running the diff plugin fails without an error message

Comparing release=namespace, chart=incubator/raw
Comparing release=shared-secets, chart=settlemint/shared-secrets
Comparing release=rabbitmq, chart=stable/rabbitmq-ha
Comparing release=launchpad, chart=settlemint/launchpad
Comparing release=ingress, chart=stable/nginx-ingress
in ./helmfile.yaml: 5 errors:
err 0: failed processing release rabbitmq: helm exited with status 1:
  
err 1: failed processing release namespace: helm exited with status 1:
  
err 2: failed processing release launchpad: helm exited with status 1:
  
err 3: failed processing release shared-secets: helm exited with status 1:
  
err 4: failed processing release ingress: helm exited with status 1:
  

Any idea what’s wrong or on how to debug this?

Andrey Nazarov avatar
Andrey Nazarov

What if you add --log-level=debug ?

:--1:1
Roderik van der Veer avatar
Roderik van der Veer
~/Development/launchpad/infrastructure helm3 !3 ❯ helm diff upgrade --reset-values --allow-unreleased launchpad settlemint/launchpad --version 1.0.0-helm3 --kube-context launchpad --namespace alpha --detailed-exitcode                   
~/Development/launchpad/infrastructure helm3 !3 ❯ echo $?                                                                                                                                                                                       
1
Roderik van der Veer avatar
Roderik van der Veer

No output but a 1 exit code

Roderik van der Veer avatar
Roderik van der Veer

if i remove namespace and context it starts to return stuff

Roderik van der Veer avatar
Roderik van der Veer

will figure out the mismatch there

Roderik van der Veer avatar
Roderik van der Veer

ok, my context was wrong

Roderik van der Veer avatar
Roderik van der Veer

Sorry for the flurry of questions, but what would be the reason that i get a ’failed to download incubator/raw` error with this helmfile

 repositories:
  - name: stable
    url: <https://kubernetes-charts.storage.googleapis.com>
  - name: incubator
    url: <https://kubernetes-charts-incubator.storage.googleapis.com>
  - name: harbor
    url: <https://helm.goharbor.io>
  - name: elastic
    url: <https://helm.elastic.co>
  - name: kiwigrid
    url: <https://kiwigrid.github.io>


helmDefaults:
  cleanupOnFail: true
  verify: true
  wait: true
  timeout: 600
  force: true
  atomic: true

releases: 
  - name: mynamespace
    chart: incubator/raw
    version: 0.2.3
    values:
      - resources:
        - apiVersion: v1
          kind: Namespace
          metadata:
            name: {{ .Namespace }}
Roderik van der Veer avatar
Roderik van der Veer
processing releases in group 1/4: alpha/mynamespace
worker 1/1 started
worker 1/1 finished
worker 1/1 started
Upgrading release=mynamespace, chart=incubator/raw
exec: helm upgrade --install --reset-values mynamespace incubator/raw --version 0.2.3 --verify --wait --timeout 600s --force --atomic --cleanup-on-fail --namespace alpha --values /var/folders/nr/3q7q00qs4hv46dwwffc9jp6h0000gn/T/values359668070 --history-max 10
exec: helm upgrade --install --reset-values mynamespace incubator/raw --version 0.2.3 --verify --wait --timeout 600s --force --atomic --cleanup-on-fail --namespace alpha --values /var/folders/nr/3q7q00qs4hv46dwwffc9jp6h0000gn/T/values359668070 --history-max 10: 
worker 1/1 finished

FAILED RELEASES:
NAME
mynamespace
err: release "mynamespace" in "helmfile.yaml" failed: failed processing release mynamespace: helm exited with status 1:
  Error: failed to download "incubator/raw" (hint: running `helm repo update` may help)
in ./helmfile.yaml: failed processing release mynamespace: helm exited with status 1:
  Error: failed to download "incubator/raw" (hint: running `helm repo update` may help)
Roderik van der Veer avatar
Roderik van der Veer

incubator is there when doing helm repo list, searching returns the raw chart and version

Roderik van der Veer avatar
Roderik van der Veer

found it:

Error: failed to fetch provenance "<https://kubernetes-charts-incubator.storage.googleapis.com/raw-0.2.3.tgz.prov>"
helm.go:76: [debug] failed to fetch provenance "<https://kubernetes-charts-incubator.storage.googleapis.com/raw-0.2.3.tgz.prov>"
Andrey Nazarov avatar
Andrey Nazarov

Yep, verify: false should fix this.

Roderik van der Veer avatar
Roderik van der Veer

is it normal that if i put it to true in the defaults, that setting it to false on the release does not work?

Andrey Nazarov avatar
Andrey Nazarov

Could be a bug:)

Juan Soto avatar
Juan Soto

Hi, thank you all for helping me to develop my solution with Helmfile

Juan Soto avatar
Juan Soto

now I need to integrate this deployment in CircleCI

Juan Soto avatar
Juan Soto

They way I am deploying my app is:

Juan Soto avatar
Juan Soto
Juan Soto avatar
Juan Soto

Anybody has experience on running helmfile pipelines on CircleCI ? I already have the source code in github.

Juan Soto avatar
Juan Soto

and the K8s cluster is running on GKE

Juan Soto avatar
Juan Soto

Anybody has tested that ?

Balaji J avatar
Balaji J

Hi all, hope you doing good. Im completely new to helm and helmfile. Can someone please clarify the difference between using helmfile vs creating a umbrella chart with dependency using requirements.yaml? Are they not solving the same usecase of deploying a group of charts for a complete deployment of applicaiton with dependant charts? Can you please clarify to understand the advantage the helmfile gives apart from environment specific deployment support like dev,production etc?

Balaji J avatar
Balaji J

anyone?

Zachary Loeber avatar
Zachary Loeber

I would highly recommend simply trying them both out to solve an issue. I think the difference feels like the difference between using a screwdriver and a powertool personally.

Zachary Loeber avatar
Zachary Loeber

you can use helmfile to template out solution stacks built from helmfiles in very innovative ways

Erik Osterman avatar
Erik Osterman

One problem with umbrella charts is you cannot easily decouple releases down the road if you want to reorganize how you deploy them. You can activate/deactivate dependencies easily, but that will effectively destroy them.

Erik Osterman avatar
Erik Osterman

With helmfile, you don’t have that problem. You control everything about the release.

Erik Osterman avatar
Erik Osterman

With umbrella charts, you control the chart and manage the SDLC. That’s a bit of investment. With helmfile, you don’t need to construct new charts just for the purpose of bundling services and dependencies.

Erik Osterman avatar
Erik Osterman

With helmfile, you can surgically target one dependency and update it.

Balaji J avatar
Balaji J

thanks @Erik Osterman for the clarification… so in umbrella chart its not possible to update single dependant chart without restarting the main chart?

Erik Osterman avatar
Erik Osterman

Right you would need to update the parent chart and it couples their life cycles then together

Erik Osterman avatar
Erik Osterman

The other major major thing that Helmfile adds is the parameterization of values

Erik Osterman avatar
Erik Osterman

In helm, the values.yaml is static. With Helmfile we have the full power of gotemplates in values

Erik Osterman avatar
Erik Osterman

And with Helmfile environments we can create any number of helm releases on the fly

Erik Osterman avatar
Erik Osterman

Helmfile is a swissarmy knife for managing helm releases of charts by dozens of different vendors, that’s proved invaluable to us. With Helmfile we have a consistent interface for deploying charts

2019-12-22

Juan Soto avatar
Juan Soto

Hi, I ’ve been trying different approaches but I am still having problems referencing Environment Values in the helmfile.yaml

Juan Soto avatar
Juan Soto
mumoshu avatar
mumoshu

Your helmfile.yaml?

Juan Soto avatar
Juan Soto
mumoshu avatar
mumoshu

Separate releases and bases with ---

mumoshu avatar
mumoshu
environments:
  apps-1:
    values:
      - ../common/defaults.yaml
      - ../environments/apps-1.yaml
    secrets:
      - ../common/secrets.sops.yaml
    missingFileHandler: Error
---
releases:
{{ range $index, $ns := .Environment.Values.applicationNamespaces }}
# ...
Juan Soto avatar
Juan Soto

Great! It worked

1
Juan Soto avatar
Juan Soto

Now I am having different issue

Juan Soto avatar
Juan Soto
mumoshu avatar
mumoshu

you need to create it with kubectl create ns dev beforehand in helm 3

mumoshu avatar
mumoshu

or use raw chart

releases:
- name: ns
  chart: incubator/raw
  values:
  - resources:
    - apiVersion: v1
       kind: Namespace
      metadata:
        name: dev
- name: yourapp
  chart: yourchart
  namespace: dev
  needs:
  - ns
Roderik van der Veer avatar
Roderik van der Veer

wow, this should really be in the README.md!

Juan Soto avatar
Juan Soto

Can’t I include the creationg of the ns using the chart?

mumoshu avatar
mumoshu

you can. see above

Juan Soto avatar
Juan Soto

ok

Juan Soto avatar
Juan Soto
Juan Soto avatar
Juan Soto

in ./helmfile.yaml: failed to read helmfile.yaml: reading document at index 1: yaml: line 7: mapping values are not allowed in this context

Juan Soto avatar
Juan Soto

humn…identation problem

mumoshu avatar
mumoshu

yup. fix it and it would work

Juan Soto avatar
Juan Soto

great. Fixed!. What’s more, both environments are being deployed to their relevant namespaces

Juan Soto avatar
Juan Soto

Both environment are working! Great

Juan Soto avatar
Juan Soto

I am having the following error

Juan Soto avatar
Juan Soto
$ helmfile -e stage apply
in ./helmfile.yaml: failed to read helmfile.yaml: reading document at index 1: yaml: unmarshal errors:
  line 31: cannot unmarshal !!int `5` into []string
Juan Soto avatar
Juan Soto
Juan Soto avatar
Juan Soto

any ideas?

mumoshu avatar
mumoshu

try seeing line 31 of your subhelmfile

mumoshu avatar
mumoshu

the error indicates it’s due to a type mismatch between your replicas defined in stage.yaml and your call-side(line 31 of sub-helmfile)

Juan Soto avatar
Juan Soto
mumoshu avatar
mumoshu

values: {{ .Environment.Values.replicas }} seems wrong

Juan Soto avatar
Juan Soto

namespace and environment values are string and they went through

Juan Soto avatar
Juan Soto

but replicas variable for some reason is not working.

Juan Soto avatar
Juan Soto

It has an integer as it value.

mumoshu avatar
mumoshu

yeah so use value: {{ .Environment.Values.replicas }} not values: {{...}

Juan Soto avatar
Juan Soto

thanks

Juan Soto avatar
Juan Soto

Now I need to put that value inside the helm chart

Juan Soto avatar
Juan Soto

in chart1/values.yml

Juan Soto avatar
Juan Soto
Juan Soto avatar
Juan Soto

I am going to change it to

Juan Soto avatar
Juan Soto
Juan Soto avatar
Juan Soto
$ helmfile -e stage apply
Building dependency release=caylent-helmfile, chart=caylent
in ./helmfile.yaml: helm exited with status 1:
  Error: cannot load values.yaml: error converting YAML to JSON: yaml: invalid map key: map[interface {}]interface {}{".Values.nreplicas":interface {}(nil)}
mumoshu avatar
mumoshu

you cant use go template expressions in yaml file

mumoshu avatar
mumoshu

change it to .gotmpl

2019-12-21

2019-12-20

Yingjun Hu avatar
Yingjun Hu

It seems like it’s the similar question I posted yesterday https://github.com/roboll/helmfile/issues/1045. I was trying to pass the environment values from my top level helmfile to sub helmfiles but wasn’t able to. Any advice?

Environment values not being used in nested states · Issue #1045 · roboll/helmfile

I&#39;ve been trying to use environments in my main helmfile and have multiple sub helmfiles in my releases folder. I wanted all the sub helmfiles to pick up whatever the value I defined in a speci…

Yingjun Hu avatar
Yingjun Hu

From what I understand after debugging yesterday, it seemed to me that whenever there’s a change of directory, the env values will be cleaned and not be passed to nested helmfiles.

Juan Soto avatar
Juan Soto

I see. I had to create the variables inside the helmfile, then I was able to reference it from the helm chart.

Yingjun Hu avatar
Yingjun Hu

Yah I did the similar thing to work around it but still curious what the best way is

Juan Soto avatar
Juan Soto

Another question. As I have two environments ready to be deployed. I would to implemente an strategy that allows me to deploy both environments at the same time. With the current configuration, each time I deploy to dev, it overwrites the stage environment. Are there any solution to have both envs up and running? What do you think?

Yingjun Hu avatar
Yingjun Hu

Don’t you use two clusters for two envs?

Juan Soto avatar
Juan Soto

No, I ony have one cluster

Juan Soto avatar
Juan Soto

A gke cluster

Juan Soto avatar
Juan Soto

What do you suggest?

Yingjun Hu avatar
Yingjun Hu

Well we use two eks clusters. Don’t think it’s common to deploy two envs in the same cluster but I could think of templating different namespaces to deploy different envs if you have to.

Juan Soto avatar
Juan Soto

ok thanks. So the question is: How do we deploy different environments in separate namespaces within the same cluster?

Yingjun Hu avatar
Yingjun Hu

IMO you can set the ns value in your environments and template it

Juan Soto avatar
Juan Soto

using the helm chart? or from helmfile?

Yingjun Hu avatar
Yingjun Hu

Both? I mainly use helmfile to choose which ns to deploy but could be different for you

Juan Soto avatar
Juan Soto

Do you have any example to share?

Yingjun Hu avatar
Yingjun Hu

Something like releases:

  • name: chart1-helmfile namespace: {{ .Environment.Values.testVersion}} chart: “./chart1”
Yingjun Hu avatar
Yingjun Hu

And define this testVersion in your helmfile as environments: dev: values: - testVersion: 1.0.0

Juan Soto avatar
Juan Soto

ok great, I will try that

mumoshu avatar
mumoshu

@Yingjun Hu fyi i’ve replied to you in https://github.com/roboll/helmfile/issues/1045

Environment values not being used in nested states · Issue #1045 · roboll/helmfile

I&#39;ve been trying to use environments in my main helmfile and have multiple sub helmfiles in my releases folder. I wanted all the sub helmfiles to pick up whatever the value I defined in a speci…

Yingjun Hu avatar
Yingjun Hu

Thanks @mumoshu! That is super helpful!

Environment values not being used in nested states · Issue #1045 · roboll/helmfile

I&#39;ve been trying to use environments in my main helmfile and have multiple sub helmfiles in my releases folder. I wanted all the sub helmfiles to pick up whatever the value I defined in a speci…

:--1:1
mumoshu avatar
mumoshu

Requesting feedbacks regarding two biggest fundamental issues in Helmfiel and corresponding enhancement proposals:

v1: Alternative name for environment/state values? · Issue #1048 · roboll/helmfile

Background We call Helmfile-specific template parameters as Environemnt Values or State Values today. Those parameters can be loaded from helmfile.yaml or another yaml and yaml template files with …

feat: Predictable Helmfile template · Issue #932 · roboll/helmfile

TL;DR; I want to add a new helmfile.yaml field to make templating helmfile configs easier. Problem Helmfile&#39;s double-rendering has opened a wide variety of use-cases that requires you to write …

Erik Osterman avatar
Erik Osterman

I’ll review these today

v1: Alternative name for environment/state values? · Issue #1048 · roboll/helmfile

Background We call Helmfile-specific template parameters as Environemnt Values or State Values today. Those parameters can be loaded from helmfile.yaml or another yaml and yaml template files with …

feat: Predictable Helmfile template · Issue #932 · roboll/helmfile

TL;DR; I want to add a new helmfile.yaml field to make templating helmfile configs easier. Problem Helmfile&#39;s double-rendering has opened a wide variety of use-cases that requires you to write …

1

2019-12-19

Juan Soto avatar
Juan Soto

Hello I am having this error when I try to reference the value of an environment variable inside the deployment

Juan Soto avatar
Juan Soto

this is the deployment file, and I want to set the value of the environment variable helm.environment into ENVIRONMENT var. The value was set from command line using helmfile -e dev lint

Juan Soto avatar
Juan Soto

Any ideas?

Juan Soto avatar
Juan Soto
Juan Soto avatar
Juan Soto
Juan Soto avatar
Juan Soto
Pierre Humberdroz avatar
Pierre Humberdroz

I have never used the stages features but should it not just be {{ .Values.helm.environment }} ?

Juan Soto avatar
Juan Soto

I changed it to .Values as you suggested but no luck.

Juan Soto avatar
Juan Soto

at <.Values.helm.environment>: nil pointer evaluating interface {}.environment

Juan Soto avatar
Juan Soto

any ideas @mumoshu ?

mumoshu avatar
mumoshu

@Juan Soto You can’t use environment values in helm chart templates

mumoshu avatar
mumoshu

Helmfile environment values are dedicated to templates processed by Helmfile, like helmfile.yaml itself, and values.yaml.gotmpl read and rendered by Helmfile.

mumoshu avatar
mumoshu

So you write it like value: {{ .Values.helm.environment }} in your “chart” template, without .Environment

Juan Soto avatar
Juan Soto

I see.

mumoshu avatar
mumoshu

and pass that specific chart values by writing like this in helmfile.yaml

releases:
- name: yourapp
   chart: ./chart1
  values:
  - helm:
       environment: {{ .Values.helm.environment }}
mumoshu avatar
mumoshu

or

releases:
- name: yourapp
  chart: ./chart1
  values:
  - somevalues.yaml containing helm.environment
Juan Soto avatar
Juan Soto
Juan Soto avatar
Juan Soto

That is what I did. And then, I referenced {{ environment_chart }} inside the helm chart.

Juan Soto avatar
Juan Soto

Is it a good practice @mumoshu?

mumoshu avatar
mumoshu

yes that looks good

Juan Soto avatar
Juan Soto

it worked

1
Juan Soto avatar
Juan Soto

2019-12-18

2019-12-17

Marjan Jordanovski avatar
Marjan Jordanovski

Hello guys, I’m trying to create one deployment.yaml file inside helm chart, that will create 2 pods when I run helmfile. So my problem is that I do not know how to properly reference values inside deployment.yaml that will point to my .yaml.gotmpl files.

Marjan Jordanovski avatar
Marjan Jordanovski

So what I want is that on every place in deployment.yaml where it’s {{ .values }} , I want helm to detect that there are 2 gotmpl inside values/ (I do not want to specify exactly the name of that gotmpl because that would mean that I need 2 deployment files) , and to somehow iterate through them in order to create 2 deployments from that one deployment.yaml (for example first deployment would get all values from first gotmpl, and second created deployment would get all values from second gotmpl)

mumoshu avatar
mumoshu


I want helm to detect that there are 2 gotmpl inside values/ (I do not want to specify exactly the name of that gotmpl because that would mean that I need 2 deployment files)
I dont really understand this but anyway

mumoshu avatar
mumoshu

In chart’s deployment yaml you should just write it like:

containers:
- name: {{ .Values.name }}
mumoshu avatar
mumoshu

apparently your es01.yaml.gotmpl isn’t a template so you can just name it es01.yaml

Marjan Jordanovski avatar
Marjan Jordanovski

That would mean that I’m searching for name value inside values.yaml

mumoshu avatar
mumoshu

and everything is just a plain helm usage

Marjan Jordanovski avatar
Marjan Jordanovski

I’ll try to explain better:

Marjan Jordanovski avatar
Marjan Jordanovski

i have two deployments that are very similar

Marjan Jordanovski avatar
Marjan Jordanovski

so I want just one deployment.yaml

mumoshu avatar
mumoshu

yeah that’s how you should use the same that to create two releases

Marjan Jordanovski avatar
Marjan Jordanovski

which will reference only different values from two different files

mumoshu avatar
mumoshu

hmm?

mumoshu avatar
mumoshu


only different values
different from what?

Marjan Jordanovski avatar
Marjan Jordanovski

different from each other..for example:

mumoshu avatar
mumoshu

isn’t it just that the release es01 should be installed from es01.yaml.gotmpl and es03 from os03.yaml.gotmpl?

Marjan Jordanovski avatar
Marjan Jordanovski

yess

mumoshu avatar
mumoshu

then your helmfile.yaml looks correct

mumoshu avatar
mumoshu

when you run helm install -f values.yaml somechart

{{ .Values.foo.bar }} results in BAR when values.yaml is:

foo:
  bar: BAR
mumoshu avatar
mumoshu

releases:

  • name: es01 chart: ./elk-chart values:
    • es01.yaml.gotmpl

is basically

helm install ./elk-chart -f es01.yaml.gotmpl

mumoshu avatar
mumoshu

so just reference your values defined in es01.yaml.gotmpl OR es03.yaml.gotmpl from within your deployment.yaml like

mumoshu avatar
mumoshu
{{ .[Values.environment.ES](http://Values.environment.ES)_JAVA_OPTS }}
{{ .Values.environment.bootstrap.memory_lock }}
{{ .Values.ports.containerPort }}
{{ .Values.ports.name }}
Marjan Jordanovski avatar
Marjan Jordanovski

so when running helm, is this last gonna check for ES_JAVA_OPTS inside both gotmpl files, and add value for ES_JAVA_OPTS from es01.yaml.gotmpl to first created deployment and add value for ES_JAVA_OPTS from es03.yaml.gotmpl to second created deployment?

Marjan Jordanovski avatar
Marjan Jordanovski

if I just reference like you said in deployment.yaml?

Marjan Jordanovski avatar
Marjan Jordanovski

sorry for probably stupid questions though

mumoshu avatar
mumoshu

yes

mumoshu avatar
mumoshu


add value for ES_JAVA_OPTS from es01.yaml.gotmpl to first created deployment and add value for ES_JAVA_OPTS from es03.yaml.gotmpl to second created deployment?
i thought this is your intention as you did write your helmfile.yaml so in https://sweetops.slack.com/archives/CE5NGCB9Q/p1576573249022400

1
Marjan Jordanovski avatar
Marjan Jordanovski

yes that’s exactly what I want, I’m rewriting my files now in order to test your suggestions

:--1:1
Marjan Jordanovski avatar
Marjan Jordanovski
Marjan Jordanovski avatar
Marjan Jordanovski
Marjan Jordanovski avatar
Marjan Jordanovski

what I think happens here is that it look for volumeMounts.name inside values.yaml inside helm chart

Marjan Jordanovski avatar
Marjan Jordanovski

isn’t it? (because with .Values you point to values.yaml)

mumoshu avatar
mumoshu

??

mumoshu avatar
mumoshu

according to the error message, the problem is that you have invalid volumes

mumoshu avatar
mumoshu
mumoshu avatar
mumoshu

this should be

volumes:

  • name: {{ .Values.volumes.name }} emptyDir: {}
mumoshu avatar
mumoshu

ah okay you were talking about this one

mumoshu avatar
mumoshu

it looks like you’re missing voulmes.name in es03.yaml.gotmpl

mumoshu avatar
mumoshu

es01 is failing due to invalid volumes

es03 is failing due to invalid es03.yaml.gotmpl

Marjan Jordanovski avatar
Marjan Jordanovski

You’re the boss, it worked! Thank you a lot!

mumoshu avatar
mumoshu

Aweeesome!! Glad to help!

1
Marjan Jordanovski avatar
Marjan Jordanovski

so to summarize, every time I put some file inside values tag in helmfile, data inside those file is accessible from chart using .Values.tag.tag.etc. ?

mumoshu avatar
mumoshu

yep

Marjan Jordanovski avatar
Marjan Jordanovski

thanks a lot!

Marjan Jordanovski avatar
Marjan Jordanovski

I have one more question, it’s not regarding previous issue: in helmfile, if there are two releases that are pointing on the same chart, and if that chart has for example service.yaml defined, that service will be created when creating first release, and by the time second release is created that service already exists, so deployment of second release would fail…do you know a way to avoid it, so that two releases can point on the same chart but service in that chart will be created only once?

Erik Osterman avatar
Erik Osterman

When you start from scratch developing a helm chart, that might be a problem

Erik Osterman avatar
Erik Osterman

if instead, you start with the helm create command, it will setup all the scaffolding - including helpers that generate unique resource names based on the release

Erik Osterman avatar
Erik Osterman

that way you won’t encounter the conflicts that you describe

Erik Osterman avatar
Erik Osterman

it looks something like:

Erik Osterman avatar
Erik Osterman
apiVersion: v1
kind: Service
metadata:
name: {{ template "fullname" . }}
Marjan Jordanovski avatar
Marjan Jordanovski

Hello Erik, thank you! But because it would take a lot of code refactoring, and also I’m still not sure how referencing exactly works, is there maybe another way? for example, since I have two releases in helmfile pointing to the same helm chart, my idea is to create .Values.service.enabled for each release, so I can set it to “true” for first release and “false” for second release, and inside service.yaml I would set: {{- if .Values.service.enabled -}}

Marjan Jordanovski avatar
Marjan Jordanovski

#content of service.yaml goes here

Marjan Jordanovski avatar
Marjan Jordanovski

{{- end }}

Marjan Jordanovski avatar
Marjan Jordanovski

This way every time new release is created from helmfile, when deploying service it will check if .Values.service.enabled is true or false , and it will be true only for one release so only once service will be deployed

Marjan Jordanovski avatar
Marjan Jordanovski

(I tried it but it still fails to execute second release with service error “provided port is already allocated”, so I’m probably wrong somewhere, so I need help in that direction, maybe I missed something? )

mumoshu avatar
mumoshu

Probably you’d better share your whole code in github if possible Reading your explanation, I can think of several possible causes

Marjan Jordanovski avatar
Marjan Jordanovski

two files that are important in this case are helmfile:

Marjan Jordanovski avatar
Marjan Jordanovski
Marjan Jordanovski avatar
Marjan Jordanovski

and service.yaml inside elk-chart/templates:

Marjan Jordanovski avatar
Marjan Jordanovski
Marjan Jordanovski avatar
Marjan Jordanovski

this .Value.server.enabled is defined in both es01.yaml.gotmpl and es03.yaml.gotmpl , in one it’s set to true and in other it is set to false

mumoshu avatar
mumoshu

And it fails for the second release, which has server.enabled: false?

mumoshu avatar
mumoshu

If so, it seems impossible to occur. I guess the problem lies in another file and that’s why I’ve asked to share the whole project

Marjan Jordanovski avatar
Marjan Jordanovski

here it is

Marjan Jordanovski avatar
Marjan Jordanovski
MarjanJordanovski/elk

Contribute to MarjanJordanovski/elk development by creating an account on GitHub.

mumoshu avatar
mumoshu

thx!

https://github.com/MarjanJordanovski/elk/blob/dba1f17ed449cc1d2c509fba1f5aef083e2f8930/deployment-helm/values/es01.yaml.gotmpl#L13

enabled: "false" is wrong. it shoul dbe enabled: false otherwise it is treated as true

MarjanJordanovski/elk

Contribute to MarjanJordanovski/elk development by creating an account on GitHub.

1
mumoshu avatar
mumoshu
MarjanJordanovski/elk

Contribute to MarjanJordanovski/elk development by creating an account on GitHub.

Marjan Jordanovski avatar
Marjan Jordanovski

ohhh

Marjan Jordanovski avatar
Marjan Jordanovski

you are right, saved a lot of my hair again!

1
Marjan Jordanovski avatar
Marjan Jordanovski

thank you a lot!!

mumoshu avatar
mumoshu

glad to help!

2
Jonathan avatar
Jonathan

Is there any way to have different values for the same property in the values yaml.gotmpl file depending on what an environment variable is set to? e.g. if $ENV is set to “enable”, service.annotations and image.tag are set to value “x” and “y” respectively, but if $ENV is “disable” they are set to value “a” and “b”?

mumoshu avatar
mumoshu

How about adding this snippet to your values.yaml.gotmpl?

{{ if env "enable" }}
service:
  annotations: x
image:
  tag: y
{{ else }}
service:
  annotations: a
image:
  tag: b
{{ end }}
Jonathan avatar
Jonathan

sadly that didn’t work for me. failed to render [../releases/values.yaml.gotmpl], because of template: stringTemplate:20: function "ENV" not defined

Jonathan avatar
Jonathan

Or is that just looking for any environment variable with the value of “enable”, rather than the variable called ‘env’ to be equal to “enable”?

mumoshu avatar
mumoshu


rather than the variable called ‘env’ to be equal to “enable”?
sry what do you mean by this?

mumoshu avatar
mumoshu

{{ env "enable" }} stands for the value of envvar named enable

mumoshu avatar
mumoshu

and function "ENV" not defined is likely due to you’ve somehow wrote it like {{ ENV "something" }}?

Jonathan avatar
Jonathan

I see. I thought that {{ if env "enable" }} was comparing the value of the environment variable named “env” with the string “enable”. So, referring to the snippet you posted above, if the envvar $enable is set to the value true, the values would be set to x and y, and if $enable is false the values would be a and b?

mumoshu avatar
mumoshu

yep

Jonathan avatar
Jonathan

That makes more sense than how I was thinking, thats great! Thank you so much for your help!

:--1:1
mumoshu avatar
mumoshu

Glad to help!

2019-12-16

Dudi Cohen avatar
Dudi Cohen

Is there a way to iterate over helmfiles contents? For example:

helmfiles:
- path: ../*/*/release-*.yaml

releases:
{{ range $key, $values := helmfiles }}
- name: {{ .name }}
  chart: {{ .chart }}
  version: {{ .version }}
  installed: {{ .installed }}
{{ end }}

@mumoshu please

mumoshu avatar
mumoshu

Nope, unfortunately

Jonathan avatar
Jonathan

Hey! Does anyone have any ideas on how one would go about allowing an additional release to be installed when setting a value/environment variable? not sure how the selector flag would handle installing the correct release, and not install it if the variable is not set. This is how I structured my helmfile, any pointers/questions regarding this would be great!

bases:
- ../commons/environments.yaml
---
releases:
- name: "release-name"
  namespace: {{ .Namespace }}
  labels:
    chart: "release-name"
  chart: stable/example-chart
  values:
  - ../releases/values.yaml.gotmpl
{{ if eq .Values.feature_branch "Y" }} # prefferably an env variable, not sure how that is done
  - extraval:
      enabled: True
- name: "feature_branch_release"
  namespace: {{ .Namespace }}
  chart: stable/other-chart
  values:
  ../releases/feature_branch_values.yaml.gotmpl
{{ end }}
Erik Osterman avatar
Erik Osterman
cloudposse/helmfiles

Comprehensive Distribution of Helmfiles for Kubernetes - cloudposse/helmfiles

Erik Osterman avatar
Erik Osterman

We generate N releases dynamically using an environment with a custom schema that we loop through

1
TBeijen avatar
TBeijen

Nice approach, exactly what I’m looking into now. However, it seems that if I define the environments in a file included in bases the helmfile doesn’t have access to them.

TBeijen avatar
TBeijen

Works:

environments:
  apps-1:
    values:
      - ../common/defaults.yaml
      - ../environments/apps-1.yaml
    secrets:
      - ../common/secrets.sops.yaml
    missingFileHandler: Error

releases:
{{ range $index, $ns := .Environment.Values.applicationNamespaces }}

\# ...
TBeijen avatar
TBeijen

Gives map has no entry for key "applicationNamespaces":

bases:
  - environments.yaml

releases:
{{ range $index, $ns := .Environment.Values.applicationNamespaces }}

\# ...
TBeijen avatar
TBeijen

Bug? Or am I overlooking something?

TBeijen avatar
TBeijen

Pinging @mumoshu, I think this thread got of sight? Just want to check first if I should create an issue.

mumoshu avatar
mumoshu

Try separating environments and releases with ---

mumoshu avatar
mumoshu

Also separate between bases and releases

mumoshu avatar
mumoshu
bases:
  - environments.yaml
---
releases:
{{ range $index, $ns := .Environment.Values.applicationNamespaces }}

\# ...
mumoshu avatar
mumoshu
environments:
  apps-1:
    values:
      - ../common/defaults.yaml
      - ../environments/apps-1.yaml
    secrets:
      - ../common/secrets.sops.yaml
    missingFileHandler: Error
---
releases:
{{ range $index, $ns := .Environment.Values.applicationNamespaces }}

\# ...
mumoshu avatar
mumoshu

You’re hitting a fundamental issue in helmfile. Please see https://github.com/roboll/helmfile/issues/932 and give me feedbacks!

feat: Predictable Helmfile template · Issue #932 · roboll/helmfile

TL;DR; I want to add a new helmfile.yaml field to make templating helmfile configs easier. Problem Helmfile&#39;s double-rendering has opened a wide variety of use-cases that requires you to write …

TBeijen avatar
TBeijen

Thx! Will try when back at home, somewhere in the weekend. Will also look into the feedback threads.

Juan Soto avatar
Juan Soto

I want to deploy a custom helm chart, but I can’t see how to reference to a local chart

Juan Soto avatar
Juan Soto

all the examples I see they suggest:

Juan Soto avatar
Juan Soto

chart: stable/prometheus references to a remote repository. I want to run the chart I have in my local directory.

Pierre Humberdroz avatar
Pierre Humberdroz
  - name: fluentd-dashboard
    namespace: monitoring
    chart: "../charts/fluentd-dashboard"
    verify: false
    values:
      - "../charts/fluentd-dashboard/Values.yaml"

you have to define a path.

:--1:1
Juan Soto avatar
Juan Soto

I want to deploy my helm chart which is in a local directory chart1/

Juan Soto avatar
Juan Soto

Any ideas?

mumoshu avatar
mumoshu

@Juan Soto Try chart: ./chart1

:--1:1

2019-12-15

2019-12-13

Bart M. avatar
Bart M.

is there any way to mark a release as ‘to be deleted’? we have the issue that we deploy multiple versions of the same microservices with helmfile, but now we need to clean up old deprecated deploys somehow…

Andrey Nazarov avatar
Andrey Nazarov

\# set `false` to uninstall on sync
    installed: true
Bart M. avatar
Bart M.

aha let me check

Bart M. avatar
Bart M.

nice! it works, thanks a lot!

:--1:1

2019-12-12

Dudi Cohen avatar
Dudi Cohen

Hey guys, i’m trying to get the name of the chart as a variable rather than the name of the release:

releases:
- name: external-dns
  chart: my-repo/external-dns
  namespace: kube-system
  version: 1.14.0
  installed: true
  values:
  - "{{`{{ .Release.Chart }}`}}/values-dev.yaml"

Any idea on how I change .Release.Chart to only include the chart name?

Jonathan avatar
Jonathan

replace Chart with Name like so: - “{{{{ .Release.Name }}}}/values-dev.yaml” That should then resolve to “external-dns/values-dev.yaml”

Dudi Cohen avatar
Dudi Cohen

Thanks @Jonathan, but I need the name of the chart, not the name of the release as these two could be different.

TBeijen avatar
TBeijen

Is helm 3 not able to create namespaces? I run into an error whenever helmfile.yaml contains a release targeted at a non-existing namespace. (Using roboll/helmfile:helm3-v0.94.1)

starets avatar
starets
Frequently Asked Questions

What are the key differences between Helm 2 and Helm 3? Visit the FAQs for insights.

TBeijen avatar
TBeijen

Was reading up on some GitHub discussions. Totally missed that one. Ah, well. Can understand the rationale.

TBeijen avatar
TBeijen

Anyone having clever methods of creating namespace in a cicd set-up?

starets avatar
starets

ns may be created by incubator/raw helm chart, for example.

Naseem avatar
Naseem

We create namespaces via Terraform’s kubernetes provider upon cluster creation.. works well. another alternative is a k apply raw yaml with prerequired namespaces

TBeijen avatar
TBeijen

Planned on some way of bootstrapping application namespaces (rbac, default limits, quota’s where applicable). Will likely end up being a mix of kubectl and helm(file).

:--1:1
Andrey Nazarov avatar
Andrey Nazarov
Helm3 doesn't automatically create namespace · Issue #891 · roboll/helmfile

Helm3 doesn&#39;t automatically create namespace - see https://v3.helm.sh/docs/faq/#automatically-creating-namespaces How can we solve this with helmfile, so that we don&#39;t have to manually crea…

:--1:1
rms1000watt avatar
rms1000watt

Lazy question :sweat_smile:, does helmfile support replace("string_here", "to_replace", "replace_with") functionality? I recall it inherits functionality from another project, but I’m spacing out on finding that information.

Cameron Boulton avatar
Cameron Boulton

What’s up Mr. Ryan. How’s you, Chris and Calm.io?

Cameron Boulton avatar
Cameron Boulton

Helmfile supports the Go template extention library Sprig. Check out the string functions specifically for replace : http://masterminds.github.io/sprig/strings.html

String Functions

Useful template functions for Go templates.

Cameron Boulton avatar
Cameron Boulton

"I Am Ryan Smith" | replace "Am" "Was"

rms1000watt avatar
rms1000watt

@Cameron Boulton the master man with the master plan. My dream team-mate!!

rms1000watt avatar
rms1000watt

sprig, got it

rms1000watt avatar
rms1000watt

hahaha

rms1000watt avatar
rms1000watt

k, found it on the readme just now

rms1000watt avatar
rms1000watt

Calm is going smooooth, Chris and I are doing well

1
rms1000watt avatar
rms1000watt

(Actually, I thought of a work around, but I’ll leave the question here in case it’s helpful for others.)

Alucas avatar
Alucas

hmm I’m having issues adding inline values to my environment files using something like:

environments:
  dev:
    values:
    - region: "westus"

- name: chart
  chart: stable/chart
    set:
      - name: config.client.external_labels.region
        value: {{ .Environment.Values.region }}

What am I missing here?

Alucas avatar
Alucas
in ./helmfile.yaml: error during helmfile.yaml.part.0 parsing: template: stringTemplate:87:30: executing "stringTemplate" at <.Environment.Values.region>: map has no entry for key "region"
Alucas avatar
Alucas

Okay, this is weird.. If I use my base: environments.yaml at the top of the helmfile it doesn’t work, if I move everything in the environments.yaml file into the helmfile it works.

mumoshu avatar
mumoshu

@Alucas hm… sounds like helmfile needs to parse your helmfile.yaml template to finally notice that it needs to load your base environments.yaml. this is kind of a chicken-and-egg problem

mumoshu avatar
mumoshu
feat: Predictable Helmfile template · Issue #932 · roboll/helmfile

TL;DR; I want to add a new helmfile.yaml field to make templating helmfile configs easier. Problem Helmfile&#39;s double-rendering has opened a wide variety of use-cases that requires you to write …

Alucas avatar
Alucas

If I load the base: and the env at the top it doesn’t work, so I cannot have the base: environment.yaml that also includes the environments.

mumoshu avatar
mumoshu

We need to resolve #932. This is a fundamental issue in helmfile

mumoshu avatar
mumoshu

ya

mumoshu avatar
mumoshu

so try this

mumoshu avatar
mumoshu
bases:
- environments.yaml
---
- name: chart
  chart: stable/chart
    set:
      - name: config.client.external_labels.region
        value: {{ .Environment.Values.region }}
mumoshu avatar
mumoshu

this would work as helmfile separates your template at ---, loading the base environments.yaml before rendering the second part

mumoshu avatar
mumoshu

just show me your full example including bases and environments and everything else if this doesnt make sense(i might be misreading something

mumoshu avatar
mumoshu

#helmfile users - do you already use needs? I’m considering to rename it to after. wdyt?

https://github.com/roboll/helmfile/issues/1018#issuecomment-565238688

'needs:' directive in the dependent release does not respect the 'installed: false' of the other release · Issue #1018 · roboll/helmfile

releases: - name: test-1 chart: test-1 installed: false namespace: namespace-1 - name: test-2 chart: test-2 namespace: namespace-2 needs: [ namespace-1/test-1 ] In this case, the release test-2 sho…

Alucas avatar
Alucas

@mumoshu I think that fixed it for now, thanks!

1

2019-12-11

TBeijen avatar
TBeijen

Q: Adding aws-iam-authenticator. Good idea? Bad idea?

TBeijen avatar
TBeijen

I could very well imagine pitch-forks being readied now.

TBeijen avatar
TBeijen

The good would be convenience for a subset of end-users.

TBeijen avatar
TBeijen

The bad would be: What if every cloud vendor’s tools will be added? Image bloat (It’s ~30Mb, not overly huge, but still).

TBeijen avatar
TBeijen

Use case would be using the quay.io images, bringing a kubec config similar to:

apiVersion: v1
preferences: {}
kind: Config
clusters:
- cluster:
    server: <https://123123123.yl4.eu-west-1.eks.amazonaws.com>
    certificate-authority-data: LOTSOFCIPHERSTUFF
  name: apps
contexts:
- context:
    cluster: apps
    user: apps
  name: apps
current-context: apps
users:
- name: apps
  user:
    exec:
      apiVersion: [client.authentication.k8s.io/v1alpha1](http://client.authentication.k8s.io/v1alpha1)
      command: aws-iam-authenticator
      args:
        - "token"
        - "-i"
        - "apps"
TBeijen avatar
TBeijen

It would be useful for EKS users. Probably a number of non-EKS AWS users also.

roth.andy avatar
roth.andy

IMO better to add AWS CLI (which lets you do aws eks get-token)

mumoshu avatar
mumoshu

yeah aws cli would be better

roth.andy avatar
roth.andy

AFAIK aws-iam-authenticator hasn’t been deprecated yet, but it should be since all of its functionality has been rolled into aws

mumoshu avatar
mumoshu

could u elaborate a bit more? how does it relate to helmfile?

TBeijen avatar
TBeijen

@mumoshu See thread

mumoshu avatar
mumoshu

sry i don’t understand… are you talking about adding aws-iam-authenticator to where?

TBeijen avatar
TBeijen

The helmfile image.

TBeijen avatar
TBeijen

When using in CI/CD. Alternative would be building own helmfile image that _has_the binary. Or add it to the project and mount it in the CI/CD container.

mumoshu avatar
mumoshu

ah gotcha

mumoshu avatar
mumoshu

you need to build your own adding the official helmfile image as FROM [quay.io/roboll/helmfiel](http://quay.io/roboll/helmfiel):...

mumoshu avatar
mumoshu

it does help aws users but helmfile isn’t aws-specific

TBeijen avatar
TBeijen

Yes, that would make most sense (but has me setting up an ECR, track helmfile releases, etc.).

mumoshu avatar
mumoshu

the next best thing would you contribute a pull request to add aws specific helmfile dockerfile

mumoshu avatar
mumoshu

and we can probably co-maintain it, freely adding aws specific tools like awscli

mumoshu avatar
mumoshu

but not sure if it works for everyone

TBeijen avatar
TBeijen

If piggy-backing on your build pipeline sounds ok to you then that would be very interesting.

mumoshu avatar
mumoshu

actually in my own job i do bundle variant(https://github.com/mumoshu/variant) along with helmfile in the same iamge

mumoshu/variant

Wrap up your bash scripts into a modern CLI today. Graduate to a full-blown golang app tomorrow. - mumoshu/variant

mumoshu avatar
mumoshu

so only adding aws-iam-authenticator/awscli doesn’t really help me. does it for you?

roth.andy avatar
roth.andy

Something I’ve been working on, a la Geodesic, that fits here, is https://github.com/dadsgarage/dadsgarage. @Erik Osterman really got me going on the idea of creating one container that has all your tools in it. So that’s what I’m going to be doing

dadsgarage/dadsgarage

Container version of Dad&#39;s garage. It&#39;s full of tools, you spend lots of time in it, and you use it to build great things. https://hub.docker.com/r/dadsgarage/dadsgarage - dadsgarage/dadsga…

roth.andy avatar
roth.andy

You better believe it will have aws and helmfile in it

mumoshu avatar
mumoshu

yeah i’d recommend that pattern, too

TBeijen avatar
TBeijen

Makes sense indeed to have a one-stop toolbox for all CI/CD tasks, be it helmfile, straight kubectl stuff, or indeed commands that need awscli. Or eksctl

mumoshu avatar
mumoshu

:–1:

TBeijen avatar
TBeijen

Lol

roth.andy avatar
roth.andy

bam. helmfile and eksctl issues created, with good-first-issue and help-wanted labels applied

TBeijen avatar
TBeijen

Other Q: I was hoping to toggle certain applications using values defined in the cluster-specific environment. Like this (but that throws a ‘map has no entry’ error):

release:
  - name: efs-provisioner
    namespace: kube-system
    chart: stable/efs-provisioner
    version: "0.8.0"
    installed: {{ .Environment.Values.efsProvisioner.installed }}
    values:
      - releases/efs-provisioner.yaml.gotmpl

Is this possible in some way?

TBeijen avatar
TBeijen

A new day, new loads of coffee, fresh perspective. Found it:

releases:
  - name: efs-provisioner
    namespace: kube-system
    chart: stable/efs-provisioner
    version: "0.8.0"
    installedTemplate: '{{`{{ .Environment.Values.efsProvisioner.installed }}`}}'
    values:
      - releases/efs-provisioner.yaml.gotmpl

2019-12-10

Vadim Bauer avatar
Vadim Bauer

What do you think?

Do the helmfile concepts make sense in the context of Pulumi with Helm? Here is a Helm example with Pulumi -> https://github.com/pulumi/pulumi-kubernetes#deploying-a-helm-chart

I am thinking about moving to Pulumi becuase of its first class k8s support. As a bonus there is also Helm support. It would be possible to bridge the terraform-provider-helmfile to Pulumi but I wonder if it will provided value to add another layer.

As much as I like the awesomeness of the Helmfile, aren’t it too much of layers? 1 -> Pulumi 2-> Helm 3-> Helmfile especially in the imperative nature of Pulumi.

mumoshu avatar
mumoshu

hey! i think pulumi works as long as you don’t use helmfile’s features like vault/ssm/secretsmanager support, sub-helmfiles, DAG, etc.

mumoshu avatar
mumoshu

bridging terraform-provider-helmfile to pulumi does sound like too much. i’d prefer implementing a kind of pulumi plugin(how is it called? module? library?) for helmfile

mumoshu avatar
mumoshu

i’ve been long wanted to play more with pulumi so please point me to some pulumi plugin impl. guide so that i can try to build a poc

Vadim Bauer avatar
Vadim Bauer
pulumi/pulumi-terraform-bridge

A library allowing providers built with the Terraform Plugin SDK to be bridged into Pulumi. - pulumi/pulumi-terraform-bridge

Vadim Bauer avatar
Vadim Bauer
mumoshu/terraform-provider-helmfile

Deploy Helmfile releases from Terraform. Contribute to mumoshu/terraform-provider-helmfile development by creating an account on GitHub.

Vadim Bauer avatar
Vadim Bauer

@mumoshu pulumi has supprt for tf providers and there is a tf provider for helmfile, so I guess the easiest way would to bridge it.

Erik Osterman avatar
Erik Osterman

@Vadim Bauer see related discussion here: https://sweetops.slack.com/archives/CQCDCLA1M/p1573246618316100

Quick question for the group, has anyone tried their hands at AWS CDK or Pulumi for IaC?

1
Erik Osterman avatar
Erik Osterman

@mumoshu adds some interesting insights as well

Erik Osterman avatar
Erik Osterman

also, see the comment from @tamsky

1
Juan Soto avatar
Juan Soto

Hi, I have a k8s cluster running, also the kubeconfig is correctly configured. I wanna start playing with helmfile. Could you suggest me a good tutorial in order to start learning ?

mumoshu avatar
mumoshu

maybe try cloudposse’s guide? https://docs.cloudposse.com/tools/helmfile/

Erik Osterman avatar
Erik Osterman

(It’s a bit dated at this point)

1

2019-12-09

Erik Osterman avatar
Erik Osterman
Feature Request: Binary Downloads to Support Terraform Cloud · Issue #5 · mumoshu/terraform-provider-helmfile

what Add flag to download kubectl and helmfile from GitHub pinned to a specific release why Running provider in terraform cloud requires binaries be installed by some other means Using local-exec w…

Erik Osterman avatar
Erik Osterman

can have @aknysh help contribute it

Erik Osterman avatar
Erik Osterman

(we’re struggling to use it e2e on terraform cloud without this)

aknysh avatar
aknysh

I can finish implementing and testing it tomorrow, if @mumoshu will not do it tonight :)

mumoshu avatar
mumoshu

Replied, i’d appreciate your contribution and will try to review it asap once submitted!

mumoshu avatar
mumoshu


Add flag to download kubectl and helmfile

mumoshu avatar
mumoshu

would those be literally flags?

Erik Osterman avatar
Erik Osterman

any implementation preferences?

mumoshu avatar
mumoshu

not sure. im not familiar with this kind of feature

Erik Osterman avatar
Erik Osterman

prior art: atlantis downloads terraform

mumoshu avatar
mumoshu

would adding helmfile_version and helmfile_download_url_template to the resource makes sense?

Erik Osterman avatar
Erik Osterman

ya, something like that

aknysh avatar
aknysh

Before executing helmfile command, we need to download kubectl and helmfile, and possibly assume a role if provided

aknysh avatar
aknysh

All optional

mumoshu avatar
mumoshu

would adding helmfile_version and helmfile_download_url_template to the resource makes sense?

aknysh avatar
aknysh

We need it for terraform cloud

:--1:1
aknysh avatar
aknysh

Do we need to also download helm? Helmfile doesn’t do it automatically?

Erik Osterman avatar
Erik Osterman

yes, good point - so I think we need helm, helmfile, and kubectl

aknysh avatar
aknysh

Yes

aknysh avatar
aknysh

And assume role code if it’s provided

mumoshu avatar
mumoshu

got it

Erik Osterman avatar
Erik Osterman

(lol, and hrm… maybe this is cascading out of control)

mumoshu avatar
mumoshu

iam assume role?

Erik Osterman avatar
Erik Osterman

need helm-diff plugin too

Erik Osterman avatar
Erik Osterman

and then helm-diff might have deps

mumoshu avatar
mumoshu

ah

Erik Osterman avatar
Erik Osterman

ugh

mumoshu avatar
mumoshu

helm-diff has no deps as it’s a single binary tool written in golang

Erik Osterman avatar
Erik Osterman

maybe we generalize it?

Erik Osterman avatar
Erik Osterman

package_deps which is a list of binaries by URL

aknysh avatar
aknysh

Yes

mumoshu avatar
mumoshu

wouldn’t assume role automatically done today by installing aws cli and using the correct kubeconfig?

mumoshu avatar
mumoshu

package_deps would worth a dedicated terraform provider, right?

mumoshu avatar
mumoshu

and i think it doesn’t match with helmfille providers use-case

mumoshu avatar
mumoshu

as it would typically require multiple versions of helmfile/helm/kubectl binaries to co-exist

aknysh avatar
aknysh

Order of operations matters

Erik Osterman avatar
Erik Osterman

possibly - just for background, we tried using null_resource with local-exec provisioner to download, but cannot get it to work due to limited ability to control when it’s executed (we need it executed prior to helmfile-provider getting invoked which depends on the binaries)

aknysh avatar
aknysh

Terraform cloud can’t even plan without those binaries

mumoshu avatar
mumoshu

yep. i wondered there was some way to explicitly draw a dependency between local-exec and helmfile resourec

mumoshu avatar
mumoshu

apparently it isn’t possible?

Erik Osterman avatar
Erik Osterman

so for example, we do this successfully for downloading aws cli in one step and calling it in another, both using local-exec

Erik Osterman avatar
Erik Osterman

however, since local-exec is out of phase with terraform-helmfile-provider, we cannot control for the order of execution

Erik Osterman avatar
Erik Osterman

so the first time we run it, it works

aknysh avatar
aknysh

Local exec is not executed on plan

:--1:1
Erik Osterman avatar
Erik Osterman

precisely

mumoshu avatar
mumoshu

ah good to know!

Erik Osterman avatar
Erik Osterman
uber/astro

Astro is a tool for managing multiple Terraform executions as a single command - uber/astro

Erik Osterman avatar
Erik Osterman

astro also donwloads terraform

:--1:1
Erik Osterman avatar
Erik Osterman

on demand based on version

mumoshu avatar
mumoshu

maybe we should add helmfile_version, helm_version, kubectl_version, helm_diff_version for starter

mumoshu avatar
mumoshu

and implement some other provider to install arbitrary pkg from e.g. apk if necessary, for use with depends_on = [that_provider.some_bin] to draw dependency

mumoshu avatar
mumoshu

ah okay depends_on = [that_provider.some_bin] won’t work anyway

mumoshu avatar
mumoshu

as the provider won’t install it on plan. that’s the same thing as local-exec/null-resource

Erik Osterman avatar
Erik Osterman

can one tie into the init phase?

mumoshu avatar
mumoshu
mumoshu/terraform-provider-helmfile

Deploy Helmfile releases from Terraform. Contribute to mumoshu/terraform-provider-helmfile development by creating an account on GitHub.

mumoshu avatar
mumoshu

but it is unlikely to provide helmfile_releaseset resources on init

aknysh avatar
aknysh

I think a better way would be not to pollute terraform-provider-helmfile with loading external packages, but using data sources b/c they are executed on terraform plan

aknysh avatar
aknysh

we can use terraform-provider-shell b/c it already implements data source together with resource https://github.com/scottwinkler/terraform-provider-shell/blob/master/examples/test.tf#L5

scottwinkler/terraform-provider-shell

Terraform provider for executing shell commands and saving output to state file - scottwinkler/terraform-provider-shell

aknysh avatar
aknysh

or we can add a similar data source to terraform-provider-helmfile

aknysh avatar
aknysh
If the query constraint arguments for a data resource refer only to constant values or values that are already known, the data resource will be read and its state updated during Terraform's "refresh" phase, which runs prior to creating a plan
aknysh avatar
aknysh
Each data source in turn belongs to a provider, which is a plugin for Terraform that offers a collection of resource types and data sources that most often belong to a single cloud or on-premises infrastructure platform.
aknysh avatar
aknysh
Each provider may offer data sources alongside its set of resource types
aknysh avatar
aknysh

@mumoshu @Erik Osterman ^

mumoshu avatar
mumoshu

@aknysh ah, so your idea is based on the fact that datasource is evaluated even at plan time, right?

mumoshu avatar
mumoshu

would it look like this?

data "helmfile_binary" "default" {
  version = "0.94.1"
}

data "helm_binary" "default" {
  version = "3.0.0"
}

resource "helmfile_releaseset" "myapp" {
  helmfile_binary = "${helmfile_binary.default.path}`
  helm_binary = "${helm_binary.default.path}`
}
aknysh avatar
aknysh

yes!

mumoshu avatar
mumoshu

excellent! that would be much much better than adding version related keys to resources.

aknysh avatar
aknysh

in the data source, we can hide the fact that it downloads the data (make it more specific, as you showed above)), or we can allow to execute any commands (as terraform-provider-shell does) and make it more generic

mumoshu avatar
mumoshu

let’s hide it for now, as it would be dedicated to helmfile provider’s use-case.

mumoshu avatar
mumoshu

adding optional key like download_url and/or apk_package would be alright

Andrey Nazarov avatar
Andrey Nazarov

At the moment we are just building a docker container with all the stuff pre-installed. Looking forward to trying a new implementation.

2019-12-05

Pierre Humberdroz avatar
Pierre Humberdroz

Does someone know how to disable validation ?

2019-12-04

Sebastian Bauer avatar
Sebastian Bauer

Hello, I would like to know what do you think about exposing absolute basePath to current yaml? https://github.com/roboll/helmfile/pull/981

Sending basePath to yamls renderer by anih · Pull Request #981 · roboll/helmfile

Hi, I would like to propose this change because we faced a problem when we have app X that depends on app Y and we wanted to put Y in X dependencies: helmfiles: - path: git://github/anih/Y>…

2019-12-03

Andrey Nazarov avatar
Andrey Nazarov

Am I right that the construction {{ .Environment.Values | get "my.var" "value" }} allows my.var not to be presented in environments files at all?

Andrey Nazarov avatar
Andrey Nazarov

I was wondering if helmfiles: could be used in helmfile referenced via helmfiles: from the main one. Will check this out.

Andrey Nazarov avatar
Andrey Nazarov

Yes, it works very well!

Shurka avatar
Shurka

Hi. Is there any guide for migrating from helm2 to helm3?

Shurka avatar
Shurka

i mean if i have existing releases in datacenters, managed by helmfile, how do i migrate to manage them with helm3.

Andrey Nazarov avatar
Andrey Nazarov

What event makes needs: succeed? Actually, I’m interested in waiting for the postsync hook to finish.

Andrey Nazarov avatar
Andrey Nazarov

It seems it respects postsync hooks. Great!

Erik Osterman avatar
Erik Osterman

@here helmfile users at #aws-reinvent should connect tonight!

davidvasandani avatar
davidvasandani

@mumoshu I’ll come say hello!

1
mumoshu avatar
mumoshu

hey! are you there? i’ll be coming in 10mins or so

mumoshu avatar
mumoshu

its super crowded!

davidvasandani avatar
davidvasandani

What color shirt are you wearing?

mumoshu avatar
mumoshu

im in navy sweater and brown bottoms

davidvasandani avatar
davidvasandani

I’ll keep an eye out. I’m grey t-shirt with a pocket.

davidvasandani avatar
davidvasandani

Sitting at a table.

mumoshu avatar
mumoshu

and a golden medal :)

1
mumoshu avatar
mumoshu

im around the beer counter

davidvasandani avatar
davidvasandani

navy sweater brown pants. I just did a loop and I don’t see you. Doing another.

mumoshu avatar
mumoshu

do you have anything to signify you? im now around the table to grab some nacho

mumoshu avatar
mumoshu

i gotta go! thx for taking!

:100:1
davidvasandani avatar
davidvasandani

No! Thank you! I can’t wait to test the docker ingratiation we discussed.

Andrey Nazarov avatar
Andrey Nazarov

Regarding of sub-helmfiles. It seems Cloud Posse uses them extensively since they have a collection of dedicated helmfiles. @Erik Osterman how do you, guys, deal with environments? I’ve noticed you don’t have environments: blocks in your dedicated helmfiles. You don’t use them at all?

Alex Siegman avatar
Alex Siegman

I would imagine that the concept of “environments” can be different from company to company, it doesn’t make much sense to limit a helmfile meant to be shared across any arbitrary number and design of environments. By excluding that section, you can easily just include the helmfile directly, and configure it through the operating system environment (notice their heavy use of env in their templates) with tools such as direnv

Alex Siegman avatar
Alex Siegman

that way, in each of your “environments” you just call those helmfiles without an environment parameter

Andrey Nazarov avatar
Andrey Nazarov

That’s true.

joshmyers avatar
joshmyers

CP use ‘stage’ IIRC which can see passed as values in some of the helmfiles

Erik Osterman avatar
Erik Osterman

Thanks @Alex Siegman and @joshmyers

Erik Osterman avatar
Erik Osterman

Yes, the back story though is we got started long before environments existed

Erik Osterman avatar
Erik Osterman

But the gist of it is we use remote helmfiles and anything that could vary by “environnent” is actually defined with environment variables

Erik Osterman avatar
Erik Osterman

this way we easily source them from chamber

Erik Osterman avatar
Erik Osterman

On the one hand, I like that we have a consistent interface for passing settings (environment variables)

Erik Osterman avatar
Erik Osterman

on the other, I miss the cleanliness of environments in helmfile where all your settings are in git vs ssm

Erik Osterman avatar
Erik Osterman

@aknysh just got the terraform-helmfile-provider working with terraform cloud

Erik Osterman avatar
Erik Osterman

So I’m not sure if we’ll end up using environments that much since we get that out-of-the-box with how we manage terraform.

Erik Osterman avatar
Erik Osterman

also, this opens up a tremendous amount of possibilities - passing settings from terraform directly into helmfile

:100:1
joshmyers avatar
joshmyers

Terraform cloud!?! Talk. To. Me.

joshmyers avatar
joshmyers

I’m yet to hear anything about it other than “it’s bloody expensive”

Erik Osterman avatar
Erik Osterman

it’s free for teams <=5

joshmyers avatar
joshmyers

How does config work? Do they solve the many states and dependencies problem?

joshmyers avatar
joshmyers

Aye but that won’t get very far for most orgs :(

joshmyers avatar
joshmyers

2 banks I’ve been at have been quoted in the millions…they are banks though….

joshmyers avatar
joshmyers

Is it very close to Atlantis?

Erik Osterman avatar
Erik Osterman

It’s “close” but just way more polished.

Erik Osterman avatar
Erik Osterman

The nice thing with Atlantis is you can deploy it with an IAM role

Erik Osterman avatar
Erik Osterman

no way to do that with TF Cloud

2019-12-02

Andrey Nazarov avatar
Andrey Nazarov

I’ve double checked with the latest v0.94.1, the behaviour is the same as I described in https://github.com/roboll/helmfile/issues/1010. I’ve decided to move it out of the thread about a tf provider as it looks bloated.

Releases from `helmfiles` are ignored when the environment is specified · Issue #1010 · roboll/helmfile

I&#39;m facing the issue when releases defined in separate .yaml files and referenced via helmfiles: are ignored when I try to install the full stack for a certain environment. Example: #environmen…

Erik Osterman avatar
Erik Osterman

@mumoshu binary releases for terraform-provider-helmfile courtesy of @aknysh

:--1:1
Erik Osterman avatar
Erik Osterman
Add `release.yml` GitHub Action by aknysh · Pull Request #4 · mumoshu/terraform-provider-helmfile

what Add release.yml GitHub Action why Build Go binaries for different architectures and attach to GitHub release To be able to download Go binaries and install them into /terraform.d/plugins to…

2019-12-01

    keyboard_arrow_up