#helmfile (2019-06)

https://github.com/roboll/helmfile

Questions and discussion around helmfile https://github.com/roboll/helmfile and https://github.com/cloudposse/helmfiles Archive: https://archive.sweetops.com/helmfile/

2019-06-29

Vadim avatar
Vadim

Hi, could someone help me with the issue? https://github.com/roboll/helmfile/issues/731

readFile reads only first line · Issue #731 · roboll/helmfile
Hi, I have the following folder structure for openldap: in common.yaml.gotmpl I have the following lines: customLdifFiles: init.ldif: {{ readFile "../files/ldif/init.ldif" }} Content of…
mumoshu avatar
mumoshu

Try | indent 6 after readFile :)

readFile reads only first line · Issue #731 · roboll/helmfile
Hi, I have the following folder structure for openldap: in common.yaml.gotmpl I have the following lines: customLdifFiles: init.ldif: {{ readFile "../files/ldif/init.ldif" }} Content of…
Vadim avatar
Vadim

It should not take much time to help me coz solutions seems to be simple

Vadim avatar
Vadim

But problem is strange and unexpected … and solution not obvious for me

2019-06-27

Ryan Richards avatar
Ryan Richards

If you wanted to run a simple script that, lets say creates a few namespaces, could you do this somewhere with helmfiles? I found that working with terraform null resource didn’t work as i expected. If there is a best-practice for running these kind of configuration processes i would love to hear about it

kamsz avatar
kamsz

what about kubernetes provider for TF? you could use it to create namespaces

Erik Osterman avatar
Erik Osterman

Helm will by default create the namespace for you. No need to do anything fancy.

Ryan Richards avatar
Ryan Richards

@Erik Osterman thanks, i was using that as an example. We have several post-setup configuration steps that need to occur once our terraform is finished with the basic cluster setup in gke. These include things like setting default limits.

Ryan Richards avatar
Ryan Richards

Most of these tasks are installing prometheus, grafana etc. Im working now to create helmfiles for these but there are a few that to be repeatable would need to be run as scripts or something

Ryan Richards avatar
Ryan Richards

hopefully my statement made sense Im simply trying to find the best manner to apply configuration changes that dont involve installing or deploying. Running gcloud commands to obtain connectivity to a specific cluster would be another example.

Erik Osterman avatar
Erik Osterman

you can trigger commands as part of hooks

Erik Osterman avatar
Erik Osterman
cloudposse/helmfiles

Comprehensive Distribution of Helmfiles. Works with helmfile.d - cloudposse/helmfiles

Ryan Richards avatar
Ryan Richards

@Erik Osterman awesome! thank you!

Ryan Richards avatar
Ryan Richards

ah hooks on helmfile - i gotcha

2019-06-26

Ryan Richards avatar
Ryan Richards

What is the recommended approach for converting a standard helm chart into a helmfile? if there is documentation on this i apologize.

Andrey Nazarov avatar
Andrey Nazarov

You don’t need to convert anything. Helmfile basically allows you to declaratively define what should be deployed to the cluster. Pretty much like docker-compose do for docker containers or a configuration management tool for os packages.

helmfile.yaml could be as simple as


\#Your Helm repositories
repositories:
- name: stable
  url: <https://kubernetes-charts.storage.googleapis.com>


\#Helm settings
helmDefaults:
  tillerNamespace: tiller-namespace


\#Your releases
releases:
  - name: ingress
    namespace: ingress
    chart: stable/nginx-ingress #chart name
    version: 1.6.13 #chart version
Andrey Nazarov avatar
Andrey Nazarov

Almost everything (except for more complicated use cases and experimental features) is covered here: https://github.com/roboll/helmfile#configuration

roboll/helmfile

Deploy Kubernetes Helm Charts. Contribute to roboll/helmfile development by creating an account on GitHub.

Erik Osterman avatar
Erik Osterman

@Ryan Richards is there a helm chart in particular? we might have an example of it.

Erik Osterman avatar
Erik Osterman
cloudposse/helmfiles

Comprehensive Distribution of Helmfiles. Works with helmfile.d - cloudposse/helmfiles

Andrey Nazarov avatar
Andrey Nazarov

I wouldn’t call it a conversion, wrapping maybe But it doesn’t really matter. Yes, cloudposse has a lot of great examples. Thank you for this, btw!

2019-06-25

2019-06-24

Erik Forsberg avatar
Erik Forsberg

Can I access the name of the current release in some variable, inside a values template file?

Andrey Nazarov avatar
Andrey Nazarov

If I got it right simple {{ .Release.Name }} should work.

Erik Forsberg avatar
Erik Forsberg

Doesn’t seem quite that simple:

in ./helmfile.yaml: helm exited with status 1:
  Error: failed to parse /var/folders/wk/kl05m8rx49x5v67ckkkq_xhxk__c0p/T/values267206567: error converting YAML to JSON: yaml: invalid map key: map[interface {}]interface {}{".Release.Name":interface {}(nil)}

{{ .Release.Name }} definitely works inside values.yaml in a helm chart, but I don’t think it does in helmfile.yaml templating.

Andrey Nazarov avatar
Andrey Nazarov

So, you want to use it in helmfile.yaml, right?

Andrey Nazarov avatar
Andrey Nazarov

What about this

{{`{{.Release.Name}}`}} 

then?

mumoshu avatar
mumoshu

Yeah, with:

{{` ... `}}

it should work.

That’s the confusing aspect of helmfile.yaml being a go template as whole.

Erik Forsberg avatar
Erik Forsberg

Ah, so by doing that, it evaluates (in helmfile) into {{.Release.Name}} which is then evaluated by helm into the release name.

That is slightly confusing indeed

mumoshu avatar
mumoshu

That’s it! Yeah, that’s why I want to make helmfile.yaml itself a plain YAML in the future

mumoshu avatar
mumoshu
feat: Change the delimiters for state templates and release templates · Issue #658 · roboll/helmfile

As @sebastien-prudhomme suggested in #515: I&#39;ve just discovered the edge case for Prometheus AlertManager and was surprised by the fact that helmfile.yaml is a template. It seems that it&#39;s …

Erik Forsberg avatar
Erik Forsberg

I.e, I have two nginx-ingress releases, ingress-test and ingress-production, and I want to set antiaffinity based on the release name, from a values template.

2019-06-23

2019-06-22

2019-06-21

Nelson Jeppesen avatar
Nelson Jeppesen

What’s the appropriate way to run a post deploy task?

I’m spinning up an elasticsearch cluster, but after i do I want helm to run a few curl commands. I’m thinking about creating a chart with a cronjob,but is there a better way?

I know about post hooks, but that runs locally where the helmfile is run, right? Id rather run this once in a pod on the k8s cluster.

mumoshu avatar
mumoshu

Try helm chart’s “post-install” hook https://github.com/helm/helm/blob/master/docs/charts_hooks.md

helm/helm

The Kubernetes Package Manager. Contribute to helm/helm development by creating an account on GitHub.

Erik Osterman avatar
Erik Osterman

#codefresh supports helm events as triggers :-)

Erik Osterman avatar
Erik Osterman
stakater/Chowkidar

A kubernetes controller that watches/observes events & then takes configured actions – [✩Star] if you’re using it! - stakater/Chowkidar

:--1:1
mumoshu avatar
mumoshu


#codefresh supports helm events as triggers :-)

I’m interested. codefresh watches for helm events so that you can hook into them?

2019-06-20

2019-06-19

Erik Forsberg avatar
Erik Forsberg

Hmm.. Does a non-default environment inherit values from the default environment? I.e, values not defined under “production”, will they inherit value defined under “default”?

mumoshu avatar
mumoshu

Are you talking about {{.Environment.Values}} ?

Erik Forsberg avatar
Erik Forsberg

Yes.

mumoshu avatar
mumoshu

Nope

mumoshu avatar
mumoshu

But {{.Values}} should. This is new, experimental one

mumoshu avatar
mumoshu

\# Probably this is what you want to call "defaults"
values:
- foo: bar

environments:
  prod:
    values:
    - bar: baz

Now in helmfile -e prod you get bar for {{.Values.foo}}

Erik Forsberg avatar
Erik Forsberg

Aha. Yes, that should solve my problem. Does that work with 0.79? It’s not documented in the example helmfile as far as I can see.

mumoshu avatar
mumoshu

i’m awaiting several good feedbacks before documenting it(as it’s experimental :)

Erik Forsberg avatar
Erik Forsberg

Right. I see it’s in 0.71 or something, so that’s cool. I’ll try it out!

mumoshu avatar
mumoshu

yep it should work in 0.79

mumoshu avatar
mumoshu

you should use v0.75.1 at minimum, which includes the fix for the latest bug in the new values feature

Erik Forsberg avatar
Erik Forsberg

I’m on 0.78, so that’s fine. The release pace is shockingly fast!

1
mumoshu avatar
mumoshu

ok good!

SeB avatar

@mumoshu just to let you know I had a good time contributing to the helmfile project but I will certainly not contribute anymore. We have decided to leverage the ansible scripts we are using for production deployment also for the developer workflow. I believe it is too big a machinery compared to helmfile but this is a pragmatical decision for now to not multiply our tools.

1
mumoshu avatar
mumoshu

sad but understandable! thanks a lot for your contributions so far!

i’ll try to keep improving helmfile, hoping it would help you someday

SeB avatar

This project is awesome and will keep an eye on it.

1
Erik Forsberg avatar
Erik Forsberg

How do I make helmfile stop if there’s trouble with one of the charts? I.e, I don’t want it to continue setting up chart B if chart A had any kind of trouble. Is that the wait parameter? I’m already running with concurrency 1 since I use tillerless.

mumoshu avatar
mumoshu

seems like a good feature request

mumoshu avatar
mumoshu

helmfile’s behaviour is tuned towards concurrency > 1 and in a concurrent usage stopping another run in a middle of the process just because the another failed seemed like a bad idea

mumoshu avatar
mumoshu

but when concurrency is 1, an option like fastfast: true may make sense

mumoshu avatar
mumoshu

in the mean time you can use helmfiles: so that each sub-helmfile under it is processed sequentially.

but probably that’s not what u want

Erik Forsberg avatar
Erik Forsberg

Well, ideally, I would like to set some dependencies, so if chart A fails, chart B can’t continue but chart C can.

Erik Forsberg avatar
Erik Forsberg

Btw, it should be possible to solve the concurrency issue with tillerless by running tiller from helmfile rather than by using the tillerless plugin, but that may not be worth it as helm 3 will be tillerless anyway.

Andrey Nazarov avatar
Andrey Nazarov

I would love to see the concurrency issue solved for tillerless workflow. But yes, since Helm 3 is coming probably it doesn’t seem worth the effort. tillerless plugin has other drawbacks as well, btw.

Nelson Jeppesen avatar
Nelson Jeppesen

I’m confused to to best use readFile with .gotmpl when pulling in a non-yaml file

I have this in a values .gotmpl file, and it works

extraConfigMaps:
  output.conf: |-
    <match **>
      @id elasticsearch
      @type elasticsearch
...

I’d like to use this, so I can pull out that large xml into it’s own file fluentd-elasticsearch.xml

extraConfigMaps:
    output.conf: |-
    {{ readFile "fluentd-elasticsearch.xml" | indent 2 }}

but I keep getting error converting YAML to JSON: yaml: line 35: could not find expected ':'

mumoshu avatar
mumoshu

Hey! Try this:

extraConfigMaps:
    output.conf: |-
{{ readFile "fluentd-elasticsearch.xml" | indent 6 }}
mumoshu avatar
mumoshu

not sure why you have 4 spaces before output.conf but you can also try:

extraConfigMaps:
  output.conf: |-
{{ readFile "fluentd-elasticsearch.xml" | indent 4 }}
mumoshu avatar
mumoshu
extraConfigMaps:
    output.conf: |-
    {{ readFile "fluentd-elasticsearch.xml" | indent 2 }}

this won’t work cuz we get

extraConfigMaps:
  output.conf: |-
    <match **>
  @id elasticsearch
  @type elasticsearch
...
Nelson Jeppesen avatar
Nelson Jeppesen

@mumoshu omg thang you - I feel very dumb, but that was it

:--1:1
mumoshu avatar
mumoshu

glad to help!

Nelson Jeppesen avatar
Nelson Jeppesen

Pierre Humberdroz avatar
Pierre Humberdroz

Can I update the requirment version of a helm chart that I am using ? So in this case I am using stable/prometheus-operator and would like to update the grafana version to 3.5 instead of 3.3

mumoshu avatar
mumoshu

Yep. Do you have version already set for that chart in your helmfile.yaml?

mumoshu avatar
mumoshu

If not, add stable to under repositories of your helmfile.yaml, then run helmfile apply or helmfile diff or whatever, so that it automatically fetches the latest version

Pierre Humberdroz avatar
Pierre Humberdroz

so the requirments would see that I already have stable/grafana installed ?

mumoshu avatar
mumoshu

what do you mean by the requirments exactly?

mumoshu avatar
mumoshu

in case you have a helmfile.lock file generated by helmfile deps, the grafana chart version should be saved there, and it’s updated only when you run helmfile deps next time

Ribhararnus Pracutiar avatar
Ribhararnus Pracutiar

hi @mumoshu is yesterday issue not resolved yet?

Ribhararnus Pracutiar avatar
Ribhararnus Pracutiar

anyway, I want to ask question here regarding multi-line string

Ribhararnus Pracutiar avatar
Ribhararnus Pracutiar

local.yaml:

config: |
  test: a
  yes: true

in helmfile chart, template/configmap.yaml

default.yaml: |
  {{ tpl .Values.config . }}
mumoshu avatar
mumoshu

Try

default.yaml: |
{{ tpl .Values.config . | indent 2 }}
mumoshu avatar
mumoshu

This

default.yaml: |
  {{ tpl .Values.config . }}

ends up(notice the indentation issue!

default.yaml: |
  config: |
  test: a
  yes: true
Ribhararnus Pracutiar avatar
Ribhararnus Pracutiar

I think it will parse to this:

yes: true
default.yaml: |
  test: a
Ribhararnus Pracutiar avatar
Ribhararnus Pracutiar

that’s weird

Ribhararnus Pracutiar avatar
Ribhararnus Pracutiar

only first key value that correctly parse into default.yaml

mumoshu avatar
mumoshu

ah yeah good catch

Ribhararnus Pracutiar avatar
Ribhararnus Pracutiar

actual file:

data:
  nodeEnv: {{ .Values.environment }}
  default.yaml: |
  {{ tpl .Values.config . | indent 2 }}
mumoshu avatar
mumoshu

Thats because

default.yaml: |
  {{ tpl .Values.config . }}

Here .Values.config is two-lines text:

test: a
yes: true
Ribhararnus Pracutiar avatar
Ribhararnus Pracutiar

if I change indent to 4 it would give me error

mumoshu avatar
mumoshu

That is rendered as

default.yaml: |
  test: 1
yes: true
Ribhararnus Pracutiar avatar
Ribhararnus Pracutiar

do you have solution for this kind of issue?

mumoshu avatar
mumoshu

Yep. so try adding | indent 2

default.yaml: |
{{ tpl .Values.config . | indent 2 }}
mumoshu avatar
mumoshu

without the first two spaces before {{ tpl ...

Ribhararnus Pracutiar avatar
Ribhararnus Pracutiar

yes, still the configmap cannot parse the tpl

mumoshu avatar
mumoshu

that sounds like you have another issue

mumoshu avatar
mumoshu

are you using the same set of files you’ve pasted here in your actual setup?

Ribhararnus Pracutiar avatar
Ribhararnus Pracutiar

here is my actual files:

template/configmap.yaml

data:
  nodeEnv: {{ .Values.environment }}
  default.yaml: |
  {{ tpl .Values.config . | indent 2 }}

values.yaml

config: |
  port: 3090
  baseURL: <http://localhost:5001/v1>
  streamTopic: sync-web
Ribhararnus Pracutiar avatar
Ribhararnus Pracutiar

and helmfile parse this as

data:
  baseURL: '<http://localhost:5001/v1>'
  default.yaml: |
    port: 3090
  nodeEnv: development
  streamTopic: sync-web
mumoshu avatar
mumoshu

Yeah, try

data:
  nodeEnv: {{ .Values.environment }}
  default.yaml: |
{{ tpl .Values.config . | indent 4 }}
mumoshu avatar
mumoshu

The golden rule is that you shouldn’t add spaces before any template expression that results in a multi-line text

Ribhararnus Pracutiar avatar
Ribhararnus Pracutiar

indent 4 give me error

mumoshu avatar
mumoshu

Instead add | indent N to it

mumoshu avatar
mumoshu

Which error?

Ribhararnus Pracutiar avatar
Ribhararnus Pracutiar

oh my bad

Ribhararnus Pracutiar avatar
Ribhararnus Pracutiar

thank you

Ribhararnus Pracutiar avatar
Ribhararnus Pracutiar

that seems resolved now

Ribhararnus Pracutiar avatar
Ribhararnus Pracutiar

indent 2 confused me

mumoshu avatar
mumoshu

Great!

mumoshu avatar
mumoshu

Hehe

Ribhararnus Pracutiar avatar
Ribhararnus Pracutiar

thank you mumoshu this is tricky actually

:--1:1
mumoshu avatar
mumoshu

Yeah tricky but it’s actually a very common gotcha of yaml templates. You’ll see bunch of indent N in helm chart templates as well

mumoshu avatar
mumoshu

I’m not happy with that either but you’ll feel better once you get used to it…

Ribhararnus Pracutiar avatar
Ribhararnus Pracutiar

but after deployment, or sync, I only have

default.yaml:
  test: a

not including yes: true

Ribhararnus Pracutiar avatar
Ribhararnus Pracutiar

is it true?

mumoshu avatar
mumoshu

@Ribhararnus Pracutiar Regarding the issue you had yesterday, yeah, I think I got it.

Try adding the below to repositories.yaml:

environments:
  local: {}
  development: {}
  staging: {}
  production: {}
mumoshu avatar
mumoshu

It’s annoying but how helmfile currently works.

I think we need to change helmfile to not complain on missing env for bases(or probably in other cases as well

This one is similar to yours https://github.com/roboll/helmfile/issues/704#issuecomment-503645728

Unable to matches specified selector() and environment(master), in any helmfile when using Helmfiles · Issue #704 · roboll/helmfile

I&#39;m trying to make my helmfiles DRY helmfile.yaml helmfiles: - apps/99_monitor.yaml apps/99_monitor.yaml bases: - ../common/environments.yaml - ../common/repositories/kubernetes.yaml - ../commo…

Ribhararnus Pracutiar avatar
Ribhararnus Pracutiar

ok thanks, maybe later I comeback with the issue, as for now, I use repositories in every single helmfile

Ribhararnus Pracutiar avatar
Ribhararnus Pracutiar

what about above issue?

Ribhararnus Pracutiar avatar
Ribhararnus Pracutiar

ah Im sorry, you replied on my message, wait

1

2019-06-18

Erik Forsberg avatar
Erik Forsberg

Is there any variable I can use from within a hook to anchor my paths to the directory of the helmfile?

The use case is that I’m modifying a namespace via an kubectl apply -f <file>, and right now I must be in the same directory as the helmfile, i.e I can’t use –file to point at the helmfile, or the <file> won’t be found. So if there is a variable, and hook arguments are templated, then.. I could.

mumoshu avatar
mumoshu

Interesting! I thought I’ve designed helmfile to execute any command including helm, exec, hooks from within the directory that the helmfile resides, regardless of which directory you are currently at. But what you observed doesn’t match it, right?

Erik Forsberg avatar
Erik Forsberg

Haha, I didn’t try running it from a different directory, I just assumed that it wouldn’t work, and have only tried with cwd being correct. I’ll check if it works

1
Sean Turner avatar
Sean Turner

Does helmfile ever get locked out? I’ve been trying to install a couple of deploys, a service, ingress, service-account, cluster-role-binding, and configmap, and I started getting the following error (after many helmfile applys). However, I switched clusters, and stopped getting the error…

$ helmfile apply --args "--dry-run"
Building dependency ../my-appV2
No requirements found in ../my-appV2/charts.

Building dependency ../my-appV2
No requirements found in ../my-appV2/charts.

Building dependency ../my-appV2
No requirements found in ../my-appV2/charts.

Building dependency ../my-appV2
No requirements found in ../my-appV2/charts.

Building dependency ../my-appV2
No requirements found in ../my-appV2/charts.

Building dependency ../my-appV2
No requirements found in ../my-appV2/charts.

Building dependency ../my-appV2
No requirements found in ../my-appV2/charts.

Comparing sean-configmap ../my-appV2
Comparing sean-deploy-green ../my-appV2
Comparing sean-service-account ../my-appV2
Comparing sean-ingress ../my-appV2
Comparing sean-service ../my-appV2
Comparing sean-deploy-blue ../my-appV2
Comparing sean-namespace ../my-appV2
in ./helmfile.yaml: 7 errors:
err 0: failed processing release sean-ingress: helm exited with status 1:
  Error: "sean-ingress" has no deployed releases
  Error: plugin "diff" exited with error
err 1: failed processing release sean-service-account: helm exited with status 1:
  Error: "sean-service-account" has no deployed releases
  Error: plugin "diff" exited with error
err 2: failed processing release sean-deploy-blue: helm exited with status 1:
  Error: "sean-deploy-blue" has no deployed releases
  Error: plugin "diff" exited with error
err 3: failed processing release sean-configmap: helm exited with status 1:
  Error: "sean-configmap" has no deployed releases
  Error: plugin "diff" exited with error
err 4: failed processing release sean-service: helm exited with status 1:
  Error: "sean-service" has no deployed releases
  Error: plugin "diff" exited with error
err 5: failed processing release sean-namespace: helm exited with status 1:
  Error: "sean-namespace" has no deployed releases
  Error: plugin "diff" exited with error
err 6: failed processing release sean-deploy-green: helm exited with status 1:
  Error: "sean-deploy-green" has no deployed releases
  Error: plugin "diff" exited with error
Sean Turner avatar
Sean Turner

I noticed these twistlock notifications new host runtime alterts (I’ve got twistlock defenders deployed on the same cluster):

Service google-ip-forwarding-daemon attempted to obtain capability NET by executing ip route add to local $IP/32 scope host dev eth0 proto 66 (parent: /usr/bin/python2.7). Low severity audit, event is automatically added to the runtime model
Sean Turner avatar
Sean Turner

Figured it out. Needed to helm del --purge $RELEASE_NAME for each release. Don’t know why this happened.

jqjq avatar

Is there a way to run a custom script everytime a helmfile command is run? One usecase is that when I run a helmfile -e ENV … I want to run a script to check my KUBECONFIG is pointing to the right cluster for the environment. I can do it as a release hook, but the release hook declaration has to be duplicated for every release. It’d be nice if I can define a hook globally.

mumoshu avatar
mumoshu

Helmfile doesn’t cover that usecase alone. As helmfile isn’t a generic task runner I’m not yet inclined to enhance helmfile for that.

Shameless plug but I built/use #variant for wrapping helmfile so that I can add any pre/post helmfile script without affecting the overall ux much!

Ribhararnus Pracutiar avatar
Ribhararnus Pracutiar

hi @mumoshu can I add repositories.yaml file to bases? it contains repositories spec

mumoshu avatar
mumoshu

yeah it should work

Ribhararnus Pracutiar avatar
Ribhararnus Pracutiar
list complaining about no releases found · Issue #315 · roboll/helmfile

err: no releases found that matches specified selector() and environment(default), in any helmfile This is running helmfile lint with the new helmfiles: [] feature.

Ribhararnus Pracutiar avatar
Ribhararnus Pracutiar

add repositories causes error

Ribhararnus Pracutiar avatar
Ribhararnus Pracutiar
err: no releases found that matches specified selector() and environment(local), in any helmfile
Ribhararnus Pracutiar avatar
Ribhararnus Pracutiar

my bases:

bases:
  - ./base/repositories.yaml
  - ./base/environments.yaml
mumoshu avatar
mumoshu

could you try moving base dir out of helmfile.d?

Ribhararnus Pracutiar avatar
Ribhararnus Pracutiar

yes, I have tried that too

mumoshu avatar
mumoshu

hmm, could you add --log-level=debug like helmfile --log-level=debug to hopefully see at which point helmfile is failing?

Ribhararnus Pracutiar avatar
Ribhararnus Pracutiar
processing file "0-system-service.yaml" in directory "helmfile.d"
changing working directory to "/home/hh/deployments/helmfile.d"
first-pass rendering starting for "0-system-service.yaml.part.0": inherited=&{local map[] map[]}, overrode=<nil>first-pass uses: &{local map[] map[]}
error in first-pass rendering: result of "0-system-service.yaml.part.0":
 0: #
 1:
 2: bases:
 3:   # - ./base/environments.yaml
 4:   - ./base/repositories.yaml
 5:   - ./base/environments.yaml
 6:
 7:
 8: releases:
 9:   - name: sb-redis
10:     chart: stable/redis
11:     labels:
12:       test: true
13:

first-pass produced: &{local map[] map[]}
first-pass rendering result of "0-system-service.yaml.part.0": {local map[] map[]}
second-pass rendering result of "0-system-service.yaml.part.0":
 0: #
 1:
 2: bases:
 3:   # - ./base/environments.yaml
 4:   - ./base/repositories.yaml
 5:   - ./base/environments.yaml
 6:
 7:
 8: releases:
 9:   - name: sb-redis
10:     chart: stable/redis
11:     labels:
12:       test: true
13:

first-pass rendering starting for "base/repositories.yaml.part.0": inherited=&{local map[] map[]}, overrode=<nil>
first-pass uses: &{local map[] map[]}
first-pass produced: &{local map[] map[]}
first-pass rendering result of "base/repositories.yaml.part.0": {local map[] map[]}
vals:
map[]
defaultVals:[]
second-pass rendering result of "base/repositories.yaml.part.0":
 0: repositories:
 1:   - name: stable
 2:     url: <https://kubernetes-charts.storage.googleapis.com/>
 3:   - name: incubator
 4:     url: <https://kubernetes-charts-incubator.storage.googleapis.com/>

changing working directory back to "/home/hh/deployments"
err: no releases found that matches specified selector() and environment(local), in any helmfile
mumoshu avatar
mumoshu

so you seem to still have base under helmfile.d

Ribhararnus Pracutiar avatar
Ribhararnus Pracutiar

oh yes I forgot to move, but when I move I still have the error

mumoshu avatar
mumoshu

if you move it out of helmfile.d do you see another errors?

Ribhararnus Pracutiar avatar
Ribhararnus Pracutiar

hold on

Ribhararnus Pracutiar avatar
Ribhararnus Pracutiar

still error

mumoshu avatar
mumoshu

which error do you see?

Ribhararnus Pracutiar avatar
Ribhararnus Pracutiar

my bases:

bases:
  - ../common/base/repositories.yaml
  - ../common/base/environments.yaml
Ribhararnus Pracutiar avatar
Ribhararnus Pracutiar

but if I remove repositories line, it works

mumoshu avatar
mumoshu

mind sharing your logs with --log-level=debug once again?

Ribhararnus Pracutiar avatar
Ribhararnus Pracutiar
processing file "0-system-service.yaml" in directory "helmfile.d"
changing working directory to "/home/hh/test/deployments/helmfile.d"
first-pass rendering starting for "0-system-service.yaml.part.0": inherited=&{local map[] map[]}, overrode=<nil>first-pass uses: &{local map[] map[]}
error in first-pass rendering: result of "0-system-service.yaml.part.0":
 0: #
 1:
 2: bases:
 3:   # - ./base/environments.yaml
 4:   - ../common/base/repositories.yaml
 5:   - ../common/base/environments.yaml
 6:
 7:
 8: releases:
 9:   - name: sb-redis
10:     chart: stable/redis
11:     labels:
12:       test: true
13:

first-pass produced: &{local map[] map[]}
first-pass rendering result of "0-system-service.yaml.part.0": {local map[] map[]}
second-pass rendering result of "0-system-service.yaml.part.0":
 0: #
 1:
 2: bases:
 3:   # - ./base/environments.yaml
 4:   - ../common/base/repositories.yaml
 5:   - ../common/base/environments.yaml
 6:
 7:
 8: releases:
 9:   - name: sb-redis
10:     chart: stable/redis
11:     labels:
12:       test: true
13:

first-pass rendering starting for "../common/base/repositories.yaml.part.0": inherited=&{local map[] map[]}, overrode=<nil>
first-pass uses: &{local map[] map[]}
first-pass produced: &{local map[] map[]}
first-pass rendering result of "../common/base/repositories.yaml.part.0": {local map[] map[]}
vals:
map[]
defaultVals:[]
second-pass rendering result of "../common/base/repositories.yaml.part.0":
 0: repositories:
 1:   - name: stable
 2:     url: <https://kubernetes-charts.storage.googleapis.com/>
 3:   - name: incubator
 4:     url: <https://kubernetes-charts-incubator.storage.googleapis.com/>

changing working directory back to "/home/hh/test/deployments"
err: no releases found that matches specified selector() and environment(local), in any helmfile
mumoshu avatar
mumoshu

@Ribhararnus Pracutiar Thanks!

Would you mind sharing --log-level=debug logs with (1) - ./base/repositories.yaml commented out and (2) - ./base/repositories.yaml moved after ../common/base/environments.yaml as well? Probably I can get what’s wrong with the info

Ribhararnus Pracutiar avatar
Ribhararnus Pracutiar

I remove ./base/repositories, and move ../common/base/repositories under environments:

$ helmfile --environment local --log-level=debug lint
processing file "0-system-service.yaml" in directory "helmfile.d"
changing working directory to "/home/hh/test/deployments/helmfile.d"
first-pass rendering starting for "0-system-service.yaml.part.0": inherited=&{local map[] map[]}, overrode=<nil>first-pass uses: &{local map[] map[]}
error in first-pass rendering: result of "0-system-service.yaml.part.0":
 0:
 1:
 2: bases:
 3:   - ../common/base/environments.yaml
 4:   - ../common/base/repositories.yaml
 5:
 6:
 7: releases:
 8:   - name: sb-redis
 9:     chart: stable/redis
10:     labels:
11:       test: true
12:

first-pass produced: &{local map[] map[]}
first-pass rendering result of "0-system-service.yaml.part.0": {local map[] map[]}
second-pass rendering result of "0-system-service.yaml.part.0":
 0:
 1:
 2: bases:
 3:   - ../common/base/environments.yaml
 4:   - ../common/base/repositories.yaml
 5:
 6:
 7: releases:
 8:   - name: sb-redis
 9:     chart: stable/redis
10:     labels:
11:       test: true
12:

first-pass rendering starting for "../common/base/environments.yaml.part.0": inherited=&{local map[] map[]}, overrode=<nil>
first-pass uses: &{local map[] map[]}
envvals_loader: loaded ../common/environments/local.yaml:map[test:true]
first-pass produced: &{local map[test:true] map[]}
first-pass rendering result of "../common/base/environments.yaml.part.0": {local map[test:true] map[]}
vals:
map[test:true]
defaultVals:[]
second-pass rendering result of "../common/base/environments.yaml.part.0":
 0: environments:
 1:   local:
 2:     values:
 3:       - ../common/environments/local.yaml
 4:   development:
 5:     values:
 6:       - ../common/environments/development.yaml
 7:   staging:
 8:     values:
 9:       - ../common/environments/staging.yaml
10:   production:
11:     values:
12:       - ../common/environments/production.yaml

envvals_loader: loaded ../common/environments/local.yaml:map[test:true]
merged environment: &{local map[test:true] map[]}
first-pass rendering starting for "../common/base/repositories.yaml.part.0": inherited=&{local map[] map[]}, overrode=<nil>
first-pass uses: &{local map[] map[]}
first-pass produced: &{local map[] map[]}
first-pass rendering result of "../common/base/repositories.yaml.part.0": {local map[] map[]}
vals:
map[]
defaultVals:[]
second-pass rendering result of "../common/base/repositories.yaml.part.0":
 0: repositories:
 1:   - name: stable
 2:     url: <https://kubernetes-charts.storage.googleapis.com/>
 3:   - name: incubator
 4:     url: <https://kubernetes-charts-incubator.storage.googleapis.com/>

changing working directory back to "/home/hh/test/deployments"
err: no releases found that matches specified selector() and environment(local), in any helmfile
Ribhararnus Pracutiar avatar
Ribhararnus Pracutiar

actually my original helmfile is complicated and debugging for hours, I don’t know what’s happening, until I am surprised that the repositories base part causes the error a while ago

1
Ribhararnus Pracutiar avatar
Ribhararnus Pracutiar

anyway, Thank you very much for your support @mumoshu

:--1:1
mumoshu avatar
mumoshu

@Ribhararnus Pracutiar according to the logs your issue isn’t resolved yet, right? I’ll take a look and try to see what can go wrong tonight thanks for reporting!

Ribhararnus Pracutiar avatar
Ribhararnus Pracutiar

No worries buddy. Right still not resolved, maybe I wanna use repositories in every file, violate DRY till the issue resolved

mumoshu avatar
mumoshu

That helps my sleep! Thanks for your cooperation I’ll report back once I managed to fix it

2019-06-17

Alexis avatar
Alexis

hey guys, just want to know if it’s possible to use reuse-values in helmfile?

Erik Osterman avatar
Erik Osterman

helmfile supports YAML anchors

Erik Osterman avatar
Erik Osterman

might that help?

Erik Osterman avatar
Erik Osterman

we also use environment variables to do this

mumoshu avatar
mumoshu

And helmfile has never intended to support helm’s --reuset-values because it defeats the one of purposes of helmfile - reproducible deployments! But I may be missing something - what’s your goal for –reuset-values @Alexis?

Alexis avatar
Alexis

it is because we don’t save the actual image tags in git. we don’t want to reset the versions for other releases

mumoshu avatar
mumoshu

How do you set the chart value for a image tag then?

Alexis avatar
Alexis

I always pass it using –set when using helm. In helmfile, I plan on using environment variables.

mumoshu avatar
mumoshu

hmm im pretty confused. do you really rely on --reset-values when you use helm?

mumoshu avatar
mumoshu

sounds like you can just place something like tag: {{ requiredEnv" IMAGE_TAG"}} in your helmfile.yaml or even helmfile --state-values-set image.tag=${IMAGE_TAG}

mumoshu avatar
mumoshu

without –reset-values

mumoshu avatar
mumoshu

We now have a Kubernetes operator for helmfile

https://github.com/roboll/helmfile/issues/153 https://github.com/mumoshu/helmfile-operator

I’m trying to build something that complements weaveworks/flux, argo-cd, brigadecore/brigade…

[In the future] Helmfiles as a chart / Helmfile as a K8S CR · Issue #153 · roboll/helmfile

So that we can leverage everything helm provides to power helmfile :) User-defined Lua functions included in charts (Perhaps in Helm v3) Helm chart repositories to distribute versioned helmfiles fo…

mumoshu/helmfile-operator

Kubernetse operator that continuously sync any set of Chart/Kustomize/Manifest fetched from S3/Git/GCS to your cluster - mumoshu/helmfile-operator

1
Erik Osterman avatar
Erik Osterman

Amazing!

Erik Osterman avatar
Erik Osterman

excited to take a look

mumoshu avatar
mumoshu

Ask me anything! I’m not yet sure how to best describe it myself

mumoshu avatar
mumoshu

one of limitations of flux, argo-cd is that it’s inability to run helmfile apply on git commit.. it works by sourcing k8s manfiests generated by helmfile template(both of em allows you to run helmfile template automatically on git commit btw)

mumoshu avatar
mumoshu

normally it makes many useful helm commands unusable. helm test, helm status, helm list, and so on.

mumoshu avatar
mumoshu

with helmfile-operator you can manage Helmfile custom resources in git, so that argo-cd or flux can reconcile helmfile crs on git commit, and then the operator runs helmfile commands

mumoshu avatar
mumoshu

brigade is basically a ci/cd system that takes JavaScript code as a pipeline definition. so just run helmfile apply from js that is triggered via git commit

mumoshu avatar
mumoshu

or create Helmfile custom resources from the pipeline and the rest is the same as argo-cd and flux.

Andrey Nazarov avatar
Andrey Nazarov

Man, I’ve been waiting for something like helmfile operator. Even thought about trying to write it by myself, but - it’s here. I’ll definitely take a look.

fast_parrot1
Andrey Nazarov avatar
Andrey Nazarov

Really great news!

Erik Osterman avatar
Erik Osterman

how are errors handled? e.g. a bad helmfile rollout?

mumoshu avatar
mumoshu

currently it just fail leaving logs in the helmfile-applier pod. as well as diff i’m considering to use slack or github issue/pr comments to surface error messages for human operators. wdyt?

Erik Osterman avatar
Erik Osterman

Aha, yes, slack would be nice

Erik Osterman avatar
Erik Osterman

i imagine helmfile diff is not really suitable for this workflow?

mumoshu avatar
mumoshu

good point!

for that i’m planning to add a feature that uses slack or github issue comments to show changes and manual approval

:--1:1
Andrey Nazarov avatar
Andrey Nazarov

What I find tricky in Argo and Flux is how they handle multi-cluster setup. I think it should be easier to do with helmfile-operator.

mumoshu avatar
mumoshu

Yeah probably. One way would be to have a multi-cluster helmfile.yaml that has different kubeContext set per release. Each release can be regular charts, or even charts for Helmfile resource…

mumoshu avatar
mumoshu

// Just recalled that I was in the process of renaming the project from appliance-operator to helmfile-operator.. Hold on

Erik Osterman avatar
Erik Osterman

aha, that is more clear to me

2019-06-15

2019-06-14

mumoshu avatar
mumoshu

Yeah! I thought I’ve changed all the logs to go stderr and made stdout dedicated to actual template output. But that’s not the case, right? If you could comment on the relevant github issue about redundant lines you saw, it would help me fixing it

timduhenchanter avatar
timduhenchanter
Cannot apply result generated by `helmfile template` · Issue #685 · roboll/helmfile
When generating template with helmfile template, some log lines appears in the output, which prevent to run kubectl apply -f - on it. helmfile template kubectl -n nucleus-preprod apply -f Buildin…
Ryan Richards avatar
Ryan Richards

Really like the content you guys publish. Also am enjoying helmfile.

2
Ryan Richards avatar
Ryan Richards

Im using terraform to build the cluster and would like to use helmfile for configuring all the post-installation tasks.

:--1:1
Erik Osterman avatar
Erik Osterman

if only we had a terraform-helmfile-provider

1

2019-06-13

SeB avatar

Hello @mumoshu Is it on purpose that there is no documentation on the new state value introduced here : https://github.com/roboll/helmfile/pull/647

feat: state values by mumoshu · Pull Request #647 · roboll/helmfile

This adds values to state files as proposed in #640. values: - key1: val1 - defaults.yaml environments: default: - values: - environments/default.yaml production: - values: - envir…

mumoshu avatar
mumoshu

Yes. I wanted to confirm that it’s working as you might have expected first

https://github.com/roboll/helmfile/pull/647#issuecomment-498558304

feat: state values by mumoshu · Pull Request #647 · roboll/helmfile

This adds values to state files as proposed in #640. values: - key1: val1 - defaults.yaml environments: default: - values: - environments/default.yaml production: - values: - envir…

mumoshu avatar
mumoshu

before writing any doc

SeB avatar

I do confirm this works as expected.

Cedric Meury avatar
Cedric Meury

Yo fellow Helmfile freaks

mumoshu avatar
mumoshu

Hey!

timduhenchanter avatar
timduhenchanter

Is it possible to get helmfile template to only spit out YAML within the executable? Right now I’m just nuking all the lines before the first --- match but I’m not familiar enough with helmfile yet to know that this will resolve all cases of non-templated output

2019-06-12

Justice London avatar
Justice London

Howdy. Not sure if this question has been asked before… but how do you get selectors when specifying an environment to work? I get an error if I use something like:

helmfiles:
   - path: environments/coreserv/production/helmfile.yaml

---
releases:
  - name: mycoolrelease
    labels:
       team: coreserv

---
`helmfile -e production -l team=coreserv diff`
mumoshu avatar
mumoshu

Hey! What are you using the production environment for? Can you omit it if you can live without it?

mumoshu avatar
mumoshu

I’d like to see the actual error message you got, but probably it’s due to that you’re missing the production environment… in under environments section and helmfile is complaining about the missing env, right?

Justice London avatar
Justice London

I do have a production environment defined, like so:

environments:
  default:
    values:
      - environments/default/values.yaml
  development:
    values:
      - environments/{{ env "TEAM_NAME" | default "coreserv" }}/development/values.yaml
  production:
    values:
      - environments/{{ env "TEAM_NAME" | default "coreserv" }}/production/values.yaml
mumoshu avatar
mumoshu

Thanks!

Would you mind sharing the full log of your helmfile execution with debug logs helmfile --debug log-level?

Justice London avatar
Justice London

Sure. I’ll post it tomorrow. Not at the office right now.

Justice London avatar
Justice London

If I don’t use an environment it is able to find the helmfile releases defined there.

mumoshu avatar
mumoshu

@Justice London Just replied in the thread

2019-06-10

SeB avatar

I’ll try but this should not be any different.

SeB avatar

and also this will fail cause value resolution does not support a dash in the values

mumoshu avatar
mumoshu

Not sure what’s causing your confusion exactly, but try:

releases:
- name: tpsvc-config
  chart: |
    {{`{{ .Environment.Values | get (printf "%sVersion" "tpsvc-config") "" | eq "" | ternary "../.." "talend" }}` }}/{{`{{.Release.Name}}`}}
SeB avatar

Hi @mumoshu thanks a lot ! this is working but I have another issue though. The version of a release seem not to accept templatization

releases:
- name: tpsvc-config
  chart: |
    {{ `{{  .Environment.Values | get (printf "%sVersion" "tpsvc-config") "" | eq "" | ternary "../.." "talend" }}` }}/{{`{{.Release.Name}}`}}
  version: |
    {{` {{ .Environment.Values | get (printf "%sVersion" "tpsvc-config") "" }}` }}
SeB avatar

it is failing with :

  Error: chart "tpsvc-config" matching {{ .Environment.Values | get (printf "%sVersion" "tpsvc-config") "" }} not found in talend index. (try 'helm repo update'). improper constraint: {{ .Environment.Values | get (printf "%sVersion" "tpsvc-config") "" }}
mumoshu avatar
mumoshu

Ah, that sounds like not implemented!

Andrey Nazarov avatar
Andrey Nazarov

Got it, nevermind the question. I’ve deleted it.

mumoshu avatar
mumoshu

// hold on..

SeB avatar

yep

SeB avatar

it seems that the version is missing here

mumoshu avatar
mumoshu

You found it!

SeB avatar

Do you want me to create an issue ?

mumoshu avatar
mumoshu

Please do so! It’s even nicer if you directly create a PR to add it

SeB avatar

yep I know

1
SeB avatar

I’ll try to find some time tonight

party_parrot1
1
SeB avatar
thank anyway for the “ ” tip, yaml is always a mystery for me.
Andrey Nazarov avatar
Andrey Nazarov

Wow, it’s great to know about “I”.

1
SeB avatar
@mumoshu can you explain why the is required for this scalar, why the initial string does not work. is it because the the ” inside the string ?
SeB avatar

I have made it work using the simple quote to surround the string like

  chart: '{{ `{{  .Environment.Values | get (printf "%sVersion" "tpsvc-config") "" | eq "" | ternary "../.." "talend" }}`'
SeB avatar

it fails using the double quote even with the inside double quote escaped ?

mumoshu avatar
mumoshu

@SeB That’s basically due to that {{ whatever }}//{{ whatever }} isn’t a valid yaml string

mumoshu avatar
mumoshu
chart: {{` {{  .Environment.Values | get (printf "%sVersion" "tpsvc-config") "" | eq "" | ternary "../.." "talend" }} ``}

evaluates to

chart: {{  .Environment.Values | get (printf "%sVersion" "tpsvc-config") "" | eq "" | ternary "../.." "talend" }}

which looks chart: {{ whatever }}//{{ whatever }} for the yaml parser

mumoshu avatar
mumoshu
chart: '{{ `{{  .Environment.Values | get (printf "%sVersion" "tpsvc-config") "" | eq "" | ternary "../.." "talend" }}`}}'

this works cuz it evaluates to:

chart: '{{  .Environment.Values | get (printf "%sVersion" "tpsvc-config") "" | eq "" | ternary "../.." "talend" }}'

which looks chart: 'str' to the yaml parser

SeB avatar

ok thanks a lot for the explanation

mumoshu avatar
mumoshu
chart: "{{` {{  .Environment.Values | get (printf "%sVersion" "tpsvc-config") "" | eq "" | ternary "../.." "talend" }} `}}"

and this is close but doesn’t work in your case as you have " in release tempaltes

mumoshu avatar
mumoshu

it evaluates to

chart: "{{  .Environment.Values | get (printf "%sVersion" "tpsvc-config") "" | eq "" | ternary "../.." "talend" }}"

and it looks chart: "{{ .Enviro...ersion" "tpsvc-config") "" | eq "" ..... to the yaml parser

) and ` eq ` should look like stray terms to the yaml parser

and probably chart: "{{ .Enviro...ersion" "tpsvc-config" is chart: "str1" "str2" which isn’t valid as well

SeB avatar

ha ok.

1
mumoshu avatar
mumoshu

my understanding may be incomplete but that’s probably it :slightly_smiling_face:

: | should always work

:--1:1
Andrey Nazarov avatar
Andrey Nazarov

Probably it would be useful to mention this trick somewhere in the docs. It’s not that obvious at first.

:--1:2

2019-06-09

SeB avatar

Hi guys, I have a template issue, I’d like to have in a release the “chart: ” value to be computed according to the version set in environment so that when :

• the version has some value like “0.1.3” the chart definition is chart: myregistry/mychart

• the version is empty or not set the chart def is a local chart chart: ../mychart I thought that the following would work

environments:
  dev:
    values:
      - tpsvc-configVersion: ""
  prod:
    values:
      - tpsvc-configVersion: 1.0


templates:
  chart: &chart
    {{- $versionKey := printf "%sVersion" `{{.Release.Name}}` }}
    chart: {{ .Environment.Values | get ($versionKey) "" | eq "" | ternary "../.." "talend" }}/{{`{{.Release.Name}}`}}
    version: {{.Environment.Values | get ($versionKey) ""}}

releases:
- name: tpsvc-config
  <<: *chart

But it does not

Andrey Nazarov avatar
Andrey Nazarov

Out of curioucity, are you sure get and ternary are supported? I’ve never used them and cannot find them mentioned anywhere.

Andrey Nazarov avatar
Andrey Nazarov

And what so you mean by “it doesn’t work”?

Andrey Nazarov avatar
Andrey Nazarov

I was wondering if something like

{{ .Environment.Values.`{{ .Release.Name }}`.version }}

works? Never checked it out. Then it could be as simple as {{ with }} and {{ if/else }} blocks.

SeB avatar

yes get and ternary are supported.

SeB avatar

the issues comes from the fact that the templates are rendered before the release is rendered.

SeB avatar

I have tried to escape the template to have the rendered late w

SeB avatar

but I have a error.

Andrey Nazarov avatar
Andrey Nazarov


yes get and ternary are supported.
Yes, I’ve just found it in Sprig.

SeB avatar

When I say it does not work I mean I always get the local chart ../../tpsvc-config

SeB avatar

it never takes the version into account.

SeB avatar

basically the versionKey is rendered to early and is probably set with the value “{{.ReleaseName}}Version” which of course is never found in the value and this explains why I always get the ../../tpsvc-config

Andrey Nazarov avatar
Andrey Nazarov


I was wondering if something like ... works?
No, this won’t work for sure as well. At least not with that syntax.

Andrey Nazarov avatar
Andrey Nazarov

But it would be cool to have some sort of embedment…

Andrey Nazarov avatar
Andrey Nazarov

Another possible solution could be range by releases. Something like:

environments:
  dev:
    values:
      - versions:
          tpsvc-config: ""
          something-else: 1.0
  prod:
    values:
      - versions:
          tpsvc-config: 1.0
          something-else: 2.0

releases:
  {{- range $name, $version := .Environment.Values.versions }}
  - name: {{ $name }}
    {{- if eq $version "" }}
    chart: ../{{ $name }}
    {{- else }}
    chart: myregistry/{{ $name }}
    {{- end }}
    version: {{ $version }}
  {{- end }}
Andrey Nazarov avatar
Andrey Nazarov


basically the versionKey is rendered to early and is probably set with the value

Yep, probably that’s the case. Does your snippet work out of release template? I.e. being defined in releases block?

SeB avatar

it does not work either in the release block

SeB avatar

I think your solution with the range should work but this is not very easy to read.

Andrey Nazarov avatar
Andrey Nazarov

Yeah, the readability is not perfect for sure.

Andrey Nazarov avatar
Andrey Nazarov


it does not work either in the release block
that’s a bit suspicious. But you removed backticks, right?What does helm template show?

Andrey Nazarov avatar
Andrey Nazarov

Btw, could you please point me out to the get definition. I was too fast saying I found it in sprig. I’ve just seen ternary.

SeB avatar

this is one of the undocumented function provided by helmfile

SeB avatar
roboll/helmfile

Deploy Kubernetes Helm Charts. Contribute to roboll/helmfile development by creating an account on GitHub.

:--1:1
SeB avatar

I was hoping to improve the doc about those but did not find the time yet.

SeB avatar

Do you have any idea to solve this, if it is even possible.

Andrey Nazarov avatar
Andrey Nazarov

During the aforementioned conversation I thought about the following. I was wondering if something like this is possible with helmfile (forgive me for the syntax I don’t know how to show embedded part): {{ .Environment.Values.{{ .Release.Name }}.version }}?

Is there any chance to have that sort of templating? Or it just makes no sense?

SeB avatar

I don’t understand why when I do this the rendering fails

releases:
- name: tpsvc-config
  chart: {{ `{{ .Environment.Values | get (printf "%sVersion" "tpsvc-config") "" | eq "" | ternary "../.." "talend" }}` }}/{{`{{.Release.Name}}`}}
SeB avatar

with an error err: failed to read helmfile.config.yaml: reading document at index 1: yaml: line 15: did not find expected key

SeB avatar

but when I do this

releases:
- name: tpsvc-config
  chart: ../{{`{{.Release.Name}}`}}

This renders properly‹

Andrey Nazarov avatar
Andrey Nazarov

What if you try without get? The same pipeline but with .Environment.Values.tpsvc-configVersion? Sorry for interfering again, I just find your approach useful for one of my use cases.

2019-06-07

Erik Forsberg avatar
Erik Forsberg

Hi! The example helmfile refers to a helm-git plugin. I can only find a helm-git plugin that takes github/gitlab URLs, not git+https - so a link to this plugin would be appreciated.

Andrey Nazarov avatar
Andrey Nazarov
aslafy-z/helm-git

Helm plugin to fetch charts from Git repositories. Contribute to aslafy-z/helm-git development by creating an account on GitHub.

:--1:1
Andrey Nazarov avatar
Andrey Nazarov

It should be bundled to the most recent helmfile’s Docker image.

Erik Forsberg avatar
Erik Forsberg

Another question: The docs says I need to disable concurrency if I run with tillerless. Is there a way to do this from configuration file, (i.e. the helmfile.yaml), or only from commandline?

Andrey Nazarov avatar
Andrey Nazarov

My two cents is that helmDefaults are all about helm calls, so these values are passed to helm command or to helm tiller run. But concurrency is the helmfile’s property.

Erik Forsberg avatar
Erik Forsberg

Makes sense.

Andrey Nazarov avatar
Andrey Nazarov

A bit more about concurrency. It would be great to have --concurrency flag defined in docs and in help output of helmfile.

Erik Forsberg avatar
Erik Forsberg

It is documented in the output of e.g. helmfile sync --help. Not that it’s super-easy to find..

Andrey Nazarov avatar
Andrey Nazarov

Oh, yes, so it’s supposed to be a non-global flag, even though almost every command uses it.

2019-06-06

Andrey Nazarov avatar
Andrey Nazarov

Does helmfile has some sort of global timeout somewhere?

I’ve got “ERROR: Job failed: execution took longer than 1h0m0s seconds”. As for helm operations the timeout is set to 1200 sec via helmDefaults. The execution of tillerless approach is much longer especially when we have failing releases.

mumoshu avatar
mumoshu

What do you mean by global - would it be the timeout per the whole helmfile run?

Andrey Nazarov avatar
Andrey Nazarov

Yes.

mumoshu avatar
mumoshu
timeout 1s helmfile -f helmfile.remote.yaml diff; status=$?; if [ $status -eq 124 ]; then echo "timed out after 1s"; fi

mind writing a bash snippet like this?

mumoshu avatar
mumoshu

// so the answer was “no” - helmfile doesn’t have the feature yet

Andrey Nazarov avatar
Andrey Nazarov

But what about this error then? “ERROR: Job failed: execution took longer than 1h0m0s seconds”

Andrey Nazarov avatar
Andrey Nazarov

Oh, sorry

mumoshu avatar
mumoshu

gitlab?

Andrey Nazarov avatar
Andrey Nazarov

I know where it is from))

:--1:1
Andrey Nazarov avatar
Andrey Nazarov

Yes, you are right. I just forgot to tune it. Sorry for the noise.

mumoshu avatar
mumoshu

glad it was solved!

1

2019-06-04

mumoshu avatar
mumoshu

The “remote helmfiles” feature is coming! I’d appreciate it if you could test it out: https://github.com/roboll/helmfile/pull/648

:--1:1
mumoshu avatar
mumoshu

especially @Erik Osterman

2019-06-03

Andrey Nazarov avatar
Andrey Nazarov

Is it expected that helmfile ignores kubeContext if it doesn’t exist? I just noticed that behaviour, haven’t doublechecked it though.

mumoshu avatar
mumoshu

not expected. helmfile just passes it to helm commands so I’d expect helm should fail if the kubeContext you’ve provided and then passed by helmfile doesn’t exist!

Andrey Nazarov avatar
Andrey Nazarov

I’m using helm tiller plugin, btw. Well, I’ll check it out again and create an issue.

Andrey Nazarov avatar
Andrey Nazarov
Wrong kubeContext is ignored · Issue #642 · roboll/helmfile

I&#39;ve noticed that if I set kubeContext to the wrong value (i.e. the value that doesn&#39;t match any existing kube contexts) it will get silently ignored and the existing one will be used. I&#3…

mumoshu avatar
mumoshu

Thanks for reporting! I’ll take a look soon

Andrey Nazarov avatar
Andrey Nazarov

Can variables taken from environments like {{ .Environment.Values.imageTag }} be rendered from helmfile.yaml? For some reason it always shows me <no value>. Am I missing something? From the docs it should work, but…

Andrey Nazarov avatar
Andrey Nazarov

However {{ .Environment.Name }} is rendered correctly.

Andrey Nazarov avatar
Andrey Nazarov

It seems it doesn’t work when I have bases: defined. Probably I’ve read about that in some thread or issue….

mumoshu avatar
mumoshu

Would you mind sharing the full example?

I suppose it’s very hard to understand the behaviour due to the chicken-and-egg problems that is fundamental to it.

mumoshu avatar
mumoshu

You should always separate environments: and bases: and other parts of your helmfile.yaml with --- if you template so much

mumoshu avatar
mumoshu

like

environments:
  default:
    values: [env.values.yaml]

---


\# some template to generate state according to the env defined in the prev part...
{{ if .Environment.Values.foo ..... }}
mumoshu avatar
mumoshu

then the behaviour should be pretty straight-forward to understand

Andrey Nazarov avatar
Andrey Nazarov

I’ll try to do my best explaining our use case. So, we’ve got several stages and we deploying some workloads in per customer manner. So we need staging and prod to be equal, but development environment is a bit different. And we also have a bunch of shared namespaces. Due to that requirements our current helmfile is huge, a lot of duplications and stuff. It’s just tricky to maintain. And it would be great if we can uninstall a particular client’s app from the environment. So, it’s time to refactor it. Well, I can share a snippet, feel free to ask more. Here is the first attempt to refactor our huge helmfile:

hemlfile.yaml

repositories:

\#snip


\# we've tried inlines and files, both work ok.
environments:
  dev:
    values:
    - environment/dev.yaml
  staging:
    values:
    - environment/staging.yaml
  prod:
    values:
    - imageTag: release-1
      instances:
        customer_1: down
        costumer_2: up

helmDefaults:           
  tillerNamespace: my-tiller
  kubeContext: some-context
  verify: false
  wait: true
  timeout: 1200
  recreatePods: false
  force: true


\# this leads to the error, we want some common code be excluded for a better readability

\#bases:

\#  - ingress.yaml

\#  - cert-manager.yaml

\#  - monitoring.yaml

releases:

\# releases in shared namespaces, split by environments
  - name: shared-{{ .Environment.Name }}-elastic
    namespace: shared-{{ .Environment.Name }}
    ...


\# a bunch of shared releases here

  {{- if eq .Environment.Name "dev" }}
  # apps in dev environment
  - name: app1-{{ .Environment.Name }}
    namespace: {{ .Environment.Name }}
    chart: chartmuseum/app1
    version: 1.0.0
    values:
      - imageTag: {{ .Environment.Values.imageTag }}
    ...

  # a bunch of apps here

  {{- else }}
  {{- $dot := . }}
  {{- range $instance, $active := .Environment.Values.instances }}
  - name: app1-{{ $instance }}-{{ $dot.Environment.Name }}
    namespace: {{ $instance }}-{{ $dot.Environment.Name }}
    chart: chartmuseum/app1
    version: 1.0.0
    values:
      - imageTag: {{ $dot.Environment.Values.imageTag }}
    ...
   {{- if eq $active "up" }}
   installed: true
   {{- else }}
   installed: false
   {{- end }}


\# a bunch of apps here

  {{- end }} # range
  {{- end }} # if
Andrey Nazarov avatar
Andrey Nazarov

There could be some issues with spaces in the snippet above, don’t pay attention to them since I wrote it on the fly.

Andrey Nazarov avatar
Andrey Nazarov

There is no templaiting in ingress.yaml, cert-manager.yaml and monitoring.yaml. They are plain yamls.

Andrey Nazarov avatar
Andrey Nazarov

Ah, probably I misunderstood the bases: feature.

Andrey Nazarov avatar
Andrey Nazarov

But no. Here is the snippet from docs. It is possible to merge releases. helmfile.yaml

bases:
- commons.yaml
- environments.yaml

releases:
- name: myapp
  chart: mychart

commons.yaml

releases:
- name: metricbaet
  chart: stable/metricbeat
mumoshu avatar
mumoshu

Ah, the doc seems to be outdated

mumoshu avatar
mumoshu

helmfile does’t “append” arrays on merging bases now, because it broke doube rendering by duplicating releases

mumoshu avatar
mumoshu
temlates:
  metricbeat: &metricbeat
  - name: metricbeat

being explicit about how to concat releases, by including this and using yaml anchor should be the only feasible way today

{{ readFile "templates.yaml" }}

releases:
- <<: *metricbeat
Andrey Nazarov avatar
Andrey Nazarov

Ok, I’ll try it out. Probably I’ll just put the content of the bases to the main helmfile. We’ve already cut off a lot.

Thank you for the great work, I think I won’t get tired to repeat this).

:--1:1
mumoshu avatar
mumoshu

Thanks as always for your feedback!

mumoshu avatar
mumoshu

FWIW, I’ve updated the doc to explicitly state that helmfile doesn’t merge arrays

https://github.com/roboll/helmfile/blob/master/docs/writing-helmfile.md#merging-arrays-in-layers

roboll/helmfile

Deploy Kubernetes Helm Charts. Contribute to roboll/helmfile development by creating an account on GitHub.

mumoshu avatar
mumoshu

Please feel free to bring up advanced use-cases like yours so that I can probably enhance the doc even more

Andrey Nazarov avatar
Andrey Nazarov

Alright, when I put all the content of bases to the main helmfile.yaml everything works quite OK. My next thoughts are about moving the environments: block to the dedicated yaml and call it from the main file via bases:. This https://github.com/roboll/helmfile/blob/master/docs/writing-helmfile.md#layering-state-template-files shows the example, but helmfile.yaml.gotml is mentioned there, not the plain yaml. And I cannot find anywhere what I should do with this helmfile.yaml.gotml to render it. If I pass it to helmfile the error is the same - it just cannot find variables:

err: error during helmfile.yaml.part.0 parsing: template: stringTemplate:152:28: executing "stringTemplate" at <.Environment.Values.app1.chartVersion>: map has no entry for key "app1"
roboll/helmfile

Deploy Kubernetes Helm Charts. Contribute to roboll/helmfile development by creating an account on GitHub.

mumoshu avatar
mumoshu

base.yaml:

environments:
  default:
    values:
    - foo: BASE_DEFAULT_ENV_VALUE

helmfile.yaml.gotmpl:

bases:
- base.yaml
---
releases:
  - name: test1
    chart: mychart-{{ .Environment.Values.foo }}

This actually works on my machine, by evaluating {{ .Environment.Values.foo }} to BASE_DEFAULT_ENV_VALUE.

Would there be any interesting logs if you run helmfile with helmfile --log-level=debug ?

mumoshu avatar
mumoshu

and helmfile works the same way regardless of the ext is yaml or gotmpl, as of today. I’ll change it to require .gotmpl in helmfile v1 to enable state templates though.

Andrey Nazarov avatar
Andrey Nazarov

Hey, it seems I’m missing ---, what a shame. I’ll check this out.

:--1:1
Andrey Nazarov avatar
Andrey Nazarov

Yes it works, cool.

1
mumoshu avatar
mumoshu

Great!

    keyboard_arrow_up