#helmfile (2019-06)
Questions and discussion around helmfile https://github.com/roboll/helmfile and https://github.com/cloudposse/helmfiles
Archive: https://archive.sweetops.com/helmfile/
2019-06-03
Is it expected that helmfile ignores kubeContext if it doesn’t exist? I just noticed that behaviour, haven’t doublechecked it though.
not expected. helmfile just passes it to helm commands so I’d expect helm should fail if the kubeContext you’ve provided and then passed by helmfile doesn’t exist!
I’m using helm tiller plugin, btw. Well, I’ll check it out again and create an issue.
Thanks for reporting! I’ll take a look soon
Can variables taken from environments like {{ .Environment.Values.imageTag }} be rendered from helmfile.yaml? For some reason it always shows me <no value>. Am I missing something? From the docs it should work, but…
However {{ .Environment.Name }} is rendered correctly.
It seems it doesn’t work when I have bases: defined. Probably I’ve read about that in some thread or issue….
Would you mind sharing the full example?
I suppose it’s very hard to understand the behaviour due to the chicken-and-egg problems that is fundamental to it.
You should always separate environments: and bases: and other parts of your helmfile.yaml with --- if you template so much
like
environments:
default:
values: [env.values.yaml]
---
# some template to generate state according to the env defined in the prev part...
{{ if .Environment.Values.foo ..... }}
then the behaviour should be pretty straight-forward to understand
I’ll try to do my best explaining our use case. So, we’ve got several stages and we deploying some workloads in per customer manner. So we need staging and prod to be equal, but development environment is a bit different. And we also have a bunch of shared namespaces. Due to that requirements our current helmfile is huge, a lot of duplications and stuff. It’s just tricky to maintain. And it would be great if we can uninstall a particular client’s app from the environment. So, it’s time to refactor it. Well, I can share a snippet, feel free to ask more. Here is the first attempt to refactor our huge helmfile:
hemlfile.yaml
repositories:
#snip
# we've tried inlines and files, both work ok.
environments:
dev:
values:
- environment/dev.yaml
staging:
values:
- environment/staging.yaml
prod:
values:
- imageTag: release-1
instances:
customer_1: down
costumer_2: up
helmDefaults:
tillerNamespace: my-tiller
kubeContext: some-context
verify: false
wait: true
timeout: 1200
recreatePods: false
force: true
# this leads to the error, we want some common code be excluded for a better readability
#bases:
# - ingress.yaml
# - cert-manager.yaml
# - monitoring.yaml
releases:
# releases in shared namespaces, split by environments
- name: shared-{{ .Environment.Name }}-elastic
namespace: shared-{{ .Environment.Name }}
...
# a bunch of shared releases here
{{- if eq .Environment.Name "dev" }}
# apps in dev environment
- name: app1-{{ .Environment.Name }}
namespace: {{ .Environment.Name }}
chart: chartmuseum/app1
version: 1.0.0
values:
- imageTag: {{ .Environment.Values.imageTag }}
...
# a bunch of apps here
{{- else }}
{{- $dot := . }}
{{- range $instance, $active := .Environment.Values.instances }}
- name: app1-{{ $instance }}-{{ $dot.Environment.Name }}
namespace: {{ $instance }}-{{ $dot.Environment.Name }}
chart: chartmuseum/app1
version: 1.0.0
values:
- imageTag: {{ $dot.Environment.Values.imageTag }}
...
{{- if eq $active "up" }}
installed: true
{{- else }}
installed: false
{{- end }}
# a bunch of apps here
{{- end }} # range
{{- end }} # if
There could be some issues with spaces in the snippet above, don’t pay attention to them since I wrote it on the fly.
There is no templaiting in ingress.yaml, cert-manager.yaml and monitoring.yaml. They are plain yamls.
Ah, probably I misunderstood the bases: feature.
But no. Here is the snippet from docs. It is possible to merge releases. helmfile.yaml
bases:
- commons.yaml
- environments.yaml
releases:
- name: myapp
chart: mychart
commons.yaml
releases:
- name: metricbaet
chart: stable/metricbeat
Ah, the doc seems to be outdated
helmfile does’t “append” arrays on merging bases now, because it broke doube rendering by duplicating releases
temlates:
metricbeat: &metricbeat
- name: metricbeat
being explicit about how to concat releases, by including this and using yaml anchor should be the only feasible way today
{{ readFile "templates.yaml" }}
releases:
- <<: *metricbeat
Ok, I’ll try it out. Probably I’ll just put the content of the bases to the main helmfile. We’ve already cut off a lot.
Thank you for the great work, I think I won’t get tired to repeat this).
Thanks as always for your feedback!
FWIW, I’ve updated the doc to explicitly state that helmfile doesn’t merge arrays
https://github.com/roboll/helmfile/blob/master/docs/writing-helmfile.md#merging-arrays-in-layers
Please feel free to bring up advanced use-cases like yours so that I can probably enhance the doc even more
Alright, when I put all the content of bases to the main helmfile.yaml everything works quite OK. My next thoughts are about moving the environments: block to the dedicated yaml and call it from the main file via bases:. This https://github.com/roboll/helmfile/blob/master/docs/writing-helmfile.md#layering-state-template-files shows the example, but helmfile.yaml.gotml is mentioned there, not the plain yaml. And I cannot find anywhere what I should do with this helmfile.yaml.gotml to render it. If I pass it to helmfile the error is the same - it just cannot find variables:
err: error during helmfile.yaml.part.0 parsing: template: stringTemplate:152:28: executing "stringTemplate" at <.Environment.Values.app1.chartVersion>: map has no entry for key "app1"
base.yaml:
environments:
default:
values:
- foo: BASE_DEFAULT_ENV_VALUE
helmfile.yaml.gotmpl:
bases:
- base.yaml
---
releases:
- name: test1
chart: mychart-{{ .Environment.Values.foo }}
This actually works on my machine, by evaluating {{ .Environment.Values.foo }} to BASE_DEFAULT_ENV_VALUE.
Would there be any interesting logs if you run helmfile with helmfile --log-level=debug ?
and helmfile works the same way regardless of the ext is yaml or gotmpl, as of today. I’ll change it to require .gotmpl in helmfile v1 to enable state templates though.
Great!
2019-06-04
The “remote helmfiles” feature is coming! I’d appreciate it if you could test it out: https://github.com/roboll/helmfile/pull/648
especially @Erik Osterman (Cloud Posse)
2019-06-05
2019-06-06
Does helmfile has some sort of global timeout somewhere?
I’ve got “ERROR: Job failed: execution took longer than 1h0m0s seconds”. As for helm operations the timeout is set to 1200 sec via helmDefaults. The execution of tillerless approach is much longer especially when we have failing releases.
What do you mean by global - would it be the timeout per the whole helmfile run?
Yes.
timeout 1s helmfile -f helmfile.remote.yaml diff; status=$?; if [ $status -eq 124 ]; then echo "timed out after 1s"; fi
mind writing a bash snippet like this?
// so the answer was “no” - helmfile doesn’t have the feature yet
But what about this error then? “ERROR: Job failed: execution took longer than 1h0m0s seconds”
Oh, sorry
gitlab?
Yes, you are right. I just forgot to tune it. Sorry for the noise.
2019-06-07
Hi! The example helmfile refers to a helm-git plugin. I can only find a helm-git plugin that takes github/gitlab URLs, not git+https - so a link to this plugin would be appreciated.
It should be bundled to the most recent helmfile’s Docker image.
Another question: The docs says I need to disable concurrency if I run with tillerless. Is there a way to do this from configuration file, (i.e. the helmfile.yaml), or only from commandline?
My two cents is that helmDefaults are all about helm calls, so these values are passed to helm command or to helm tiller run. But concurrency is the helmfile’s property.
Makes sense.
A bit more about concurrency. It would be great to have --concurrency flag defined in docs and in help output of helmfile.
It is documented in the output of e.g. helmfile sync --help. Not that it’s super-easy to find..
Oh, yes, so it’s supposed to be a non-global flag, even though almost every command uses it.
2019-06-09
Hi guys,
I have a template issue, I’d like to have in a release the “chart: ” value to be computed according to the version set in environment so that when :
• the version has some value like “0.1.3” the chart definition is chart: myregistry/mychart
• the version is empty or not set the chart def is a local chart chart: ../mychart
I thought that the following would work
environments:
dev:
values:
- tpsvc-configVersion: ""
prod:
values:
- tpsvc-configVersion: 1.0
templates:
chart: &chart
{{- $versionKey := printf "%sVersion" `{{.Release.Name}}` }}
chart: {{ .Environment.Values | get ($versionKey) "" | eq "" | ternary "../.." "talend" }}/{{`{{.Release.Name}}`}}
version: {{.Environment.Values | get ($versionKey) ""}}
releases:
- name: tpsvc-config
<<: *chart
But it does not
Out of curioucity, are you sure get and ternary are supported? I’ve never used them and cannot find them mentioned anywhere.
And what so you mean by “it doesn’t work”?
I was wondering if something like
{{ .Environment.Values.`{{ .Release.Name }}`.version }}
works? Never checked it out. Then it could be as simple as {{ with }} and {{ if/else }} blocks.
yes get and ternary are supported.
the issues comes from the fact that the templates are rendered before the release is rendered.
I have tried to escape the template to have the rendered late w
but I have a error.
yes get and ternary are supported.
Yes, I’ve just found it in Sprig.
When I say it does not work I mean I always get the local chart ../../tpsvc-config
it never takes the version into account.
basically the versionKey is rendered to early and is probably set with the value “{{.ReleaseName}}Version” which of course is never found in the value and this explains why I always get the ../../tpsvc-config
I was wondering if something like ... works?
No, this won’t work for sure as well. At least not with that syntax.
But it would be cool to have some sort of embedment…
Another possible solution could be range by releases. Something like:
environments:
dev:
values:
- versions:
tpsvc-config: ""
something-else: 1.0
prod:
values:
- versions:
tpsvc-config: 1.0
something-else: 2.0
releases:
{{- range $name, $version := .Environment.Values.versions }}
- name: {{ $name }}
{{- if eq $version "" }}
chart: ../{{ $name }}
{{- else }}
chart: myregistry/{{ $name }}
{{- end }}
version: {{ $version }}
{{- end }}
basically the versionKey is rendered to early and is probably set with the value
Yep, probably that’s the case.
Does your snippet work out of release template? I.e. being defined in releases block?
it does not work either in the release block
I think your solution with the range should work but this is not very easy to read.
Yeah, the readability is not perfect for sure.
it does not work either in the release block
that’s a bit suspicious. But you removed backticks, right?What does helm template show?
Btw, could you please point me out to the get definition. I was too fast saying I found it in sprig. I’ve just seen ternary.
this is one of the undocumented function provided by helmfile
I was hoping to improve the doc about those but did not find the time yet.
Do you have any idea to solve this, if it is even possible.
During the aforementioned conversation I thought about the following.
I was wondering if something like this is possible with helmfile (forgive me for the syntax I don’t know how to show embedded part):
{{ .Environment.Values.{{ .Release.Name }}.version }}?
Is there any chance to have that sort of templating? Or it just makes no sense?
I don’t understand why when I do this the rendering fails
releases:
- name: tpsvc-config
chart: {{ `{{ .Environment.Values | get (printf "%sVersion" "tpsvc-config") "" | eq "" | ternary "../.." "talend" }}` }}/{{`{{.Release.Name}}`}}
with an error err: failed to read helmfile.config.yaml: reading document at index 1: yaml: line 15: did not find expected key
but when I do this
releases:
- name: tpsvc-config
chart: ../{{`{{.Release.Name}}`}}
This renders properly‹
What if you try without get? The same pipeline but with .Environment.Values.tpsvc-configVersion? Sorry for interfering again, I just find your approach useful for one of my use cases.
2019-06-10
I’ll try but this should not be any different.
and also this will fail cause value resolution does not support a dash in the values
Not sure what’s causing your confusion exactly, but try:
releases:
- name: tpsvc-config
chart: |
{{`{{ .Environment.Values | get (printf "%sVersion" "tpsvc-config") "" | eq "" | ternary "../.." "talend" }}` }}/{{`{{.Release.Name}}`}}
Hi @mumoshu thanks a lot ! this is working but I have another issue though. The version of a release seem not to accept templatization
releases:
- name: tpsvc-config
chart: |
{{ `{{ .Environment.Values | get (printf "%sVersion" "tpsvc-config") "" | eq "" | ternary "../.." "talend" }}` }}/{{`{{.Release.Name}}`}}
version: |
{{` {{ .Environment.Values | get (printf "%sVersion" "tpsvc-config") "" }}` }}
it is failing with :
Error: chart "tpsvc-config" matching {{ .Environment.Values | get (printf "%sVersion" "tpsvc-config") "" }} not found in talend index. (try 'helm repo update'). improper constraint: {{ .Environment.Values | get (printf "%sVersion" "tpsvc-config") "" }}
Ah, that sounds like not implemented!
Got it, nevermind the question. I’ve deleted it.
// hold on..
yep
it seems that the version is missing here
You found it!
Do you want me to create an issue ?
Please do so! It’s even nicer if you directly create a PR to add it
| thank anyway for the “ | ” tip, yaml is always a mystery for me. |
| @mumoshu can you explain why the | is required for this scalar, why the initial string does not work. is it because the the ” inside the string ? |
I have made it work using the simple quote to surround the string like
chart: '{{ `{{ .Environment.Values | get (printf "%sVersion" "tpsvc-config") "" | eq "" | ternary "../.." "talend" }}`'
it fails using the double quote even with the inside double quote escaped ?
@SeB That’s basically due to that {{ whatever }}//{{ whatever }} isn’t a valid yaml string
chart: {{` {{ .Environment.Values | get (printf "%sVersion" "tpsvc-config") "" | eq "" | ternary "../.." "talend" }} ``}
evaluates to
chart: {{ .Environment.Values | get (printf "%sVersion" "tpsvc-config") "" | eq "" | ternary "../.." "talend" }}
which looks chart: {{ whatever }}//{{ whatever }} for the yaml parser
chart: '{{ `{{ .Environment.Values | get (printf "%sVersion" "tpsvc-config") "" | eq "" | ternary "../.." "talend" }}`}}'
this works cuz it evaluates to:
chart: '{{ .Environment.Values | get (printf "%sVersion" "tpsvc-config") "" | eq "" | ternary "../.." "talend" }}'
which looks chart: 'str' to the yaml parser
ok thanks a lot for the explanation
chart: "{{` {{ .Environment.Values | get (printf "%sVersion" "tpsvc-config") "" | eq "" | ternary "../.." "talend" }} `}}"
and this is close but doesn’t work in your case as you have " in release tempaltes
it evaluates to
chart: "{{ .Environment.Values | get (printf "%sVersion" "tpsvc-config") "" | eq "" | ternary "../.." "talend" }}"
and it looks chart: "{{ .Enviro...ersion" "tpsvc-config") "" | eq "" ..... to the yaml parser
) and ` | eq ` should look like stray terms to the yaml parser |
and probably chart: "{{ .Enviro...ersion" "tpsvc-config" is chart: "str1" "str2" which isn’t valid as well
my understanding may be incomplete but that’s probably it :slightly_smiling_face:
: | should always work
Probably it would be useful to mention this trick somewhere in the docs. It’s not that obvious at first.
2019-06-12
Howdy. Not sure if this question has been asked before… but how do you get selectors when specifying an environment to work? I get an error if I use something like:
helmfiles:
- path: environments/coreserv/production/helmfile.yaml
---
releases:
- name: mycoolrelease
labels:
team: coreserv
---
`helmfile -e production -l team=coreserv diff`
Hey! What are you using the production environment for? Can you omit it if you can live without it?
I’d like to see the actual error message you got, but probably it’s due to that you’re missing the production environment… in under environments section and helmfile is complaining about the missing env, right?
I do have a production environment defined, like so:
environments:
default:
values:
- environments/default/values.yaml
development:
values:
- environments/{{ env "TEAM_NAME" | default "coreserv" }}/development/values.yaml
production:
values:
- environments/{{ env "TEAM_NAME" | default "coreserv" }}/production/values.yaml
Thanks!
Would you mind sharing the full log of your helmfile execution with debug logs helmfile --debug log-level?
Sure. I’ll post it tomorrow. Not at the office right now.
If I don’t use an environment it is able to find the helmfile releases defined there.
@Justice London Just replied in the thread
2019-06-13
Hello @mumoshu Is it on purpose that there is no documentation on the new state value introduced here : https://github.com/roboll/helmfile/pull/647
Yes. I wanted to confirm that it’s working as you might have expected first
https://github.com/roboll/helmfile/pull/647#issuecomment-498558304
before writing any doc
I do confirm this works as expected.
Yo fellow Helmfile freaks
Hey! 
Is it possible to get helmfile template to only spit out YAML within the executable? Right now I’m just nuking all the lines before the first --- match but I’m not familiar enough with helmfile yet to know that this will resolve all cases of non-templated output
2019-06-14
Yeah! I thought I’ve changed all the logs to go stderr and made stdout dedicated to actual template output.
But that’s not the case, right? If you could comment on the relevant github issue about redundant lines you saw, it would help me fixing it
Tacked it onto https://github.com/roboll/helmfile/issues/685
Im using terraform to build the cluster and would like to use helmfile for configuring all the post-installation tasks.
2019-06-15
2019-06-17
hey guys, just want to know if it’s possible to use reuse-values in helmfile?
helmfile supports YAML anchors
might that help?
we also use environment variables to do this
And helmfile has never intended to support helm’s --reuset-values because it defeats the one of purposes of helmfile - reproducible deployments!
But I may be missing something - what’s your goal for –reuset-values @Alexis?
it is because we don’t save the actual image tags in git. we don’t want to reset the versions for other releases
How do you set the chart value for a image tag then?
I always pass it using –set when using helm. In helmfile, I plan on using environment variables.
hmm im pretty confused. do you really rely on --reset-values when you use helm?
sounds like you can just place something like tag: {{ requiredEnv" IMAGE_TAG"}} in your helmfile.yaml or even helmfile --state-values-set image.tag=${IMAGE_TAG}
without –reset-values
We now have a Kubernetes operator for helmfile
https://github.com/roboll/helmfile/issues/153 https://github.com/mumoshu/helmfile-operator
I’m trying to build something that complements weaveworks/flux, argo-cd, brigadecore/brigade…
Amazing!
excited to take a look
Ask me anything! I’m not yet sure how to best describe it myself
one of limitations of flux, argo-cd is that it’s inability to run helmfile apply on git commit.. it works by sourcing k8s manfiests generated by helmfile template(both of em allows you to run helmfile template automatically on git commit btw)
normally it makes many useful helm commands unusable. helm test, helm status, helm list, and so on.
with helmfile-operator you can manage Helmfile custom resources in git, so that argo-cd or flux can reconcile helmfile crs on git commit, and then the operator runs helmfile commands
brigade is basically a ci/cd system that takes JavaScript code as a pipeline definition. so just run helmfile apply from js that is triggered via git commit
or create Helmfile custom resources from the pipeline and the rest is the same as argo-cd and flux.
Man, I’ve been waiting for something like helmfile operator. Even thought about trying to write it by myself, but - it’s here. I’ll definitely take a look.
1
Really great news!
how are errors handled? e.g. a bad helmfile rollout?
currently it just fail leaving logs in the helmfile-applier pod. as well as diff i’m considering to use slack or github issue/pr comments to surface error messages for human operators. wdyt?
Aha, yes, slack would be nice
i imagine helmfile diff is not really suitable for this workflow?
good point!
for that i’m planning to add a feature that uses slack or github issue comments to show changes and manual approval
What I find tricky in Argo and Flux is how they handle multi-cluster setup. I think it should be easier to do with helmfile-operator.
Yeah probably. One way would be to have a multi-cluster helmfile.yaml that has different kubeContext set per release. Each release can be regular charts, or even charts for Helmfile resource…
// Just recalled that I was in the process of renaming the project from appliance-operator to helmfile-operator.. Hold on
aha, that is more clear to me
2019-06-18
Is there any variable I can use from within a hook to anchor my paths to the directory of the helmfile?
The use case is that I’m modifying a namespace via an kubectl apply -f <file>, and right now I must be in the same directory as the helmfile, i.e I can’t use –file to point at the helmfile, or the <file> won’t be found. So if there is a variable, and hook arguments are templated, then.. I could.
Interesting! I thought I’ve designed helmfile to execute any command including helm, exec, hooks from within the directory that the helmfile resides, regardless of which directory you are currently at. But what you observed doesn’t match it, right?
Haha, I didn’t try running it from a different directory, I just assumed that it wouldn’t work, and have only tried with cwd being correct. I’ll check if it works
Does helmfile ever get locked out? I’ve been trying to install a couple of deploys, a service, ingress, service-account, cluster-role-binding, and configmap, and I started getting the following error (after many helmfile applys). However, I switched clusters, and stopped getting the error…
$ helmfile apply --args "--dry-run"
Building dependency ../my-appV2
No requirements found in ../my-appV2/charts.
Building dependency ../my-appV2
No requirements found in ../my-appV2/charts.
Building dependency ../my-appV2
No requirements found in ../my-appV2/charts.
Building dependency ../my-appV2
No requirements found in ../my-appV2/charts.
Building dependency ../my-appV2
No requirements found in ../my-appV2/charts.
Building dependency ../my-appV2
No requirements found in ../my-appV2/charts.
Building dependency ../my-appV2
No requirements found in ../my-appV2/charts.
Comparing sean-configmap ../my-appV2
Comparing sean-deploy-green ../my-appV2
Comparing sean-service-account ../my-appV2
Comparing sean-ingress ../my-appV2
Comparing sean-service ../my-appV2
Comparing sean-deploy-blue ../my-appV2
Comparing sean-namespace ../my-appV2
in ./helmfile.yaml: 7 errors:
err 0: failed processing release sean-ingress: helm exited with status 1:
Error: "sean-ingress" has no deployed releases
Error: plugin "diff" exited with error
err 1: failed processing release sean-service-account: helm exited with status 1:
Error: "sean-service-account" has no deployed releases
Error: plugin "diff" exited with error
err 2: failed processing release sean-deploy-blue: helm exited with status 1:
Error: "sean-deploy-blue" has no deployed releases
Error: plugin "diff" exited with error
err 3: failed processing release sean-configmap: helm exited with status 1:
Error: "sean-configmap" has no deployed releases
Error: plugin "diff" exited with error
err 4: failed processing release sean-service: helm exited with status 1:
Error: "sean-service" has no deployed releases
Error: plugin "diff" exited with error
err 5: failed processing release sean-namespace: helm exited with status 1:
Error: "sean-namespace" has no deployed releases
Error: plugin "diff" exited with error
err 6: failed processing release sean-deploy-green: helm exited with status 1:
Error: "sean-deploy-green" has no deployed releases
Error: plugin "diff" exited with error
I noticed these twistlock notifications new host runtime alterts (I’ve got twistlock defenders deployed on the same cluster):
Service google-ip-forwarding-daemon attempted to obtain capability NET by executing ip route add to local $IP/32 scope host dev eth0 proto 66 (parent: /usr/bin/python2.7). Low severity audit, event is automatically added to the runtime model
Figured it out. Needed to helm del --purge $RELEASE_NAME for each release. Don’t know why this happened.
Is there a way to run a custom script everytime a helmfile command is run? One usecase is that when I run a helmfile -e ENV … I want to run a script to check my KUBECONFIG is pointing to the right cluster for the environment. I can do it as a release hook, but the release hook declaration has to be duplicated for every release. It’d be nice if I can define a hook globally.
Helmfile doesn’t cover that usecase alone. As helmfile isn’t a generic task runner I’m not yet inclined to enhance helmfile for that.
Shameless plug but I built/use #variant for wrapping helmfile so that I can add any pre/post helmfile script without affecting the overall ux much!
hi @mumoshu can I add repositories.yaml file to bases? it contains repositories spec
yeah it should work
add repositories causes error
err: no releases found that matches specified selector() and environment(local), in any helmfile
my bases:
bases:
- ./base/repositories.yaml
- ./base/environments.yaml
could you try moving base dir out of helmfile.d?
yes, I have tried that too
hmm, could you add --log-level=debug like helmfile --log-level=debug to hopefully see at which point helmfile is failing?
processing file "0-system-service.yaml" in directory "helmfile.d"
changing working directory to "/home/hh/deployments/helmfile.d"
first-pass rendering starting for "0-system-service.yaml.part.0": inherited=&{local map[] map[]}, overrode=<nil>first-pass uses: &{local map[] map[]}
error in first-pass rendering: result of "0-system-service.yaml.part.0":
0: #
1:
2: bases:
3: # - ./base/environments.yaml
4: - ./base/repositories.yaml
5: - ./base/environments.yaml
6:
7:
8: releases:
9: - name: sb-redis
10: chart: stable/redis
11: labels:
12: test: true
13:
first-pass produced: &{local map[] map[]}
first-pass rendering result of "0-system-service.yaml.part.0": {local map[] map[]}
second-pass rendering result of "0-system-service.yaml.part.0":
0: #
1:
2: bases:
3: # - ./base/environments.yaml
4: - ./base/repositories.yaml
5: - ./base/environments.yaml
6:
7:
8: releases:
9: - name: sb-redis
10: chart: stable/redis
11: labels:
12: test: true
13:
first-pass rendering starting for "base/repositories.yaml.part.0": inherited=&{local map[] map[]}, overrode=<nil>
first-pass uses: &{local map[] map[]}
first-pass produced: &{local map[] map[]}
first-pass rendering result of "base/repositories.yaml.part.0": {local map[] map[]}
vals:
map[]
defaultVals:[]
second-pass rendering result of "base/repositories.yaml.part.0":
0: repositories:
1: - name: stable
2: url: <https://kubernetes-charts.storage.googleapis.com/>
3: - name: incubator
4: url: <https://kubernetes-charts-incubator.storage.googleapis.com/>
changing working directory back to "/home/hh/deployments"
err: no releases found that matches specified selector() and environment(local), in any helmfile
so you seem to still have base under helmfile.d
oh yes I forgot to move, but when I move I still have the error
if you move it out of helmfile.d do you see another errors?
hold on
still error
which error do you see?
my bases:
bases:
- ../common/base/repositories.yaml
- ../common/base/environments.yaml
but if I remove repositories line, it works
mind sharing your logs with --log-level=debug once again?
processing file "0-system-service.yaml" in directory "helmfile.d"
changing working directory to "/home/hh/test/deployments/helmfile.d"
first-pass rendering starting for "0-system-service.yaml.part.0": inherited=&{local map[] map[]}, overrode=<nil>first-pass uses: &{local map[] map[]}
error in first-pass rendering: result of "0-system-service.yaml.part.0":
0: #
1:
2: bases:
3: # - ./base/environments.yaml
4: - ../common/base/repositories.yaml
5: - ../common/base/environments.yaml
6:
7:
8: releases:
9: - name: sb-redis
10: chart: stable/redis
11: labels:
12: test: true
13:
first-pass produced: &{local map[] map[]}
first-pass rendering result of "0-system-service.yaml.part.0": {local map[] map[]}
second-pass rendering result of "0-system-service.yaml.part.0":
0: #
1:
2: bases:
3: # - ./base/environments.yaml
4: - ../common/base/repositories.yaml
5: - ../common/base/environments.yaml
6:
7:
8: releases:
9: - name: sb-redis
10: chart: stable/redis
11: labels:
12: test: true
13:
first-pass rendering starting for "../common/base/repositories.yaml.part.0": inherited=&{local map[] map[]}, overrode=<nil>
first-pass uses: &{local map[] map[]}
first-pass produced: &{local map[] map[]}
first-pass rendering result of "../common/base/repositories.yaml.part.0": {local map[] map[]}
vals:
map[]
defaultVals:[]
second-pass rendering result of "../common/base/repositories.yaml.part.0":
0: repositories:
1: - name: stable
2: url: <https://kubernetes-charts.storage.googleapis.com/>
3: - name: incubator
4: url: <https://kubernetes-charts-incubator.storage.googleapis.com/>
changing working directory back to "/home/hh/test/deployments"
err: no releases found that matches specified selector() and environment(local), in any helmfile
@Ribhararnus Pracutiar Thanks!
Would you mind sharing --log-level=debug logs with (1) - ./base/repositories.yaml commented out and (2) - ./base/repositories.yaml moved after ../common/base/environments.yaml as well?
Probably I can get what’s wrong with the info
I remove ./base/repositories, and move ../common/base/repositories under environments:
$ helmfile --environment local --log-level=debug lint
processing file "0-system-service.yaml" in directory "helmfile.d"
changing working directory to "/home/hh/test/deployments/helmfile.d"
first-pass rendering starting for "0-system-service.yaml.part.0": inherited=&{local map[] map[]}, overrode=<nil>first-pass uses: &{local map[] map[]}
error in first-pass rendering: result of "0-system-service.yaml.part.0":
0:
1:
2: bases:
3: - ../common/base/environments.yaml
4: - ../common/base/repositories.yaml
5:
6:
7: releases:
8: - name: sb-redis
9: chart: stable/redis
10: labels:
11: test: true
12:
first-pass produced: &{local map[] map[]}
first-pass rendering result of "0-system-service.yaml.part.0": {local map[] map[]}
second-pass rendering result of "0-system-service.yaml.part.0":
0:
1:
2: bases:
3: - ../common/base/environments.yaml
4: - ../common/base/repositories.yaml
5:
6:
7: releases:
8: - name: sb-redis
9: chart: stable/redis
10: labels:
11: test: true
12:
first-pass rendering starting for "../common/base/environments.yaml.part.0": inherited=&{local map[] map[]}, overrode=<nil>
first-pass uses: &{local map[] map[]}
envvals_loader: loaded ../common/environments/local.yaml:map[test:true]
first-pass produced: &{local map[test:true] map[]}
first-pass rendering result of "../common/base/environments.yaml.part.0": {local map[test:true] map[]}
vals:
map[test:true]
defaultVals:[]
second-pass rendering result of "../common/base/environments.yaml.part.0":
0: environments:
1: local:
2: values:
3: - ../common/environments/local.yaml
4: development:
5: values:
6: - ../common/environments/development.yaml
7: staging:
8: values:
9: - ../common/environments/staging.yaml
10: production:
11: values:
12: - ../common/environments/production.yaml
envvals_loader: loaded ../common/environments/local.yaml:map[test:true]
merged environment: &{local map[test:true] map[]}
first-pass rendering starting for "../common/base/repositories.yaml.part.0": inherited=&{local map[] map[]}, overrode=<nil>
first-pass uses: &{local map[] map[]}
first-pass produced: &{local map[] map[]}
first-pass rendering result of "../common/base/repositories.yaml.part.0": {local map[] map[]}
vals:
map[]
defaultVals:[]
second-pass rendering result of "../common/base/repositories.yaml.part.0":
0: repositories:
1: - name: stable
2: url: <https://kubernetes-charts.storage.googleapis.com/>
3: - name: incubator
4: url: <https://kubernetes-charts-incubator.storage.googleapis.com/>
changing working directory back to "/home/hh/test/deployments"
err: no releases found that matches specified selector() and environment(local), in any helmfile
actually my original helmfile is complicated and debugging for hours, I don’t know what’s happening, until I am surprised that the repositories base part causes the error a while ago
@Ribhararnus Pracutiar according to the logs your issue isn’t resolved yet, right? I’ll take a look and try to see what can go wrong tonight thanks for reporting!
No worries buddy. Right still not resolved, maybe I wanna use repositories in every file, violate DRY till the issue resolved
That helps my sleep! Thanks for your cooperation I’ll report back once I managed to fix it
2019-06-19
Hmm.. Does a non-default environment inherit values from the default environment? I.e, values not defined under “production”, will they inherit value defined under “default”?
Are you talking about {{.Environment.Values}} ?
Yes.
Nope
But {{.Values}} should. This is new, experimental one
# Probably this is what you want to call "defaults"
values:
- foo: bar
environments:
prod:
values:
- bar: baz
Now in helmfile -e prod you get bar for {{.Values.foo}}
Aha. Yes, that should solve my problem. Does that work with 0.79? It’s not documented in the example helmfile as far as I can see.
i’m awaiting several good feedbacks before documenting it(as it’s experimental :)
Right. I see it’s in 0.71 or something, so that’s cool. I’ll try it out!
yep it should work in 0.79
you should use v0.75.1 at minimum, which includes the fix for the latest bug in the new values feature
ok good!
@mumoshu just to let you know I had a good time contributing to the helmfile project but I will certainly not contribute anymore. We have decided to leverage the ansible scripts we are using for production deployment also for the developer workflow. I believe it is too big a machinery compared to helmfile but this is a pragmatical decision for now to not multiply our tools.
sad but understandable! thanks a lot for your contributions so far!
i’ll try to keep improving helmfile, hoping it would help you someday
How do I make helmfile stop if there’s trouble with one of the charts? I.e, I don’t want it to continue setting up chart B if chart A had any kind of trouble. Is that the wait parameter? I’m already running with concurrency 1 since I use tillerless.
seems like a good feature request
helmfile’s behaviour is tuned towards concurrency > 1 and in a concurrent usage stopping another run in a middle of the process just because the another failed seemed like a bad idea
but when concurrency is 1, an option like fastfast: true may make sense
in the mean time you can use helmfiles: so that each sub-helmfile under it is processed sequentially.
but probably that’s not what u want
Well, ideally, I would like to set some dependencies, so if chart A fails, chart B can’t continue but chart C can.
Btw, it should be possible to solve the concurrency issue with tillerless by running tiller from helmfile rather than by using the tillerless plugin, but that may not be worth it as helm 3 will be tillerless anyway.
I would love to see the concurrency issue solved for tillerless workflow. But yes, since Helm 3 is coming probably it doesn’t seem worth the effort. tillerless plugin has other drawbacks as well, btw.
I’m confused to to best use readFile with .gotmpl when pulling in a non-yaml file
I have this in a values .gotmpl file, and it works
extraConfigMaps:
output.conf: |-
<match **>
@id elasticsearch
@type elasticsearch
...
I’d like to use this, so I can pull out that large xml into it’s own file fluentd-elasticsearch.xml
extraConfigMaps:
output.conf: |-
{{ readFile "fluentd-elasticsearch.xml" | indent 2 }}
but I keep getting error converting YAML to JSON: yaml: line 35: could not find expected ':'
Hey! Try this:
extraConfigMaps:
output.conf: |-
{{ readFile "fluentd-elasticsearch.xml" | indent 6 }}
not sure why you have 4 spaces before output.conf but you can also try:
extraConfigMaps:
output.conf: |-
{{ readFile "fluentd-elasticsearch.xml" | indent 4 }}
extraConfigMaps:
output.conf: |-
{{ readFile "fluentd-elasticsearch.xml" | indent 2 }}
this won’t work cuz we get
extraConfigMaps:
output.conf: |-
<match **>
@id elasticsearch
@type elasticsearch
...
@mumoshu omg thang you - I feel very dumb, but that was it
glad to help!
Can I update the requirment version of a helm chart that I am using ?
So in this case I am using stable/prometheus-operator and would like to update the grafana version to 3.5 instead of 3.3
Yep. Do you have version already set for that chart in your helmfile.yaml?
If not, add stable to under repositories of your helmfile.yaml, then run helmfile apply or helmfile diff or whatever, so that it automatically fetches the latest version
so the requirments would see that I already have stable/grafana installed ?
what do you mean by the requirments exactly?
in case you have a helmfile.lock file generated by helmfile deps, the grafana chart version should be saved there, and it’s updated only when you run helmfile deps next time
hi @mumoshu is yesterday issue not resolved yet?
anyway, I want to ask question here regarding multi-line string
local.yaml:
config: |
test: a
yes: true
in helmfile chart, template/configmap.yaml
default.yaml: |
{{ tpl .Values.config . }}
Try
default.yaml: |
{{ tpl .Values.config . | indent 2 }}
This
default.yaml: |
{{ tpl .Values.config . }}
ends up(notice the indentation issue!
default.yaml: |
config: |
test: a
yes: true
I think it will parse to this:
yes: true
default.yaml: |
test: a
that’s weird
only first key value that correctly parse into default.yaml
ah yeah good catch
actual file:
data:
nodeEnv: {{ .Values.environment }}
default.yaml: |
{{ tpl .Values.config . | indent 2 }}
Thats because
default.yaml: |
{{ tpl .Values.config . }}
Here .Values.config is two-lines text:
test: a
yes: true
if I change indent to 4 it would give me error
That is rendered as
default.yaml: |
test: 1
yes: true
do you have solution for this kind of issue?
Yep. so try adding | indent 2
default.yaml: |
{{ tpl .Values.config . | indent 2 }}
without the first two spaces before {{ tpl ...
yes, still the configmap cannot parse the tpl
that sounds like you have another issue
are you using the same set of files you’ve pasted here in your actual setup?
here is my actual files:
template/configmap.yaml
data:
nodeEnv: {{ .Values.environment }}
default.yaml: |
{{ tpl .Values.config . | indent 2 }}
values.yaml
config: |
port: 3090
baseURL: <http://localhost:5001/v1>
streamTopic: sync-web
and helmfile parse this as
data:
baseURL: '<http://localhost:5001/v1>'
default.yaml: |
port: 3090
nodeEnv: development
streamTopic: sync-web
Yeah, try
data:
nodeEnv: {{ .Values.environment }}
default.yaml: |
{{ tpl .Values.config . | indent 4 }}
The golden rule is that you shouldn’t add spaces before any template expression that results in a multi-line text
indent 4 give me error
Instead add | indent N to it
Which error?
oh my bad
thank you
that seems resolved now
indent 2 confused me
Great!
Hehe
Yeah tricky but it’s actually a very common gotcha of yaml templates. You’ll see bunch of indent N in helm chart templates as well
I’m not happy with that either but you’ll feel better once you get used to it…
but after deployment, or sync, I only have
default.yaml:
test: a
not including yes: true
is it true?
@Ribhararnus Pracutiar Regarding the issue you had yesterday, yeah, I think I got it.
Try adding the below to repositories.yaml:
environments:
local: {}
development: {}
staging: {}
production: {}
It’s annoying but how helmfile currently works.
I think we need to change helmfile to not complain on missing env for bases(or probably in other cases as well
This one is similar to yours https://github.com/roboll/helmfile/issues/704#issuecomment-503645728
ok thanks, maybe later I comeback with the issue, as for now, I use repositories in every single helmfile
what about above issue?
2019-06-20
2019-06-21
What’s the appropriate way to run a post deploy task?
I’m spinning up an elasticsearch cluster, but after i do I want helm to run a few curl commands. I’m thinking about creating a chart with a cronjob,but is there a better way?
I know about post hooks, but that runs locally where the helmfile is run, right? Id rather run this once in a pod on the k8s cluster.
Try helm chart’s “post-install” hook https://github.com/helm/helm/blob/master/docs/charts_hooks.md
#codefresh supports helm events as triggers :-)
#codefresh supports helm events as triggers :-)
I’m interested. codefresh watches for helm events so that you can hook into them?
2019-06-22
2019-06-23
2019-06-24
Can I access the name of the current release in some variable, inside a values template file?
If I got it right simple {{ .Release.Name }} should work.
Doesn’t seem quite that simple:
in ./helmfile.yaml: helm exited with status 1:
Error: failed to parse /var/folders/wk/kl05m8rx49x5v67ckkkq_xhxk__c0p/T/values267206567: error converting YAML to JSON: yaml: invalid map key: map[interface {}]interface {}{".Release.Name":interface {}(nil)}
{{ .Release.Name }} definitely works inside values.yaml in a helm chart, but I don’t think it does in helmfile.yaml templating.
So, you want to use it in helmfile.yaml, right?
What about this
{{`{{.Release.Name}}`}}
then?
Yeah, with:
{{` ... `}}
it should work.
That’s the confusing aspect of helmfile.yaml being a go template as whole.
Ah, so by doing that, it evaluates (in helmfile) into {{.Release.Name}} which is then evaluated by helm into the release name.
That is slightly confusing indeed
That’s it! Yeah, that’s why I want to make helmfile.yaml itself a plain YAML in the future
I.e, I have two nginx-ingress releases, ingress-test and ingress-production, and I want to set antiaffinity based on the release name, from a values template.
2019-06-25
2019-06-26
What is the recommended approach for converting a standard helm chart into a helmfile? if there is documentation on this i apologize.
You don’t need to convert anything. Helmfile basically allows you to declaratively define what should be deployed to the cluster. Pretty much like docker-compose do for docker containers or a configuration management tool for os packages.
helmfile.yaml could be as simple as
#Your Helm repositories
repositories:
- name: stable
url: <https://kubernetes-charts.storage.googleapis.com>
#Helm settings
helmDefaults:
tillerNamespace: tiller-namespace
#Your releases
releases:
- name: ingress
namespace: ingress
chart: stable/nginx-ingress #chart name
version: 1.6.13 #chart version
Almost everything (except for more complicated use cases and experimental features) is covered here: https://github.com/roboll/helmfile#configuration
@Ryan Richards is there a helm chart in particular? we might have an example of it.
I wouldn’t call it a conversion, wrapping maybe But it doesn’t really matter. Yes, cloudposse has a lot of great examples. Thank you for this, btw!
2019-06-27
If you wanted to run a simple script that, lets say creates a few namespaces, could you do this somewhere with helmfiles? I found that working with terraform null resource didn’t work as i expected. If there is a best-practice for running these kind of configuration processes i would love to hear about it
what about kubernetes provider for TF? you could use it to create namespaces
Helm will by default create the namespace for you. No need to do anything fancy.
@Erik Osterman (Cloud Posse) thanks, i was using that as an example. We have several post-setup configuration steps that need to occur once our terraform is finished with the basic cluster setup in gke. These include things like setting default limits.
Most of these tasks are installing prometheus, grafana etc. Im working now to create helmfiles for these but there are a few that to be repeatable would need to be run as scripts or something
hopefully my statement made sense Im simply trying to find the best manner to apply configuration changes that dont involve installing or deploying. Running gcloud commands to obtain connectivity to a specific cluster would be another example.
you can trigger commands as part of hooks
@Erik Osterman (Cloud Posse) awesome! thank you!
ah hooks on helmfile - i gotcha
2019-06-29
Hi, could someone help me with the issue? https://github.com/roboll/helmfile/issues/731
Try | indent 6 after readFile :)
It should not take much time to help me coz solutions seems to be simple
But problem is strange and unexpected … and solution not obvious for me