#helmfile (2019-11)

https://github.com/helmfile/helmfile

Questions and discussion around helmfile https://github.com/roboll/helmfile and https://github.com/cloudposse/helmfiles

Archive: https://archive.sweetops.com/helmfile/

2019-11-01

2019-11-02

mumoshu avatar
mumoshu

I’d like to deprecate “double-rendering” in favor of this feature: https://github.com/roboll/helmfile/issues/932

WDYT?

feat: Double-render-free Helmfile config template · Issue #932 · roboll/helmfile

TL;DR; I want to add a new helmfile.yaml field to make templating helmfile configs easier and more powerful than ever. Problem Helmfile's double-rendering has opened a wide variety of use-cases…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

How reddit uses Helmfile (2018). https://youtu.be/7Qxuo9W5SlY

5

2019-11-04

Bart M. avatar
Bart M.

hmm getOrNil can only return as string?

mumoshu avatar
mumoshu

nope. what do you see though?

Bart M. avatar
Bart M.

well I’m trying to loop over a dict that might not exist…

Bart M. avatar
Bart M.

so I’m trying to do something like this:

{{ range $version, $semver := (getOrNil .Values.deploys "somerelease") | default {} }}
Bart M. avatar
Bart M.

the | default {} doesn’t seem to work, so I left it out

Bart M. avatar
Bart M.

but then I get: wrong type for value: expected string, got map[string]interface{}

Bart M. avatar
Bart M.

on that line

Bart M. avatar
Bart M.

core reason for this is that we need to deploy the same stack to quite a few namespaces, all different versions, so now we have a central base that includes a deploys map like this:

deploys:
  service1:
    v0: ">=0.0.1-alpha.1"
    v1: "1.0.2"
  service2:
    v2: "2.0.0"

The file containing this is loaded depending on the environment that’s being deployed

mumoshu avatar
mumoshu

@Bart M. It should be getOrNil "somerelease" .Values.deploys

Bart M. avatar
Bart M.

ow

Bart M. avatar
Bart M.

right damn

mumoshu avatar
mumoshu

and {} isnt a valid symbol in go template so probably you want (getOrNil "somerelease" .Values.deploys) | default (dict) or even shorter get "somerelease" .Values.deploys (dict)

Bart M. avatar
Bart M.

hmm ic

Bart M. avatar
Bart M.

allright - thanks, seems to work like that

mumoshu avatar
mumoshu

awesome!

2019-11-05

pjbecotte avatar
pjbecotte

Another question- how does values templating work in helmfile? Like, I can’t use .Values from a helmfile in .gotmpl files…but I can use values passed in from higher level helmfiles using the ‘helmfiles’ key. Butniflt feels like there is a way I’m missing to not need multiple layers. The kinds of things I want to do are declare a value in one place, and then use that value later in to build up other values.

pjbecotte avatar
pjbecotte

Like ‘tag: 123’ and then ‘image: tag: {{.Values.tag}}’

pjbecotte avatar
pjbecotte

Hmm, and it looks like I can also you values from the environment. Interestingly, env variables override values passed in

mumoshu avatar
mumoshu

Hey!
I can’t use .Values from a helmfile in .gotmpl files.

I thought this is actually opposite. You can access .Values from within .gomtpl files listed under releases[].values[]

mumoshu avatar
mumoshu

But I’d say it’s the only exception

mumoshu avatar
mumoshu

To pass .Values down to sub-helmfiles, you need to explicitly state so in helmfiles[].values

mumoshu avatar
mumoshu


it looks like I can also you values from the environment. Interestingly, env variables override values passed in

I don’t understand this. Would you mind providing an example?

mumoshu avatar
mumoshu

helmfile doesn’t override values with env variables except that you explicitly stated so

2019-11-07

Gourav avatar

Hello All.. Is there any document/image which depicts the kiam, reloader and cert-manager? Which tries to show how they are interrelated. If anyone is having such link, can you please share with me?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Nothing visual

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we have our implementation of them here: https://github.com/cloudposse/helmfiles/tree/master/releases

cloudposse/helmfiles

Comprehensive Distribution of Helmfiles. Works with helmfile.d - cloudposse/helmfiles

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

kiam is to obtain IAM roles needed by cert-manager for Let’s Encrypt validation for wildcard domains

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

cert-manager is needed for the PKI certs that kiam uses between agents/servers

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

reloader is needed to restart the pods for kiam when the certificates change (or else you have a forced outage)

Gourav avatar

@Erik Osterman (Cloud Posse) Thank you very much

Frank avatar

Hi all! I have some helm charts that have to be installed in the right order, e.g. 1) install helm chart 1 which created some CRDs and so on 2) wait some time until helm chart 1 is settled 3) install helm chart 2

Pierre Humberdroz avatar
Pierre Humberdroz

https://github.com/helm-notifier/Kubernetes-Infrastructure

Check the helmfile.d they get executed in alphabetical order

helm-notifier/Kubernetes-Infrastructure

Contribute to helm-notifier/Kubernetes-Infrastructure development by creating an account on GitHub.

Frank avatar

I work with references from one (master) helmfile to (child) helmfiles. like this I think the order is also preserved plus additionally I can use common values and so on. But what doesn’t work is this wait time…

Frank avatar

e.g.

bases:
  - common.yaml
---
helmfiles:
  - path: "storageclass.yaml"
  - path: "reloader.yaml"
  - path: "certmanager.yaml"
#  - path: "appscode.yaml"
  - path: "others.yaml"
    values:
    - environment: {{ .Values.environment }}  
    - domain: {{ .Values.domain }} 
Frank avatar

I tried to do this with wait(on helmfile and release level) and atomic , but both seem not to work stable in my case. (By the way I use helm3 , latest release, that might be important )

mumoshu avatar
mumoshu

what is your definition of the chart is settled?

I thought wait: true i.e. helm’s --wait, more or less, wait for all the pods under deployments are ready in terms of pod readiness.

is that what you want? or perhaps you have an another criteria to be met before helmfile proceeds to the next release?

Balaji J avatar
Balaji J

Hi @Frank, were u able to achieve the ordering of starting the helm chart? if so, can you pls clarify how the issue was solved?

2019-11-09

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Wow Helmfile getting spammed in issues! Hope this is an isolated incident. GitHub has been amazingly spam free.

1
1
mumoshu avatar
mumoshu

just finished reporting those all

1

2019-11-10

Andrew Nazarov avatar
Andrew Nazarov

Is it possible to prevent a release from updating when doing helmfile apply? I don’t see any flag allowing me to do so. The use case: some charts have some random values which are recalculated every time we run helm diff. So there are always some diffs. Hence this leads to unnecessary upgrades. This ignore is not the best solution for sure. Like it will also ignore all “positive changes” that should be syncronized. But at the moment I don’t see anything else that could have been done to solve this. Any thoughts on this?

Andrew Nazarov avatar
Andrew Nazarov

No, it was a bad idea. It’s better to upgrade a release every time.

pjbecotte avatar
pjbecotte

What are the random numbers for? I generate secrets- but i use hooks so that I only generate them if they don’t already exist. If you’re generating random values in manifests that you apply, then you are certainly changing stuff.

Andrew Nazarov avatar
Andrew Nazarov

Usually this comes from certs, like tls.crt, tls.key, caBundle, etc fields. Artifactory and Stash charts are know for doing that. Also I’ve seen something similar from Elasticsearch, but not anymore. Cannot find any examples with generated strings and random numbers right now, but they definitely existed before).

2019-11-11

pjbecotte avatar
pjbecotte

Any way to run helmfile hooks without helmfile sync?

pjbecotte avatar
pjbecotte

(am investigating using our helmfile setup with our argocd deployment…but I made extensive use of helmfile hooks)

mumoshu avatar
mumoshu

@pjbecotte Unfortunately it isn’t possible today. But a feature request is welcomed. If you could also include your specific use-cases for hooks(fetching charts and modifying it, creating namespaces by calling kubectl, etc) that would be great

pjbecotte avatar
pjbecotte

Sure- helmfile is golang?

mumoshu avatar
mumoshu

Yep

pjbecotte avatar
pjbecotte

Probably not gonna be a PR then :)

pjbecotte avatar
pjbecotte

I’ll put in an issue though

mumoshu avatar
mumoshu

Thanks!

2019-11-12

Thomas Burton avatar
Thomas Burton

Afternoon. Question:

Thomas Burton avatar
Thomas Burton

I am running into an error with installing any charts through helmfile. I get the following error:

sung kang avatar
sung kang

That’s an underlying helm issue

sung kang avatar
sung kang

Happens when you fail the first release

Andrew Nazarov avatar
Andrew Nazarov

indeed

Andrew Nazarov avatar
Andrew Nazarov

atomic: true could help here

Thomas Burton avatar
Thomas Burton

thanks for your responses

Thomas Burton avatar
Thomas Burton
Thomas Burton avatar
Thomas Burton

This is my helmfile

Thomas Burton avatar
Thomas Burton
Thomas Burton avatar
Thomas Burton

any help would be greatly appreciated

Thomas Burton avatar
Thomas Burton

managed to fix by running helmfile destroy then helmfile apply

mumoshu avatar
mumoshu

I periodically see folks being trapped by the fact that atomic isnt enabled by default. Hopefully we are able to cut v1 and start turning it on by default since then https://github.com/roboll/helmfile/issues/776

plan: Helmfile v1 · Issue #776 · roboll/helmfile

Helmfile v1 may be the first helmfile release may introduce small backward-incompatible changes to provide long-term benefit for users. It won't break anything badly as Helmfile is already adop…

mumoshu avatar
mumoshu

@Erik Osterman (Cloud Posse) I’d appreciate your comment here! https://github.com/roboll/helmfile/issues/932#issuecomment-553188582

feat: Predictable Helmfile template · Issue #932 · roboll/helmfile

TL;DR; I want to add a new helmfile.yaml field to make templating helmfile configs easier. Problem Helmfile's double-rendering has opened a wide variety of use-cases that requires you to write …

2019-11-13

Thomas Burton avatar
Thomas Burton

Anyone able to help here

After running helmfile --environment staging apply successfully I go to check my releases and they’re not listed even though part of the logs suggests otherwise.

Thomas Burton avatar
Thomas Burton

This is the output from the logs

Thomas Burton avatar
Thomas Burton

I am running a tiller deployment in the staging namespace

Thomas Burton avatar
Thomas Burton

However, if I run helm list --tiller-namespace staging it returns absolutely nothing

Thomas Burton avatar
Thomas Burton

Another weird thing is that the pods start correctly

Thomas Burton avatar
Thomas Burton
Andrew Nazarov avatar
Andrew Nazarov

What’s in helmDefaults.tillerNamespace in your helmfile.yaml?

Thomas Burton avatar
Thomas Burton

@Andrew Nazarov it was originally default. Changed it to {{ .Environment.Name }} but still no luck.

Andrew Nazarov avatar
Andrew Nazarov

oh, you are using tillerless plugin)

Andrew Nazarov avatar
Andrew Nazarov

Then try helm tiller run staging -- helm ls

Thomas Burton avatar
Thomas Burton

A-HA!

Thomas Burton avatar
Thomas Burton

Thanks v much

Thomas Burton avatar
Thomas Burton

Worked

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
helm/helm

The Kubernetes Package Manager. Contribute to helm/helm development by creating an account on GitHub.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

3.0 official!!l?

yuri avatar

seems official, whats the support status of helmfile with helm v3? sorry was not following lately

pjbecotte avatar
pjbecotte

I’ve been using v3 for the last month

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

With Helmfile?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I know there have been a few PRs to add support

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Haven’t tested it yet

pjbecotte avatar
pjbecotte

Yeah with helmfile

4
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Daren @chris FYI

1
chris avatar
chris
10:15:37 PM

@chris has joined the channel

pjbecotte avatar
pjbecotte

Something I just thought of…the first time I had to set an env variables…butbinhavent been doing that

2019-11-15

Shikhar Goel avatar
Shikhar Goel

Hi…i am not able to run helmfile using helm 3…i am getting the following error….. Error: context deadline exceeded Error: plugin "diff" exited with error

mumoshu avatar
mumoshu

Which version of helmfile are u using?

mumoshu avatar
mumoshu

Do you have tillerless: true set?

mumoshu avatar
mumoshu

You need to turn it off

Shikhar Goel avatar
Shikhar Goel

Thanks it worked with removing tiller plugin from dockerfile

1
Shikhar Goel avatar
Shikhar Goel

Hi mumoshu…if i set tillerless: false using helm3 then i get the same error

Shikhar Goel avatar
Shikhar Goel

bash-5.0# helmfile --environment=singlenode apply --concurrency=1 Building dependency ../../helm-installer/rook-operator Comparing rook-operator ../../helm-installer/rook-operator in ./helmfile.yaml: in .helmfiles[0]: in environments/singlenode/00-helmfile.yaml: failed processing release rook-operator: helm exited with status 1: Error: context deadline exceeded Error: plugin "diff" exited with error

mumoshu avatar
mumoshu
Fully support helm 3 in docker image (helm diff) · Issue #980 · roboll/helmfile

Is it possible to add helm 3 to the docker image of helmfile? So we can just use the helmfile docker image in CI/CD without needing to install helm3. Things to do: Update helm diff plugin to versio…

mumoshu avatar
mumoshu

tl;dr; pls try reinstalling helm-diff with –version flag

Shikhar Goel avatar
Shikhar Goel

diff 3.0.0-rc.7 Preview helm upgrade changes as a diff

mumoshu avatar
mumoshu

yeah but you may still have onld helm-diff binary!

Shikhar Goel avatar
Shikhar Goel

I think diff is of correct version

mumoshu avatar
mumoshu

nope

mumoshu avatar
mumoshu

surprisingly.

mumoshu avatar
mumoshu

helm plugin install` clones the repo. the version number is taken from the the plugin.yaml contained in the repo. but installation of helm-diff binary happens independently

mumoshu avatar
mumoshu
databus23/helm-diff

A helm plugin that shows a diff explaining what a helm upgrade would change - databus23/helm-diff

Shikhar Goel avatar
Shikhar Goel

Thanks it worked…that was the only issue….

mumoshu avatar
mumoshu

and this seems to have an issue(?) that, when --version is omitted, installs the latest “non-rc” release

mumoshu avatar
mumoshu

glad it worked!

Shikhar Goel avatar
Shikhar Goel

So do you mind to add this fix in the latest Dockerfile for v3.

mumoshu avatar
mumoshu

did we miss that in dockerfile?

mumoshu avatar
mumoshu
roboll/helmfile

Deploy Kubernetes Helm Charts. Contribute to roboll/helmfile development by creating an account on GitHub.

mumoshu avatar
mumoshu

@Shikhar Goel would you mind submitting a pr for that?

Shikhar Goel avatar
Shikhar Goel

Sure

1

2019-11-18

Shikhar Goel avatar
Shikhar Goel

Hi…….I am seeing this issue. I have also referred this ‘https://github.com/roboll/helmfile/issues/863’ but it didn’t work. I took pull from master today only.Upgrading ../../helm-installer/rook-operator in ./helmfile.yaml: in .helmfiles[0]: in environments/singlenode/00-helmfile.yaml: failed processing release rook-operator: helm exited with status 1: Error: invalid argument "300" for "--timeout" flag: time: missing unit in duration 300

Timeout issue with Helm v3 mode · Issue #863 · roboll/helmfile

in Helm v3 –timeout option in upgrade command has a type of duration, i.e. 60s or 5m –timeout duration time to wait for any individual Kubernetes operation (like Jobs for hooks) (default 5m0s) he…

Shikhar Goel avatar
Shikhar Goel

When i make 300s then again i am facing issue.

Timeout issue with Helm v3 mode · Issue #863 · roboll/helmfile

in Helm v3 –timeout option in upgrade command has a type of duration, i.e. 60s or 5m –timeout duration time to wait for any individual Kubernetes operation (like Jobs for hooks) (default 5m0s) he…

Shikhar Goel avatar
Shikhar Goel

could not deduce environment: block, configuring only .Environment.Name. error: failed to read 00-helmfile.yaml.part.0: reading document at index 1: yaml: unmarshal errors: line 14: cannot unmarshal !!str 300s into int in ./helmfile.yaml: in .helmfiles[0]: in environments/singlenode/00-helmfile.yaml: failed to read 00-helmfile.yaml: reading document at index 1: yaml: unmarshal errors: line 14: cannot unmarshal !!str 300s into int

mumoshu avatar
mumoshu
roboll/helmfile

Deploy Kubernetes Helm Charts. Contribute to roboll/helmfile development by creating an account on GitHub.

mumoshu avatar
mumoshu

it should be a number so 300 should be okay

Shikhar Goel avatar
Shikhar Goel

But it is giving error when used 300

mumoshu avatar
mumoshu

@Shikhar Goel which helmfile command are you running?

i thought i’ve fixed it for helm upgrade

Shikhar Goel avatar
Shikhar Goel

helmfile apply

mumoshu avatar
mumoshu

Hmm

Shikhar Goel avatar
Shikhar Goel

helm3 wants it in form of 300s and helmfile accepts it in form of 300(int)

mumoshu avatar
mumoshu

yes and helmfile automatically append s in helm3 mode

Shikhar Goel avatar
Shikhar Goel

but it is not in our case

mumoshu avatar
mumoshu

interesting

mumoshu avatar
mumoshu

could you provide a smallest possible example to reproduce it?

mumoshu avatar
mumoshu

i have no idea how that would happen..

Shikhar Goel avatar
Shikhar Goel

Sure one minute

Shikhar Goel avatar
Shikhar Goel

`releases: #ROOK #ROOK-INIT {{ if .Environment.Values.isRook }}

  • name: rook-operator chart: ../../helm-installer/rook-operator values:
    • /home/dmp/helmfile/{{ requiredEnv “env_file_name” }} wait: true timeout: 300 atomic: true {{ end }}

#PREREQUISITE-PROMETHEUS-GRAFANA {{ if .Environment.Values.isPrometheusGrafana }}

  • name: prerequisite-prometheus chart: ../../helm-installer/prerequisite-prometheus values:
    • /home/dmp/helmfile/{{ requiredEnv “env_file_name” }} wait: true timeout: 300 atomic: true`
Shikhar Goel avatar
Shikhar Goel

This is my helmfile and when i am running helmfile apply it is giving error Upgrading ../../helm-installer/rook-operator in ./helmfile.yaml: in .helmfiles[0]: in environments/singlenode/00-helmfile.yaml: failed processing release rook-operator: helm exited with status 1: Error: invalid argument "300" for "--timeout" flag: time: missing unit in duration 300

mumoshu avatar
mumoshu

can we generalize it to this?

releases:
  - name: rook-operator
    chart: stable/envoy
    wait: true
    timeout: 300
    atomic: true
  - name: prerequisite-prometheus
    chart: stable/envoy
    wait: true
    timeout: 300
    atomic: true

`

mumoshu avatar
mumoshu

so that i can run it locally

Shikhar Goel avatar
Shikhar Goel

actually we have local helm charts that we are running…but yes we can regenrate like this also

mumoshu avatar
mumoshu

thanks!

mumoshu avatar
mumoshu

okay reproduced

mumoshu avatar
mumoshu

HELMFILE_HELM3 helmfile apply still works

mumoshu avatar
mumoshu
feat: Automatically enable Helm v3 mode by drakedevel · Pull Request #963 · roboll/helmfile

Runs helm version in helmexec.New, and exposes a method on Interface to allow other packages to use the detected version. Preserves compatibility with previous HELMFILE_HELM3 mechanism. Resolves #923

Shikhar Goel avatar
Shikhar Goel

so i have to use HELMFILE_HELM3 helmfile apply explicitily in helmfile apply command

mumoshu avatar
mumoshu

yeah for now

mumoshu avatar
mumoshu

once the bug is fixed it isn’t needed

Shikhar Goel avatar
Shikhar Goel

ok cool but do i have to add some plugin or anything to make it work

Shikhar Goel avatar
Shikhar Goel

because i am getting bash-5.0# HELMFILE_HELM3 helmfile --environment=singlenode apply --concurrency=1 bash: HELMFILE_HELM3: command not found

mumoshu avatar
mumoshu

i thought it’s something posix-ish

mumoshu avatar
mumoshu

HELMFILE_HELM3=1 helmfile

mumoshu avatar
mumoshu

try adding =1

Shikhar Goel avatar
Shikhar Goel

that is already set in env

mumoshu avatar
mumoshu

then you can omit it from the command

Shikhar Goel avatar
Shikhar Goel

still got the same error

Shikhar Goel avatar
Shikhar Goel

Upgrading ../../helm-installer/rook-operator in ./helmfile.yaml: in .helmfiles[0]: in environments/singlenode/00-helmfile.yaml: failed processing release rook-operator: helm exited with status 1: Error: invalid argument “300” for “–timeout” flag: time: missing unit in duration 300

mumoshu avatar
mumoshu

what do you get from echo $HELMFILE_HELM3?

mumoshu avatar
mumoshu

or export | grep HELMFILE

Shikhar Goel avatar
Shikhar Goel

bash-5.0# env | grep -i HELMFILE_HELM HELMFILE_HELM3=1 bash-5.0# echo $HELMFILE_HELM3 1

mumoshu avatar
mumoshu

hm then your helmfile must be outdated..?

Shikhar Goel avatar
Shikhar Goel

i have just downloaded from your master branch today

mumoshu avatar
mumoshu

but it seems to be working for me

mumoshu avatar
mumoshu

could you double-check your path and timestamp on the helmfile binary being used?

Shikhar Goel avatar
Shikhar Goel

yes i have done that….

Shikhar Goel avatar
Shikhar Goel

i have build using docker build -t helmfile:0.1 .

Shikhar Goel avatar
Shikhar Goel

is there anything else we should do…

mumoshu avatar
mumoshu

..?

mumoshu avatar
mumoshu

could you provide me a full procedure to reproduce it?

mumoshu avatar
mumoshu

what do you run exactly after docker building it?

Shikhar Goel avatar
Shikhar Goel

Actually we have dockerfile inside helmfile repo so when we run docker build command then a docker image is build using that docker file…

Shikhar Goel avatar
Shikhar Goel

so for this i have removed your Dockerfile and renamed Dockerfile.v3 with Dockerfiule

Shikhar Goel avatar
Shikhar Goel

and then ran the command

mumoshu avatar
mumoshu

so how do you run helmfile after that?

Shikhar Goel avatar
Shikhar Goel

i have made a shell script that clone the helmcharts inside the docker image and call helmfile apply method after that

Shikhar Goel avatar
Shikhar Goel

Is there any other way to make docker image for helmfile

Shikhar Goel avatar
Shikhar Goel

??

mumoshu avatar
mumoshu

no it seems good

mumoshu avatar
mumoshu

but i cant reproduce it easily

mumoshu avatar
mumoshu

could you try just running make install to install helmfile locally and see if it works as intended?

mumoshu avatar
mumoshu

from what i see, all i can say atm is you may have outdated helmfile binary (or source repo)…

mumoshu avatar
mumoshu

also be sure to run go mod vendor before docker-build

mumoshu avatar
mumoshu

if you’ve outdated vendor/ under the helmfile repo

Shikhar Goel avatar
Shikhar Goel

Sorry i had previos cloned repo…but when i do docker build on your latest pulled master i get the following error

Shikhar Goel avatar
Shikhar Goel
mumoshu avatar
mumoshu

did you run go mod vendor?

Shikhar Goel avatar
Shikhar Goel

Nope let me try that

Shikhar Goel avatar
Shikhar Goel

Is this expected?

Shikhar Goel avatar
Shikhar Goel
Shikhar Goel avatar
Shikhar Goel

after building with latest image….still this issue persist

mumoshu avatar
mumoshu

no

mumoshu avatar
mumoshu

did you manually removed .git after cloning?

Shikhar Goel avatar
Shikhar Goel

Nope…

mumoshu avatar
mumoshu

no idea…

mumoshu avatar
mumoshu

where the error is coming from?

Shikhar Goel avatar
Shikhar Goel

Can you do one thing…can you please build helmfile docker image and make it pubic

mumoshu avatar
mumoshu

you can just grab it from quay

Shikhar Goel avatar
Shikhar Goel

ok…

mumoshu avatar
mumoshu

but why were you buijding it manually? did you have to do so??

Shikhar Goel avatar
Shikhar Goel

Actually i was adding sleep into it so that pod is not complete after execution

Shikhar Goel avatar
Shikhar Goel

can you please send me the quay url with helm3 and latest helmfile

mumoshu avatar
mumoshu

use tags prefixed with helm3

Shikhar Goel avatar
Shikhar Goel

ok thanks

Shikhar Goel avatar
Shikhar Goel

after using your image also same issue persist

Shikhar Goel avatar
Shikhar Goel

Upgrading ../../helm-installer/rook-operator in ./helmfile.yaml: in .helmfiles[0]: in environments/singlenode/00-helmfile.yaml: failed processing release rook-operator: helm exited with status 1: Error: invalid argument "300" for "--timeout" flag: time: missing unit in duration 300

mumoshu avatar
mumoshu

@Shikhar Goel I’ve tried to reproduce it but had no luck so far

#
# On my macOS machine:
#

$ docker run -it --net host -v $HOME:$HOME --rm quay.io/roboll/helmfile:helm3-v0.93.1 sh

#
# In the docker container:
#

# helmfile -v
helmfile version v0.93.1
# helm version
version.BuildInfo{Version:"v3.0.0", GitCommit:"e29ce2a54e96cd02ccfce88bee4f58bb6e2a28b6", GitTreeState:"clean", GoVersion:"go1.13.4"}
# cat helmfile.timeout.yaml
releases:
- name: envoy
  chart: stable/envoy
  timeout: 10
# helm repo add stable <https://kubernetes-charts.storage.googleapis.com/>
# helmfile -f helmfile.timeout.yaml sync
<succeeds>
mumoshu avatar
mumoshu

It just works as expected for me.

Shikhar Goel avatar
Shikhar Goel

ok thanks mumoshu…i will try it once again

Shikhar Goel avatar
Shikhar Goel

Thanks momushu..it worked with 0.93.1…earliar i was using 0.93.0

Shikhar Goel avatar
Shikhar Goel

Thanks Alot!…

mumoshu avatar
mumoshu

glad it worked!

Shikhar Goel avatar
Shikhar Goel

Just one more thing how do you think of implementing namespace feature that helm3 has.

Shikhar Goel avatar
Shikhar Goel

Like each helm chart will have its all info stored in its namespace

Shikhar Goel avatar
Shikhar Goel

in the form of secrets

mumoshu avatar
mumoshu

just specify namespace: yourns under releases[] and it should work

Shikhar Goel avatar
Shikhar Goel

Yup…that it works but i was thinking that there can be some namespace variable in helmdefaults so that if the user wants to send all the info in one namespace then he or she can do that easily.

mumoshu avatar
mumoshu

yeah can you just use helmDefaults.namespace?

Shikhar Goel avatar
Shikhar Goel

ok cool..

mumoshu avatar
mumoshu

Ah!

mumoshu avatar
mumoshu

So you mean helmfile doesn’t have support for that yet, right?

mumoshu avatar
mumoshu

seems so. i have never realized that we missed it under helmDefaults

mumoshu avatar
mumoshu

This should be a good feature request

Shikhar Goel avatar
Shikhar Goel

yup…actually i tested that…and i didn’t find that…thats why i was asking

mumoshu avatar
mumoshu

got it! thx

mumoshu avatar
mumoshu

would you mind writing a feature request?

Shikhar Goel avatar
Shikhar Goel

Yup i will start it now only.

mumoshu avatar
mumoshu

Much appreciated!

Shikhar Goel avatar
Shikhar Goel

2019-11-19

Shikhar Goel avatar
Shikhar Goel

Hi…–concurrency=1 is not working in helm3…

Shikhar Goel avatar
Shikhar Goel

I want to run my charts sequentially is there any other way i can do that

mumoshu avatar
mumoshu

how did you verify that?

mumoshu avatar
mumoshu

i thought we had tests for that so need to investigate how and where it’s happening

Shikhar Goel avatar
Shikhar Goel

I ran my helmfile and there was no execution order

mumoshu avatar
mumoshu

is it actually running concurrently? or the ordering is just random?

Shikhar Goel avatar
Shikhar Goel

order is just random

mumoshu avatar
mumoshu

ah

mumoshu avatar
mumoshu

maybe it’s alphabetically sorted, right?

mumoshu avatar
mumoshu

or is it really random?

mumoshu avatar
mumoshu

random, like each run results in different ordering?

Shikhar Goel avatar
Shikhar Goel

i think it is alphabetically sorted

Shikhar Goel avatar
Shikhar Goel

but i want to execute in the order of defination

mumoshu avatar
mumoshu

okay i think that’s a regression since 0.90.0 or so

mumoshu avatar
mumoshu

as the workaround, you can add needs to specify the order

mumoshu avatar
mumoshu
release:
- name: foo
- name: bar
  needs:
  - foo

this makes bar processed after foo

mumoshu avatar
mumoshu

regardless of concurrency.

Shikhar Goel avatar
Shikhar Goel

yup i think that is the only option now…and also regarding timeout…it didn’t work i have to remove explicitly from the helmfile

mumoshu avatar
mumoshu

probably i can try to bring back the old behavior in coming days but not today

Shikhar Goel avatar
Shikhar Goel

Ya sure thanks…that will be really helpfull

mumoshu avatar
mumoshu

will take a look into that as well. thx for your patience and support!

Shikhar Goel avatar
Shikhar Goel

Do i file a bug for that…or you will be notify me once done

mumoshu avatar
mumoshu

filing a bug would be super helpful, as i’m always overwhelmed by todos and the github issues are the only way to keep me reminded

Andrew Nazarov avatar
Andrew Nazarov

Are you sure about alphabetical order? For me it seems random even with v0.87.0.

mumoshu avatar
mumoshu

@Andrew Nazarov Does it seem random even when you’ve --concurrency -1 ?

mumoshu avatar
mumoshu

If it’s not set or set to 2 or bigger, the ordering should be random, regardless of helmfile is pre v0.90 or not

Andrew Nazarov avatar
Andrew Nazarov

I ran helmfile --log-level=debug --environment=alpha --selector type!=ingress apply --concurrency=1 . helm diff and helm upgrade calls looked pretty random. I used the helm-tiller plugin, btw.

mumoshu avatar
mumoshu

oh… good to know. thx!

Andrew Nazarov avatar
Andrew Nazarov

Haven’t checked the most recent version yet. And I’m in a way to switch to Helm 3.

mumoshu avatar
mumoshu

based on how it’s implemented, i thought it should be the order of definition.

perhaps that’s my mlsconception, or perhaps it’s conditional.

mumoshu avatar
mumoshu

anyways i’ve added some test case for that

Andrew Nazarov avatar
Andrew Nazarov

And my examples were related to Helm 2. The initial question was about Helm 3.

1

2019-11-20

rms1000watt avatar
rms1000watt

https://github.com/helm/helm/issues/3701 Does anyone hit this still? For some reason updating (adding) environment variables doesn’t get properly applied to the PodTemplateSpec of the deployment.

Updating the value of a computed environment variable did not take · Issue #3701 · helm/helm

I had a working chart and template that I accidentally broke; The problem was that an environment variable for a pod contained an incorrect value. &quot;Break&quot; here is taken to mean the pod co…

rms1000watt avatar
rms1000watt

might just helm render then kubectl apply

Updating the value of a computed environment variable did not take · Issue #3701 · helm/helm

I had a working chart and template that I accidentally broke; The problem was that an environment variable for a pod contained an incorrect value. &quot;Break&quot; here is taken to mean the pod co…

mumoshu avatar
mumoshu

wow this looks like an interesting issue

mumoshu avatar
mumoshu

do you have any chart or helmfile.yaml i can use to reproduce?

rms1000watt avatar
rms1000watt

So the process that got me here (sort of impossible to reproduce)

  1. kubectl apply -f generic-deployment-only.yaml
  2. let developers do a bunch of kubectl commands against this deployment
  3. Import this deployment to helm without destroying it first
  4. Add/update env vars and notice they don’t get updated

Honestly, I doubt it’s a helmfile issue. Probably just helm being helm

mumoshu avatar
mumoshu

Thanks! How did you import the deployment to helm?

2019-11-21

TBeijen avatar
TBeijen

Seem to run into a strange situation where failing to decrypt sops secrets (because not having KMS permissions) fails silently, resulting in an error due to missing variable, hiding the actual problem. In other projects error clearly states sops failing. So… probably overlooking some silly thing. Does this ring any bells?

mumoshu avatar
mumoshu

Hmmm, no idea unfortunately. All Helmfile does is to shell out to helm-secrets expecting it would exit with a non-zero code when something went wrong

mumoshu avatar
mumoshu

Reading Helmfile code to see if I’ve made silly mistake in it like ignoring exit code from helm-secrets at all..

TBeijen avatar
TBeijen

Haven’t pinpointed it. We now know that something with co-workers AWS config was incorrect, causing decrypt to fail. However helmfile was silent about it. In another project decrypt errors were clearly stated.

TBeijen avatar
TBeijen

Only difference between projects was that the latter secrets file had a fallback-pgp key configured as well, presenting an interactive screen for pgp password if KMS decryption failed.

TBeijen avatar
TBeijen

So maybe that’s the difference: Interactive screen triggering a decrypt error in Helmfile, where simply failing to decrypt goes unnoticed (and then results in some missing var error).

TBeijen avatar
TBeijen

Now need to focus on something else but will try to pinpoint if this difference indeed is what causes the different behaviour.

mumoshu avatar
mumoshu

Does it fail as intended when you call helm-secrets dec $FILE directly?

TBeijen avatar
TBeijen

Additional complexity: We run tiller-less. Need to look into that later as it ‘worked on my machine’.

TBeijen avatar
TBeijen

Will later try to set-up scenario where it doesn’t decrypt. Then work my way up: sops, helm secrets, helm tiller, helmfile.

2019-11-22

jedineeper avatar
jedineeper

are there any special changes to helmfile for the helmv3 upgrade? my helm installation seems to be fine but the latest version of helmfile isn’t happy? Is there a change in syntax?

jedineeper avatar
jedineeper

does look like a helm issue which is frustrating

jedineeper avatar
jedineeper
Comparing release=drone, chart=stable/drone
in ./helmfile.yaml: failed processing release drone: helm exited with status 1:
  Error: Couldn't load repositories file (repository/repositories.yaml).
  You might need to run `helm init` (or `helm init --client-only` if tiller is already installed)
  Error: plugin "diff" exited with error
jedineeper avatar
jedineeper
Helm 3: Error: Couldn't load repositories file (repository\repositories.yaml). You might need to run `helm init` · Issue #161 · databus23/helm-diff

I&#39;m currently trying to setup helm 3 with helmfile and helm-diff but I run in some weird issues. When running helm file with debug mode I notice that the following command cause issues for helm…

Andrew Nazarov avatar
Andrew Nazarov

Yes, it was a similar issue in the helmfile repo.

https://github.com/roboll/helmfile/issues/980

Fully support helm 3 in docker image (helm diff) · Issue #980 · roboll/helmfile

Is it possible to add helm 3 to the docker image of helmfile? So we can just use the helmfile docker image in CI/CD without needing to install helm3. Things to do: Update helm diff plugin to versio…

Andrew Nazarov avatar
Andrew Nazarov
Not working with helm3 · Issue #994 · roboll/helmfile

I&#39;m trying to install simple environment with helmfile using helm3 on brand new 1.16 cluster and it fails with following error: worker 1/1 finished err: release &quot;cert-manager&quot; in &quo…

2019-11-23

2019-11-24

mumoshu avatar
mumoshu

it’s still rough around the edges but…

https://github.com/mumoshu/terraform-provider-helmfile

mumoshu/terraform-provider-helmfile

Deploy Helmfile releases from Terraform. Contribute to mumoshu/terraform-provider-helmfile development by creating an account on GitHub.

3
2
3
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Omg @mumoshu you are the greatest

mumoshu/terraform-provider-helmfile

Deploy Helmfile releases from Terraform. Contribute to mumoshu/terraform-provider-helmfile development by creating an account on GitHub.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Thank you for this! @Andriy Knysh (Cloud Posse)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Andriy Knysh (Cloud Posse) Let’s try this this week with terraform cloud

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Daren @Alex Siegman @Rhooker @btai heads up

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@coreygale start using Helmfile yet? :)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Shane think you mentioned some interest in this

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

will try, great work @mumoshu

btai avatar

nice nice

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

What’s cool about this “ value = helmfile_release_set.mystack.apply_output” is we can use a regex to extract parts of the output (e.g. NOTES.txt) which can contain metadata. Or for custom charts, output JSON in notes and use terraform interpolations to unpack

Andrew Nazarov avatar
Andrew Nazarov

Is it possible to reference remote helmfile.yaml?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Good question! I assume so. @mumoshu ?

mumoshu avatar
mumoshu

Yes it should work. path is basically -f in Helmfile and it should accept remote urls.

Andrew Nazarov avatar
Andrew Nazarov

How does it work with environments? I mean environmnent values are taken from the file referenced in the main helmfile.yaml. Or I could somehow define them in tf config directly. I’m figuring out the best way to use the plugin.

We are using terraform to create clusters on GKE and we are using a repo per stage to restrict the access. At the same time we think it would be better to separate cluster creation and app deployment. Moreover we want to keep the existing helmfile.yaml and don’t repeat its configuration. Our helmfile.yaml is highly templatized and we are using environments.yaml as a simple interface for our developers. All they need to do is to change or add some values in this file and that’s it.

Andrew Nazarov avatar
Andrew Nazarov

So if somebody had some thoughts to share I would really appreciate this.

mumoshu avatar
mumoshu

it really depends on the use-case but generally speaking, i’d use terraform variables for a few parameters like the name of environment, region, availability zone.

1
mumoshu avatar
mumoshu

and set it via helmfile_release_set values

values = {
      # Corresponds to --state-values-set region=whatever
      region = "whatever"
    }
mumoshu avatar
mumoshu

btw it’ll likely to be changed to

values = [
  <<EOS
{
  "region": "whatever"
}
EOS
]
mumoshu avatar
mumoshu

terraform native maps doesn’t really support arbitrary nested maps that we cant do this:

values = {
      # Corresponds to --state-values-set region=whatever
      foo = {
        bar = {
          region = "whatever"
        }
      }
    }
mumoshu avatar
mumoshu

a good news is that terraform 0.12 is able to produce a structured diff against these json-object-looking strings

mumoshu avatar
mumoshu

so it isn’t a huge u/x breaker

mumoshu avatar
mumoshu


Moreover we want to keep the existing helmfile.yaml and don’t repeat its configuration

how do you provide helmfile environments.yaml file today?

mumoshu avatar
mumoshu

if you’re passing it via helmfile --state-values-file environments.yaml, you can just point that file from values_file in tf helmfile_release_set resource

Andrew Nazarov avatar
Andrew Nazarov

Now it’s just like that

bases:
- environments.yaml
---
mumoshu avatar
mumoshu

got it. would you consider to specify the path to environments.yaml from tf-side?

mumoshu avatar
mumoshu

or the whole contents of the environments.yaml, or only some part of it?

Andrew Nazarov avatar
Andrew Nazarov

Yes, something like that

mumoshu avatar
mumoshu

in any case, the general recommendation would be that u should do as much as possible in helmfile

Andrew Nazarov avatar
Andrew Nazarov

Probably only a part of it, based on the environment passed to tf.

Andrew Nazarov avatar
Andrew Nazarov

Ok)

mumoshu avatar
mumoshu

i was considering to allow inlining the whole helmfile.yaml content in tf, it it helps

mumoshu avatar
mumoshu

you could use tf string interpolation syntax to write whatever helmfile.yaml you like in it

Andrew Nazarov avatar
Andrew Nazarov

Alright, thank you very much)

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
values = [
  <<EOS
{
  "region": "whatever"
}
EOS
]

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Why move to HEREDOC?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(string) vs structured data in HCL?

Daren avatar

Interesting. Unfortunately its not compatible with Terraform Cloud as it relies on cli tools + config

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Can’t those be installed just like custom terraform provider?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

…pretty sure

mumoshu avatar
mumoshu

@Erik Osterman (Cloud Posse) That’s due to the limitation in HCL/TF resource

https://sweetops.slack.com/archives/CE5NGCB9Q/p1574841839216000?thread_ts=1574585241.174500&cid=CE5NGCB9Q

mumoshu avatar
mumoshu

I think that’s why the helm provider doesn’t support maps for setting values as well https://www.terraform.io/docs/providers/helm/release.html#example-usage

mumoshu avatar
mumoshu

Should we move to a set syntax similar to the helm provider, something like:

set {
  name = "metrics.enabled"
  bool = true
}

set {
  name = "image.tag"
  string = "sometag"
}

set {
  name = "deployment.replicas"
  int = 3
}
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I follow what you are suggesting, but don’t understand how if one is writing a custom provider how it would be limited to using certain data types, of which map is not supported. (Not speaking from experience)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Is that what you are saying?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Hrmm I follow a little bit more

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Though I thought this is possible with locals

mumoshu avatar
mumoshu

yeah i believe you’re following me correctly

mumoshu avatar
mumoshu

from my experience, terraform resource spec’s is restricted to have

mumoshu avatar
mumoshu

a static and predefined schema

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

So for consistency with helm provider maybe makes sense to do the same

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Otherwise it could be a flat map with dot notation

mumoshu avatar
mumoshu

i’ll try taking a deeper look but probably the set syntax would be better

mumoshu avatar
mumoshu

a flat map wit dot nation seems not work as we need to give a specific type(string, int, bool, etc.) to the schema

mumoshu avatar
mumoshu
values = {
  "foo.bar" = 1,
  "bar.baz = true
}

probably this can’t be expressed in today’s tf resource schema

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I wonder if this syntax would work:

values = yamlencode({
      # Corresponds to --state-values-set region=whatever
      foo = {
        bar = {
          region = "whatever"
        }
      }
    })
mumoshu avatar
mumoshu

o

mumoshu avatar
mumoshu

looks good

mumoshu avatar
mumoshu

then we don’t need to change anything

mumoshu avatar
mumoshu

from the helmfile provider’s perspective, values is just a array of strings

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

oh cool

mumoshu avatar
mumoshu

where you use yamlencode to convert the HCL map into string

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I confirmed the syntax is acceptable

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

…with nested maps

mumoshu avatar
mumoshu

cool!

mumoshu avatar
mumoshu

actually, when i start working on the helmfile provider, i was blindly believing that it’s possible without yamlencode

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

…i would have assumed the same!

mumoshu avatar
mumoshu

pretty disappointed when it turned out i had to change maps to strings. hopefully this will be fixed in tf in the future..

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Out of curiosity, would would happen if one did this?

values = yamlencode({
      # Corresponds to --state-values-set region=whatever
      foo = {
        bar = {
          region = "{{ .Env.REGION }}"
        }
      }
    })
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

would helmfile render the gotmpl stuff?

mumoshu avatar
mumoshu

i have not tried yet but it should

1
mumoshu avatar
mumoshu

just a moment..

mumoshu avatar
mumoshu

ah sry it won’t work. under the hood it is just that the provider produces a temporary values.yaml to be passed to --state-values-file

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

aha, ok - was just curious. don’t have a use-case yet.

mumoshu avatar
mumoshu

you’d need to make it .gotmpl` to let gotmpl expressions evaluated

mumoshu avatar
mumoshu

ok

mumoshu avatar
mumoshu

maybe we can add values_templates for that

Andrew Nazarov avatar
Andrew Nazarov

I was wondering if terraform apply works correctly with presync hooks installing crds… I mean in situations like this one: https://github.com/roboll/helmfile/issues/538#issuecomment-499237514

[doc] Recommended way of handling non conventional install such as cert-manager · Issue #538 · roboll/helmfile

HI, When reading the process to install the cert-manager chart (https://hub.helm.sh/charts/jetstack/cert-manager), you can see two steps before installing the chart: installing some CRD prior to th…

mumoshu avatar
mumoshu

can i ask why you think so?

mumoshu avatar
mumoshu

my theory was that it should work if vanilla helmfile apply works, as the tf provider just run helmfile apply under the hood

Andrew Nazarov avatar
Andrew Nazarov

Haven’t checked it yet with the most recent versions of helmfile and tf provider (currently dealing with some different troubles with cert-manager), but considering the provided example I had to run helmfile sync first and only then apply started working.

mumoshu avatar
mumoshu

thx. do you see any specifc error from the first helmfile apply?

Andrew Nazarov avatar
Andrew Nazarov

It was some application specific error. I don’t remember exactly. I will try to check this out a bit later.

Andrew Nazarov avatar
Andrew Nazarov

It seems now it works quite ok.

Andrew Nazarov avatar
Andrew Nazarov

Another problem I have is that I always get the following error when running terraform apply, but helmfile apply works ok:

Error: could not get apiVersions from Kubernetes: unable to retrieve the complete list of server APIs: admission.stash.appscode.com/v1alpha1: the server could not find the requested resource, admission.stash.appscode.com/v1beta1: the server could not find the requested resource, repositories.stash.appscode.com/v1alpha1: the server could not find the requested resource
Andrew Nazarov avatar
Andrew Nazarov

Probably it’s related with the installation order. So, since stash-operator always shows diff helmfile always tries to reinstall it, hence for some time the api are not available. When this happens another services cannot be installed due to the https://github.com/helm/helm/issues/6361

unable to retrieve the complete list of server APIs · Issue #6361 · helm/helm

Output of helm version: version.BuildInfo{Version:&quot;v3.0+unreleased&quot;, GitCommit:&quot;180db556aaf45f34516f8ddb9ddac28d71736a3e&quot;, GitTreeState:&quot;clean&quot;, GoVersion:&quot;go1.13…

Andrew Nazarov avatar
Andrew Nazarov

But somehow it always works for helmfile apply, but never for terraform apply. Any hints? The order is different? Concurrency is different?

mumoshu avatar
mumoshu

hmm, could you share your tf file?

mumoshu avatar
mumoshu

does any actual upgrades via terraform apply works after the successful helmfile apply?

Andrew Nazarov avatar
Andrew Nazarov

It’s as simple as that

variable "project_id" {
  type = string
  description = "Google Cloud project ID"
}

variable "region" {
  type = string
  description = "Default Google Cloud region"
}

provider "google" {
  version = "~> 2.18.0"
  project = var.project_id
  region = var.region
  credentials = "${file("./credentials/account.json")}"
}

provider "helmfile" {}

resource "helmfile_release_set" "mystack" {
  path = "git::https://******/helmfile.yaml"

  working_directory = path.module

  environment = "staging"
}

data "google_compute_address" "dev_ip_address" {
  name = "***-dev"
}

output "external-ip" {
  description = "External static IP adress for dev"
  value = data.google_compute_address.dev_ip_address.address
}

output "mystack_diff" {
  value = helmfile_release_set.mystack.diff_output
}

output "mystack_apply" {
  value = helmfile_release_set.mystack.apply_output
}
mumoshu avatar
mumoshu

ohh… it does look simple

Andrew Nazarov avatar
Andrew Nazarov

Some updates happen initially as well, but since the whole terraform apply fails with the error, the state is not updated. So we cannot use terraform to manage that half-baked infra). Except for giving another try to apply.

mumoshu avatar
mumoshu

i reread the relevant part of the providers code. unfortunately i have no clue so far. it is really just calling helmfile apply.

mumoshu avatar
mumoshu

would it be easily reproducible for me if i tried to install stash-operatory?

Andrew Nazarov avatar
Andrew Nazarov

I’m not sure actually). I had the same errors some time ago even with helmfile apply, but subsequent applying helped. So, I think this is just a coincidence.

Andrew Nazarov avatar
Andrew Nazarov

I will play with it more. Don’t waste your time exploring this right now).

1
Andrew Nazarov avatar
Andrew Nazarov

I’ve decided to move stash-operator to the dedicated helmfile and reference it via helmfiles: from the main one. But for some reason it’s ignored during installation. What I’m doing wrong?

bases:
- environments.yaml
---
helmfiles:
  - stash-operator/helmfile.yaml

repositories:
....

releases:
....
Andrew Nazarov avatar
Andrew Nazarov

I can see the rendering though

first-pass rendering result of "helmfile.yaml.part.0": {staging map[] map[]}
vals:
map[]
defaultVals:[]
second-pass rendering result of "helmfile.yaml.part.0":
 0: repositories:
 1:  - name: incubator
 2:    url: <https://kubernetes-charts-incubator.storage.googleapis.com/>
 3:  - name: appscode
 4:    url: <https://charts.appscode.com/stable/>
 5: 
 6: releases:
 7:   - name: namespace-stash
 8:     namespace: default
 9:     chart: incubator/raw
10:     values:
11:       - resources:
12:         - apiVersion: v1
13:           kind: Namespace
14:           metadata:
15:             name: stash
16:           spec:
17: 
18:   - name: stash-operator
19:     namespace: stash
20:     chart: appscode/stash
21:     version: v0.9.0-rc.2
22:     needs:
23:       - default/namespace-stash
24:     labels:
25:       type: backup
mumoshu avatar
mumoshu
$ cat helmfile.all.yaml
helmfiles:
- helmfile.stashop.yaml

$ cat helmfile.stashop.yaml
repositories:
- name: incubator
  url: <https://kubernetes-charts-incubator.storage.googleapis.com/>
- name: appscode
  url: <https://charts.appscode.com/stable/>
releases:
- name: namespace-stash
  namespace: default
  chart: incubator/raw
  values:
  - resources:
    - apiVersion: v1
      kind: Namespace
      metadata:
        name: stash
      spec:
- name: stash-operator
  namespace: stash
  chart: appscode/stash
  version: v0.9.0-rc.2
  needs:
  - default/namespace-stash
  labels:
    type: backup
mumoshu avatar
mumoshu

helmfile --helm-binary helm3 -f helmfile.all.yaml apply worked for me. but on helmfile destroy it failed with the error we talked about:

FAILED RELEASES:
NAME
stash-operator
in ./helmfile.all.yaml: in .helmfiles[0]: in ./helmfile.stashop.yaml: release "stash-operator" failed: helm3 exited with status 1:
  Error: uninstallation completed with 1 error(s): could not get apiVersions from Kubernetes: could not get apiVersions from Kubernetes: unable to retrieve the complete list of server APIs: admission.stash.appscode.com/v1alpha1: the server is currently unable to handle the request, admission.stash.appscode.com/v1beta1: the server is currently unable to handle the request, repositories.stash.appscode.com/v1alpha1: the server is currently unable to handle the request
mumoshu avatar
mumoshu

if helmfile apply doesn’t install stash-operator for you, probably something in your top-level helmfile.yaml relates

mumoshu avatar
mumoshu
bases in helmfile.yaml will make environment disappeared · Issue #1008 · roboll/helmfile

Helmfile version: v0.94.0 Hi I run into a problem when use advanced configuration Layering State Files If I include bases into helmfile.yaml, and then use helm list, err: no releases found that mat…

Andrew Nazarov avatar
Andrew Nazarov

Hm, maybe. But my situation is a little bit different. In my case helmfile ignores releases defined in helmfiles:. I’ll open up a new issue with the example to reproduce.

Andrew Nazarov avatar
Andrew Nazarov

Hm, or you’ve already fixed that)). Let me double check that we the latest release.

Andrew Nazarov avatar
Andrew Nazarov

It seems no. I’ll file the issue. Probably I’m doing something wrong.

Andrew Nazarov avatar
Andrew Nazarov
Releases from `helmfiles` are ignored when the environment is specified · Issue #1010 · roboll/helmfile

I&#39;m facing the issue when releases defined in separate .yaml files and referenced via helmfiles: are ignored when I try to install the full stack for a certain environment. Example: #environmen…

mumoshu avatar
mumoshu

@Andrew Nazarov Thx! I was flying to re:Invent. I’ll take a look soon. Curious but did it work before?

Andrew Nazarov avatar
Andrew Nazarov

Haven’t used helmfiles: block before.

mumoshu avatar
mumoshu

got it! thanks

Andrew Nazarov avatar
Andrew Nazarov

Hm, it seems I need to define environments in subhelmfile for this to work. But shouldn’t they be taken from the main one?

mumoshu avatar
mumoshu

@Andrew Nazarov Replied to you in https://github.com/roboll/helmfile/issues/1010#issuecomment-561364261

I’m gonna do two things - wdyt?

Releases from `helmfiles` are ignored when the environment is specified · Issue #1010 · roboll/helmfile

I&#39;m facing the issue when releases defined in separate .yaml files and referenced via helmfiles: are ignored when I try to install the full stack for a certain environment. Example: #environmen…

Corey Gale avatar
Corey Gale

@Erik Osterman (Cloud Posse) Still getting our feet wet with Helmfile but yes it’s in our new k8s stack Thanks again for showing up the light

1
1
mumoshu avatar
mumoshu

just realized that there was another way to call helmfile from tf:

https://kubedex.com/using-helmfile-with-terraform/ https://github.com/Kubedex/terraform-helmfile

it’s interesting as well

Using Helmfile with Terraform - kubedex.comattachment image

Come and read Using Helmfile with Terraform on Kubedex.com. The number one site to Discover, Compare and Share Kubernetes Applications.

Kubedex/terraform-helmfile

Run Helmfile from Terraform. Contribute to Kubedex/terraform-helmfile development by creating an account on GitHub.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

It’s interesting but assumes you can call it from Docker

Using Helmfile with Terraform - kubedex.comattachment image

Come and read Using Helmfile with Terraform on Kubedex.com. The number one site to Discover, Compare and Share Kubernetes Applications.

Kubedex/terraform-helmfile

Run Helmfile from Terraform. Contribute to Kubedex/terraform-helmfile development by creating an account on GitHub.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

But we run inside of Docker already with geodesic

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

So it’s a non starter. Docker in Docker would be overkill.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

That said it was an interesting demonstration

1
mumoshu avatar
mumoshu

one advantage of my terraform-helmfile-provider would be that it exploits terraform plan output to show diff from helmfile diff

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

This has huge potential

mumoshu avatar
mumoshu

and you can even use terraform output to view helmfile apply results

Rhooker avatar
Rhooker
02:36:24 PM

@Rhooker has joined the channel

dustinvb avatar
dustinvb

This is still rough but I am setting up a demo for it as we speak… https://github.com/codefresh-contrib/cfstep-helmfile If there is interest in putting CI in Codefresh for the project that might be something that would help keep the Dockerfile contents updated. Also, is this ready for Helm 3.0 can I simply supply the newest helm binary and expect this to work?

2
mumoshu avatar
mumoshu


Also, is this ready for Helm 3.0 can I simply supply the newest helm binary and expect this to work?

Yes, it should work as helmfile detects helm version automatically

dustinvb avatar
dustinvb

Okay I think I am behind a couple version so will slip in the helm 3.0 binary with the 0.93.2 or something else over the holidays.

mumoshu avatar
mumoshu

Looking forward to it! JFYI, we have images for the helm3 setup in [quay.io/roboll/helmfile:helm3-](http://quay.io/roboll/helmfile:helm3-){ver} for reference

mumoshu avatar
mumoshu
Quay

Quay is the best place to build, store, and distribute your containers. Public repositories are always free.

roboll/helmfile

Deploy Kubernetes Helm Charts. Contribute to roboll/helmfile development by creating an account on GitHub.

2019-11-25

Roderik van der Veer avatar
Roderik van der Veer

I’ve got a quick question. I have a nice big Helmfile deploying an application stack. It is completely deployed within a namespace to separate it from other things running on the cluster. I now want to install a complete second instance of the stack, in a different namespace but on the same cluster. I imagined that because it is namespaced, it would just work, except it modified my original deploy. What would be the best way to enable such a usecase? Prefixing the release name with the namespace? (helm 2, latest helmfile, GKE)

jafow avatar


Prefixing the release name with the namespace?
I stumbled into this same problem and got around it by doing just that: prefixing release with Namespace and passing that a value with the --namespace option

Roderik van der Veer avatar
Roderik van der Veer

Thx!

2019-11-26

Roderik van der Veer avatar
Roderik van der Veer

I’m struggling a bit with –namespace and namespace definition in releases. I run helmfile with --namespace namespaceA. All my releases have namespace: {{ .Namespace }} and one (ingress controller) has namespace: something in the release. But it gets deloyed to namespaceA . Anyone have an idea on how to do this (apart from running helmfile twice)?

mumoshu avatar
mumoshu

Hi! It sounds like what you might need is --state-values-set namespace=namespaceA and {{.Values.namespace}}

mumoshu avatar
mumoshu

--namespace is for overriding namespaces for all the releases, regardless of what their “default” values are in helmfile.yaml

mumoshu avatar
mumoshu

So you’d better use a more general template .Values and the corresponding flag to inject it at runtime --state-values-set to code your convention into helmfile.yaml

2019-11-27

Josh McDade avatar
Josh McDade

Hi, I’m having issues running helmfile -n my-ns apply from v0.93.1 (I have set HELMFILE_HELM3=1 for my helm3 releases). Every release errors:

helm exited with status 1:
  Error: context deadline exceeded
  Error: plugin "diff" exited with error

The version of diff plugin is the same as the one in the Dockerfile.helm3 diff 3.0.0-rc.7

mumoshu avatar
mumoshu

it may be super surprising but you actually have an outdated helm-diff binary installed

mumoshu avatar
mumoshu

even you see diff 3.0.0-r.7 in helm plugin list`

mumoshu avatar
mumoshu
Fully support helm 3 in docker image (helm diff) · Issue #980 · roboll/helmfile

Is it possible to add helm 3 to the docker image of helmfile? So we can just use the helmfile docker image in CI/CD without needing to install helm3. Things to do: Update helm diff plugin to versio…

mumoshu avatar
mumoshu

try this

2019-11-28

mumoshu avatar
mumoshu

@Erik Osterman (Cloud Posse) btw, are you still using atlantis? just wanted to ask if it can be a good idea to use atlantis for deploying k8s apps, as we now have the tf provider for helmfile

https://github.com/roboll/helmfile/issues/386

Github PR workflow support similar to Atlantis · Issue #386 · roboll/helmfile

A workflow similar to Atlantis for terraform would be a killer feature for helmfile - https://github.com/runatlantis/atlantis/

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yes, we are still using atlantis (and starting to poc terraform cloud)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

which is why all this is extra exciting.

1
mumoshu avatar
mumoshu

(wow a year old issue

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we’re using helmfile with atlantis, using a custom step

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

…but would rather just keep it vanilla terraform

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

also, excited about the possibility of relying on fewer ENVs

mumoshu avatar
mumoshu

that’s cool. i tried it but ended up building my own chatops solution w/ a github app that triggers circleci job according to the body of comments

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

and mod + variant?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

did you do the github app b/c for #chatops using #github-actions is too slow?

mumoshu avatar
mumoshu

the chatops solution was github app + variant + helmfile/eksctl, but we have an another pipeline that continually updates things like base docker image tags with mod + variant

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ok

mumoshu avatar
mumoshu

did you do the github app b/c for #chatops using #github-actions is too slow? no. but i think it works. we’re planning to move everything to github actions

mumoshu avatar
mumoshu

btw i was wondering if we can create an “oneshot” version of atlantis, that is executed on each github action run

mumoshu avatar
mumoshu

so that we don’t need to host atlantis..

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

hah! that would be cool

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think it might be worth opening a feature request for that one

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

see what the receptiveness will be

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

The key is to get buy-in from luke

1
mumoshu avatar
mumoshu

okay i will! thanks for the input

2019-11-29

    keyboard_arrow_up