#helmfile (2019-03)

https://github.com/roboll/helmfile

Questions and discussion around helmfile https://github.com/roboll/helmfile and https://github.com/cloudposse/helmfiles Archive: https://archive.sweetops.com/helmfile/

2019-03-31

2019-03-30

mumoshu avatar
mumoshu

thx! i’ll add helmfile destroy in my next sprint on helmfile

Erik Osterman avatar
Erik Osterman

….which would operate only on (installed: true) ?

mumoshu avatar
mumoshu

no, i wasn’t considering to do so! im interested in what would be the benefit, though

Erik Osterman avatar
Erik Osterman

was thinking that was maybe the closest to terraform destroy?

mumoshu avatar
mumoshu

trying to understand

Erik Osterman avatar
Erik Osterman

so “that with a state of installed should be destroyed”

mumoshu avatar
mumoshu

ah thats an interesting idea! and sounds feasible

mumoshu avatar
mumoshu

i’m thinking if we need ignored: true that is different than installed: true to express it

mumoshu avatar
mumoshu

hm, maybe an overkill…

Erik Osterman avatar
Erik Osterman

since the state in helmfile is not external (e.g. helmfile.state) but instead defined directly inside of the helmfile.yaml, that makes the installed state determined by installed: true

Erik Osterman avatar
Erik Osterman

but yea, it gets confusing

Erik Osterman avatar
Erik Osterman

just for flavor….

Erik Osterman avatar
Erik Osterman
Erik Osterman avatar
Erik Osterman

terraform plan has an argument for -destroy which generates a plan to destroy

Erik Osterman avatar
Erik Osterman

that’s kind of like installed: false + helmfile apply

:--1:1
Erik Osterman avatar
Erik Osterman

Erik Osterman avatar
Erik Osterman

or another way of thinking about it:

Erik Osterman avatar
Erik Osterman

helmfile apply + installed: false == helmfile destroy + installed: true

mumoshu avatar
mumoshu

that equation does help me get the idea

Erik Osterman avatar
Erik Osterman

now, i don’t know if that’s what the community wants

Erik Osterman avatar
Erik Osterman

just logically that makes sense to me

mumoshu avatar
mumoshu

so am i. and i think theres no downside of making it as such

Erik Osterman avatar
Erik Osterman

the nice thing is then terraform destroy undoes exactly what was created from terraform apply

Erik Osterman avatar
Erik Osterman

… which is a nice way to iterate

mumoshu avatar
mumoshu

agreed

Erik Osterman avatar
Erik Osterman

you realize what you’re doing…. you’re making it so we can use something like https://github.com/gosuri/terraform-exec-provider

gosuri/terraform-exec-provider

Contribute to gosuri/terraform-exec-provider development by creating an account on GitHub.

Erik Osterman avatar
Erik Osterman
samsung-cnct/terraform-provider-execute

execute arbitrary commands on Terraform create and destroy - samsung-cnct/terraform-provider-execute

Erik Osterman avatar
Erik Osterman

@Shane ever look into something like this?

Erik Osterman avatar
Erik Osterman

Shane avatar
Shane

I’m familiar with it, but never caught on in my designs.

Erik Osterman avatar
Erik Osterman

we could have a generic provider in terraform that calls command with apply, or destroy and maybe we can use helmfile then with terraform

:--1:1
mumoshu avatar
mumoshu

thought if installed: true better be ignored: false or enabled: true but turned out either of them fits my mental model

2019-03-28

mumoshu avatar
mumoshu

shouldn’t helmfile diff and helm diff hide diffs of secrets by default? please :–1: on this issue if you agree https://github.com/databus23/helm-diff/issues/128

Suppressing secrets by default? · Issue #128 · databus23/helm-diff

helm-diff as of today show diffs of secrets, which results in exposing your secret values to stdout. This seems problematic as when used in CI, the user ends up unexpectedly leaves the secret value…

mumoshu avatar
mumoshu

considering to add helmfile destroy while deprecating helmfile delete. wdyt? https://github.com/roboll/helmfile/issues/511

helmfile delete shoud use --purge as default · Issue #511 · roboll/helmfile

I never understood why you would not want to purge everything upon delete. it seems that is may be used for troubleshooting but if you want to delete a release why would you want to troubleshoot an…

:--1:1
aknysh avatar
aknysh

agree about --purge

2019-03-27

2019-03-26

2019-03-25

Abhsihek Jaisingh avatar
Abhsihek Jaisingh

Is there any performance difference between helmfile sync & helmfile apply, like if I am using helmfile in my CI running on every commit. Which should be more preferable?

mumoshu avatar
mumoshu

apply is preferable, as it syncs only when there are changes to be applied also see https://github.com/roboll/helmfile/issues/205#issuecomment-411811549

Only sync release if there is a difference · Issue #205 · roboll/helmfile

Any possibility of adding a feature to run the sync for releases in the helmfile that have differences. With the feature of helm diff I would imagine this would be something that would fit?

Abhsihek Jaisingh avatar
Abhsihek Jaisingh

Agreed, but is helmfile apply any different from just helmfile diff && helmfile sync?

mumoshu avatar
mumoshu

yep, the latter creates new helm releases even for unchanged ones, afaik

Abhsihek Jaisingh avatar
Abhsihek Jaisingh

@mumoshu got it, thanks github-check-mark

Erik Osterman avatar
Erik Osterman

in a CI/CD context, helmfile apply coupled with the intalled flag is the way to go

:--1:1

2019-03-24

2019-03-22

mumoshu avatar
mumoshu

@Erik Osterman Can I add a link to this channel in the helmfile readme?

Erik Osterman avatar
Erik Osterman

@mumoshu please do!

Erik Osterman avatar
Erik Osterman

We also have an archive of the channel publicly available here: https://archive.sweetops.com/helmfile/

helmfile

SweetOps is a collaborative DevOps community. We welcome engineers from around the world of all skill levels, backgrounds, and experience to join us! This is the best place to talk shop, ask questions, solicit feedback, and work together as a community to build sweet infrastructure.

Shane avatar
Shane

@mumoshu or anyone else I’m getting more interested in this or something similar due to some requests from my teams - https://github.com/roboll/helmfile/issues/483

Lock file · Issue #483 · roboll/helmfile

Summary In order to maintain predictable deployments, as developer I want to generate and use lock file for all chart versions retrieved from a helmfile. Rationale We have an environment git repo f…

mumoshu avatar
mumoshu
Lock file · Issue #483 · roboll/helmfile

Summary In order to maintain predictable deployments, as developer I want to generate and use lock file for all chart versions retrieved from a helmfile. Rationale We have an environment git repo f…

mumoshu avatar
mumoshu

Perhaps we can import the relevant helm pkg from the upstream, so that we don’t need to shell-out to helm search..? But it may not worth efforts

Shane avatar
Shane

If we could decide on a design that also achieves what I need to do I can probably do the PR.

Erik Osterman avatar
Erik Osterman

I think a locking mechanism with helm is desperately needed

Erik Osterman avatar
Erik Osterman

i think a locking mechanism in helmfile would help, but where we need it most is when we deploy our own apps, which can happen dozens of times a day

Erik Osterman avatar
Erik Osterman

(E.g. unlimited staging environments)

Erik Osterman avatar
Erik Osterman

the problem is if helm is executed twice in short succession to “upgade” an environment, the outcome is unpredictable

Erik Osterman avatar
Erik Osterman

we use helmfile for those deployments as well, however the pipeline is different

Erik Osterman avatar
Erik Osterman

and the pipeline isn’t always executed by atlantis; sometime codefresh

Erik Osterman avatar
Erik Osterman

so I see the actual fix as being some kind of locking mechanism that is in k8s ,not on the filesystem

Erik Osterman avatar
Erik Osterman

@Shane

Shane avatar
Shane

a helm upgrade is predictable if you don’t run repo update

Shane avatar
Shane

but ya in codefresh you would need to get a new index

Erik Osterman avatar
Erik Osterman

to concurrent helm upgrades on the same release is predictable?

Erik Osterman avatar
Erik Osterman

where is the locking happening?

Shane avatar
Shane

Whenever we run helm standalone we always specify a version so we don’t have your issues.

Erik Osterman avatar
Erik Osterman

e.g. helm upgrade --wait, i see that affected

Shane avatar
Shane

interesting never had a usecase for a concurrent upgrade against the same helm release

Erik Osterman avatar
Erik Osterman

well, we do not want concurrent upgrades

Erik Osterman avatar
Erik Osterman

just if we are deploying an app on every commit to a branch, those can happen very close to each other

Erik Osterman avatar
Erik Osterman

one upgrade starts before the other one finishes and they both are in --wait state

Shane avatar
Shane

ahh ya, that’s an entirely different issue than what a lockfile would accomplish

Shane avatar
Shane

You want helm to block.

Erik Osterman avatar
Erik Osterman

(or helmfile)

Erik Osterman avatar
Erik Osterman

but in a distributed manner

Shane avatar
Shane

That will also be a issue with helm v3

1
Shane avatar
Shane

I do agree a change lock should be available.

Shane avatar
Shane

which would only make sense inside of k8s

:--1:1
Erik Osterman avatar
Erik Osterman

we do all our app deploys also with helmfile; we call it the “helm cartridge” strategy. basically with every app, we ship a helmfile.yaml inside of it that describes how to deploy the app.

Erik Osterman avatar
Erik Osterman

“inside of it” = “inside of the docker image”

Erik Osterman avatar
Erik Osterman

Erik Osterman avatar
Erik Osterman

that said, i don’t personally have any complaints about adding a lock file

Erik Osterman avatar
Erik Osterman

…and we’ll probably use the flag if it exists

Erik Osterman avatar
Erik Osterman

just it doesn’t solve all our headaches

Erik Osterman avatar
Erik Osterman

Erik Osterman avatar
Erik Osterman

I think what would solve our problems is terraform-helmfile-provider

Erik Osterman avatar
Erik Osterman

then we can piggy back on all the capabilities of terraform

Erik Osterman avatar
Erik Osterman

and all the power/flexibility of helmfile

Erik Osterman avatar
Erik Osterman

and we’d get locking to boot (since terraform already supports state locking)

Erik Osterman avatar
Erik Osterman

@mumoshu first planted this seed in my head

Erik Osterman avatar
Erik Osterman

@Shane how’s that for a weekend project?

Shane avatar
Shane

haha

Shane avatar
Shane

terraform solves your locking problem, but I personally don’t know if I see the value in other things terraform can do.

Erik Osterman avatar
Erik Osterman

soooooo what about creating that IAM role for your service so it can write to S3

Erik Osterman avatar
Erik Osterman

what about creating that DNS zone for your external-dns

Erik Osterman avatar
Erik Osterman

what about creating that ElasticSearch cluster for your kibana UI

Erik Osterman avatar
Erik Osterman

it’s interesting if we can provision the backing services needed to deploy things in k8s

Erik Osterman avatar
Erik Osterman

k8s handles a very narrow slice of automation, the rest is handled by tf

Erik Osterman avatar
Erik Osterman
07:52:41 PM
Erik Osterman avatar
Erik Osterman
07:52:46 PM
Erik Osterman avatar
Erik Osterman
07:52:53 PM
Erik Osterman avatar
Erik Osterman
07:53:05 PM
Erik Osterman avatar
Erik Osterman

there are a few examples of where we wrote the terraform modules for provisioning the backing services needed by apps deployed with `helmfile

Shane avatar
Shane

Sure, but no specific reason to couple terraform infrastructure and terraform for services. You certainly can, but does not mean they need to be in one location.

Shane avatar
Shane

Essentially we have a process that runs terraform and a second process that runs helmfile.

Erik Osterman avatar
Erik Osterman

yea, we do something similar

Shane avatar
Shane

certainly coupling them is another option.

Erik Osterman avatar
Erik Osterman

just orchestrating the relationships between them is not “solved”

Shane avatar
Shane

But from our design I don’t view it as preferred to couple them.

Shane avatar
Shane

If anything I would rather helmfile pull from a terraform state output.

Erik Osterman avatar
Erik Osterman

what do you think about Terraform Operators?

Erik Osterman avatar
Erik Osterman
rancher/terraform-operator

Use K8s to Run Terraform. Contribute to rancher/terraform-operator development by creating an account on GitHub.

Erik Osterman avatar
Erik Osterman

(experimental)

Shane avatar
Shane

I think they are a massive ton of work to actually get functional

Shane avatar
Shane

due to all of the migration edge cases where it would destroy your infrastructure.

Erik Osterman avatar
Erik Osterman

lol

Erik Osterman avatar
Erik Osterman

yes

Shane avatar
Shane

I essentially wrote one for provisioning kubernetes, rds, s3 and a few other resources at my last job.

Shane avatar
Shane

It was essentially a operator to provision cloud resources for our platform

Shane avatar
Shane

It was …..painful

Shane avatar
Shane

We essentially used terraform in the background, but we had to have specific migration plans when terraform was not backwards compatible between 2 versions.

Shane avatar
Shane

overall we decided that using terraform to do that type of work was a disaster for us.

Erik Osterman avatar
Erik Osterman

wow, interesting you already went down that path

Shane avatar
Shane

it works fine if you have well worn modules, but it’s not very good for adhoc oh I need to change X

Shane avatar
Shane

not like you can just rename resources and apply the terraform

Shane avatar
Shane

you will break everything at least for the time being

Erik Osterman avatar
Erik Osterman

i was concerned about those points. so basically, having purpose built operators in k8s might work, but a generalized operator for tf might be impractical for the reasons you mentioned

Erik Osterman avatar
Erik Osterman

purpose built operator = etcd operator, mysql operator, etc

Shane avatar
Shane

I took a interview with these guys recently and it sounds like they are doing exactly what you are talking about just not in terraform - https://crossplane.io/

Crossplane

The open source multicloud control plane.

Erik Osterman avatar
Erik Osterman

that looks interesting

Erik Osterman avatar
Erik Osterman

I’m torn. I think what we want to achive is hard to achieve e2e in terraform

Shane avatar
Shane

ya, they are very very early. I hope they can accomplish what they are aiming for it would be great.

Erik Osterman avatar
Erik Osterman

a higher order language is probably required.

Shane avatar
Shane

ya, which is why they are not doing it in terraform as the evolution is to hard.

Erik Osterman avatar
Erik Osterman
Stakater

Kubernetes Expert! Stakater offers companies a highway to Kubernetes adoption for their DevSecOps automation - [✩Star] if you’re using it! - Stakater

Erik Osterman avatar
Erik Osterman

have you seen them?

Shane avatar
Shane

npe

Erik Osterman avatar
Erik Osterman

they are also like cloudposse

Erik Osterman avatar
Erik Osterman

more k8s focused

Erik Osterman avatar
Erik Osterman

some interesting projects, all open source

2019-03-21

mumoshu avatar
mumoshu

FYI: Use raw/incubator to add adhoc templated resources to your helm release

https://github.com/roboll/helmfile/issues/494#issuecomment-474697430

[Feature Request] Add local Helm Chart templates to repository chart release · Issue #494 · roboll/helmfile

Let me start with describing the issue I am trying to solve. Many (official) Helm charts allow you to specify annotations on object descriptions (e.g. ingress descriptions). Ingress controllers sup…

Erik Osterman avatar
Erik Osterman

Yes love that chart!

Erik Osterman avatar
Erik Osterman

We use it too

:--1:1
Erik Osterman avatar
Erik Osterman
cloudposse/helmfiles

Comprehensive Distribution of Helmfiles. Works with helmfile.d - cloudposse/helmfiles

mumoshu avatar
mumoshu

I think I’ve firstly discovered the chart reading your commits made somewhere in cloudposse-related repo thx about that

mumoshu avatar
mumoshu

i’ve added templates setting to the chart just now, which helps creating an adhoc chart thats creates a k8s secret with helm-secrets https://github.com/mumoshu/charts/blob/56d5b63d2998bcd519b52639b586d896b9016633/incubator/raw/README.md#templated-resources

mumoshu/charts

Curated applications for Kubernetes. Contribute to mumoshu/charts development by creating an account on GitHub.

Erik Osterman avatar
Erik Osterman

Brilliant!

Erik Osterman avatar
Erik Osterman

@Igor Rodionov

Erik Osterman avatar
Erik Osterman

So this came up again this week. We are now running Helmfile under Atlantis for a customer

Erik Osterman avatar
Erik Osterman

For now, we are downloading a tarball from GitHub, but would love to do remote sources

Erik Osterman avatar
Erik Osterman

Thinking maybe to have @aknysh submit a PR

Erik Osterman avatar
Erik Osterman

Any suggestions for him?

mumoshu avatar
mumoshu

I think we have potential need to support a lot of sources

Erik Osterman avatar
Erik Osterman

Our objective is to version pin Helmfiles like we do terraform modules

mumoshu avatar
mumoshu

Sounds great :–1:

Erik Osterman avatar
Erik Osterman

The risk is of course it is incompatible with some capabilities of Helmfile

Erik Osterman avatar
Erik Osterman

Is that acceptable?

mumoshu avatar
mumoshu

Do you have any concrete example(s) about the incompatibility? (I had none

Erik Osterman avatar
Erik Osterman

No… not immediately

1
Erik Osterman avatar
Erik Osterman
Erik Osterman avatar
Erik Osterman

This was what I was thinking of…

mumoshu avatar
mumoshu

Let me recall..

mumoshu avatar
mumoshu

Ah, so we usually have a companion values.yaml(.gotmpl) files that is intended to be stored and used along with the referring helmfile.yaml

mumoshu avatar
mumoshu

in certain cases you want to read a remote helmfile.yaml, you also want companion values.yaml files to be fetched and used altogether

Erik Osterman avatar
Erik Osterman

Could we generalize what it means to “open and read” a file?

Erik Osterman avatar
Erik Osterman

So that a file could be really just anywhere

Erik Osterman avatar
Erik Osterman

E.g. use curl

Erik Osterman avatar
Erik Osterman

Which supports every scheme imaginable

Erik Osterman avatar
Erik Osterman

libcurl

Erik Osterman avatar
Erik Osterman

If no scheme specified, we default to file://

mumoshu avatar
mumoshu

sounds good

Erik Osterman avatar
Erik Osterman

@aknysh project for you

mumoshu avatar
mumoshu

so if you specify <https://example.com/foo/bar/helmfile.yaml>, what might happen?

mumoshu avatar
mumoshu

downloading only helmfile.yaml results in you’re unable to read any dependent values.yaml(.gotmpl) from the remote helmfile.yaml

Erik Osterman avatar
Erik Osterman

Then then it is up to the remote Helmfile to use a well formed url if it should be a companion

Erik Osterman avatar
Erik Osterman

Or if not, then it would use the local file

mumoshu avatar
mumoshu

so do we need to download <https://example.com/foo/bar>, extract everything into a local dir, then reads helmfile.yaml in it?

aknysh avatar
aknysh

Thanks Erik :)

Erik Osterman avatar
Erik Osterman

So for example: referencing “values.yaml.gotmpl” maps to “file://./values.yaml.gotmpl”

Erik Osterman avatar
Erik Osterman

(On my phone)

mumoshu avatar
mumoshu


Then then it is up to the remote Helmfile to use a well formed url if it should be a companion

does that mean you need to, in your remote helmfile.yaml, turn every path to values.yaml to something like <https://path/to/your/values.yaml> ?

Erik Osterman avatar
Erik Osterman

Yes, if the implication is that all deps are remote

Erik Osterman avatar
Erik Osterman

But for values, that’s a perfect example of what we would usually customize per customer

Erik Osterman avatar
Erik Osterman

We don’t want that to be remote

Erik Osterman avatar
Erik Osterman

That’s their flavor

mumoshu avatar
mumoshu

hmm.. i’m still unsure

Erik Osterman avatar
Erik Osterman
cloudposse/helmfiles

Comprehensive Distribution of Helmfiles. Works with helmfile.d - cloudposse/helmfiles

Erik Osterman avatar
Erik Osterman

So here is how we use it

mumoshu avatar
mumoshu

would your customer customizes the remote helmfile.yaml fetched by helmfile?

Erik Osterman avatar
Erik Osterman

We provide these sample overrides

mumoshu avatar
mumoshu

or just values.yaml?

Erik Osterman avatar
Erik Osterman

But we would never want these defined by us

Erik Osterman avatar
Erik Osterman

These are what end users ultimate set to customize

mumoshu avatar
mumoshu

if that’s the latter, how do your customer pass the custom values.yaml to the remote helmfile.yaml without forking it?

Erik Osterman avatar
Erik Osterman

So the remote Helmfile loads a local values file

Erik Osterman avatar
Erik Osterman

(I don’t fully understand the code implications of what I am saying)

mumoshu avatar
mumoshu

helmfile finds values.yaml relative to the helmfile.yaml

mumoshu avatar
mumoshu

as of today.

Erik Osterman avatar
Erik Osterman

Oh

Erik Osterman avatar
Erik Osterman

Yes yes

mumoshu avatar
mumoshu

for reproducibility of helmfile deployment

Erik Osterman avatar
Erik Osterman

But the root Helmfile or each included Helmfile?

Erik Osterman avatar
Erik Osterman

I see what you are getting at.

Erik Osterman avatar
Erik Osterman

I just always assumed it was relative to the root Helmfile or pwd

Erik Osterman avatar
Erik Osterman

I look at the values.yaml like terraform.tfvars

Erik Osterman avatar
Erik Osterman

So I guess I see it all relative to my current working directory

mumoshu avatar
mumoshu


But the root Helmfile or each included Helmfile?

each included helmfile.yaml

Erik Osterman avatar
Erik Osterman

Oh

Erik Osterman avatar
Erik Osterman

Yea not sure how to reconcile that

Erik Osterman avatar
Erik Osterman

That gets messy

Erik Osterman avatar
Erik Osterman

I mean, I get why it works that way today for consistency

mumoshu avatar
mumoshu

that way, you can even helmfile sync the sub-helmfile alone

Erik Osterman avatar
Erik Osterman

Yes

Erik Osterman avatar
Erik Osterman

Maybe a new keyword

Erik Osterman avatar
Erik Osterman

Like base_url

mumoshu avatar
mumoshu

but i do understand it isn’t intuitive to some people, and it doesn’t work straightforward when you want a library helmfile.yaml

Erik Osterman avatar
Erik Osterman

(Thinking like the base tag in html)

mumoshu avatar
mumoshu

yep, a new keyword may work

Erik Osterman avatar
Erik Osterman

@aknysh sounds like we have some more thinking to do on this

aknysh avatar
aknysh

ok let me re-read all the above

1
mumoshu avatar
mumoshu

do we probably need a way to override helmfile env..?

helmfiles:
- source: <https://path/to/helmfile.yaml>
   overrides:
      environments:
         customprod:
         - myownprod.yaml

so that helmfile --environment customprod helmfile.yaml works against the remote helmfile.yaml, even the customprod isn’t defined in the remote helmfile.yaml?

mumoshu avatar
mumoshu

this defeats the reproducibility of sub-helmfiles. but does allow us to use the remote helmfile.yaml as a reusable, customizable package

mumoshu avatar
mumoshu

like a helm chart

Erik Osterman avatar
Erik Osterman

@Igor Rodionov

mumoshu avatar
mumoshu

Created an issue to track it before I forget https://github.com/roboll/helmfile/issues/523

feat: Allow environment values to be overrode when referring sub-helmfile · Issue #523 · roboll/helmfile

This is similar to #361, but for helmfile.yaml. You can write helmfile.yaml containing one or more sub-helmfiles that is processed before releases, in helmfiles: environments: default: values: - en…

mumoshu avatar
mumoshu

Anyway, what I think recently is: We generally need abilities to (1)pin and (2)update any dependencies. a semver in the source spec, and a command to fetch the latest ver that accomodates the version constraint would be useful

mumoshu avatar
mumoshu

after that, we can add any datasource users want…

mumoshu avatar
mumoshu

as long as it supports version pinning and updating based on semver

mumoshu avatar
mumoshu

anydep was my attempt to achieve that https://github.com/mumoshu/anydep

mumoshu/anydep

General-purpose project/application dependency manager - mumoshu/anydep

Erik Osterman avatar
Erik Osterman

Oh yes, I recall you mentioning this to me

    keyboard_arrow_up