#helmfile (2019-03)
Questions and discussion around helmfile https://github.com/roboll/helmfile and https://github.com/cloudposse/helmfiles
Archive: https://archive.sweetops.com/helmfile/
2019-03-21
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
FYI: Use raw/incubator
to add adhoc templated resources to your helm release
https://github.com/roboll/helmfile/issues/494#issuecomment-474697430
Let me start with describing the issue I am trying to solve. Many (official) Helm charts allow you to specify annotations on object descriptions (e.g. ingress descriptions). Ingress controllers sup…
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Yes love that chart!
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Comprehensive Distribution of Helmfiles. Works with helmfile.d
- cloudposse/helmfiles
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
I think I’ve firstly discovered the chart reading your commits made somewhere in cloudposse-related repo thx about that
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
i’ve added templates
setting to the chart just now, which helps creating an adhoc chart thats creates a k8s secret with helm-secrets https://github.com/mumoshu/charts/blob/56d5b63d2998bcd519b52639b586d896b9016633/incubator/raw/README.md#templated-resources
Curated applications for Kubernetes. Contribute to mumoshu/charts development by creating an account on GitHub.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Brilliant!
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
@Igor Rodionov
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
So this came up again this week. We are now running Helmfile under Atlantis for a customer
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
For now, we are downloading a tarball from GitHub, but would love to do remote sources
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Thinking maybe to have @Andriy Knysh (Cloud Posse) submit a PR
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Any suggestions for him?
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
I think we have potential need to support a lot of sources
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Our objective is to version pin Helmfiles like we do terraform modules
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
Sounds great
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
The risk is of course it is incompatible with some capabilities of Helmfile
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Is that acceptable?
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
Do you have any concrete example(s) about the incompatibility? (I had none
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
This was what I was thinking of…
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
Let me recall..
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
Ah, so we usually have a companion values.yaml(.gotmpl)
files that is intended to be stored and used along with the referring helmfile.yaml
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
in certain cases you want to read a remote helmfile.yaml, you also want companion values.yaml files to be fetched and used altogether
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Could we generalize what it means to “open and read” a file?
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
So that a file could be really just anywhere
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
E.g. use curl
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Which supports every scheme imaginable
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
libcurl
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
libcurl go bingding
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
If no scheme specified, we default to file://
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
sounds good
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
@Andriy Knysh (Cloud Posse) project for you
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
so if you specify <https://example.com/foo/bar/helmfile.yaml>
, what might happen?
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
downloading only helmfile.yaml
results in you’re unable to read any dependent values.yaml(.gotmpl)
from the remote helmfile.yaml
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Then then it is up to the remote Helmfile to use a well formed url if it should be a companion
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Or if not, then it would use the local file
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
so do we need to download <https://example.com/foo/bar>
, extract everything into a local dir, then reads helmfile.yaml
in it?
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
Thanks Erik :)
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
So for example: referencing “values.yaml.gotmpl” maps to “file://./values.yaml.gotmpl”
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
(On my phone)
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
Then then it is up to the remote Helmfile to use a well formed url if it should be a companion
does that mean you need to, in your remote helmfile.yaml, turn every path to values.yaml to something like <https://path/to/your/values.yaml>
?
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Yes, if the implication is that all deps are remote
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
But for values, that’s a perfect example of what we would usually customize per customer
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
We don’t want that to be remote
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
That’s their flavor
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
hmm.. i’m still unsure
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Comprehensive Distribution of Helmfiles. Works with helmfile.d
- cloudposse/helmfiles
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
So here is how we use it
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
would your customer customizes the remote helmfile.yaml fetched by helmfile?
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
We provide these sample overrides
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
or just values.yaml?
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
But we would never want these defined by us
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
These are what end users ultimate set to customize
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
if that’s the latter, how do your customer pass the custom values.yaml to the remote helmfile.yaml without forking it?
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
So the remote Helmfile loads a local values file
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
(I don’t fully understand the code implications of what I am saying)
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
helmfile finds values.yaml relative to the helmfile.yaml
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
as of today.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Oh
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Yes yes
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
for reproducibility of helmfile deployment
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
But the root Helmfile or each included Helmfile?
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
I see what you are getting at.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
I just always assumed it was relative to the root Helmfile or pwd
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
I look at the values.yaml like terraform.tfvars
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
So I guess I see it all relative to my current working directory
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
But the root Helmfile or each included Helmfile?
each included helmfile.yaml
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Oh
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Yea not sure how to reconcile that
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
That gets messy
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
I mean, I get why it works that way today for consistency
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
that way, you can even helmfile sync
the sub-helmfile alone
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Yes
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Maybe a new keyword
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Like base_url
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
but i do understand it isn’t intuitive to some people, and it doesn’t work straightforward when you want a library
helmfile.yaml
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
(Thinking like the base tag in html)
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
yep, a new keyword may work
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
@Andriy Knysh (Cloud Posse) sounds like we have some more thinking to do on this
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
do we probably need a way to override helmfile env..?
helmfiles:
- source: <https://path/to/helmfile.yaml>
overrides:
environments:
customprod:
- myownprod.yaml
so that helmfile --environment customprod helmfile.yaml
works against the remote helmfile.yaml, even the customprod
isn’t defined in the remote helmfile.yaml?
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
this defeats the reproducibility of sub-helmfiles. but does allow us to use the remote helmfile.yaml as a reusable, customizable package
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
like a helm chart
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
@Igor Rodionov
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
Created an issue to track it before I forget https://github.com/roboll/helmfile/issues/523
This is similar to #361, but for helmfile.yaml. You can write helmfile.yaml containing one or more sub-helmfiles that is processed before releases, in helmfiles: environments: default: values: - en…
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
Anyway, what I think recently is: We generally need abilities to (1)pin and (2)update any dependencies. a semver in the source spec, and a command to fetch the latest ver that accomodates the version constraint would be useful
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
after that, we can add any datasource users want…
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
as long as it supports version pinning and updating based on semver
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
anydep was my attempt to achieve that https://github.com/mumoshu/anydep
General-purpose project/application dependency manager - mumoshu/anydep
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Oh yes, I recall you mentioning this to me
2019-03-22
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
@Erik Osterman (Cloud Posse) Can I add a link to this channel in the helmfile readme?
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
@mumoshu please do!
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
We also have an archive of the channel publicly available here: https://archive.sweetops.com/helmfile/
SweetOps is a collaborative DevOps community. We welcome engineers from around the world of all skill levels, backgrounds, and experience to join us! This is the best place to talk shop, ask questions, solicit feedback, and work together as a community to build sweet infrastructure.
![Shane avatar](https://avatars.slack-edge.com/2018-11-16/481069875217_6fbbee537c0736f89e7f_72.png)
@mumoshu or anyone else I’m getting more interested in this or something similar due to some requests from my teams - https://github.com/roboll/helmfile/issues/483
Summary In order to maintain predictable deployments, as developer I want to generate and use lock file for all chart versions retrieved from a helmfile. Rationale We have an environment git repo f…
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
I’ve made a quick proposal for it!
https://github.com/roboll/helmfile/issues/483#issuecomment-476030604
Summary In order to maintain predictable deployments, as developer I want to generate and use lock file for all chart versions retrieved from a helmfile. Rationale We have an environment git repo f…
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
Perhaps we can import the relevant helm pkg from the upstream, so that we don’t need to shell-out to helm search
..? But it may not worth efforts
![Shane avatar](https://avatars.slack-edge.com/2018-11-16/481069875217_6fbbee537c0736f89e7f_72.png)
If we could decide on a design that also achieves what I need to do I can probably do the PR.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
I think a locking mechanism with helm is desperately needed
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
i think a locking mechanism in helmfile
would help, but where we need it most is when we deploy our own apps, which can happen dozens of times a day
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
(E.g. unlimited staging environments)
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
the problem is if helm
is executed twice in short succession to “upgade” an environment, the outcome is unpredictable
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
we use helmfile
for those deployments as well, however the pipeline is different
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
and the pipeline isn’t always executed by atlantis
; sometime codefresh
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
so I see the actual fix as being some kind of locking mechanism that is in k8s ,not on the filesystem
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
@Shane
![Shane avatar](https://avatars.slack-edge.com/2018-11-16/481069875217_6fbbee537c0736f89e7f_72.png)
a helm upgrade
is predictable if you don’t run repo update
![Shane avatar](https://avatars.slack-edge.com/2018-11-16/481069875217_6fbbee537c0736f89e7f_72.png)
but ya in codefresh you would need to get a new index
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
to concurrent helm upgrades on the same release is predictable?
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
where is the locking happening?
![Shane avatar](https://avatars.slack-edge.com/2018-11-16/481069875217_6fbbee537c0736f89e7f_72.png)
Whenever we run helm
standalone we always specify a version so we don’t have your issues.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
e.g. helm upgrade --wait
, i see that affected
![Shane avatar](https://avatars.slack-edge.com/2018-11-16/481069875217_6fbbee537c0736f89e7f_72.png)
interesting never had a usecase for a concurrent upgrade against the same helm release
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
well, we do not want concurrent upgrades
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
just if we are deploying an app on every commit to a branch, those can happen very close to each other
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
one upgrade starts before the other one finishes and they both are in --wait
state
![Shane avatar](https://avatars.slack-edge.com/2018-11-16/481069875217_6fbbee537c0736f89e7f_72.png)
ahh ya, that’s an entirely different issue than what a lockfile would accomplish
![Shane avatar](https://avatars.slack-edge.com/2018-11-16/481069875217_6fbbee537c0736f89e7f_72.png)
You want helm to block.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
(or helmfile
)
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
but in a distributed manner
![Shane avatar](https://avatars.slack-edge.com/2018-11-16/481069875217_6fbbee537c0736f89e7f_72.png)
![Shane avatar](https://avatars.slack-edge.com/2018-11-16/481069875217_6fbbee537c0736f89e7f_72.png)
I do agree a change lock should be available.
![Shane avatar](https://avatars.slack-edge.com/2018-11-16/481069875217_6fbbee537c0736f89e7f_72.png)
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
we do all our app deploys also with helmfile
; we call it the “helm cartridge” strategy. basically with every app, we ship a helmfile.yaml
inside of it that describes how to deploy the app.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
“inside of it” = “inside of the docker image”
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
that said, i don’t personally have any complaints about adding a lock file
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
…and we’ll probably use the flag if it exists
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
just it doesn’t solve all our headaches
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
I think what would solve our problems is terraform-helmfile-provider
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
then we can piggy back on all the capabilities of terraform
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
and all the power/flexibility of helmfile
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
and we’d get locking to boot (since terraform already supports state locking)
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
@mumoshu first planted this seed in my head
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
@Shane how’s that for a weekend project?
![Shane avatar](https://avatars.slack-edge.com/2018-11-16/481069875217_6fbbee537c0736f89e7f_72.png)
haha
![Shane avatar](https://avatars.slack-edge.com/2018-11-16/481069875217_6fbbee537c0736f89e7f_72.png)
terraform solves your locking problem, but I personally don’t know if I see the value in other things terraform can do.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
soooooo what about creating that IAM role for your service so it can write to S3
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
what about creating that DNS zone for your external-dns
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
what about creating that ElasticSearch cluster for your kibana UI
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
it’s interesting if we can provision the backing services needed to deploy things in k8s
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
k8s handles a very narrow slice of automation, the rest is handled by tf
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
there are a few examples of where we wrote the terraform modules for provisioning the backing services needed by apps deployed with `helmfile
![Shane avatar](https://avatars.slack-edge.com/2018-11-16/481069875217_6fbbee537c0736f89e7f_72.png)
Sure, but no specific reason to couple terraform infrastructure and terraform for services. You certainly can, but does not mean they need to be in one location.
![Shane avatar](https://avatars.slack-edge.com/2018-11-16/481069875217_6fbbee537c0736f89e7f_72.png)
Essentially we have a process that runs terraform and a second process that runs helmfile.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
yea, we do something similar
![Shane avatar](https://avatars.slack-edge.com/2018-11-16/481069875217_6fbbee537c0736f89e7f_72.png)
certainly coupling them is another option.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
just orchestrating the relationships between them is not “solved”
![Shane avatar](https://avatars.slack-edge.com/2018-11-16/481069875217_6fbbee537c0736f89e7f_72.png)
But from our design I don’t view it as preferred to couple them.
![Shane avatar](https://avatars.slack-edge.com/2018-11-16/481069875217_6fbbee537c0736f89e7f_72.png)
If anything I would rather helmfile pull from a terraform state output.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
what do you think about Terraform Operators?
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Use K8s to Run Terraform. Contribute to rancher/terraform-operator development by creating an account on GitHub.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
(experimental)
![Shane avatar](https://avatars.slack-edge.com/2018-11-16/481069875217_6fbbee537c0736f89e7f_72.png)
I think they are a massive ton of work to actually get functional
![Shane avatar](https://avatars.slack-edge.com/2018-11-16/481069875217_6fbbee537c0736f89e7f_72.png)
due to all of the migration edge cases where it would destroy your infrastructure.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
lol
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
yes
![Shane avatar](https://avatars.slack-edge.com/2018-11-16/481069875217_6fbbee537c0736f89e7f_72.png)
I essentially wrote one for provisioning kubernetes, rds, s3 and a few other resources at my last job.
![Shane avatar](https://avatars.slack-edge.com/2018-11-16/481069875217_6fbbee537c0736f89e7f_72.png)
It was essentially a operator to provision cloud resources for our platform
![Shane avatar](https://avatars.slack-edge.com/2018-11-16/481069875217_6fbbee537c0736f89e7f_72.png)
It was …..painful
![Shane avatar](https://avatars.slack-edge.com/2018-11-16/481069875217_6fbbee537c0736f89e7f_72.png)
We essentially used terraform in the background, but we had to have specific migration plans when terraform was not backwards compatible between 2 versions.
![Shane avatar](https://avatars.slack-edge.com/2018-11-16/481069875217_6fbbee537c0736f89e7f_72.png)
overall we decided that using terraform to do that type of work was a disaster for us.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
wow, interesting you already went down that path
![Shane avatar](https://avatars.slack-edge.com/2018-11-16/481069875217_6fbbee537c0736f89e7f_72.png)
it works fine if you have well worn modules, but it’s not very good for adhoc oh I need to change X
![Shane avatar](https://avatars.slack-edge.com/2018-11-16/481069875217_6fbbee537c0736f89e7f_72.png)
not like you can just rename resources and apply the terraform
![Shane avatar](https://avatars.slack-edge.com/2018-11-16/481069875217_6fbbee537c0736f89e7f_72.png)
you will break everything at least for the time being
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
i was concerned about those points. so basically, having purpose built operators in k8s might work, but a generalized operator for tf might be impractical for the reasons you mentioned
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
purpose built operator = etcd operator, mysql operator, etc
![Shane avatar](https://avatars.slack-edge.com/2018-11-16/481069875217_6fbbee537c0736f89e7f_72.png)
I took a interview with these guys recently and it sounds like they are doing exactly what you are talking about just not in terraform - https://crossplane.io/
The open source multicloud control plane.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
that looks interesting
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
I’m torn. I think what we want to achive is hard to achieve e2e in terraform
![Shane avatar](https://avatars.slack-edge.com/2018-11-16/481069875217_6fbbee537c0736f89e7f_72.png)
ya, they are very very early. I hope they can accomplish what they are aiming for it would be great.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
a higher order language is probably required.
![Shane avatar](https://avatars.slack-edge.com/2018-11-16/481069875217_6fbbee537c0736f89e7f_72.png)
ya, which is why they are not doing it in terraform as the evolution is to hard.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Kubernetes Expert! Stakater offers companies a highway to Kubernetes adoption for their DevSecOps automation - [✩Star] if you’re using it! - Stakater
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
have you seen them?
![Shane avatar](https://avatars.slack-edge.com/2018-11-16/481069875217_6fbbee537c0736f89e7f_72.png)
npe
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
they are also like cloudposse
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
more k8s focused
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
some interesting projects, all open source
2019-03-24
2019-03-25
![Abhsihek Jaisingh avatar](https://secure.gravatar.com/avatar/3f11355e49bca53dab7b593dee15cf4c.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0005-72.png)
Is there any performance difference between helmfile sync
& helmfile apply
, like if I am using helmfile in my CI running on every commit. Which should be more preferable?
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
apply
is preferable, as it sync
s only when there are changes to be applied
also see https://github.com/roboll/helmfile/issues/205#issuecomment-411811549
Any possibility of adding a feature to run the sync for releases in the helmfile that have differences. With the feature of helm diff I would imagine this would be something that would fit?
![Abhsihek Jaisingh avatar](https://secure.gravatar.com/avatar/3f11355e49bca53dab7b593dee15cf4c.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0005-72.png)
Agreed, but is helmfile apply
any different from just helmfile diff && helmfile sync
?
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
yep, the latter creates new helm releases even for unchanged ones, afaik
![Abhsihek Jaisingh avatar](https://secure.gravatar.com/avatar/3f11355e49bca53dab7b593dee15cf4c.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0005-72.png)
@mumoshu got it, thanks
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
in a CI/CD context, helmfile apply
coupled with the intalled
flag is the way to go
2019-03-26
2019-03-27
2019-03-28
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
shouldn’t helmfile diff
and helm diff
hide diffs of secrets by default? please on this issue if you agree
https://github.com/databus23/helm-diff/issues/128
helm-diff as of today show diffs of secrets, which results in exposing your secret values to stdout. This seems problematic as when used in CI, the user ends up unexpectedly leaves the secret value…
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
considering to add helmfile destroy
while deprecating helmfile delete
. wdyt?
https://github.com/roboll/helmfile/issues/511
I never understood why you would not want to purge everything upon delete. it seems that is may be used for troubleshooting but if you want to delete a release why would you want to troubleshoot an…
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
agree about --purge
2019-03-30
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
thx! i’ll add helmfile destroy
in my next sprint on helmfile
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
….which would operate only on (installed: true) ?
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
no, i wasn’t considering to do so! im interested in what would be the benefit, though
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
was thinking that was maybe the closest to terraform destroy
?
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
trying to understand
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
so “that with a state of installed should be destroyed”
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
ah thats an interesting idea! and sounds feasible
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
i’m thinking if we need ignored: true
that is different than installed: true
to express it
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
hm, maybe an overkill…
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
since the state in helmfile
is not external (e.g. helmfile.state
) but instead defined directly inside of the helmfile.yaml
, that makes the installed state determined by installed: true
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
but yea, it gets confusing
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
just for flavor….
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
terraform plan
has an argument for -destroy
which generates a plan to destroy
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
or another way of thinking about it:
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
helmfile apply
+ installed: false
== helmfile destroy
+ installed: true
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
that equation does help me get the idea
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
now, i don’t know if that’s what the community wants
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
just logically that makes sense to me
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
so am i. and i think theres no downside of making it as such
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
the nice thing is then terraform destroy
undoes exactly what was created from terraform apply
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
… which is a nice way to iterate
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
agreed
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
you realize what you’re doing…. you’re making it so we can use something like https://github.com/gosuri/terraform-exec-provider
Contribute to gosuri/terraform-exec-provider development by creating an account on GitHub.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
execute arbitrary commands on Terraform create and destroy - samsung-cnct/terraform-provider-execute
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
@Shane ever look into something like this?
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
![Shane avatar](https://avatars.slack-edge.com/2018-11-16/481069875217_6fbbee537c0736f89e7f_72.png)
I’m familiar with it, but never caught on in my designs.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
we could have a generic provider in terraform that calls command
with apply
, or destroy
and maybe we can use helmfile
then with terraform
![mumoshu avatar](https://secure.gravatar.com/avatar/8e045bf747ca7a90b1d955dc30217271.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png)
thought if installed: true
better be ignored: false
or enabled: true
but turned out either of them fits my mental model