#helmfile (2019-12)
Questions and discussion around helmfile https://github.com/roboll/helmfile and https://github.com/cloudposse/helmfiles
Archive: https://archive.sweetops.com/helmfile/
2019-12-01
2019-12-02
I’ve double checked with the latest v0.94.1, the behaviour is the same as I described in https://github.com/roboll/helmfile/issues/1010. I’ve decided to move it out of the thread about a tf provider as it looks bloated.
@mumoshu binary releases for terraform-provider-helmfile courtesy of @Andriy Knysh (Cloud Posse)
2019-12-03
Am I right that the construction {{ .Environment.Values | get "my.var" "value" }} allows my.var not to be presented in environments files at all?
I was wondering if helmfiles: could be used in helmfile referenced via helmfiles: from the main one. Will check this out.
Yes, it works very well!
Hi. Is there any guide for migrating from helm2 to helm3?
i mean if i have existing releases in datacenters, managed by helmfile, how do i migrate to manage them with helm3.
What event makes needs: succeed? Actually, I’m interested in waiting for the postsync hook to finish.
It seems it respects postsync hooks. Great!
@here helmfile users at #aws-reinvent should connect tonight!
@mumoshu I’ll come say hello!
hey! are you there? i’ll be coming in 10mins or so
its super crowded!
What color shirt are you wearing?
im in navy sweater and brown bottoms
I’ll keep an eye out. I’m grey t-shirt with a pocket.
Sitting at a table.
im around the beer counter
navy sweater brown pants. I just did a loop and I don’t see you. Doing another.
do you have anything to signify you? im now around the table to grab some nacho
No! Thank you! I can’t wait to test the docker ingratiation we discussed.
Regarding of sub-helmfiles. It seems Cloud Posse uses them extensively since they have a collection of dedicated helmfiles. @Erik Osterman (Cloud Posse) how do you, guys, deal with environments? I’ve noticed you don’t have environments: blocks in your dedicated helmfiles. You don’t use them at all?
I would imagine that the concept of “environments” can be different from company to company, it doesn’t make much sense to limit a helmfile meant to be shared across any arbitrary number and design of environments. By excluding that section, you can easily just include the helmfile directly, and configure it through the operating system environment (notice their heavy use of env in their templates) with tools such as direnv
that way, in each of your “environments” you just call those helmfiles without an environment parameter
That’s true.
CP use ‘stage’ IIRC which can see passed as values in some of the helmfiles
Thanks @Alex Siegman and @joshmyers
Yes, the back story though is we got started long before environments existed
But the gist of it is we use remote helmfiles and anything that could vary by “environnent” is actually defined with environment variables
this way we easily source them from chamber
On the one hand, I like that we have a consistent interface for passing settings (environment variables)
on the other, I miss the cleanliness of environments in helmfile where all your settings are in git vs ssm
@Andriy Knysh (Cloud Posse) just got the terraform-helmfile-provider working with terraform cloud
So I’m not sure if we’ll end up using environments that much since we get that out-of-the-box with how we manage terraform.
also, this opens up a tremendous amount of possibilities - passing settings from terraform directly into helmfile
Terraform cloud!?! Talk. To. Me.
I’m yet to hear anything about it other than “it’s bloody expensive”
it’s free for teams <=5
How does config work? Do they solve the many states and dependencies problem?
Aye but that won’t get very far for most orgs :(
2 banks I’ve been at have been quoted in the millions…they are banks though….
Is it very close to Atlantis?
It’s “close” but just way more polished.
The nice thing with Atlantis is you can deploy it with an IAM role
no way to do that with TF Cloud
2019-12-04
Hello, I would like to know what do you think about exposing absolute basePath to current yaml? https://github.com/roboll/helmfile/pull/981
2019-12-05
Does someone know how to disable validation ?
2019-12-09
@mumoshu any thoughts on this? https://github.com/mumoshu/terraform-provider-helmfile/issues/5
can have @Andriy Knysh (Cloud Posse) help contribute it
(we’re struggling to use it e2e on terraform cloud without this)
I can finish implementing and testing it tomorrow, if @mumoshu will not do it tonight :)
Replied, i’d appreciate your contribution and will try to review it asap once submitted!
Add flag to download kubectl and helmfile
would those be literally flags?
any implementation preferences?
not sure. im not familiar with this kind of feature
prior art: atlantis downloads terraform
would adding helmfile_version and helmfile_download_url_template to the resource makes sense?
ya, something like that
Before executing helmfile command, we need to download kubectl and helmfile, and possibly assume a role if provided
All optional
yeah that makes sense. https://sweetops.slack.com/archives/CE5NGCB9Q/p1575937428326200 will allow that
Do we need to also download helm? Helmfile doesn’t do it automatically?
yes, good point - so I think we need helm, helmfile, and kubectl
Yes
And assume role code if it’s provided
got it
(lol, and hrm… maybe this is cascading out of control)
iam assume role?
need helm-diff plugin too
and then helm-diff might have deps
ah
ugh
helm-diff has no deps as it’s a single binary tool written in golang
maybe we generalize it?
package_deps which is a list of binaries by URL
Yes
wouldn’t assume role automatically done today by installing aws cli and using the correct kubeconfig?
package_deps would worth a dedicated terraform provider, right?
and i think it doesn’t match with helmfille providers use-case
as it would typically require multiple versions of helmfile/helm/kubectl binaries to co-exist
Order of operations matters
possibly - just for background, we tried using null_resource with local-exec provisioner to download, but cannot get it to work due to limited ability to control when it’s executed (we need it executed prior to helmfile-provider getting invoked which depends on the binaries)
Terraform cloud can’t even plan without those binaries
yep. i wondered there was some way to explicitly draw a dependency between local-exec and helmfile resourec
apparently it isn’t possible?
so for example, we do this successfully for downloading aws cli in one step and calling it in another, both using local-exec
however, since local-exec is out of phase with terraform-helmfile-provider, we cannot control for the order of execution
so the first time we run it, it works
precisely
ah good to know!
on demand based on version
maybe we should add helmfile_version, helm_version, kubectl_version, helm_diff_version for starter
and implement some other provider to install arbitrary pkg from e.g. apk if necessary, for use with depends_on = [that_provider.some_bin] to draw dependency
ah okay depends_on = [that_provider.some_bin] won’t work anyway
as the provider won’t install it on plan. that’s the same thing as local-exec/null-resource
can one tie into the init phase?
probably. the relevant code would go into https://github.com/mumoshu/terraform-provider-helmfile/blob/2c55c39d5b8d494729f09d2a4ba74f0d187f5452/pkg/tfhelmfile/provider.go#L24
but it is unlikely to provide helmfile_releaseset resources on init
I think a better way would be not to pollute terraform-provider-helmfile with loading external packages, but using data sources b/c they are executed on terraform plan
we can use terraform-provider-shell b/c it already implements data source together with resource https://github.com/scottwinkler/terraform-provider-shell/blob/master/examples/test.tf#L5
or we can add a similar data source to terraform-provider-helmfile
If the query constraint arguments for a data resource refer only to constant values or values that are already known, the data resource will be read and its state updated during Terraform's "refresh" phase, which runs prior to creating a plan
Each data source in turn belongs to a provider, which is a plugin for Terraform that offers a collection of resource types and data sources that most often belong to a single cloud or on-premises infrastructure platform.
Each provider may offer data sources alongside its set of resource types
@mumoshu @Erik Osterman (Cloud Posse) ^
@Andriy Knysh (Cloud Posse) ah, so your idea is based on the fact that datasource is evaluated even at plan time, right?
would it look like this?
data "helmfile_binary" "default" {
version = "0.94.1"
}
data "helm_binary" "default" {
version = "3.0.0"
}
resource "helmfile_releaseset" "myapp" {
helmfile_binary = "${helmfile_binary.default.path}`
helm_binary = "${helm_binary.default.path}`
}
yes!
excellent! that would be much much better than adding version related keys to resources.
this is a good example https://github.com/scottwinkler/terraform-provider-shell/blob/master/examples/test.tf#L5
in the data source, we can hide the fact that it downloads the data (make it more specific, as you showed above)), or we can allow to execute any commands (as terraform-provider-shell does) and make it more generic
let’s hide it for now, as it would be dedicated to helmfile provider’s use-case.
adding optional key like download_url and/or apk_package would be alright
At the moment we are just building a docker container with all the stuff pre-installed. Looking forward to trying a new implementation.
2019-12-10
What do you think?
Do the helmfile concepts make sense in the context of Pulumi with Helm? Here is a Helm example with Pulumi -> https://github.com/pulumi/pulumi-kubernetes#deploying-a-helm-chart
I am thinking about moving to Pulumi becuase of its first class k8s support. As a bonus there is also Helm support.
It would be possible to bridge the terraform-provider-helmfile to Pulumi but I wonder if it will provided value to add another layer.
As much as I like the awesomeness of the Helmfile, aren’t it too much of layers? 1 -> Pulumi 2-> Helm 3-> Helmfile especially in the imperative nature of Pulumi.
hey! i think pulumi works as long as you don’t use helmfile’s features like vault/ssm/secretsmanager support, sub-helmfiles, DAG, etc.
bridging terraform-provider-helmfile to pulumi does sound like too much. i’d prefer implementing a kind of pulumi plugin(how is it called? module? library?) for helmfile
i’ve been long wanted to play more with pulumi so please point me to some pulumi plugin impl. guide so that i can try to build a poc
@mumoshu pulumi has supprt for tf providers and there is a tf provider for helmfile, so I guess the easiest way would to bridge it.
@Vadim Bauer see related discussion here: https://sweetops.slack.com/archives/CQCDCLA1M/p1573246618316100
@mumoshu adds some interesting insights as well
Hi, I have a k8s cluster running, also the kubeconfig is correctly configured. I wanna start playing with helmfile. Could you suggest me a good tutorial in order to start learning ?
maybe try cloudposse’s guide? https://docs.cloudposse.com/tools/helmfile/
2019-12-11
Q: Adding aws-iam-authenticator. Good idea? Bad idea?
I could very well imagine pitch-forks being readied now.
The good would be convenience for a subset of end-users.
The bad would be: What if every cloud vendor’s tools will be added? Image bloat (It’s ~30Mb, not overly huge, but still).
Use case would be using the quay.io images, bringing a kubec config similar to:
apiVersion: v1
preferences: {}
kind: Config
clusters:
- cluster:
server: <https://123123123.yl4.eu-west-1.eks.amazonaws.com>
certificate-authority-data: LOTSOFCIPHERSTUFF
name: apps
contexts:
- context:
cluster: apps
user: apps
name: apps
current-context: apps
users:
- name: apps
user:
exec:
apiVersion: client.authentication.k8s.io/v1alpha1
command: aws-iam-authenticator
args:
- "token"
- "-i"
- "apps"
It would be useful for EKS users. Probably a number of non-EKS AWS users also.
IMO better to add AWS CLI (which lets you do aws eks get-token)
yeah aws cli would be better
AFAIK aws-iam-authenticator hasn’t been deprecated yet, but it should be since all of its functionality has been rolled into aws
could u elaborate a bit more? how does it relate to helmfile?
@mumoshu See thread
sry i don’t understand… are you talking about adding aws-iam-authenticator to where?
The helmfile image.
When using in CI/CD. Alternative would be building own helmfile image that _has_the binary. Or add it to the project and mount it in the CI/CD container.
ah gotcha
you need to build your own adding the official helmfile image as FROM [quay.io/roboll/helmfiel](http://quay.io/roboll/helmfiel):...
it does help aws users but helmfile isn’t aws-specific
Yes, that would make most sense (but has me setting up an ECR, track helmfile releases, etc.).
the next best thing would you contribute a pull request to add aws specific helmfile dockerfile
and we can probably co-maintain it, freely adding aws specific tools like awscli
but not sure if it works for everyone
If piggy-backing on your build pipeline sounds ok to you then that would be very interesting.
actually in my own job i do bundle variant(https://github.com/mumoshu/variant) along with helmfile in the same iamge
so only adding aws-iam-authenticator/awscli doesn’t really help me. does it for you?
Something I’ve been working on, a la Geodesic, that fits here, is https://github.com/dadsgarage/dadsgarage. @Erik Osterman (Cloud Posse) really got me going on the idea of creating one container that has all your tools in it. So that’s what I’m going to be doing
You better believe it will have aws and helmfile in it
yeah i’d recommend that pattern, too
Makes sense indeed to have a one-stop toolbox for all CI/CD tasks, be it helmfile, straight kubectl stuff, or indeed commands that need awscli. Or eksctl …
Lol
bam. helmfile and eksctl issues created, with good-first-issue and help-wanted labels applied
Other Q: I was hoping to toggle certain applications using values defined in the cluster-specific environment. Like this (but that throws a ‘map has no entry’ error):
release:
- name: efs-provisioner
namespace: kube-system
chart: stable/efs-provisioner
version: "0.8.0"
installed: {{ .Environment.Values.efsProvisioner.installed }}
values:
- releases/efs-provisioner.yaml.gotmpl
Is this possible in some way?
A new day, new loads of coffee, fresh perspective. Found it:
releases:
- name: efs-provisioner
namespace: kube-system
chart: stable/efs-provisioner
version: "0.8.0"
installedTemplate: '{{`{{ .Environment.Values.efsProvisioner.installed }}`}}'
values:
- releases/efs-provisioner.yaml.gotmpl
2019-12-12
Hey guys, i’m trying to get the name of the chart as a variable rather than the name of the release:
releases:
- name: external-dns
chart: my-repo/external-dns
namespace: kube-system
version: 1.14.0
installed: true
values:
- "{{`{{ .Release.Chart }}`}}/values-dev.yaml"
Any idea on how I change .Release.Chart to only include the chart name?
replace Chart with Name like so: - “{{{{ .Release.Name }}}}/values-dev.yaml” That should then resolve to “external-dns/values-dev.yaml”
Thanks @Jonathan, but I need the name of the chart, not the name of the release as these two could be different.
Is helm 3 not able to create namespaces? I run into an error whenever helmfile.yaml contains a release targeted at a non-existing namespace. (Using roboll/helmfile:helm3-v0.94.1)
Was reading up on some GitHub discussions. Totally missed that one. Ah, well. Can understand the rationale.
Anyone having clever methods of creating namespace in a cicd set-up?
ns may be created by incubator/raw helm chart, for example.
We create namespaces via Terraform’s kubernetes provider upon cluster creation.. works well. another alternative is a k apply raw yaml with prerequired namespaces
Planned on some way of bootstrapping application namespaces (rbac, default limits, quota’s where applicable). Will likely end up being a mix of kubectl and helm(file).
Lazy question :sweat_smile:, does helmfile support replace("string_here", "to_replace", "replace_with") functionality? I recall it inherits functionality from another project, but I’m spacing out on finding that information.
What’s up Mr. Ryan. How’s you, Chris and Calm.io?
Helmfile supports the Go template extention library Sprig. Check out the string functions specifically for replace : http://masterminds.github.io/sprig/strings.html
"I Am Ryan Smith" | replace "Am" "Was"
@Cameron Boulton the master man with the master plan. My dream team-mate!!
sprig, got it
hahaha
k, found it on the readme just now
(Actually, I thought of a work around, but I’ll leave the question here in case it’s helpful for others.)
hmm I’m having issues adding inline values to my environment files using something like:
environments:
dev:
values:
- region: "westus"
- name: chart
chart: stable/chart
set:
- name: config.client.external_labels.region
value: {{ .Environment.Values.region }}
What am I missing here?
in ./helmfile.yaml: error during helmfile.yaml.part.0 parsing: template: stringTemplate:87:30: executing "stringTemplate" at <.Environment.Values.region>: map has no entry for key "region"
Okay, this is weird.. If I use my base: environments.yaml at the top of the helmfile it doesn’t work, if I move everything in the environments.yaml file into the helmfile it works.
@Alucas hm… sounds like helmfile needs to parse your helmfile.yaml template to finally notice that it needs to load your base environments.yaml. this is kind of a chicken-and-egg problem
If I load the base: and the env at the top it doesn’t work, so I cannot have the base: environment.yaml that also includes the environments.
We need to resolve #932. This is a fundamental issue in helmfile
ya
so try this
bases:
- environments.yaml
---
- name: chart
chart: stable/chart
set:
- name: config.client.external_labels.region
value: {{ .Environment.Values.region }}
this would work as helmfile separates your template at ---, loading the base environments.yaml before rendering the second part
just show me your full example including bases and environments and everything else if this doesnt make sense(i might be misreading something
#helmfile users - do you already use needs? I’m considering to rename it to after. wdyt?
https://github.com/roboll/helmfile/issues/1018#issuecomment-565238688
2019-12-13
is there any way to mark a release as ‘to be deleted’? we have the issue that we deploy multiple versions of the same microservices with helmfile, but now we need to clean up old deprecated deploys somehow…
# set `false` to uninstall on sync
installed: true
aha let me check
2019-12-15
2019-12-16
Is there a way to iterate over helmfiles contents? For example:
helmfiles:
- path: ../*/*/release-*.yaml
releases:
{{ range $key, $values := helmfiles }}
- name: {{ .name }}
chart: {{ .chart }}
version: {{ .version }}
installed: {{ .installed }}
{{ end }}
@mumoshu please
Nope, unfortunately
Hey! Does anyone have any ideas on how one would go about allowing an additional release to be installed when setting a value/environment variable? not sure how the selector flag would handle installing the correct release, and not install it if the variable is not set. This is how I structured my helmfile, any pointers/questions regarding this would be great!
bases:
- ../commons/environments.yaml
---
releases:
- name: "release-name"
namespace: {{ .Namespace }}
labels:
chart: "release-name"
chart: stable/example-chart
values:
- ../releases/values.yaml.gotmpl
{{ if eq .Values.feature_branch "Y" }} # prefferably an env variable, not sure how that is done
- extraval:
enabled: True
- name: "feature_branch_release"
namespace: {{ .Namespace }}
chart: stable/other-chart
values:
../releases/feature_branch_values.yaml.gotmpl
{{ end }}
We generate N releases dynamically using an environment with a custom schema that we loop through
Nice approach, exactly what I’m looking into now. However, it seems that if I define the environments in a file included in bases the helmfile doesn’t have access to them.
Works:
environments:
apps-1:
values:
- ../common/defaults.yaml
- ../environments/apps-1.yaml
secrets:
- ../common/secrets.sops.yaml
missingFileHandler: Error
releases:
{{ range $index, $ns := .Environment.Values.applicationNamespaces }}
# ...
Gives map has no entry for key "applicationNamespaces":
bases:
- environments.yaml
releases:
{{ range $index, $ns := .Environment.Values.applicationNamespaces }}
# ...
Bug? Or am I overlooking something?
Pinging @mumoshu, I think this thread got of sight? Just want to check first if I should create an issue.
Try separating environments and releases with ---
Also separate between bases and releases
bases:
- environments.yaml
---
releases:
{{ range $index, $ns := .Environment.Values.applicationNamespaces }}
# ...
environments:
apps-1:
values:
- ../common/defaults.yaml
- ../environments/apps-1.yaml
secrets:
- ../common/secrets.sops.yaml
missingFileHandler: Error
---
releases:
{{ range $index, $ns := .Environment.Values.applicationNamespaces }}
# ...
You’re hitting a fundamental issue in helmfile. Please see https://github.com/roboll/helmfile/issues/932 and give me feedbacks!
Thx! Will try when back at home, somewhere in the weekend. Will also look into the feedback threads.
I want to deploy a custom helm chart, but I can’t see how to reference to a local chart
all the examples I see they suggest:
chart: stable/prometheus references to a remote repository. I want to run the chart I have in my local directory.
- name: fluentd-dashboard
namespace: monitoring
chart: "../charts/fluentd-dashboard"
verify: false
values:
- "../charts/fluentd-dashboard/Values.yaml"
you have to define a path.
I want to deploy my helm chart which is in a local directory chart1/
Any ideas?
2019-12-17
Hello guys, I’m trying to create one deployment.yaml file inside helm chart, that will create 2 pods when I run helmfile. So my problem is that I do not know how to properly reference values inside deployment.yaml that will point to my .yaml.gotmpl files.
So what I want is that on every place in deployment.yaml where it’s {{ .values }} , I want helm to detect that there are 2 gotmpl inside values/ (I do not want to specify exactly the name of that gotmpl because that would mean that I need 2 deployment files) , and to somehow iterate through them in order to create 2 deployments from that one deployment.yaml (for example first deployment would get all values from first gotmpl, and second created deployment would get all values from second gotmpl)
I want helm to detect that there are 2 gotmpl inside values/ (I do not want to specify exactly the name of that gotmpl because that would mean that I need 2 deployment files)
I dont really understand this but anyway
In chart’s deployment yaml you should just write it like:
containers:
- name: {{ .Values.name }}
apparently your es01.yaml.gotmpl isn’t a template so you can just name it es01.yaml
That would mean that I’m searching for name value inside values.yaml
and everything is just a plain helm usage
I’ll try to explain better:
i have two deployments that are very similar
so I want just one deployment.yaml
yeah that’s how you should use the same that to create two releases
which will reference only different values from two different files
hmm?
only different values
different from what?
different from each other..for example:
isn’t it just that the release es01 should be installed from es01.yaml.gotmpl and es03 from os03.yaml.gotmpl?
yess
then your helmfile.yaml looks correct
when you run helm install -f values.yaml somechart
{{ .Values.foo.bar }} results in BAR when values.yaml is:
foo:
bar: BAR
releases:
- name: es01
chart: ./elk-chart
values:
- es01.yaml.gotmpl
is basically
helm install ./elk-chart -f es01.yaml.gotmpl
so just reference your values defined in es01.yaml.gotmpl OR es03.yaml.gotmpl from within your deployment.yaml like
{{ .Values.environment.ES_JAVA_OPTS }}
{{ .Values.environment.bootstrap.memory_lock }}
{{ .Values.ports.containerPort }}
{{ .Values.ports.name }}
so when running helm, is this last gonna check for ES_JAVA_OPTS inside both gotmpl files, and add value for ES_JAVA_OPTS from es01.yaml.gotmpl to first created deployment and add value for ES_JAVA_OPTS from es03.yaml.gotmpl to second created deployment?
if I just reference like you said in deployment.yaml?
sorry for probably stupid questions though
yes
add value for ES_JAVA_OPTS from es01.yaml.gotmpl to first created deployment and add value for ES_JAVA_OPTS from es03.yaml.gotmpl to second created deployment?
i thought this is your intention as you did write your helmfile.yaml so in https://sweetops.slack.com/archives/CE5NGCB9Q/p1576573249022400
yes that’s exactly what I want, I’m rewriting my files now in order to test your suggestions
what I think happens here is that it look for volumeMounts.name inside values.yaml inside helm chart
isn’t it? (because with .Values you point to values.yaml)
??
according to the error message, the problem is that you have invalid volumes
this should be
volumes:
- name: {{ .Values.volumes.name }} emptyDir: {}
ah okay you were talking about this one
it looks like you’re missing voulmes.name in es03.yaml.gotmpl
es01 is failing due to invalid volumes
es03 is failing due to invalid es03.yaml.gotmpl
You’re the boss, it worked! Thank you a lot!
so to summarize, every time I put some file inside values tag in helmfile, data inside those file is accessible from chart using .Values.tag.tag.etc. ?
yep
thanks a lot!
I have one more question, it’s not regarding previous issue: in helmfile, if there are two releases that are pointing on the same chart, and if that chart has for example service.yaml defined, that service will be created when creating first release, and by the time second release is created that service already exists, so deployment of second release would fail…do you know a way to avoid it, so that two releases can point on the same chart but service in that chart will be created only once?
When you start from scratch developing a helm chart, that might be a problem
if instead, you start with the helm create command, it will setup all the scaffolding - including helpers that generate unique resource names based on the release
that way you won’t encounter the conflicts that you describe
it looks something like:
apiVersion: v1
kind: Service
metadata:
name: {{ template "fullname" . }}
Hello Erik, thank you! But because it would take a lot of code refactoring, and also I’m still not sure how referencing exactly works, is there maybe another way? for example, since I have two releases in helmfile pointing to the same helm chart, my idea is to create .Values.service.enabled for each release, so I can set it to “true” for first release and “false” for second release, and inside service.yaml I would set: {{- if .Values.service.enabled -}}
#content of service.yaml goes here
{{- end }}
This way every time new release is created from helmfile, when deploying service it will check if .Values.service.enabled is true or false , and it will be true only for one release so only once service will be deployed
(I tried it but it still fails to execute second release with service error “provided port is already allocated”, so I’m probably wrong somewhere, so I need help in that direction, maybe I missed something? )
Probably you’d better share your whole code in github if possible Reading your explanation, I can think of several possible causes
two files that are important in this case are helmfile:
and service.yaml inside elk-chart/templates:
this .Value.server.enabled is defined in both es01.yaml.gotmpl and es03.yaml.gotmpl , in one it’s set to true and in other it is set to false
And it fails for the second release, which has server.enabled: false?
If so, it seems impossible to occur. I guess the problem lies in another file and that’s why I’ve asked to share the whole project
here it is
thx!
enabled: "false" is wrong. it shoul dbe enabled: false otherwise it is treated as true
ohhh
thank you a lot!!
Is there any way to have different values for the same property in the values yaml.gotmpl file depending on what an environment variable is set to? e.g. if $ENV is set to “enable”, service.annotations and image.tag are set to value “x” and “y” respectively, but if $ENV is “disable” they are set to value “a” and “b”?
How about adding this snippet to your values.yaml.gotmpl?
{{ if env "enable" }}
service:
annotations: x
image:
tag: y
{{ else }}
service:
annotations: a
image:
tag: b
{{ end }}
sadly that didn’t work for me. failed to render [../releases/values.yaml.gotmpl], because of template: stringTemplate:20: function "ENV" not defined
Or is that just looking for any environment variable with the value of “enable”, rather than the variable called ‘env’ to be equal to “enable”?
rather than the variable called ‘env’ to be equal to “enable”?
sry what do you mean by this?
{{ env "enable" }} stands for the value of envvar named enable
and function "ENV" not defined is likely due to you’ve somehow wrote it like {{ ENV "something" }}?
I see. I thought that {{ if env "enable" }} was comparing the value of the environment variable named “env” with the string “enable”. So, referring to the snippet you posted above, if the envvar $enable is set to the value true, the values would be set to x and y, and if $enable is false the values would be a and b?
yep
That makes more sense than how I was thinking, thats great! Thank you so much for your help!
Glad to help!
2019-12-18
2019-12-19
Hello I am having this error when I try to reference the value of an environment variable inside the deployment
this is the deployment file, and I want to set the value of the environment variable helm.environment into ENVIRONMENT var. The value was set from command line using helmfile -e dev lint
Any ideas?
I have never used the stages features but should it not just be {{ .Values.helm.environment }} ?
I changed it to .Values as you suggested but no luck.
at <.Values.helm.environment>: nil pointer evaluating interface {}.environment
any ideas @mumoshu ?
@Juan Soto You can’t use environment values in helm chart templates
Helmfile environment values are dedicated to templates processed by Helmfile, like helmfile.yaml itself, and values.yaml.gotmpl read and rendered by Helmfile.
So you write it like value: {{ .Values.helm.environment }} in your “chart” template, without .Environment
I see.
and pass that specific chart values by writing like this in helmfile.yaml
releases:
- name: yourapp
chart: ./chart1
values:
- helm:
environment: {{ .Values.helm.environment }}
or
releases:
- name: yourapp
chart: ./chart1
values:
- somevalues.yaml containing helm.environment
That is what I did. And then, I referenced {{ environment_chart }} inside the helm chart.
Is it a good practice @mumoshu?
yes that looks good
2019-12-20
It seems like it’s the similar question I posted yesterday https://github.com/roboll/helmfile/issues/1045. I was trying to pass the environment values from my top level helmfile to sub helmfiles but wasn’t able to. Any advice?
From what I understand after debugging yesterday, it seemed to me that whenever there’s a change of directory, the env values will be cleaned and not be passed to nested helmfiles.
I see. I had to create the variables inside the helmfile, then I was able to reference it from the helm chart.
Yah I did the similar thing to work around it but still curious what the best way is
Another question. As I have two environments ready to be deployed. I would to implemente an strategy that allows me to deploy both environments at the same time. With the current configuration, each time I deploy to dev, it overwrites the stage environment. Are there any solution to have both envs up and running? What do you think?
Don’t you use two clusters for two envs?
No, I ony have one cluster
A gke cluster
What do you suggest?
Well we use two eks clusters. Don’t think it’s common to deploy two envs in the same cluster but I could think of templating different namespaces to deploy different envs if you have to.
ok thanks. So the question is: How do we deploy different environments in separate namespaces within the same cluster?
IMO you can set the ns value in your environments and template it
using the helm chart? or from helmfile?
Both? I mainly use helmfile to choose which ns to deploy but could be different for you
Do you have any example to share?
Something like releases:
- name: chart1-helmfile namespace: {{ .Environment.Values.testVersion}} chart: “./chart1”
And define this testVersion in your helmfile as environments: dev: values: - testVersion: 1.0.0
ok great, I will try that
Requesting feedbacks regarding two biggest fundamental issues in Helmfiel and corresponding enhancement proposals:
2019-12-21
2019-12-22
Hi, I ’ve been trying different approaches but I am still having problems referencing Environment Values in the helmfile.yaml
Your helmfile.yaml?
Separate releases and bases with ---
Now I am having different issue
you need to create it with kubectl create ns dev beforehand in helm 3
or use raw chart
releases:
- name: ns
chart: incubator/raw
values:
- resources:
- apiVersion: v1
kind: Namespace
metadata:
name: dev
- name: yourapp
chart: yourchart
namespace: dev
needs:
- ns
wow, this should really be in the README.md!
Can’t I include the creationg of the ns using the chart?
you can. see above
ok
in ./helmfile.yaml: failed to read helmfile.yaml: reading document at index 1: yaml: line 7: mapping values are not allowed in this context
humn…identation problem
yup. fix it and it would work
great. Fixed!. What’s more, both environments are being deployed to their relevant namespaces
Both environment are working! Great
I am having the following error
$ helmfile -e stage apply
in ./helmfile.yaml: failed to read helmfile.yaml: reading document at index 1: yaml: unmarshal errors:
line 31: cannot unmarshal !!int `5` into []string
any ideas?
try seeing line 31 of your subhelmfile
the error indicates it’s due to a type mismatch between your replicas defined in stage.yaml and your call-side(line 31 of sub-helmfile)
values: {{ .Environment.Values.replicas }} seems wrong
namespace and environment values are string and they went through
but replicas variable for some reason is not working.
It has an integer as it value.
yeah so use value: {{ .Environment.Values.replicas }} not values: {{...}
thanks
Now I need to put that value inside the helm chart
in chart1/values.yml
I am going to change it to
$ helmfile -e stage apply
Building dependency release=caylent-helmfile, chart=caylent
in ./helmfile.yaml: helm exited with status 1:
Error: cannot load values.yaml: error converting YAML to JSON: yaml: invalid map key: map[interface {}]interface {}{".Values.nreplicas":interface {}(nil)}
you cant use go template expressions in yaml file
change it to .gotmpl
2019-12-23
Do sub-helmfiles inherit helmDefaults?
wondering the same thing. Still want to find a way to let sub-helmfiles inherit values from upstream
no they don’t. the general rule is that nothing other than Environment.Name should be inherited
I’m trying to run helmfile with helm 3, (3.0.2) and i installed helm diff 3.0.0-rc.7 but when running the diff plugin fails without an error message
Comparing release=namespace, chart=incubator/raw
Comparing release=shared-secets, chart=settlemint/shared-secrets
Comparing release=rabbitmq, chart=stable/rabbitmq-ha
Comparing release=launchpad, chart=settlemint/launchpad
Comparing release=ingress, chart=stable/nginx-ingress
in ./helmfile.yaml: 5 errors:
err 0: failed processing release rabbitmq: helm exited with status 1:
err 1: failed processing release namespace: helm exited with status 1:
err 2: failed processing release launchpad: helm exited with status 1:
err 3: failed processing release shared-secets: helm exited with status 1:
err 4: failed processing release ingress: helm exited with status 1:
Any idea what’s wrong or on how to debug this?
~/Development/launchpad/infrastructure helm3 !3 ❯ helm diff upgrade --reset-values --allow-unreleased launchpad settlemint/launchpad --version 1.0.0-helm3 --kube-context launchpad --namespace alpha --detailed-exitcode
~/Development/launchpad/infrastructure helm3 !3 ❯ echo $?
1
No output but a 1 exit code
if i remove namespace and context it starts to return stuff
will figure out the mismatch there
ok, my context was wrong
Sorry for the flurry of questions, but what would be the reason that i get a ’failed to download incubator/raw` error with this helmfile
repositories:
- name: stable
url: <https://kubernetes-charts.storage.googleapis.com>
- name: incubator
url: <https://kubernetes-charts-incubator.storage.googleapis.com>
- name: harbor
url: <https://helm.goharbor.io>
- name: elastic
url: <https://helm.elastic.co>
- name: kiwigrid
url: <https://kiwigrid.github.io>
helmDefaults:
cleanupOnFail: true
verify: true
wait: true
timeout: 600
force: true
atomic: true
releases:
- name: mynamespace
chart: incubator/raw
version: 0.2.3
values:
- resources:
- apiVersion: v1
kind: Namespace
metadata:
name: {{ .Namespace }}
processing releases in group 1/4: alpha/mynamespace
worker 1/1 started
worker 1/1 finished
worker 1/1 started
Upgrading release=mynamespace, chart=incubator/raw
exec: helm upgrade --install --reset-values mynamespace incubator/raw --version 0.2.3 --verify --wait --timeout 600s --force --atomic --cleanup-on-fail --namespace alpha --values /var/folders/nr/3q7q00qs4hv46dwwffc9jp6h0000gn/T/values359668070 --history-max 10
exec: helm upgrade --install --reset-values mynamespace incubator/raw --version 0.2.3 --verify --wait --timeout 600s --force --atomic --cleanup-on-fail --namespace alpha --values /var/folders/nr/3q7q00qs4hv46dwwffc9jp6h0000gn/T/values359668070 --history-max 10:
worker 1/1 finished
FAILED RELEASES:
NAME
mynamespace
err: release "mynamespace" in "helmfile.yaml" failed: failed processing release mynamespace: helm exited with status 1:
Error: failed to download "incubator/raw" (hint: running `helm repo update` may help)
in ./helmfile.yaml: failed processing release mynamespace: helm exited with status 1:
Error: failed to download "incubator/raw" (hint: running `helm repo update` may help)
incubator is there when doing helm repo list, searching returns the raw chart and version
found it:
Error: failed to fetch provenance "<https://kubernetes-charts-incubator.storage.googleapis.com/raw-0.2.3.tgz.prov>"
helm.go:76: [debug] failed to fetch provenance "<https://kubernetes-charts-incubator.storage.googleapis.com/raw-0.2.3.tgz.prov>"
Yep, verify: false should fix this.
is it normal that if i put it to true in the defaults, that setting it to false on the release does not work?
Could be a bug:)
Hi, thank you all for helping me to develop my solution with Helmfile
now I need to integrate this deployment in CircleCI
They way I am deploying my app is:
Anybody has experience on running helmfile pipelines on CircleCI ? I already have the source code in github.
and the K8s cluster is running on GKE
Anybody has tested that ?
Hi all, hope you doing good. Im completely new to helm and helmfile. Can someone please clarify the difference between using helmfile vs creating a umbrella chart with dependency using requirements.yaml? Are they not solving the same usecase of deploying a group of charts for a complete deployment of applicaiton with dependant charts? Can you please clarify to understand the advantage the helmfile gives apart from environment specific deployment support like dev,production etc?
anyone?
I would highly recommend simply trying them both out to solve an issue. I think the difference feels like the difference between using a screwdriver and a powertool personally.
you can use helmfile to template out solution stacks built from helmfiles in very innovative ways
One problem with umbrella charts is you cannot easily decouple releases down the road if you want to reorganize how you deploy them. You can activate/deactivate dependencies easily, but that will effectively destroy them.
With helmfile, you don’t have that problem. You control everything about the release.
With umbrella charts, you control the chart and manage the SDLC. That’s a bit of investment. With helmfile, you don’t need to construct new charts just for the purpose of bundling services and dependencies.
With helmfile, you can surgically target one dependency and update it.
thanks @Erik Osterman (Cloud Posse) for the clarification… so in umbrella chart its not possible to update single dependant chart without restarting the main chart?
Right you would need to update the parent chart and it couples their life cycles then together
The other major major thing that Helmfile adds is the parameterization of values
In helm, the values.yaml is static. With Helmfile we have the full power of gotemplates in values
And with Helmfile environments we can create any number of helm releases on the fly
Helmfile is a swissarmy knife for managing helm releases of charts by dozens of different vendors, that’s proved invaluable to us. With Helmfile we have a consistent interface for deploying charts
2019-12-24
2019-12-25
2019-12-26
2019-12-30
We are looking into adopting helmfile for our existing helm use cases. We currently have 15-20 active clusters at any given time, but only about 10 are permanent (we spin up clusters several times throughout the day). These clusters are spread accross aws accounts, on prem and across regions. We don’t have the concept of a linear dev -> stage -> prod. To get to the point, can anyone point me to an example helmfile that might fit this use case?
we use helmfile much the same way. there isn’t necessarily a linear dev->stage->prod SDLC. We also use it to manage releases in multiple clusters, across multiple aws accounts, across multiple customers
Perhaps this is why we rely more on “environment variables” than the construct of “environments”
here are all of our helmfiles
We store a lot of settings in SSM and use chamber to call helmfile and pass those settings
we write a lot of settings to SSM using terraform
(there’s also now a terraform-helmfile-provider )
thanks for the links. Where do you keep your environment variables? Big fans of IaC, ie if we multiple charts that reference a specific vpc depending on account and region, where would that vpc id go?
You can probably refer to my earlier discussion https://github.com/roboll/helmfile/issues/1045. But I guess the only way (and the way I’m using) is to define environments in helmfile and inject values to your releases.
In our case, the VPC ID would be an output from a Terraform module, that would be written to SSM
As much as possible, we try to keep dynamically generated values (e.g. vpc_id out of code because and into the parameter store so that it can be dynamic and easily automated).
vpc id was maybe a poor example, maybe chart version? ie a different chart version depending on aws account/region.
2019-12-31
It seems like the releases defined in helmfile are executed async?
I think there’s a --concurrency flag you can use to set parallelism to 1 and then it would be synchronous
I see, thanks!