#release-engineering (2020-08)

cc @Erik Osterman (Cloud Posse). I’m sure he’ll be stoked.

thanks @Matt Gowie! this sounds very interesting indeed.

Still gotta digest how this helps us. we were kind of wishing for the ability to “bless” PRs or forks.

sounds like pull_request_target is what we want, but still don’t get it.

@Andriy Knysh (Cloud Posse)

Yeah, I like the comment keyword approach, from someone with write privs to the repo, to kick off a build. And I want to be sure secrets remain protected so they can’t just be echoed in logs or captured and sent elsewhere

ya exactly… even for debugging I’ve added a printenv

This doesn’t yet get there, I don’t think, for public projects… But glad they’re actively iterating and releasing features!

have you seen their public roadmap? it’s pretty exciting how much investment is going into it

and in the nearterm

Saw it, but haven’t yet spent any time poking around, or trying to gauge velocity

Went ahead and followed the project, we’ll see if I need an email filter too

okay, so using this, I thikn what we can do for infra tests is first check if a blessed label is set (or something like that) - otherwise skip. Then checkout their branch. Then run the tests.

it works basically the same as dispatched events, that also always trigger on the default branch, but then checkout the merge commit using settings from the payload

yep. what’s your question?

Awesome. One question I have had is: How are you dealing with boilerplate AWS CLI code in your repos that doesn’t really change from app to app?

I just spent last Friday setting up a separate repo of bash scripts that I can clone in by evaling a CI variable….but this seems overly complex and eval is icky.

Looking at using remote template includes now as an alternative. https://docs.gitlab.com/ee/ci/yaml/#includeremote

You’ll have better luck if you boil things down to something more generic. You are trying to have reusable code in different pipelines. Are you trying to reuse whole pipeline stages, or just a bash script? Or something else?

Good questions…. So there are a few things that keep popping up. We are using ECS for most deploys now. We also have 2 AWS accounts (prod and pre-prod). Things I am trying to abstract away is:

• Selection of the correct ECR URL to use when publishing docker images.

• Forcing the deploy of the ECS cluster using the AWS CLI

For the CLI, I’m pulling the official docker image and creating an alias.

ECS has great support in Terraform. Have you looked at using that rather than coming up with a custom flow in Bash?

I have all the code for building out the cluster in TF. For the CI/CD, it’s mostly just building the docker image, pushing it to the right ECR repo, and then triggering the ECS cluster to re-deploy via CLI

@DJ have you check https://docs.gitlab.com/ee/ci/cloud_deployment/#deploy-your-application-to-the-aws-elastic-container-service-ecs ?

I had not @Maycon Santos! Thanks. That looks like a pretty easy way to push a deploy to ECS. Might borrow some of those techniques..

FWIW, I’m currently leaning toward the Helm operator. Maybe do something like this?

1. Install the operator using the Terraform Helm provider
2. Use local-exec or the Shell provider to kubectl apply the CRDs Advantages to this seem to be that if the Terraform Helm provider does have any issues, it is only being used to deploy the operator, rather than all the other deployments I need

For things that need to be run locally, I should be able to run whatever I want with docker run rather than having to have actual dependencies installed on the machine running terraform (though, that would complicate running terraform inside a container, which is very frequently done, so maybe that isn’t the best approach)

kubectl apply is nicely idempotent, so I don’t have to worry if Terraform wants to run it every time I terraform apply

@roth.andy Hey! I’m currently researching towards building something similar to yours, but based on a custom terraform provider(https://github.com/mumoshu/terraform-provider-eksctl). there should be some nuance but i think i can share a few things that might be common

for avoiding local-exec for helmfile, i can definitely recommend using https://github.com/mumoshu/terraform-provider-helmfile as you’ve noted

you can even use the helmfile provider install the helm operator and HelmRelease custom resources, if you want to defer/delegate application deployment to the helm operator

Does it require helm, helmfile, helm-diff, helm-git, unfortunately yes. however, i’m going to integrate a binary package manager(https://github.com/mumoshu/shoal/) into the helmfile provider so that it can install those binaries on terraform apply

Thanks @mumoshu, I saw a note about the package manager in an issue on the helmfile provider as well, sounds cool.

Do you think the helmfile provider is in a place where you would feel comfortable using it in production?

I have 2 repos.

infrastructure-terraform with 2 layers and infrastructure-kuberentes which has uses helmfile .. Maybe I can join next office hours and show it .. I wanted to get feedback on it anyways for quite some time.

@roth.andy I’m working for a client now where I’m building a v0.1 of a very similar project to what you’re targeting.

Couple things of note:

1. Running on EKS, we run 1 local-exec to update the local Kube Configuration after the cluster is created.
2. We decided to install Flux and HelmOperator via the Helm provider and then have a separate repo where Flux / HO looks for our HelmReleases for all other CRDs / in-house Applications that we’re deploying. This is in a very early stage, but so far seems to be doing the trick. I also wanted to use Helmfile + the provider with similar reasoning on why you wanted to use them, but Flux was already chosen as the tool of choice by an colleague before I joined the project. Didn’t want to rock the boat too much.

Definitely interested in where you land — be sure to provide updates!

@Matt Gowie how are you guys liking the helm operator?

• stable?

• easy to update?

• easy to use?

• Stable — Unseen as of right now. Still in pre-prod environments with this new toolset.

• Easy to update — Haven’t updated the Helm Operator itself yet, but updating the Helm Releases that it uses is easy enough since it’s also GitOps and it does a good job of tracking changes there and doing deployments.

• Easy to use — Easy to install for sure. HelmReleases definitely seem less DRY than something like Helmfile, but they are fairly straight forward and it’s easy to customized Flux/HO to do what we want so far. Again, super early stage, but so far good experience.

:thumbsup: for terraform-provider-helmfile supporting shoal for dependency management. we want to use it with terraform cloud, but lack of automatic dependencies has made it very hard to utilize for this use-case.

i’ve been using the terraform helm provider since late 2018 and it has worked fine. unless things have changed recently, it does require helm on the machine you run terraform from — annoying if you use TF cloud.

but terraform helm provider also doesn’t diff

@Erik Osterman (Cloud Posse) what do you mean by diff? It is aware that changes are made to the helm values but probably just uses the terraform state to show that diff

I’ll be honest I only use it to deploy a cluster-wide services charts (i.e cert-manager) and logging/monitoring helm charts and that bit of terraform has not changed much — the only things that i ever changed there are the helm values and chart versions and that use case has worked fine for me.

@mumoshu should the Helmfile provider be able to work in the same lifecycle as the creation of a cluster? I’m set up to do that now, but when I run terraform apply the Helmfile provider immediately tries to run helm diff though the cluster doesn’t even exist yet.

The log says Config not found (the config is a local_file resource that gets created after the cluster gets provisioned)

Wait hold the phone. I might have a bug in my helmfile

possible ID10T error

Okay now I’m back to thinking it is an issue with the helmfile provider. I do not see any issue with the actual helmfile itself

I’m starting from a completely empty terraform state, and Terraform needs to create the cluster, and then deploy to it using the helmfile provider.

It doesn’t seem like that is something the Helmfile provider supports, since it is immediately trying to connect to a nonexistent cluster

You probably need to do this in your terraform code:

https://github.com/cloudposse/terraform-aws-eks-cluster/blob/df8b991bef53fcab8f01c542cd1c3ccc6242b61c/auth.tf#L49-L60

This is to defer the attempt to connect to the cluster until after the cluster exists

and responds to healthcheck

then…

@Erik Osterman (Cloud Posse) I think the issue is that the helmfile provider tries to read cluster state before it even starts applying anything. Since a cluster doesn’t even exist yet it obviously can’t do that

I do already have a depends_on on the helmfile release

yes, this is true even of the kubernetes provider for terraform

so that’s why this hack exists to get around that limitation. probably the same thing would work with the helmfile provider.

but the way null_resource works and the curl command means it won’t complete until the cluster is online and availble, which means the depends on blocks the providers from connecting to the cluster until the cluster is online.

okay I’ll take a look. thanks

let me know how it goes - if you get stuck can probably unblock

still happening. happens even on terraform plan

@Andriy Knysh (Cloud Posse) look familiar?

The helmfile provider is trying to run helm diff as part of Terraform’s “Refreshing state…” process, but it can’t because there is no cluster yet and it doesn’t seem like it is able to handle that.

is the cluster on localhost?

no

that’s what kubectl defaults to if it is not pointed at a valid kubeconfig file

so looks like kubeconfig is not read or is incorrect

There is no kubeconfig yet. This terraform project creates the cluster, then the intention is to use the helmfile provider on the brand new cluster immediately

This, for example, works fine with a local-exec that calls helmfile apply, but the helmfile provider wants to run helmfile diff inside an as-yet nonexistent cluster when you run terraform plan

It may be that this is just not something that the helmfile provider supports, but I suspect otherwise since @mumoshu said this in a comment on one of the github issues: https://github.com/mumoshu/terraform-provider-helmfile/issues/20#issuecomment-681598191

I’m thinking this situation is one of the following:

1. I’m doing something wrong - If so I’ll put in a PR to add to the docs once I figure it out
2. I’m experiencing a bug - If so I’ll write up a github issue
3. The Helmfile provider is not meant to be used in the same terraform apply as the creation of the cluster it needs to connect to - If so I’ll put in a PR to add to the docs

I’ll assume #2 for now and write up a github issue

I think start with #2. @mumoshu might have some ideas how to mitigate it. i did some more digging too and don’t see a way right now around it.

@mumoshu should the Helmfile provider be able to work in the same lifecycle as the creation of a cluster? I’m set up to do that now, but when I run terraform apply the Helmfile provider immediately tries to run helm diff though the cluster doesn’t even exist yet. @roth.andy Hey! I just started reading this thread, but anyway- It should work since recent versions of the provider I’ve released yesterday

This was the fix(or a workaround for the underlying terraform limitation) required to make it work: https://github.com/mumoshu/terraform-provider-helmfile/commit/681ecb39707823cd3e76ed16c8351aa58f0d11b6 It’s included since v0.4.3

honestly, i dont have much experience with terraform-aws-eks-cluster or rke_rancher_master_cluster modules, but, since v0.4.3, the helmfile provider should work as long as you can somehow pass helmfile_releaset_set a kubeconfig path to the working k8s cluster.

so looks like kubeconfig is not read or is incorrect This is mostly correct, but to be extra clear - I think this as rather a Terraform issue.

Terraform tries to call the provider for the diff operation(part of plan) EVEN if the target resource has a missing value that depends on another, not-yet-created tf resource, replacing the missing value as an empty value.

Example:

resource "eksctl_cluster" "mycluster" {
   ...
}

resource "helmfile_release_set" "mystack" {
  kubeconfig_path = eksctl_cluster.mycluster.kubeconfig_path
}

In this case, you’ll probably expect Terraform to trigger diff on helmfile_release_set if and only if the eksctl_cluster is created. But that’s not how Terraform works. Terraform tries to run diff anyway replacing eksctl_cluster.mycluster.kubeconfig_path with an empty value

resource "helmfile_release_set" "mystack" {
  kubeconfig_path = ""
}

0.4.3 fixes that, by skipping diff when kubeconfig_path is empty.

kubeconfig_path is new to me. Sounds like that would fix my issue. I’ve been adding KUBECONFIG to the environment variables since that is what the README does. I will submit a PR tomorrow to improve the docs

I wasn’t clear but you can use either. environment_variables.KUBECONFIG and kubeconfig_path works exactly the same way internally

Oh. Hmm. I was using the latest version today and getting the results above. Could a later version have caused a regression? I will try v0.4.3 tomorrow

The good news is that the provider worked really well when there was a cluster to point at. Im looking forward to using it more. I had some issues with the new shoal stuff but I only spent a few minutes with it so nothing significant to report yet

Ah okay then it might be another issue. I’m now wondering if it’s just that-

You’ve actually passed non-empty kubeconfig path to the helmfile provider, but the path does not exist yet when Terraform calls the provider for diff?

I think you’ve tried to use wait_cluster null_resource to let Terraform defer applying helmfile_release_set until the cluster is created. I believe it would work for apply, but not for diff. diff happens regardless of dependencies among tf resources, as described above.

It’s definitely possible. I will investigate more tomorrow and report back

environment_variables = {
    KUBECONFIG = abspath(local_file.kubeconfig.filename)
  }

Is what I am using now. Tomorroq I will try changing it to a ternary that more explicitly sets the value to an empty string if the file doesn’t exist

Thanks!

The important and not well known point of this issue is that Terraform runs internal diff operations on resources regardless of resource dependencies, with emptying any resource attributes if the dependent resources are not created.

So we’ll probably end up needing to somehow “signal” helmfile provider if the dependent resource(=cluster) is already created or not.

environment_variables = {
    KUBECONFIG = abspath(local_file.kubeconfig.filename)
  }

Is kubeconfig.filename static?

I take it you haven’t yet tried terraform-eksctl-provider with terraform-helmfile-provider in the same terraform workspace?

…on a coldstart

No! I did try it and it works

hrmmmmmmm that’s promising then! so this should definitely be possible

Yes it is static. It comes from a local_file resource

Gotcha, so that’s the issue I think

Let’s try making the filename dynamic so that it becomes empty on Terraform-internal diff phase, which signals the helmfile provider to not fail on invalid cluster.

Definitely sounds like it. I definitely have some avenues to explore when I get back to it. Thanks!

Possible enhancement could be to skip the diff if empty string OR the file specified doesn’t exist. Might even try getting a PR in, though my golang skills are nothing to be proud of

Sounds like a great idea!

A PR is definitely welcomed. I’ll also try adding it when I have a chance(Currently working on stabilizing the R53 + NLB cluster canary thing so maybe after that

@roth.andy fyi i’ve added the enhancement that we discussed above in v0.5.3

@barak can probably answer this one :-)

Hi Yoni :)

Hi :)

Checkov is probably the most rich in content and frameworks. You can integrate it in various stages on the CI/CD pipeline and get a hunch of misconfigs in IaC (k8, serverless, terraform, cfn, arm) without the need to provision them. It enables you to shift security testing left in the lifecycle.

For a wider solution that can shift left and right (production security testing) you can try the bridgecrew.io SaaS platform.

P.s. I’m one of the the developers of both checkov and bridgecrew :)

@barak is it expected that Checkov doesn’t ‘deep scan’ from a composition (ie, root module) down into its declared modules? I was trying it out and if I scan my module itself it gives me detailed results on the resources in the module. If I go up to the composition where I’m passing in various config vars to do things, the checkov result was only verifying that I didn’t have hardcoded access keys. Should it only be scanning my modules? But then, how do I verify that someone has not passed in configs that would otherwise violate a check?

Hi @Zach take a look on “scanning third party modules” here: https://www.checkov.io/2.Concepts/Evaluations.html

it’s a workaround , but some found it usefull

In this case these are my modules, just pulled in from a different repository, though I suppose that distinction doesn’t really matter since its still an ‘external’ module?

Ohhh hm. Also we have our modules in a ‘mono-repo’ so scanning the entire .terraform might be a mess

this is probably a ‘me’ problem more than Checkov. We have our compositions in one mono repo with all the environments/stages, and all the modules in a second repo that we tag for module ‘releases’

While a little manual (potentially useful with flags/vars available in your build pipeline for which modules you currently care about?) you don’t have to stop at -d .terraform, can narrow the scope further to individual -d .terraform/module paths. ie checkov -d .terraform/modules/eks

I used tfscan on several projects. It was pretty good but limited in scope.