#release-engineering (2021-02)

Did you see the PR at https://github.com/deliverybot/deliverybot/pull/56 - provides quite some ideas on self-hosting

On the same train here - githubs deployment api is barely usable in the current state. And the approval for the github actions pushes you into enterprise tier for private repositories

Thanks for the link! Not sure I understand about the GitHub actions approval issue - is that something distinct from PR review approval?

It’s this feature https://github.blog/changelog/2020-12-15-github-actions-environments-environment-protection-rules-and-environment-secrets-beta/

So that for example when you have staging and prod, staging is autodeployed and then waits for an approval to run the next step to deploy to prod

Ah I had no idea about that - well I’m going to investigate self hosting deliverybot for now.

I completely agree. Our Infrastructure code generally goes untouched after go live.

We will deploy infra, deploy app, and then deploy infra again if needed.

these steps you mention are all done manually right? I am using a similar approach, using path triggers for Infrastructure execution. This is something I am trying to improve to make it simple as possible for all level of developers to adopt.

We have the processes automated in Jenkins. I’ll go into more detail after this meeting..

For infrastructure projects when a PR is created we run a simple tf lint check and validate - after success some teams run a plan against staging, and some teams also run a plan against prod - no output (tfplan) is saved.

After the PR is merged into master/main we run a plan and save the output to $ENVIRONMENT.tfplan. Someone must manually approve the deployment to the staging environment, and subsequently prod.

A similar plan for app whether it’s ECS or Lambda right now - lint, unit test, static analysis, etc… on PR. The artifact we deploy (container, zip, etc…) is created on the master branch and has a gated deploy to first staging, then prod. Some teams prefer auto deploy to staging, nobody is auto deploying to prod currently

Good one for #office-hours

Thanks @MattyB, we have a similar path for our projects too. One question, for the app, do you run any app related infrastructure pipeline or step? like for ECS, create the ECR or Task definition? Is it together with the app pipeline?

About #4. I agree. We indeed have several pipelines for infra and apps, we split everything up to 3 tiers/level very similar to what Eric presented once.

We played a bit with everything baked into the same repo/pipeline and we didn’t like it at all. Most of all we suffered from long terraform apply, spoiled state (since tooling was immature), a wide blast radius, etc.

Now for us it’s much more convenient with separated code/repos/pipelines: infra/platform layer is managed by Terraform in a dedicated repo; non-app services or shared services are deployed as Helm releases managed by Helmfile in a dedicated repo; apps also have a dedicated pipeline for Helmfile.

But it’s not completely painless. The problem we have now sometime is that when your app depends on some cloud service (a bucket for instance) or something else you have to figure out at what level to create this bucket. Especially if you don’t want to use TF on the apps level. Luckily helmfile has hooks, but sometimes it feels too hack-ish. Now we are looking forward to trying Config Connector (we are on GCP) which allows you to define Cloud services using Kubernetes objects.

@Maycon Santos - I believe Terraform created the task def on the second apply. A new task def is created with any subsequent deployments on the app side due to a new image name (tag) as well as any changes to env vars, secret refs, etc.. I was hoping to find a library that was already written for this, but we didn’t have any such luck. Most things we found weren’t enterprise ready..

@Erik Osterman (Cloud Posse) I am the author of the blog post. I will try to join the office hours this week. Happy to discuss more

Hello @Kostis (Codefresh), great blog post, it is nice to have you here.

What are your thoughts on how to avoid race conditions when you have dependencies like ECR for one application? would you consider that as part of the application build pipeline?

@Maycon Santos Hey. I would consider ECR as infrastructure and not application. Regarding conflicts the easiest way is to simply enforce ordering in your CI/CD system and say that the “application” pipeline and the “infra” pipeline should never run together at the same time.

I am checking ways to enforce that with my current setup without merging both workflows.

Thanks guys for the discussion. Will let you know how it goes for me.

are you looking for something like https://stackoverflow.com/questions/19891491/linux-shell-script-string-comparison-with-wildcards?

Any reason the single quotes are seemingly used at random? See the open set of brackets but not the closing set, and around the second part of the comparison but not the first.

please don’t; that kind of snark is generally not something that happens here - if you’ve got nothing to add then don’t add anything.

It wasn’t intended as a snark comment but I understand

cool