#release-engineering (2021-02)

jenkins_ci All things CI/CD. Specific emphasis on Codefresh and CodeBuild with CodePipeline.

CI/CD Discussions

Archive: https://archive.sweetops.com/release-engineering/

2021-02-28

walicolc avatar
walicolc
Chris Fowles avatar
Chris Fowles

please don’t; that kind of snark is generally not something that happens here - if you’ve got nothing to add then don’t add anything.

1
1
walicolc avatar
walicolc

It wasn’t intended as a snark comment but I understand

Chris Fowles avatar
Chris Fowles

cool

2021-02-23

KEN avatar

Hello, how to design the CICD on Firebase? Dose Jenkins need to be installed on Mac? Or GitLab is easier to design the flow?

2021-02-16

jason einon avatar
jason einon
jenkinsfile if statement not using wildcard statement

Im trying to work out the correct syntax based on the below if statement in a jenkinsfile, however its not passing the wildcard as expected and trying to match the syntax including the wildcard sym…

joey avatar
Linux Shell Script - String Comparison with wildcards

I am trying to see if a string is part of another string in shell script (#!bin/sh). The code i have now is: #!/bin/sh #Test scriptje to test string comparison! testFoo () { t1=$1 …

MattyB avatar
MattyB

Any reason the single quotes are seemingly used at random? See the open set of brackets but not the closing set, and around the second part of the comparison but not the first.

jason einon avatar
jason einon

been working on it for a couple of hours and driving me mad

2021-02-11

2021-02-09

2021-02-08

2021-02-06

2021-02-05

Maycon Santos avatar
Maycon Santos

Hey guys, maybe you already discussed this here. But what you guys thing about this blog post: https://codefresh.io/kubernetes-tutorial/kubernetes-antipatterns-1/ specific about the 4th pattern, Mixing application deployment with infrastructure deployment

I am facing a similar concern with ECS were, I am adding 2 workflows in Github Actions, one for build and deploy the app, and another one for app-specific infra, like ECR or ECR task definition and service. My issue is to improve how we deal with race conditions, like ECR is not created yet and the app is being deployed. How do you guys deal with it?

Kubernetes Deployment Antipatterns - part 1 - Codefresh

In our previous guide, we documented 10 Docker anti-patterns. This guide has been very popular as it can help you in your first steps with container images. Creating container images for your application, however, is only half the story. You still need a way to deploy these containers in production, and the de facto solution … Continued

MattyB avatar
MattyB

I completely agree. Our Infrastructure code generally goes untouched after go live.

Kubernetes Deployment Antipatterns - part 1 - Codefresh

In our previous guide, we documented 10 Docker anti-patterns. This guide has been very popular as it can help you in your first steps with container images. Creating container images for your application, however, is only half the story. You still need a way to deploy these containers in production, and the de facto solution … Continued

MattyB avatar
MattyB

We will deploy infra, deploy app, and then deploy infra again if needed.

Maycon Santos avatar
Maycon Santos

these steps you mention are all done manually right? I am using a similar approach, using path triggers for Infrastructure execution. This is something I am trying to improve to make it simple as possible for all level of developers to adopt.

MattyB avatar
MattyB

We have the processes automated in Jenkins. I’ll go into more detail after this meeting..

MattyB avatar
MattyB

For infrastructure projects when a PR is created we run a simple tf lint check and validate - after success some teams run a plan against staging, and some teams also run a plan against prod - no output (tfplan) is saved.

After the PR is merged into master/main we run a plan and save the output to $ENVIRONMENT.tfplan. Someone must manually approve the deployment to the staging environment, and subsequently prod.

MattyB avatar
MattyB

A similar plan for app whether it’s ECS or Lambda right now - lint, unit test, static analysis, etc… on PR. The artifact we deploy (container, zip, etc…) is created on the master branch and has a gated deploy to first staging, then prod. Some teams prefer auto deploy to staging, nobody is auto deploying to prod currently

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Good one for #office-hours

1
Maycon Santos avatar
Maycon Santos

Thanks @MattyB, we have a similar path for our projects too. One question, for the app, do you run any app related infrastructure pipeline or step? like for ECS, create the ECR or Task definition? Is it together with the app pipeline?

Andrey Nazarov avatar
Andrey Nazarov

About #4. I agree. We indeed have several pipelines for infra and apps, we split everything up to 3 tiers/level very similar to what Eric presented once.

We played a bit with everything baked into the same repo/pipeline and we didn’t like it at all. Most of all we suffered from long terraform apply, spoiled state (since tooling was immature), a wide blast radius, etc.

Now for us it’s much more convenient with separated code/repos/pipelines: infra/platform layer is managed by Terraform in a dedicated repo; non-app services or shared services are deployed as Helm releases managed by Helmfile in a dedicated repo; apps also have a dedicated pipeline for Helmfile.

But it’s not completely painless. The problem we have now sometime is that when your app depends on some cloud service (a bucket for instance) or something else you have to figure out at what level to create this bucket. Especially if you don’t want to use TF on the apps level. Luckily helmfile has hooks, but sometimes it feels too hack-ish. Now we are looking forward to trying Config Connector (we are on GCP) which allows you to define Cloud services using Kubernetes objects.

MattyB avatar
MattyB

@Maycon Santos - I believe Terraform created the task def on the second apply. A new task def is created with any subsequent deployments on the app side due to a new image name (tag) as well as any changes to env vars, secret refs, etc.. I was hoping to find a library that was already written for this, but we didn’t have any such luck. Most things we found weren’t enterprise ready..

Kostis (Codefresh) avatar
Kostis (Codefresh)

@Erik Osterman (Cloud Posse) I am the author of the blog post. I will try to join the office hours this week. Happy to discuss more

2
1
Maycon Santos avatar
Maycon Santos

Hello @Kostis (Codefresh), great blog post, it is nice to have you here.

What are your thoughts on how to avoid race conditions when you have dependencies like ECR for one application? would you consider that as part of the application build pipeline?

Kostis (Codefresh) avatar
Kostis (Codefresh)

@Maycon Santos Hey. I would consider ECR as infrastructure and not application. Regarding conflicts the easiest way is to simply enforce ordering in your CI/CD system and say that the “application” pipeline and the “infra” pipeline should never run together at the same time.

Maycon Santos avatar
Maycon Santos

I am checking ways to enforce that with my current setup without merging both workflows.

Thanks guys for the discussion. Will let you know how it goes for me.

Yoni Leitersdorf (Indeni Cloudrail) avatar
Yoni Leitersdorf (Indeni Cloudrail)

This drift detection tool was mentioned here a few times already. What I’m wondering is: Anybody looking to include this as part of CI? For example, like you’d have test coverage. If the “Terraform coverage” of your cloud environment decreased in this build vs the last build, fail the build.

2021-02-04

vixus0 avatar
vixus0

Hey, I noticed cloudposse forked deliverybot – are there any plans for making it easier to self-host? I haven’t found anything that provides quite the same functionality.

Patrick Jahns avatar
Patrick Jahns

Did you see the PR at https://github.com/deliverybot/deliverybot/pull/56 - provides quite some ideas on self-hosting

On the same train here - githubs deployment api is barely usable in the current state. And the approval for the github actions pushes you into enterprise tier for private repositories

Feature: Self-Hosted Deployment by ramene · Pull Request #56 · deliverybot/deliverybot

This feature adds the ability to run deliverybot locally and exposed through ngrok via public internet. Runs on Kubernetes anywhere (Azure, minikube, AWS, GCP and more!). The Dockerfile consists of…

vixus0 avatar
vixus0

Thanks for the link! Not sure I understand about the GitHub actions approval issue - is that something distinct from PR review approval?

Patrick Jahns avatar
Patrick Jahns

So that for example when you have staging and prod, staging is autodeployed and then waits for an approval to run the next step to deploy to prod

vixus0 avatar
vixus0

Ah I had no idea about that - well I’m going to investigate self hosting deliverybot for now.

2021-02-02

    keyboard_arrow_up