#release-engineering (2018-11)

jenkins_ci All things CI/CD. Specific emphasis on Codefresh and CodeBuild with CodePipeline.

CI/CD Discussions

Archive: https://archive.sweetops.com/release-engineering/

2018-11-01

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@dustinvb just joined! he dreams in pipelines

wave1
dustinvb avatar
dustinvb

Ah bah, no :superfresh: emoji.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

haha

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we can add that

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

for those that don’t know dustin, he is an awesome support engineer at Codefresh, which is what we use for all of our CI/CD

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

hey @dustinvb

dustinvb avatar
dustinvb

Hi Andriy.

dustinvb avatar
dustinvb

Happy to help out in anyway I can.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

thanks

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

we all love Codefresh

dustinvb avatar
dustinvb

I do as well. Was a customer for a year before joining.

2018-11-06

pecigonzalo avatar
pecigonzalo

Howdy releasers

pecigonzalo avatar
pecigonzalo

I wanted to discuss one topic that is spinning in my head for a while:

  • Reusing docker or other artifacts from PRs or not

Basically, so speed up the build/release cycle, I was thinking on reusing the tested/built artifact in a PR when we deploy to the QA environment and ofcourse subsequently

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yes, reusing the docker image is ideal for deployment to production and preproduction

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we tend to treat preproduction and staging slightly different

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

staging ~ master

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

preproduction is a tagged release

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

that tagged release gets promoted to production repos

pecigonzalo avatar
pecigonzalo

but do you build it on PR as well? if you dont, do you rerun all tests on the merged commit?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yes, we rerun on merge

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so not quite what you say i guess

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

the key for us is to reuse the image on deploy to production

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but not necessarily for all other steps

pecigonzalo avatar
pecigonzalo

Yep, that part I have “pinned” i was wondering about the ohter part

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we have a pipeline step to promote images and helm charts to production registry

pecigonzalo avatar
pecigonzalo

We have basically 2 “stable” branches

  • stage (staging/preprod)
  • master (prod)
pecigonzalo avatar
pecigonzalo

and I wanted to reuse the artifact from the PR to staging, to save some more time

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

gotcha - yea, we don’t have any thing for that. we also tend to do squash+merge, so the commit sha wouldn’t be the same

pecigonzalo avatar
pecigonzalo

Yep, you dont even get a parent commit that way do you?

pecigonzalo avatar
pecigonzalo

TBH, the core of the issue is not even on the pipeline, but mostly that some steps are just too slow (thanks Java/SBT dependancies)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, i can see why you’d want to do that

pecigonzalo avatar
pecigonzalo

So to clean up:

  • feature/xyz -> staging (PR): build,test
  • staging (commit from merge): build, test, release, deploy
pecigonzalo avatar
pecigonzalo

Is that your workflow?

pecigonzalo avatar
pecigonzalo

(maybe you do direct to master and use tags or some other CD workflow for prod)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
  • feature/1234/xyz -> staging (pr), build image, run compose integration tests, push image to registry, deploy helm chart to pr-namespace in kubernetes cluster (e.g. pr-1234)
  • master (squash merge): build, test, release, deploy
  • tag release - build, test, release, deploy to preproduction
  • production deploy - promote artifacts to production, deploy
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

honestly, every customer engagement is slightly different

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but it’s more or less like that

pecigonzalo avatar
pecigonzalo

Yeah, indeed, im just trying to get some ideas

pecigonzalo avatar
pecigonzalo

and most of the examples from “blogs” are simple stuff that would not work/fly in production

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea… i can show you some examples if you want to zoom sometime

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

what are you using for cicd? are you deploying to kubernetes?

pecigonzalo avatar
pecigonzalo

We are still on ECS, looking at EKS right now as we reached that point were ECS is just… annoying and we have enough people to support an EKS

pecigonzalo avatar
pecigonzalo

At the moment? Travis, but forget about it as we are migrating to either GitlabCI or Buildkite before EoY hopefully, as we as well outgrew travis

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

What I don’t like about GitLabCI is it only supports one pipeline per repo (last i checked)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

have you looked at Codefresh?

pecigonzalo avatar
pecigonzalo

Yeah, tbh im right now leaning to BuildKite as you can even dynamically load “pipeline”

pecigonzalo avatar
pecigonzalo

I did, but found it a bit less flexible

pecigonzalo avatar
pecigonzalo

Maybe I just did not understood it completly

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

dynamically load pipeline? what’s that?

pecigonzalo avatar
pecigonzalo

Lets say you have a pipeline like:

- build:
  - run x
- if build == commit:
  - loadpipeline pecigonzalo/this/.pipeline.yml
pecigonzalo avatar
pecigonzalo

let me find the docs

pecigonzalo avatar
pecigonzalo
Defining Your Pipeline Steps | Buildkite Documentationattachment image

Automate your team’s software development processes, from testing through to delivery, no matter the language, environment or toolchain.

pecigonzalo avatar
pecigonzalo
Trigger Step | Buildkite Documentationattachment image

Automate your team’s software development processes, from testing through to delivery, no matter the language, environment or toolchain.

pecigonzalo avatar
pecigonzalo

In many things is like a nice buildbot (https://buildbot.net/) but pre done

Buildbot

Buildbot - The Continuous Integration Framework

pecigonzalo avatar
pecigonzalo

Now, this is not an easy challenge as its “hard” to identify the built artifact

pecigonzalo avatar
pecigonzalo

since we enforce “up to date branches” im testing using git rev-parse --short HEAD^2 on the merge|commit job to identify the artifact by the tag (we tag with the commit) of the branch that did the PR that was merged and created that commit.

pecigonzalo avatar
pecigonzalo
*   2341145 (HEAD -> master, tag: 0.2.1, origin/master, workingset) Merge pull request #15 from thithat/staging
|\  
| *   d1bd417 Merge pull request #14 from thithat/feature/this-that
| |\  
| | * 43ba953 (feature/this-that) Test Version Flow release
* | |   16c043a Merge pull request #13 from thithat/staging
pecigonzalo avatar
pecigonzalo
❯ git rev-parse --short HEAD^2
d1bd417

❯ git rev-parse --short HEAD^1
16c043a
pecigonzalo avatar
pecigonzalo

thoughts? ideas? what is your current CI workflow? What do you build on PR and on merge?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so our workflow is slightly different

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we have “unlimited staging environments” which correspond to PRs - one per PR; those just build images pinned to 0.0.0-gitsha

pecigonzalo avatar
pecigonzalo

entire environments or just images?

2018-11-07

pecigonzalo avatar
pecigonzalo

In a similar tone to my previous question, anyone has a working workflow for using terraform plan from PR build or similar? Because while its the recommended way for terraform, I have yet to see a working workflow for it

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think the “interactive pull requests” model is ideally suited for terraform

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

the slides linked below show a number of companies (recognizable brands) who use atlantis

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

our fork of atlantis addresses the immediate shortcomings until they are fixed upstream

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@pecigonzalo if you are asking about how to trigger terraform plan/apply from an open PR, we recently used atlantis for a GitOps workflow. See #atlantis channel. Also @Erik Osterman (Cloud Posse) recently held a meetup during #connectweek in Pasadena (CA) where he gave a live demo using Atlantis with Terraform to provision AWS user accounts using only Pull Requests

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
DevOps Mastermind Group (Pasadena, CA)attachment image

This group is targeted to those interested in cloud automation & management, with a specific emphasis on Kubernetes/Docker, Helm, Prometheus, CI/CD, Microservices, etc. All skill levels are welcome. I

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
How to use Terraform with Teams using Atlantis (#GitOps)attachment image

GitOps is where everything, including infrastructure, is maintained in Git and controlled via a combination of Pull Requests and CI/CD pipelines. Reduce the learning curve for new devs by providing a familiar, repeatable process. Use Code Reviews to catch bugs and increase operational competency. Pr

2018-11-08

pecigonzalo avatar
pecigonzalo

No, not exactly like that, im not a big fan of what atlantis does

pecigonzalo avatar
pecigonzalo

It applies before merge, to me that is anti-pattern

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Except for terraform plans are poor

pecigonzalo avatar
pecigonzalo

I was thinking appling the plan generated on PR, on the merge

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

They are optimistic at best

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

So if you merge and apply now what is in git is not deployed

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

If others are developing against it, they are just as blocked

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

So what we’ve reconciled is auto merge on successful apply

pecigonzalo avatar
pecigonzalo

Well, but that is the same case for a CD container release

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

But for container releases it’s more stable

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

You have more under your control

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Less value based errors

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Most terraform failures in my experience are due to bad values

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Containers get their values at runtime

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Not at compile time (not generally but ideally)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

You can achieve what you want easily with Codefresh.

pecigonzalo avatar
pecigonzalo

Yeah, that is true, and a fair point, but not this one:
So if you merge and apply now what is in git is not deployed
If others are developing against it, they are just as blocked

pecigonzalo avatar
pecigonzalo

In theory in the container you have the same “posibility” of failure

pecigonzalo avatar
pecigonzalo

but I agree, is not as likely, as easiert to caught on test

pecigonzalo avatar
pecigonzalo

TF has a lot of sideeffect failure scenarios, that plan does not catch

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

To me atlantis is a practical approach

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Not the theoretical ideal

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

So rather than clutter the master commit history with a bunch of patch releases, rather get it in clean. We preserve a full transcript in the git comments so we have a record of what is deployed.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Even if the PR is closed but half applied, there’s a record of that’s

pecigonzalo avatar
pecigonzalo

Yeah, its not a bad approach and im not saying Atlantis sucks, much the contrary, even Hashi bought them

pecigonzalo avatar
pecigonzalo

I just feel that is a “hack” around a bigger issue

pecigonzalo avatar
pecigonzalo

a great hack, but a hack anyway

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Agreed

pecigonzalo avatar
pecigonzalo

In my previous company we had a rake looper to get around TF Modules not having count, it worked, and it saved us a lot of time, but in “terraform” it was a hack

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yea I have heard of similar hacks… basically around code generation

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Slippery slope

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think within a closed ecosystem of a corporate environment that might fly, but it makes it very difficult to write portable code for open source

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
antonbabenko/terrible

Let’s orchestrate Terraform configuration files with Ansible! Terrible! - antonbabenko/terrible

pecigonzalo avatar
pecigonzalo

Yeah indeed, im not so keen on it anymore. Good thing TF 0.12 is around the corner . This was a long time ago

joshmyers avatar
joshmyers

I wouldn’t touch 0.12 for a while…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I just don’t see a way around it that’s practical to solve (unless your HashiCorp and have 100mil in fresh green)

pecigonzalo avatar
pecigonzalo

pecigonzalo avatar
pecigonzalo

Yeah true

pecigonzalo avatar
pecigonzalo

I might give it 2nd chance, at least it will avoid those fix PRs for a stupid terraform value problem

pecigonzalo avatar
pecigonzalo

BTW, its great to have a place to ping-pong this ideas

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yea totally!! That what this place is for

pecigonzalo avatar
pecigonzalo

Yep, thanks !

2018-11-10

joshmyers avatar
joshmyers

Run a plan, push an artefact, namely the plan output. Run an apply of said build artefact number. This could be a gitsha, PR number, Jenkins build number. I don’t see a huge advantage of Atlantis… although I caveat with not having used so opinions maybe wrong. Have been doing CI/CD of terraform plan + apply for a long time in Jenkins. Not a massive fan of having output in the PR for history. Your git history lives forever, github PRs may not. Running pre merge isnt ideal for team scenario’s. Have seen a lot of failed apply due to vars, state, lots of TF bugs, race conditions…

joshmyers avatar
joshmyers

Things like tflint can help with some of these

joshmyers avatar
joshmyers

E.g. checking a type of AMI is even available in a region

joshmyers avatar
joshmyers

Won’t catch bugs though.

joshmyers avatar
joshmyers

Is Atlantis just like serverless Jenkins ci/cd for terraform with output posted back to the PR?

joshmyers avatar
joshmyers

Sounds nice if you don’t already have a CI/CD solution/don’t need one for other things?

2018-11-11

yurchenko avatar
yurchenko

ok

2018-11-29

Gabe avatar

anyone here use github actions yet? do you know if there is a way to cache dependencies between builds?

Gabe avatar

haven’t seen anything in the docs

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

no one @cloudposse has been invited to the beta

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i reached out to some peeps at GitHub but we don’t have the klout =0

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Gabe are you in the beta?

Gabe avatar

yeah we just got accepted in

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

just checked my inbox

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

nothing yet

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Gabe avatar

i don’t think i got an email… just saw the new actions button on our repo

mrwacky avatar
mrwacky

I got the email after the actions button appeared

Gabe avatar
Gabe
01:24:13 AM
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
01:25:15 AM
Gabe avatar

ohh… they are also only available on private repos

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Ohhhhhhhhh snap

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We have 250 public repos and like 5 private

1
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

yep only private

Gabe avatar

hmm yeah… it looks pretty cool so far but a few things i’ve noticed is not being able to cache dependencies between builds, no control over the size of the machine it runs on (1 cpu 3.75 gb), and only two concurrent workflows running at one time per repo

Gabe avatar

pros are that it seems simpler than circle/jenkins and you can create actions that take environment variables so it’s easier to reuse/share actions between repos

Gabe avatar

… and we just got the email saying they have enabled it for us

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

i just got the same email a few minutes ago, and Actions on my personal GitHub account

2
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
01:42:39 AM
Steven avatar

I’m not so lucky. Still waiting

2018-11-30

davidvasandani avatar
davidvasandani
04:13:50 PM

@Andriy Knysh (Cloud Posse) this screenshot from https://github.com/cloudposse/github-status-updater how do you release to a namespace?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

every PR is a new k8s namespace. It’s how we do unlimited staging environments

davidvasandani avatar
davidvasandani

Thanks!

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

see @Erik Osterman (Cloud Posse) presentation https://cloudposse.com/devops/unlimited-staging-environments/

Unlimited Staging Environmentsattachment image

How to run complete, disposable apps on Kubernetes for Staging and Development What if you could rapidly spin up new environments in a matter of minutes entirely from scratch, triggered simply by the push of a button or automatically for every Pull Request or Branch. Would that be cool? That’s

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@davidvasandani if you have questions or need more info, we can provide it

davidvasandani avatar
davidvasandani
05:04:12 PM

@Andriy Knysh (Cloud Posse) do you happen to have an example where all these pieces are glued together?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@davidvasandani here’s a simpler and complete working example:

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Add helmfile and codefresh by osterman · Pull Request #2 · cloudposse/statup

what Add helmfile for deployment with monochart Add codefresh build manifest why Easy deployment to kubernetes

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I use it for my demos

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

basically I took a random app called statup (self-hosted statuspage.io clone) and deploy it on kubernetes using our monochart with helmfile and helm using codefresh

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

this supports unlimited staging environments

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

and automatic destruction when the PR is closed using the pull-request-closed.yaml pipeline

davidvasandani avatar
davidvasandani

Thanks @Erik Osterman (Cloud Posse) can’t wait to dig into this.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Crap I realized you need to know all the ENV

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@davidvasandani if you PM me I can get them to you

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

give me a few minutes

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

re: Self-hosted Helm Chart Registry - Codefresh added Managed Helm Repositories after the presentation, so we use it now instead of deploying our own chart museum

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
05:29:26 PM
2
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@davidvasandani ^

    keyboard_arrow_up