#release-engineering (2018-11)
All things CI/CD. Specific emphasis on Codefresh and CodeBuild with CodePipeline.
CI/CD Discussions
Archive: https://archive.sweetops.com/release-engineering/
2018-11-01


Ah bah, no :superfresh:
emoji.

haha

we can add that

for those that don’t know dustin, he is an awesome support engineer at Codefresh, which is what we use for all of our CI/CD

hey @dustinvb

Hi Andriy.

Happy to help out in anyway I can.

thanks

we all love Codefresh

I do as well. Was a customer for a year before joining.
2018-11-06

Howdy releasers

I wanted to discuss one topic that is spinning in my head for a while:
- Reusing docker or other artifacts from PRs or not
Basically, so speed up the build/release cycle, I was thinking on reusing the tested/built artifact in a PR when we deploy to the QA environment and ofcourse subsequently

yes, reusing the docker image is ideal for deployment to production and preproduction

we tend to treat preproduction and staging slightly different

staging ~ master

preproduction is a tagged release

that tagged release gets promoted to production repos

but do you build it on PR as well? if you dont, do you rerun all tests on the merged commit?

yes, we rerun on merge

so not quite what you say i guess

the key for us is to reuse the image on deploy to production

but not necessarily for all other steps

Yep, that part I have “pinned” i was wondering about the ohter part

we have a pipeline step to promote images and helm charts to production registry

We have basically 2 “stable” branches
- stage (staging/preprod)
- master (prod)

and I wanted to reuse the artifact from the PR to staging, to save some more time

gotcha - yea, we don’t have any thing for that. we also tend to do squash+merge, so the commit sha wouldn’t be the same

Yep, you dont even get a parent commit that way do you?

TBH, the core of the issue is not even on the pipeline, but mostly that some steps are just too slow (thanks Java/SBT dependancies)

yea, i can see why you’d want to do that

So to clean up:
- feature/xyz -> staging (PR): build,test
- staging (commit from merge): build, test, release, deploy

Is that your workflow?

(maybe you do direct to master and use tags or some other CD workflow for prod)

feature/1234/xyz
-> staging (pr), build image, run compose integration tests, push image to registry, deploy helm chart to pr-namespace in kubernetes cluster (e.g. pr-1234)master
(squash merge): build, test, release, deploy- tag release - build, test, release, deploy to preproduction
- production deploy - promote artifacts to production, deploy

honestly, every customer engagement is slightly different

but it’s more or less like that

Yeah, indeed, im just trying to get some ideas

and most of the examples from “blogs” are simple stuff that would not work/fly in production

yea… i can show you some examples if you want to zoom sometime

what are you using for cicd? are you deploying to kubernetes?

We are still on ECS, looking at EKS right now as we reached that point were ECS is just… annoying and we have enough people to support an EKS

At the moment? Travis, but forget about it as we are migrating to either GitlabCI or Buildkite before EoY hopefully, as we as well outgrew travis

What I don’t like about GitLabCI is it only supports one pipeline per repo (last i checked)

have you looked at Codefresh?

Yeah, tbh im right now leaning to BuildKite as you can even dynamically load “pipeline”

I did, but found it a bit less flexible

Maybe I just did not understood it completly

dynamically load pipeline? what’s that?

Lets say you have a pipeline like:
- build:
- run x
- if build == commit:
- loadpipeline pecigonzalo/this/.pipeline.yml

let me find the docs


Automate your team’s software development processes, from testing through to delivery, no matter the language, environment or toolchain.

and also, seamless triggers: https://buildkite.com/docs/pipelines/trigger-step

Automate your team’s software development processes, from testing through to delivery, no matter the language, environment or toolchain.

In many things is like a nice buildbot (https://buildbot.net/) but pre done
Buildbot - The Continuous Integration Framework

Now, this is not an easy challenge as its “hard” to identify the built artifact

since we enforce “up to date branches” im testing using git rev-parse --short HEAD^2
on the merge|commit
job to identify the artifact by the tag (we tag with the commit) of the branch that did the PR that was merged and created that commit.

* 2341145 (HEAD -> master, tag: 0.2.1, origin/master, workingset) Merge pull request #15 from thithat/staging
|\
| * d1bd417 Merge pull request #14 from thithat/feature/this-that
| |\
| | * 43ba953 (feature/this-that) Test Version Flow release
* | | 16c043a Merge pull request #13 from thithat/staging

❯ git rev-parse --short HEAD^2
d1bd417
❯ git rev-parse --short HEAD^1
16c043a

thoughts? ideas? what is your current CI workflow? What do you build on PR and on merge?

so our workflow is slightly different

we have “unlimited staging environments” which correspond to PRs - one per PR; those just build images pinned to 0.0.0-gitsha

entire environments or just images?
2018-11-07

In a similar tone to my previous question, anyone has a working workflow for using terraform plan from PR build or similar? Because while its the recommended way for terraform, I have yet to see a working workflow for it

I think the “interactive pull requests” model is ideally suited for terraform

the slides linked below show a number of companies (recognizable brands) who use atlantis

our fork of atlantis addresses the immediate shortcomings until they are fixed upstream

@pecigonzalo if you are asking about how to trigger terraform plan/apply
from an open PR, we recently used atlantis
for a GitOps workflow. See #atlantis channel. Also @Erik Osterman (Cloud Posse) recently held a meetup during #connectweek in Pasadena (CA) where he gave a live demo using Atlantis with Terraform to provision AWS user accounts using only Pull Requests


This group is targeted to those interested in cloud automation & management, with a specific emphasis on Kubernetes/Docker, Helm, Prometheus, CI/CD, Microservices, etc. All skill levels are welcome. I


GitOps is where everything, including infrastructure, is maintained in Git and controlled via a combination of Pull Requests and CI/CD pipelines. Reduce the learning curve for new devs by providing a familiar, repeatable process. Use Code Reviews to catch bugs and increase operational competency. Pr
2018-11-08

No, not exactly like that, im not a big fan of what atlantis does

It applies before merge, to me that is anti-pattern

Except for terraform plans are poor

I was thinking appling the plan
generated on PR, on the merge

They are optimistic at best

So if you merge and apply now what is in git is not deployed

If others are developing against it, they are just as blocked

So what we’ve reconciled is auto merge on successful apply

Well, but that is the same case for a CD container release

But for container releases it’s more stable

You have more under your control


Most terraform failures in my experience are due to bad values

Containers get their values at runtime

Not at compile time (not generally but ideally)

You can achieve what you want easily with Codefresh.

Yeah, that is true, and a fair point, but not this one:
So if you merge and apply now what is in git is not deployed
If others are developing against it, they are just as blocked

In theory in the container you have the same “posibility” of failure

but I agree, is not as likely, as easiert to caught on test

TF has a lot of sideeffect failure scenarios, that plan
does not catch

To me atlantis is a practical approach

Not the theoretical ideal

So rather than clutter the master commit history with a bunch of patch releases, rather get it in clean. We preserve a full transcript in the git comments so we have a record of what is deployed.

Even if the PR is closed but half applied, there’s a record of that’s

Yeah, its not a bad approach and im not saying Atlantis sucks, much the contrary, even Hashi bought them

I just feel that is a “hack” around a bigger issue

a great hack, but a hack anyway

Agreed

In my previous company we had a rake
looper to get around TF Modules not having count
, it worked, and it saved us a lot of time, but in “terraform” it was a hack

Yea I have heard of similar hacks… basically around code generation

Slippery slope

I think within a closed ecosystem of a corporate environment that might fly, but it makes it very difficult to write portable code for open source

Let’s orchestrate Terraform configuration files with Ansible! Terrible! - antonbabenko/terrible

Yeah indeed, im not so keen on it anymore. Good thing TF 0.12 is around the corner . This was a long time ago

I wouldn’t touch 0.12 for a while…

I just don’t see a way around it that’s practical to solve (unless your HashiCorp and have 100mil in fresh green)


Yeah true

I might give it 2nd chance, at least it will avoid those fix PRs for a stupid terraform value problem

BTW, its great to have a place to ping-pong this ideas

Yea totally!! That what this place is for

Yep, thanks !
2018-11-10

Run a plan, push an artefact, namely the plan output. Run an apply of said build artefact number. This could be a gitsha, PR number, Jenkins build number. I don’t see a huge advantage of Atlantis… although I caveat with not having used so opinions maybe wrong. Have been doing CI/CD of terraform plan + apply for a long time in Jenkins. Not a massive fan of having output in the PR for history. Your git history lives forever, github PRs may not. Running pre merge isnt ideal for team scenario’s. Have seen a lot of failed apply due to vars, state, lots of TF bugs, race conditions…

Things like tflint can help with some of these

E.g. checking a type of AMI is even available in a region

Won’t catch bugs though.

Is Atlantis just like serverless Jenkins ci/cd for terraform with output posted back to the PR?

Sounds nice if you don’t already have a CI/CD solution/don’t need one for other things?
2018-11-11

ok
2018-11-29

anyone here use github actions yet? do you know if there is a way to cache dependencies between builds?

haven’t seen anything in the docs

no one @cloudposse has been invited to the beta

i reached out to some peeps at GitHub but we don’t have the klout =0

@Gabe are you in the beta?

yeah we just got accepted in

just checked my inbox

nothing yet


i don’t think i got an email… just saw the new actions button on our repo

I got the email after the actions button appeared



ohh… they are also only available on private repos

Ohhhhhhhhh snap


yep only private

hmm yeah… it looks pretty cool so far but a few things i’ve noticed is not being able to cache dependencies between builds, no control over the size of the machine it runs on (1 cpu 3.75 gb), and only two concurrent workflows running at one time per repo

pros are that it seems simpler than circle/jenkins and you can create actions that take environment variables so it’s easier to reuse/share actions between repos

… and we just got the email saying they have enabled it for us

i just got the same email a few minutes ago, and Actions on my personal GitHub account


I’m not so lucky. Still waiting
2018-11-30

@Andriy Knysh (Cloud Posse) this screenshot from https://github.com/cloudposse/github-status-updater how do you release to a namespace?

every PR is a new k8s namespace. It’s how we do unlimited staging environments

Thanks!

see @Erik Osterman (Cloud Posse) presentation https://cloudposse.com/devops/unlimited-staging-environments/

How to run complete, disposable apps on Kubernetes for Staging and Development What if you could rapidly spin up new environments in a matter of minutes entirely from scratch, triggered simply by the push of a button or automatically for every Pull Request or Branch. Would that be cool? That’s

@davidvasandani if you have questions or need more info, we can provide it

@Andriy Knysh (Cloud Posse) do you happen to have an example where all these pieces are glued together?

@davidvasandani here’s a simpler and complete working example:

what Add helmfile for deployment with monochart Add codefresh build manifest why Easy deployment to kubernetes

I use it for my demos

basically I took a random app called statup
(self-hosted statuspage.io clone) and deploy it on kubernetes using our monochart
with helmfile
and helm
using codefresh

this supports unlimited staging environments

and automatic destruction when the PR is closed using the pull-request-closed.yaml
pipeline

Thanks @Erik Osterman (Cloud Posse) can’t wait to dig into this.

Crap I realized you need to know all the ENV

@davidvasandani if you PM me I can get them to you

give me a few minutes

re: Self-hosted Helm Chart Registry - Codefresh added Managed Helm Repositories after the presentation, so we use it now instead of deploying our own chart museum


@davidvasandani ^