#release-engineering (2018-11)
All things CI/CD. Specific emphasis on Codefresh and CodeBuild with CodePipeline.
CI/CD Discussions
Archive: https://archive.sweetops.com/release-engineering/
2018-11-01
Ah bah, no :superfresh: emoji.
haha
we can add that
for those that don’t know dustin, he is an awesome support engineer at Codefresh, which is what we use for all of our CI/CD
hey @dustinvb
Hi Andriy.
Happy to help out in anyway I can.
thanks
we all love Codefresh
I do as well. Was a customer for a year before joining.
2018-11-06
Howdy releasers
I wanted to discuss one topic that is spinning in my head for a while:
- Reusing docker or other artifacts from PRs or not
Basically, so speed up the build/release cycle, I was thinking on reusing the tested/built artifact in a PR when we deploy to the QA environment and ofcourse subsequently
yes, reusing the docker image is ideal for deployment to production and preproduction
we tend to treat preproduction and staging slightly different
staging ~ master
preproduction is a tagged release
that tagged release gets promoted to production repos
but do you build it on PR as well? if you dont, do you rerun all tests on the merged commit?
yes, we rerun on merge
so not quite what you say i guess
the key for us is to reuse the image on deploy to production
but not necessarily for all other steps
Yep, that part I have “pinned” i was wondering about the ohter part
we have a pipeline step to promote images and helm charts to production registry
We have basically 2 “stable” branches
- stage (staging/preprod)
- master (prod)
and I wanted to reuse the artifact from the PR to staging, to save some more time
gotcha - yea, we don’t have any thing for that. we also tend to do squash+merge, so the commit sha wouldn’t be the same
Yep, you dont even get a parent commit that way do you?
TBH, the core of the issue is not even on the pipeline, but mostly that some steps are just too slow (thanks Java/SBT dependancies)
yea, i can see why you’d want to do that
So to clean up:
- feature/xyz -> staging (PR): build,test
- staging (commit from merge): build, test, release, deploy
Is that your workflow?
(maybe you do direct to master and use tags or some other CD workflow for prod)
feature/1234/xyz-> staging (pr), build image, run compose integration tests, push image to registry, deploy helm chart to pr-namespace in kubernetes cluster (e.g. pr-1234)master(squash merge): build, test, release, deploy- tag release - build, test, release, deploy to preproduction
- production deploy - promote artifacts to production, deploy
honestly, every customer engagement is slightly different
but it’s more or less like that
Yeah, indeed, im just trying to get some ideas
and most of the examples from “blogs” are simple stuff that would not work/fly in production
yea… i can show you some examples if you want to zoom sometime
what are you using for cicd? are you deploying to kubernetes?
We are still on ECS, looking at EKS right now as we reached that point were ECS is just… annoying and we have enough people to support an EKS
At the moment? Travis, but forget about it as we are migrating to either GitlabCI or Buildkite before EoY hopefully, as we as well outgrew travis
What I don’t like about GitLabCI is it only supports one pipeline per repo (last i checked)
have you looked at Codefresh?
Yeah, tbh im right now leaning to BuildKite as you can even dynamically load “pipeline”
I did, but found it a bit less flexible
Maybe I just did not understood it completly
dynamically load pipeline? what’s that?
Lets say you have a pipeline like:
- build:
- run x
- if build == commit:
- loadpipeline pecigonzalo/this/.pipeline.yml
let me find the docs
and also, seamless triggers: https://buildkite.com/docs/pipelines/trigger-step
In many things is like a nice buildbot (https://buildbot.net/) but pre done
Now, this is not an easy challenge as its “hard” to identify the built artifact
since we enforce “up to date branches” im testing using git rev-parse --short HEAD^2 on the merge|commit job to identify the artifact by the tag (we tag with the commit) of the branch that did the PR that was merged and created that commit.
* 2341145 (HEAD -> master, tag: 0.2.1, origin/master, workingset) Merge pull request #15 from thithat/staging
|\
| * d1bd417 Merge pull request #14 from thithat/feature/this-that
| |\
| | * 43ba953 (feature/this-that) Test Version Flow release
* | | 16c043a Merge pull request #13 from thithat/staging
❯ git rev-parse --short HEAD^2
d1bd417
❯ git rev-parse --short HEAD^1
16c043a
thoughts? ideas? what is your current CI workflow? What do you build on PR and on merge?
so our workflow is slightly different
we have “unlimited staging environments” which correspond to PRs - one per PR; those just build images pinned to 0.0.0-gitsha
entire environments or just images?
2018-11-07
In a similar tone to my previous question, anyone has a working workflow for using terraform plan from PR build or similar? Because while its the recommended way for terraform, I have yet to see a working workflow for it
I think the “interactive pull requests” model is ideally suited for terraform
the slides linked below show a number of companies (recognizable brands) who use atlantis
our fork of atlantis addresses the immediate shortcomings until they are fixed upstream
@pecigonzalo if you are asking about how to trigger terraform plan/apply from an open PR, we recently used atlantis for a GitOps workflow. See #atlantis channel. Also @Erik Osterman (Cloud Posse) recently held a meetup during #connectweek in Pasadena (CA) where he gave a live demo using Atlantis with Terraform to provision AWS user accounts using only Pull Requests
2018-11-08
No, not exactly like that, im not a big fan of what atlantis does
It applies before merge, to me that is anti-pattern
Except for terraform plans are poor
I was thinking appling the plan generated on PR, on the merge
They are optimistic at best
So if you merge and apply now what is in git is not deployed
If others are developing against it, they are just as blocked
So what we’ve reconciled is auto merge on successful apply
Well, but that is the same case for a CD container release
But for container releases it’s more stable
You have more under your control
Most terraform failures in my experience are due to bad values
Containers get their values at runtime
Not at compile time (not generally but ideally)
You can achieve what you want easily with Codefresh.
Yeah, that is true, and a fair point, but not this one:
So if you merge and apply now what is in git is not deployed
If others are developing against it, they are just as blocked
In theory in the container you have the same “posibility” of failure
but I agree, is not as likely, as easiert to caught on test
TF has a lot of sideeffect failure scenarios, that plan does not catch
To me atlantis is a practical approach
Not the theoretical ideal
So rather than clutter the master commit history with a bunch of patch releases, rather get it in clean. We preserve a full transcript in the git comments so we have a record of what is deployed.
Even if the PR is closed but half applied, there’s a record of that’s
Yeah, its not a bad approach and im not saying Atlantis sucks, much the contrary, even Hashi bought them
I just feel that is a “hack” around a bigger issue
a great hack, but a hack anyway
Agreed
In my previous company we had a rake looper to get around TF Modules not having count, it worked, and it saved us a lot of time, but in “terraform” it was a hack
Yea I have heard of similar hacks… basically around code generation
Slippery slope
I think within a closed ecosystem of a corporate environment that might fly, but it makes it very difficult to write portable code for open source
Yeah indeed, im not so keen on it anymore. Good thing TF 0.12 is around the corner . This was a long time ago
I wouldn’t touch 0.12 for a while…
I just don’t see a way around it that’s practical to solve (unless your HashiCorp and have 100mil in fresh green)
Yeah true
I might give it 2nd chance, at least it will avoid those fix PRs for a stupid terraform value problem
BTW, its great to have a place to ping-pong this ideas
Yea totally!! That what this place is for
Yep, thanks !
2018-11-10
Run a plan, push an artefact, namely the plan output. Run an apply of said build artefact number. This could be a gitsha, PR number, Jenkins build number. I don’t see a huge advantage of Atlantis… although I caveat with not having used so opinions maybe wrong. Have been doing CI/CD of terraform plan + apply for a long time in Jenkins. Not a massive fan of having output in the PR for history. Your git history lives forever, github PRs may not. Running pre merge isnt ideal for team scenario’s. Have seen a lot of failed apply due to vars, state, lots of TF bugs, race conditions…
Things like tflint can help with some of these
E.g. checking a type of AMI is even available in a region
Won’t catch bugs though.
Is Atlantis just like serverless Jenkins ci/cd for terraform with output posted back to the PR?
Sounds nice if you don’t already have a CI/CD solution/don’t need one for other things?
2018-11-11
ok
2018-11-29
anyone here use github actions yet? do you know if there is a way to cache dependencies between builds?
haven’t seen anything in the docs
no one @cloudposse has been invited to the beta
i reached out to some peeps at GitHub but we don’t have the klout =0
@Gabe are you in the beta?
yeah we just got accepted in
just checked my inbox
nothing yet
i don’t think i got an email… just saw the new actions button on our repo
I got the email after the actions button appeared
ohh… they are also only available on private repos
Ohhhhhhhhh snap
yep only private
hmm yeah… it looks pretty cool so far but a few things i’ve noticed is not being able to cache dependencies between builds, no control over the size of the machine it runs on (1 cpu 3.75 gb), and only two concurrent workflows running at one time per repo
pros are that it seems simpler than circle/jenkins and you can create actions that take environment variables so it’s easier to reuse/share actions between repos
… and we just got the email saying they have enabled it for us
i just got the same email a few minutes ago, and Actions on my personal GitHub account
I’m not so lucky. Still waiting
2018-11-30
@Andriy Knysh (Cloud Posse) this screenshot from https://github.com/cloudposse/github-status-updater how do you release to a namespace?
every PR is a new k8s namespace. It’s how we do unlimited staging environments
Thanks!
see @Erik Osterman (Cloud Posse) presentation https://cloudposse.com/devops/unlimited-staging-environments/
@davidvasandani if you have questions or need more info, we can provide it
@Andriy Knysh (Cloud Posse) do you happen to have an example where all these pieces are glued together?
@davidvasandani here’s a simpler and complete working example:
I use it for my demos
basically I took a random app called statup (self-hosted statuspage.io clone) and deploy it on kubernetes using our monochart with helmfile and helm using codefresh
this supports unlimited staging environments
and automatic destruction when the PR is closed using the pull-request-closed.yaml pipeline
Thanks @Erik Osterman (Cloud Posse) can’t wait to dig into this.
Crap I realized you need to know all the ENV
@davidvasandani if you PM me I can get them to you
give me a few minutes
re: Self-hosted Helm Chart Registry - Codefresh added Managed Helm Repositories after the presentation, so we use it now instead of deploying our own chart museum
@davidvasandani ^