#release-engineering (2019-01)

jenkins_ci All things CI/CD. Specific emphasis on Codefresh and CodeBuild with CodePipeline.

CI/CD Discussions

Archive: https://archive.sweetops.com/release-engineering/

2019-01-21

2019-01-17

Igor Rodionov avatar
Igor Rodionov

Nice

joshmyers avatar
joshmyers
jessfraz/junk

A place for everything without a home. Contribute to jessfraz/junk development by creating an account on GitHub.

pecigonzalo avatar
pecigonzalo

Hey we are evaluating diff tools for CI/CD (codefresh, buildkite, codepipeline, etc). I have a question for the users of codepipeline, how do you CD your pipeline definitions? EG: in codefresh or buildkite, this is read from VCS when a run is triggered

Close I could see is have the pipeline, update the pipeline itself, but im a bit concerned how would this effect the run itself that updated the build. Lets say you have a PR that updates the pipeline and some code to acomodate the pipeline update, you PR CI runs using the old pipeline. Lets say you merge, it deploys the new pipeline update first, would the subsequent steps run with the updated pipeline?

aknysh avatar
aknysh

@pecigonzalo we have an example of using CodePipeline here https://github.com/cloudposse/terraform-aws-jenkins/blob/master/main.tf#L115

cloudposse/terraform-aws-jenkins

Terraform module to build Docker image with Jenkins, save it to an ECR repo, and deploy to Elastic Beanstalk running Docker stack - cloudposse/terraform-aws-jenkins

aknysh avatar
aknysh

it deploys this repo (by default) https://github.com/cloudposse/jenkins

cloudposse/jenkins

Contribute to cloudposse/jenkins development by creating an account on GitHub.

pecigonzalo avatar
pecigonzalo

Yeah I have seen that @aknysh thanks

aknysh avatar
aknysh
cloudposse/jenkins

Contribute to cloudposse/jenkins development by creating an account on GitHub.

aknysh avatar
aknysh

which is in the repo itself

aknysh avatar
aknysh

i mean you can add any commends in there

pecigonzalo avatar
pecigonzalo

Yeah,but I dont know if that answers my question tho

aknysh avatar
aknysh

yea, I don’t know either was long time when we tested it

pecigonzalo avatar
pecigonzalo

pecigonzalo avatar
pecigonzalo

I mean, the module deploys a pipeline, that is great, and then the jenkins repo has its own build definitions

pecigonzalo avatar
pecigonzalo

which is also good

pecigonzalo avatar
pecigonzalo

but then, what deploys the pipeline itself? another pipeline?

pecigonzalo avatar
pecigonzalo

(talking about AWS codepipeline)

aknysh avatar
aknysh

the module deploys it

pecigonzalo avatar
pecigonzalo

when using codefresh, buildkite, etc you can define the pipeline itself in the repo

pecigonzalo avatar
pecigonzalo

the module defines the code for it, but does not deploy it

pecigonzalo avatar
pecigonzalo

you have to “apply” that module somewhere

aknysh avatar
aknysh

ah i see now what you mean

pecigonzalo avatar
pecigonzalo

I have seen some other CI/CD tools/companies have a repo containing all the pipelines, and that itself had its own pipeline, manually deployed as it was simple enought

Erik Osterman avatar
Erik Osterman

We have a strategy of provisioning the pipeline creation via GitOps via a centralized repo

Erik Osterman avatar
Erik Osterman

but then keep the pipelines themselves in the repos they manage.

pecigonzalo avatar
pecigonzalo

Im not sure I follow, you have repo with the list of repos, clone all, parse/deploy the pipelines in them?

Erik Osterman avatar
Erik Osterman

so, in most CICD systems (other than CodeBuild/CodePipeline with terraform), the act of adding the repo, configuring the pipeline to use a manifest (e.g. circle.yml, travis.yml, codefresh.yml, Jenkinsfile) is a manual process

Erik Osterman avatar
Erik Osterman

We’ve automated that provisioning using GitOps style best practices

Erik Osterman avatar
Erik Osterman

For #codefresh, we stick that in a org/codefresh repo

pecigonzalo avatar
pecigonzalo

Well actually, for travis and I believe also circle its just enabling it like, travis enable from the repo, then it automatically pics the file

pecigonzalo avatar
pecigonzalo

But I think I got the idea, and if I understood correctly it is similar to the workflow I was describing on the top of the thread I think

pecigonzalo avatar
pecigonzalo

but meh, not a fan of that workflow

aknysh avatar
aknysh

so for a workflow to update the pipeline itself, maybe consider atlantis

aknysh avatar
aknysh

(which of cause needs its own pipeline to be deployed or you can just apply it to deploy on fargate as @Erik Osterman is doing now)

pecigonzalo avatar
pecigonzalo

Exactly! its the chicken and egg

pecigonzalo avatar
pecigonzalo

Yeah fargate seems perfect for atlantis

pecigonzalo avatar
pecigonzalo

TBH, some of the things that we dont deploy often, I dont mind deploying “automanually”

pecigonzalo avatar
pecigonzalo

as a side note, I think I mentioned this before, as much as I like the idea of atlantis I dont like the apply before PR merge workflow. It would be like deploying before merge

pecigonzalo avatar
pecigonzalo

it can even have issues on non up to date branches

2019-01-16

Erik Osterman avatar
Erik Osterman
pnikosis/semtag

A sematic tag script for Git. Contribute to pnikosis/semtag development by creating an account on GitHub.

:--1:1
antonbabenko avatar
antonbabenko

Awesome! I was looking for this myself!

pnikosis/semtag

A sematic tag script for Git. Contribute to pnikosis/semtag development by creating an account on GitHub.

Erik Osterman avatar
Erik Osterman

(via @mumoshu)

Erik Osterman avatar
Erik Osterman

(saw he was using it in #variant)

pecigonzalo avatar
pecigonzalo

We do manual version bumping, i have never found a way of doing directly from CI that “knows” when to bump each part, unless the team is strict about idk, commit messages or similar

pecigonzalo avatar
pecigonzalo

do you guys integrate this into your pipeline?

Erik Osterman avatar
Erik Osterman

sooooo one way I’ve seen this done that I like:

Erik Osterman avatar
Erik Osterman

add a file to the repo called VERSION or something

Erik Osterman avatar
Erik Osterman

as a human, you stick the version in there.

Erik Osterman avatar
Erik Osterman

you can use a tool like semtag to do that.

Erik Osterman avatar
Erik Osterman

Then the CI process will look at that file on a merge to master

Erik Osterman avatar
Erik Osterman

and call out to github to do a release tag with VERSION.

Erik Osterman avatar
Erik Osterman

I think this is a pretty elegant way of doing things.

pecigonzalo avatar
pecigonzalo

Yeah, that is basically what we do, minus the tool for bumping, as tbh its not that much extra work to bump manually

pecigonzalo avatar
pecigonzalo

then we trigger a GH release if VERSION > (existing tags)

pecigonzalo avatar
pecigonzalo

for that we do use a tool to do semver comparison, actually python

pecigonzalo avatar
pecigonzalo

for our branch model, a PR with a released version already existing, breaks the build

pecigonzalo avatar
pecigonzalo

in a master only branch model, you can just version-commitsha if version already exists (for the docker/artifact tag)

pecigonzalo avatar
pecigonzalo

so you dont release again, until a version bump basically

:--1:1
michal.matyjek avatar
michal.matyjek

we’re considering using something like https://github.com/mkj28/semversions for CI/CD where each commit to, say, master gets unique, sequential semver-like tag. Uses vYYYY.MMDD.nnnn format, so allows us to avoid “what does semver mean for my project” discusssion

mkj28/semversions

For testing semantic versions in git. Contribute to mkj28/semversions development by creating an account on GitHub.

1
michal.matyjek avatar
michal.matyjek

(it is Codefresh-specific)

michal.matyjek avatar
michal.matyjek

(but can be easily repurposed as needed)

Erik Osterman avatar
Erik Osterman

@Igor Rodionov

    keyboard_arrow_up