#release-engineering (2019-01)

jenkins_ci All things CI/CD. Specific emphasis on Codefresh and CodeBuild with CodePipeline.

CI/CD Discussions

Archive: https://archive.sweetops.com/release-engineering/

2019-01-21

2019-01-17

Igor Rodionov avatar
Igor Rodionov

Nice

joshmyers avatar
joshmyers
jessfraz/junk

A place for everything without a home. Contribute to jessfraz/junk development by creating an account on GitHub.

pecigonzalo avatar
pecigonzalo

Hey we are evaluating diff tools for CI/CD (codefresh, buildkite, codepipeline, etc). I have a question for the users of codepipeline, how do you CD your pipeline definitions? EG: in codefresh or buildkite, this is read from VCS when a run is triggered

Close I could see is have the pipeline, update the pipeline itself, but im a bit concerned how would this effect the run itself that updated the build. Lets say you have a PR that updates the pipeline and some code to acomodate the pipeline update, you PR CI runs using the old pipeline. Lets say you merge, it deploys the new pipeline update first, would the subsequent steps run with the updated pipeline?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@pecigonzalo we have an example of using CodePipeline here https://github.com/cloudposse/terraform-aws-jenkins/blob/master/main.tf#L115

cloudposse/terraform-aws-jenkins

Terraform module to build Docker image with Jenkins, save it to an ECR repo, and deploy to Elastic Beanstalk running Docker stack - cloudposse/terraform-aws-jenkins

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

it deploys this repo (by default) https://github.com/cloudposse/jenkins

cloudposse/jenkins

Contribute to cloudposse/jenkins development by creating an account on GitHub.

pecigonzalo avatar
pecigonzalo

Yeah I have seen that @Andriy Knysh (Cloud Posse) thanks

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
cloudposse/jenkins

Contribute to cloudposse/jenkins development by creating an account on GitHub.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

which is in the repo itself

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

i mean you can add any commends in there

pecigonzalo avatar
pecigonzalo

Yeah,but I dont know if that answers my question tho

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

yea, I don’t know either was long time when we tested it

pecigonzalo avatar
pecigonzalo

pecigonzalo avatar
pecigonzalo

I mean, the module deploys a pipeline, that is great, and then the jenkins repo has its own build definitions

pecigonzalo avatar
pecigonzalo

which is also good

pecigonzalo avatar
pecigonzalo

but then, what deploys the pipeline itself? another pipeline?

pecigonzalo avatar
pecigonzalo

(talking about AWS codepipeline)

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

the module deploys it

pecigonzalo avatar
pecigonzalo

when using codefresh, buildkite, etc you can define the pipeline itself in the repo

pecigonzalo avatar
pecigonzalo

the module defines the code for it, but does not deploy it

pecigonzalo avatar
pecigonzalo

you have to “apply” that module somewhere

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

ah i see now what you mean

pecigonzalo avatar
pecigonzalo

I have seen some other CI/CD tools/companies have a repo containing all the pipelines, and that itself had its own pipeline, manually deployed as it was simple enought

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We have a strategy of provisioning the pipeline creation via GitOps via a centralized repo

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but then keep the pipelines themselves in the repos they manage.

pecigonzalo avatar
pecigonzalo

Im not sure I follow, you have repo with the list of repos, clone all, parse/deploy the pipelines in them?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so, in most CICD systems (other than CodeBuild/CodePipeline with terraform), the act of adding the repo, configuring the pipeline to use a manifest (e.g. circle.yml, travis.yml, codefresh.yml, Jenkinsfile) is a manual process

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We’ve automated that provisioning using GitOps style best practices

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

For #codefresh, we stick that in a org/codefresh repo

pecigonzalo avatar
pecigonzalo

Well actually, for travis and I believe also circle its just enabling it like, travis enable from the repo, then it automatically pics the file

pecigonzalo avatar
pecigonzalo

But I think I got the idea, and if I understood correctly it is similar to the workflow I was describing on the top of the thread I think

pecigonzalo avatar
pecigonzalo

but meh, not a fan of that workflow

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

so for a workflow to update the pipeline itself, maybe consider atlantis

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

(which of cause needs its own pipeline to be deployed or you can just apply it to deploy on fargate as @Erik Osterman (Cloud Posse) is doing now)

pecigonzalo avatar
pecigonzalo

Exactly! its the chicken and egg

pecigonzalo avatar
pecigonzalo

Yeah fargate seems perfect for atlantis

pecigonzalo avatar
pecigonzalo

TBH, some of the things that we dont deploy often, I dont mind deploying “automanually”

pecigonzalo avatar
pecigonzalo

as a side note, I think I mentioned this before, as much as I like the idea of atlantis I dont like the apply before PR merge workflow. It would be like deploying before merge

pecigonzalo avatar
pecigonzalo

it can even have issues on non up to date branches

2019-01-16

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
pnikosis/semtag

A sematic tag script for Git. Contribute to pnikosis/semtag development by creating an account on GitHub.

:--1:1
antonbabenko avatar
antonbabenko

Awesome! I was looking for this myself!

pnikosis/semtag

A sematic tag script for Git. Contribute to pnikosis/semtag development by creating an account on GitHub.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(via @mumoshu)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(saw he was using it in #variant)

pecigonzalo avatar
pecigonzalo

We do manual version bumping, i have never found a way of doing directly from CI that “knows” when to bump each part, unless the team is strict about idk, commit messages or similar

pecigonzalo avatar
pecigonzalo

do you guys integrate this into your pipeline?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

sooooo one way I’ve seen this done that I like:

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

add a file to the repo called VERSION or something

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

as a human, you stick the version in there.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

you can use a tool like semtag to do that.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Then the CI process will look at that file on a merge to master

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

and call out to github to do a release tag with VERSION.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think this is a pretty elegant way of doing things.

pecigonzalo avatar
pecigonzalo

Yeah, that is basically what we do, minus the tool for bumping, as tbh its not that much extra work to bump manually

pecigonzalo avatar
pecigonzalo

then we trigger a GH release if VERSION > (existing tags)

pecigonzalo avatar
pecigonzalo

for that we do use a tool to do semver comparison, actually python

pecigonzalo avatar
pecigonzalo

for our branch model, a PR with a released version already existing, breaks the build

pecigonzalo avatar
pecigonzalo

in a master only branch model, you can just version-commitsha if version already exists (for the docker/artifact tag)

pecigonzalo avatar
pecigonzalo

so you dont release again, until a version bump basically

:--1:1
michal.matyjek avatar
michal.matyjek

we’re considering using something like https://github.com/mkj28/semversions for CI/CD where each commit to, say, master gets unique, sequential semver-like tag. Uses vYYYY.MMDD.nnnn format, so allows us to avoid “what does semver mean for my project” discusssion

mkj28/semversions

For testing semantic versions in git. Contribute to mkj28/semversions development by creating an account on GitHub.

1
michal.matyjek avatar
michal.matyjek

(it is Codefresh-specific)

michal.matyjek avatar
michal.matyjek

(but can be easily repurposed as needed)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Igor Rodionov

    keyboard_arrow_up