#release-engineering

CI/CD Discussions

Archive: https://archive.sweetops.com/release-engineering/

2019-10-20

2019-10-18

davidvasandani

Does anyone pay for Serverless Enterprise?

2019-10-17

trying to wrap my head around releasing node apps vs semantic versioning (version change in package.json) and making it simple for developers. Couple ideas, but open for more: 1) with release branch - release is merging from master to release, then use https://github.com/semantic-release/semantic-release to bump version, tag git, and package 2) when developer bumps version in package.json- detect that and release then? Any tooling for this style though?

jafow


Any tooling for this style though?
we do something similar and built some custom scripts that track all versions of apps in release with a manifest.

jafow

the release captain or lead is responsible for deciding if there’s a semver minor or major bump. the versioning script picks it up and updates

jafow

mostly i’m commenting here to get updates on this thread because i’d be interested in hearing more about this.

Erik Osterman

Mostly agree with @jafow

Erik Osterman

One other thing I have seen and like is simply to define the release in the code. That way it can be reviewed. Using GitHub actions you can easily enforce that it is modified. Then on merge a release will be cut based on that file

Erik Osterman

I think developing a heuristic is difficult and mileage may vary

Erik Osterman

This approach means that you get full control over the semver and easily enforced

Erik Osterman

Couple this with CODEOWNERS and you also have a way to ensure release captains are involved

With CircleCi, we can use either a merge or a specific tag to get a release going. It can do the version bump on its own so develop doesn’t have to think about it

2019-10-16

davidvasandani

@aknysh I’d love to hear more about this.

aknysh

i understand this is in a private repo implemented for a customer. We need to discuss with @Adam how to share it

1
davidvasandani

@Brij S we use the serverless framework a lot and are also struggling with CI/CD. I’d love to hear what you’ve come up with so far and how you’re doing branching.

Brij S

as of now, the best ive got for you is to use github actions and codepipeline

Brij S

its definitely not the best idea, but it ‘works’

davidvasandani

Alex Siegman

I actually jotted down some notes on serverless and CICD, it’s on my list for an upcoming project coming from one of my dev teams. I’m not sure what part of CICD you’re struggling with, but I think it’s key to decide what resources live with what development lifecycle.

My notes aren’t really good for public consumption since it’s all shorthand, but, the general concept is that you should figure out the demarcation of what serverless manages, and what your other IaC manages. Then your CICD for serverless can become much easier, as it just pulls in dependencies through SSM, or CloudFormation exports, or any number of things.

Here’s a link talking about this by using cloudformation exports as an intermediary: https://theburningmonk.com/2019/03/making-terraform-and-serverless-framework-work-together/

Making Terraform and Serverless framework work together | theburningmonk.com

The Serverless framework is the most popular deployment framework for serverless applications. It gives you a convenient abstraction over CloudFormation and some best practices out-of-the-box: Filters out dev dependencies for Node.js function. Update deployment packages to S3, which lets you work around the default 50MB limit on deployment packages. Enforces a consistent naming convention for … Making Terraform and Serverless framework work together Read More »

1

2019-10-08

Brij S

does anyone here use the serverless framework? If so, do you have a CICD pipeline for it(looking for a way to do automated rollbacks upon failure, manual gates for going to prod etc)

Erik Osterman

We implemented this with Codefresh

Erik Osterman

We used approval steps for gates

Erik Osterman

And implemented unlimited staging environments

Erik Osterman

@aknysh can share more

Erik Osterman

@Adam is one happy camper

Adam
02:59:00 AM

@Adam has joined the channel

2019-10-03

Sharanya

Create Jenkinsfile to deploy UI code to S3 bucket.

1

2019-09-26

Erik Osterman

@ you run some argo, no?

yeah…. let me ping the argo person

wookasz
06:34:39 PM

@wookasz has joined the channel

2019-09-24

anyone use self-hosted argo in production? if so, what are your thoughts? https://argoproj.github.io/

Get stuff done with Kubernetes | Argo

Kubernetes native workflows, deployments, CI, events

@wookasz see here

Get stuff done with Kubernetes | Argo

Kubernetes native workflows, deployments, CI, events

wookasz

we do, we run some ML training pipelines as well as some very simple workflows that perform a bit of data extraction and dump it into S3

personally I really like it and it’s been fairly reliable and very flexible

wookasz

happy to answer any questions

@wookasz awesome! im spinning it up on k8s soon and will be testing it out

wookasz

we have not used it for CI/CD

wookasz

also, some developers find the workflow.yaml configuration a bit cumbersome, but I think for someone used to k8s manifests it shouldn’t be problem

2019-09-14

davidvasandani

@ Are you able to manually trigger the Jenkins job with a webhook?

2019-09-13

GitHub -> Jenkins webhooks for PR builds - mine stopped working, although the hook is recieved and jenkins replies 200, no logs in system log that are helpful besides it recieving the call. Any ideas? I know it can get pretty complex

Received PULL_REQUEST for https://<github repo> from 140.82.115.247 ⇒ https://<myjenkins>/github-webhook/
Sep 13, 2019 6<i class="em em-52"></i>04 PM INFO com.squareup.okhttp.internal.Platform$JdkWithJettyBootPlatform getSelectedProtocol
ALPN callback dropped: SPDY and HTTP/2 are disabled. Is alpn-boot on the boot class path?

ive tried changing log levels but no additional logs are added

2019-09-09

2019-09-08

Erik Osterman

@Callum Robertson ask @ about spinnaker

Callum Robertson

Hi @, thanks for the intro @Erik Osterman. I luckily got my artifacts working well.

Callum Robertson

Sometimes taking a break and coming back to the problem is the best way to solve it haha

Callum Robertson

Good to know that you’re Spinnaker weapon. I’ll keep that in mind

Callum Robertson
01:57:35 AM

@Callum Robertson has joined the channel

2019-09-02

Gocho

Hi all, forgive me if my question is stupid : I opened a PR for https://github.com/cloudposse/terraform-aws-lambda-elasticsearch-cleanup, but it failed on terraform lint. I’m trying to understand why…

But when I read the makefile, I don’t really understand how the lint works (I use TFLint, and it says everything is ok) … what does the terraform/install terraform/lint refers to ?

cloudposse/terraform-aws-lambda-elasticsearch-cleanup

Terraform module to provision a scheduled Lambda function which will delete old AWS ElasticSearch indices - cloudposse/terraform-aws-lambda-elasticsearch-cleanup

curious deviant
cloudposse/terraform-aws-lambda-elasticsearch-cleanup

Terraform module to provision a scheduled Lambda function which will delete old AWS ElasticSearch indices - cloudposse/terraform-aws-lambda-elasticsearch-cleanup

Gocho

Thanks for your reply. I tried terraform fmt, but it gave me nothing I don’t know if that is relevant, but I’m using terrraform 0.11, not 0.12 yet

curious deviant

yup.. just realized that

curious deviant

I cloned your repo.. however for me it did format the <http://outputs.tf>. I am using 0.12.3 on my local

curious deviant

Also if we look at the old build that was passing, looks to me the version shouldn’t be the issue

Gocho

I just realized the format did something… it added a comma after the actions/resources values if http://main.tf

Gocho

I try to commit that

Gocho

ok, that was that. Build has passed the lint step Thanks !

curious deviant

so glad it worked out

Erik Osterman

@Gocho we will take a look tomorrow

1
Erik Osterman

@aknysh

2019-08-31

Erik Osterman

That’s the HCL format which is already deprecated

Erik Osterman

Here’s an example of yaml

Erik Osterman
Label Pull Requests by osterman · Pull Request #515 · cloudposse/geodesic

what Automatically label PRs based on what they modify why Easily filter PRs based on what they manage

Erik Osterman
Add GitHub Actions by osterman · Pull Request #165 · cloudposse/build-harness

what Add action to automatically rebuild readme and push upstream Respond to /readme command. Note: issue_comment workflows use master workflow for security. See https://developer.github.com/actio

Erik Osterman

@Brij S the second link has links to what I found useful when learning about them

2019-08-29

Brij S

anyone tryout github actions yet? Ive created a .github/main.workflow file to be invoked for each PR, but when I create a PR it doesnt do anything

2019-08-28

Sharanya

Did anyone Come across NPM memory Issues ?

2019-07-10

Milos Backonja
08:13:12 AM

@Milos Backonja has joined the channel

2019-06-11

Erik Osterman

Also love #codefresh

dustinvb
11:42:03 PM

2019-06-07

been using codefresh for about a week. its the most feature rich one out there for k8s that ive used

4
2

2019-06-06

deftunix

hi all, we are looking for a cicd toolchain that support deployment on vm and kubernetes containers. Any thought?

oscarsullivan_old
3

2019-05-08

Erik Osterman
Erik Osterman

we are on now

cabrinha

cant talk but will listen

2019-05-06

cabrinha

What is the go-to or standard approach to SDLC for infrastructure code?

cabrinha

Anyone got some articles/docs? I’m also wondering what CloudPosse uses and suggests.

Erik Osterman

@cabrinha i don’t have it written down, but have lots and lots to say about it

3
Erik Osterman

if you want to join this wednesday office hours at 11:30 am

Erik Osterman

i’ll share

cabrinha

i’ll try to make it to that, let me know if you throw a blog post out there in the meantime

2019-05-04

evgmoskalenko

Thanks)))

2019-05-03

evgmoskalenko

Hello, Could you tell me please, how are you deploy services to the AWS? Just describe by Terraform the infrastructure in a separate repository?

evgmoskalenko

Or describe the service in Terraform and put terraform code in the repository of the service?

aknysh

@evgmoskalenko usually #1 - describe infrastructure in a separate repo. We do multi-account setup, one AWS account per environment (prod, staging, dev, testing etc.). Each environment/account is represented by a GitHub repo, which usually provides all the variables (from Dockerfile, ENV vars, TF vars, etc.)

aknysh

take a look here:

aknysh
cloudposse/reference-architectures

Get up and running quickly with one of our reference architecture using our fully automated cold-start process. - cloudposse/reference-architectures

aknysh

All TF code is in one central catalog that is reused across all repos https://github.com/cloudposse/terraform-root-modules

cloudposse/terraform-root-modules

Example Terraform service catalog of “root module” blueprints for provisioning reference architectures - cloudposse/terraform-root-modules

aknysh
cloudposse/root.cloudposse.co

Example Terraform Reference Architecture for Geodesic Module Parent (“Root” or “Identity”) Organization in AWS. - cloudposse/root.cloudposse.co

aknysh
cloudposse/prod.cloudposse.co

Example Terraform/Kubernetes Reference Infrastructure for Cloud Posse Production Organization in AWS - cloudposse/prod.cloudposse.co

2019-04-29

vishnu.shukla

Issue with the authentication from your source code

vishnu.shukla

check the build error

Erik Osterman
Jenkins is Getting Old

Jenkins has become a victim of its own success.

2

2019-04-26

chirag kalal

I was trying to run AWS codebuild with ECR and include buildspeac.yml file in my project but still getting error like:

chirag kalal
[Container] 2019/04/26 06<i class="em em-27"></i>27 Waiting for agent ping
[Container] 2019/04/26 06<i class="em em-27"></i>32 Waiting for DOWNLOAD_SOURCE
[Container] 2019/04/26 06<i class="em em-27"></i>36 Phase is DOWNLOAD_SOURCE
[Container] 2019/04/26 06<i class="em em-27"></i>36 CODEBUILD_SRC_DIR=/codebuild/output/src310808022/src/bitbucket.org/xyz/demo
[Container] 2019/04/26 06<i class="em em-27"></i>36 Phase complete: DOWNLOAD_SOURCE State: FAILED
[Container] 2019/04/26 06<i class="em em-27"></i>36 Phase context status code: YAML_FILE_ERROR Message: YAML file does not exist

2019-04-23

Titus Rapid

anyone have massive mysql databases? I am looking for other methods to release 1-2TB datasets

2019-04-19

dustinvb

I am looking for a simple tool that is maybe already in a Docker image that can take server/credentials vars and remote execute commands on that server with just SSH. I’ve found fabric and sup both seem a little advanced for what I am looking for since I’d ideally like to just provide the vars there and then specify the commands maybe another var for the command? Nice-to-have would be the ability to specify a file to send over and execute. Anyone have any suggestions?

Erik Osterman

I am a bit confused. The SSH cli can do that out of the box. You just need to pass the credentials to the Docker image.

Erik Osterman

Similarly scp would let you copy

dustinvb

Which SSH image is this?

Erik Osterman

Any image that ships with open SSH

Erik Osterman

(E.g. geodesic)

Erik Osterman

There might be an official openSSH image too

dustinvb

Thanks

Erik Osterman

I can work through it with you if it would help

dustinvb

I was trying to reduce commands like

    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh
    - echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
    - echo "$DEV_SSH_PRIVATE_KEY" > ~/.ssh/id_rsa
    - chmod 600 ~/.ssh/id_rsa
    - ssh $DEV_SSH_HOST 'git --work-tree=/X/Y --git-dir=/X/Y/.git pull origin'

With some type of entrypoint scripting. No worries.

2019-04-18

vishnu.shukla

any one using AWS codepipeline for Ruby framework??

vishnu.shukla
vishnu.shukla

can someone help me on this error

vishnu.shukla

?

Alex Co

it does not find any available runtime for js

Alex Co

you may want to check the bundle setup

Alex Co
rails/execjs

Run JavaScript code from Ruby. Contribute to rails/execjs development by creating an account on GitHub.

Alex Co
rails/execjs

Run JavaScript code from Ruby. Contribute to rails/execjs development by creating an account on GitHub.

Alex Co

the simplest way should be install a version of nodejs alongside with the build environment

Alex Co
rails/execjs

Run JavaScript code from Ruby. Contribute to rails/execjs development by creating an account on GitHub.

2019-04-11

@ do you use the open source or enterprise?

@btai we are using the open source one, we use the official 1.x builds as we are under gmv limits. Also we are currently testing knative build to possibly replace drone.

Gmv limits? what’s knative build?

Interesting

2019-04-08

we are using drone for quiet some time. Apart from licensing it works pretty great and the author is quiet responsive on gitter to address your issues

2019-04-01

anyone use https://docs.drone.io/ on k8s before? if so, how was setting it up with the helm chart and how stable is it?

2019-03-27

oscarsullivan_old

@Erik Osterman I’ve got Codefresh pushing to ECR (and apparently codefreshe’s own registry too). How does one go about now pulling that ECR image?

Have tried: aws-vault exec acme-mgmt-iac aws ecr get-login aws-vault exec acme-mgmt-iac docker pull <http<i class="em em-//xxxx.dkr.ecr.eu-west-2.amazonaws.com/acme/foo"></i>develop>

But both have their errors

oscarsullivan_old

I can see a quick workaround actually by having a default credentials so I can avoid aws-vault

oscarsullivan_old

Well that workaround worked. But would be to find out the answer.

oscarsullivan_old

Likewise how to avoid:

REPOSITORY                                                   TAG                        IMAGE ID            CREATED             SIZE
<http://xxx.dkr.ecr.eu-west-2.amazonaws.com/acme/pod>   develop                    xxx        41 minutes ago      498MB

so I can just run docker run acme/pod:develop instead of docker run <http<i class="em em-//xxx.dkr.ecr.eu-west-2.amazonaws.com/acme/pod"></i>develop>

aknysh
Using the AWS CLI with Amazon ECR - Amazon ECR

The following steps help you install the AWS Command Line Interface and then log in to Amazon ECR. From there, you can create an image repository, push an image to that repository, and perform other common scenarios in Amazon ECR.

aknysh

docker needs a registry to pull an image (DockerHub by default), so docker run acme/pod:develop will not work with ECR

Tim Malone

…unless you hack your DNS to make http://hub.docker.com (or whatever domain it connects to when pulling) return your ECR endpoint instead

meh, I seem to recall some ssl explosions doing that.

1
oscarsullivan_old

Ooo Tim I kind of fancy doing that but I just know I’ll need the standard http://hub.docker.com at some point ;)

1
oscarsullivan_old

Going to use variant instead

2019-02-14

Erik Osterman
05:22:15 AM

@Erik Osterman set the channel purpose: CI/CD Discussions

Archive: https://archive.sweetops.com/release-engineering/

2019-02-05

Erik Osterman

Erik Osterman

Codepipeline is primitive

Erik Osterman

for example, they just added webhooks, but even with webhooks you cannot access data from the webhooks

Erik Osterman

@F3D3M2C0R I’d be happy to give you a demo of #codefresh so you can judge for yourself

2
joshmyers

Anything is better than codepipeline

joshmyers

For anything non trivial

F3D3M2C0R

you’re right, however i’m looking for some functional, there are several tools but they have pain to setting up and mantaince

F3D3M2C0R

spinnaker looks good

2019-02-04

F3D3M2C0R

hi, do you think that codefresh it’s better than codepipeline for CI/CD in aws eks? I’m new with aws eks

2019-01-21

2019-01-17

Igor Rodionov

Nice

joshmyers
jessfraz/junk

A place for everything without a home. Contribute to jessfraz/junk development by creating an account on GitHub.

Hey we are evaluating diff tools for CI/CD (codefresh, buildkite, codepipeline, etc). I have a question for the users of codepipeline, how do you CD your pipeline definitions? EG: in codefresh or buildkite, this is read from VCS when a run is triggered

Close I could see is have the pipeline, update the pipeline itself, but im a bit concerned how would this effect the run itself that updated the build. Lets say you have a PR that updates the pipeline and some code to acomodate the pipeline update, you PR CI runs using the old pipeline. Lets say you merge, it deploys the new pipeline update first, would the subsequent steps run with the updated pipeline?

aknysh
cloudposse/terraform-aws-jenkins

Terraform module to build Docker image with Jenkins, save it to an ECR repo, and deploy to Elastic Beanstalk running Docker stack - cloudposse/terraform-aws-jenkins

aknysh

it deploys this repo (by default) https://github.com/cloudposse/jenkins

cloudposse/jenkins

Contribute to cloudposse/jenkins development by creating an account on GitHub.

Yeah I have seen that @aknysh thanks

aknysh
cloudposse/jenkins

Contribute to cloudposse/jenkins development by creating an account on GitHub.

aknysh

which is in the repo itself

aknysh

i mean you can add any commends in there

Yeah,but I dont know if that answers my question tho

aknysh

yea, I don’t know either was long time when we tested it

I mean, the module deploys a pipeline, that is great, and then the jenkins repo has its own build definitions

which is also good

but then, what deploys the pipeline itself? another pipeline?

(talking about AWS codepipeline)

aknysh

the module deploys it

when using codefresh, buildkite, etc you can define the pipeline itself in the repo

the module defines the code for it, but does not deploy it

you have to “apply” that module somewhere

aknysh

ah i see now what you mean

I have seen some other CI/CD tools/companies have a repo containing all the pipelines, and that itself had its own pipeline, manually deployed as it was simple enought

Erik Osterman

We have a strategy of provisioning the pipeline creation via GitOps via a centralized repo

Erik Osterman

but then keep the pipelines themselves in the repos they manage.

Im not sure I follow, you have repo with the list of repos, clone all, parse/deploy the pipelines in them?

Erik Osterman

so, in most CICD systems (other than CodeBuild/CodePipeline with terraform), the act of adding the repo, configuring the pipeline to use a manifest (e.g. circle.yml, travis.yml, codefresh.yml, Jenkinsfile) is a manual process

Erik Osterman

We’ve automated that provisioning using GitOps style best practices

Erik Osterman

For #codefresh, we stick that in a org/codefresh repo

Well actually, for travis and I believe also circle its just enabling it like, travis enable from the repo, then it automatically pics the file

But I think I got the idea, and if I understood correctly it is similar to the workflow I was describing on the top of the thread I think

but meh, not a fan of that workflow

aknysh

so for a workflow to update the pipeline itself, maybe consider atlantis

aknysh

(which of cause needs its own pipeline to be deployed or you can just apply it to deploy on fargate as @Erik Osterman is doing now)

Exactly! its the chicken and egg

Yeah fargate seems perfect for atlantis

TBH, some of the things that we dont deploy often, I dont mind deploying “automanually”

as a side note, I think I mentioned this before, as much as I like the idea of atlantis I dont like the apply before PR merge workflow. It would be like deploying before merge

it can even have issues on non up to date branches

2019-01-16

Erik Osterman
pnikosis/semtag

A sematic tag script for Git. Contribute to pnikosis/semtag development by creating an account on GitHub.

1
antonbabenko

Awesome! I was looking for this myself!

pnikosis/semtag

A sematic tag script for Git. Contribute to pnikosis/semtag development by creating an account on GitHub.

1
Erik Osterman

(via @mumoshu)

1
Erik Osterman

(saw he was using it in #variant)

1

We do manual version bumping, i have never found a way of doing directly from CI that “knows” when to bump each part, unless the team is strict about idk, commit messages or similar

1

do you guys integrate this into your pipeline?

1
Erik Osterman

sooooo one way I’ve seen this done that I like:

1
Erik Osterman

add a file to the repo called VERSION or something

1
Erik Osterman

as a human, you stick the version in there.

1
Erik Osterman

you can use a tool like semtag to do that.

1
Erik Osterman

Then the CI process will look at that file on a merge to master

1
Erik Osterman

and call out to github to do a release tag with VERSION.

1
Erik Osterman

I think this is a pretty elegant way of doing things.

1

Yeah, that is basically what we do, minus the tool for bumping, as tbh its not that much extra work to bump manually

1

then we trigger a GH release if VERSION > (existing tags)

1

for that we do use a tool to do semver comparison, actually python

1

for our branch model, a PR with a released version already existing, breaks the build

1

in a master only branch model, you can just version-commitsha if version already exists (for the docker/artifact tag)

1

so you dont release again, until a version bump basically

1

we’re considering using something like https://github.com/mkj28/semversions for CI/CD where each commit to, say, master gets unique, sequential semver-like tag. Uses vYYYY.MMDD.nnnn format, so allows us to avoid “what does semver mean for my project” discusssion

mkj28/semversions

For testing semantic versions in git. Contribute to mkj28/semversions development by creating an account on GitHub.

1

(it is Codefresh-specific)

1

(but can be easily repurposed as needed)

1
Erik Osterman

@Igor Rodionov

2018-12-19

mumoshu
08:37:11 PM

@mumoshu has joined the channel

2018-12-18

Enabling Actions on Public Repositories

Get started with one of our guides, or jump straight into the API documentation.

Erik Osterman

@mumoshu

Enabling Actions on Public Repositories

Get started with one of our guides, or jump straight into the API documentation.

Erik Osterman

this would also be pretty nice

aknysh

nice

aknysh

has anyone created a useful GitHub Action?

antonbabenko

I have not, but I saw Actions yesterday in action with one of my customers for real terraform commands… looks very cool, though not yet ready for many real use-cases

i was messing around to make on for terraform fmt

davidvasandani
hashicorp/terraform-github-actions

GitHub Actions For Terraform. Contribute to hashicorp/terraform-github-actions development by creating an account on GitHub.

Nikola Velkovski

Niiice , what about docker layer caching and actions, if that works it would be just awesome.

2018-12-13

joshmyers

hmm, I’ve stored the root tokens for auto unseal before in an HSM, that the Vault nodes have read access to

joshmyers

This wasn’t a kubernetes/helm deployment.

joshmyers

It was a complex PCI environment (AWS) and Vault was in usage for the PKI backend. HA Vault instances came up, auto init, unseal, create their own certs for TLS between clients and them, and then start doling out certs to instances in the environment as they come up

2018-12-12

endofcake

Anyone working with Hashicorp Vault? Keen to hear how you tackle the deployment pipeline.

Erik Osterman

(me too)

joshmyers

deployment of vault itself?

Erik Osterman

also, vault+helm

Erik Osterman

we did a poc earlier this year and did envconsul as PID1 in our containers

Erik Osterman

(envconsul is like chamber for vault)

endofcake

I deployed Vault itself with the community Helm chart (had to heavily modify it, which wasn’t an ideal experience). The first issue is that Vault needs to be initialised (if it’s the first time you are deploying it), and also unsealed. These steps are tricky to automate, which makes having ephemeral Vaults a challenge (cloud backed auto-unseal is not an option, HSM might be, but not right now).

Erik Osterman


cloud backed auto-unseal is not an option

Erik Osterman

it is now - was just released to CE

Erik Osterman

maybe not supported by terraform modules yet

Erik Osterman

supports KMS based unsealing

endofcake

The second issue is deploying configuration such as policies, auth methods etc. Right now we’re using Terraform, but authenticating is a problem.

endofcake

Yeah, I know it exists, that’s why I’m saying it’s not an option for us - we’re running on prem.

Erik Osterman

ohhhhhhhh

Erik Osterman

on prem

endofcake

Heavily regulated financial environment. It’s a thing.

Erik Osterman

heh, I bet.

Erik Osterman

@ might have more to contribute

2018-11-30

davidvasandani
04:13:50 PM

@aknysh this screenshot from https://github.com/cloudposse/github-status-updater how do you release to a namespace?

aknysh

every PR is a new k8s namespace. It’s how we do unlimited staging environments

davidvasandani

Thanks!

aknysh
Unlimited Staging Environments

How to run complete, disposable apps on Kubernetes for Staging and Development What if you could rapidly spin up new environments in a matter of minutes entirely from scratch, triggered simply by the push of a button or automatically for every Pull Request or Branch. Would that be cool? That’s

aknysh

@davidvasandani if you have questions or need more info, we can provide it

davidvasandani
05:04:12 PM

@aknysh do you happen to have an example where all these pieces are glued together?

Erik Osterman

@davidvasandani here’s a simpler and complete working example:

Erik Osterman
Add helmfile and codefresh by osterman · Pull Request #2 · cloudposse/statup

what Add helmfile for deployment with monochart Add codefresh build manifest why Easy deployment to kubernetes

Erik Osterman

I use it for my demos

Erik Osterman

basically I took a random app called statup (self-hosted http://statuspage.io clone) and deploy it on kubernetes using our monochart with helmfile and helm using codefresh

Erik Osterman

this supports unlimited staging environments

Erik Osterman

and automatic destruction when the PR is closed using the pull-request-closed.yaml pipeline

davidvasandani

Thanks @Erik Osterman can’t wait to dig into this.

Erik Osterman

Crap I realized you need to know all the ENV

Erik Osterman

@davidvasandani if you PM me I can get them to you

aknysh

give me a few minutes

aknysh

re: Self-hosted Helm Chart Registry - Codefresh added Managed Helm Repositories after the presentation, so we use it now instead of deploying our own chart museum

aknysh
05:29:26 PM
2
aknysh

@davidvasandani ^

2018-11-29

anyone here use github actions yet? do you know if there is a way to cache dependencies between builds?

haven’t seen anything in the docs

Erik Osterman

no one @cloudposse has been invited to the beta

Erik Osterman

i reached out to some peeps at GitHub but we don’t have the klout =0

Erik Osterman

@Gabe are you in the beta?

yeah we just got accepted in

Erik Osterman

just checked my inbox

Erik Osterman

nothing yet

Erik Osterman

i don’t think i got an email… just saw the new actions button on our repo

mrwacky

I got the email after the actions button appeared

Gabe
01:24:13 AM
Erik Osterman
01:25:15 AM

ohh… they are also only available on private repos

Erik Osterman

Ohhhhhhhhh snap

Erik Osterman

We have 250 public repos and like 5 private

aknysh

yep only private

hmm yeah… it looks pretty cool so far but a few things i’ve noticed is not being able to cache dependencies between builds, no control over the size of the machine it runs on (1 cpu 3.75 gb), and only two concurrent workflows running at one time per repo

pros are that it seems simpler than circle/jenkins and you can create actions that take environment variables so it’s easier to reuse/share actions between repos

… and we just got the email saying they have enabled it for us

aknysh

i just got the same email a few minutes ago, and Actions on my personal GitHub account

2
Erik Osterman
01:42:39 AM
Steven

I’m not so lucky. Still waiting

2018-11-11

yurchenko

ok

2018-11-10

joshmyers

Run a plan, push an artefact, namely the plan output. Run an apply of said build artefact number. This could be a gitsha, PR number, Jenkins build number. I don’t see a huge advantage of Atlantis… although I caveat with not having used so opinions maybe wrong. Have been doing CI/CD of terraform plan + apply for a long time in Jenkins. Not a massive fan of having output in the PR for history. Your git history lives forever, github PRs may not. Running pre merge isnt ideal for team scenario’s. Have seen a lot of failed apply due to vars, state, lots of TF bugs, race conditions…

joshmyers

Things like tflint can help with some of these

joshmyers

E.g. checking a type of AMI is even available in a region

joshmyers

Won’t catch bugs though.

joshmyers

Is Atlantis just like serverless Jenkins ci/cd for terraform with output posted back to the PR?

joshmyers

Sounds nice if you don’t already have a CI/CD solution/don’t need one for other things?

2018-11-08

No, not exactly like that, im not a big fan of what atlantis does

It applies before merge, to me that is anti-pattern

Erik Osterman

Except for terraform plans are poor

I was thinking appling the plan generated on PR, on the merge

Erik Osterman

They are optimistic at best

Erik Osterman

So if you merge and apply now what is in git is not deployed

Erik Osterman

If others are developing against it, they are just as blocked

Erik Osterman

So what we’ve reconciled is auto merge on successful apply

Well, but that is the same case for a CD container release

Erik Osterman

But for container releases it’s more stable

Erik Osterman

You have more under your control

Erik Osterman

Less value based errors

1
Erik Osterman

Most terraform failures in my experience are due to bad values

1
Erik Osterman

Containers get their values at runtime

Erik Osterman

Not at compile time (not generally but ideally)

Erik Osterman

You can achieve what you want easily with Codefresh.

Yeah, that is true, and a fair point, but not this one:
So if you merge and apply now what is in git is not deployed
If others are developing against it, they are just as blocked

In theory in the container you have the same “posibility” of failure

but I agree, is not as likely, as easiert to caught on test

TF has a lot of sideeffect failure scenarios, that plan does not catch

Erik Osterman

To me atlantis is a practical approach

Erik Osterman

Not the theoretical ideal

Erik Osterman

So rather than clutter the master commit history with a bunch of patch releases, rather get it in clean. We preserve a full transcript in the git comments so we have a record of what is deployed.

Erik Osterman

Even if the PR is closed but half applied, there’s a record of that’s

Yeah, its not a bad approach and im not saying Atlantis sucks, much the contrary, even Hashi bought them

I just feel that is a “hack” around a bigger issue

a great hack, but a hack anyway

Erik Osterman

Agreed

In my previous company we had a rake looper to get around TF Modules not having count, it worked, and it saved us a lot of time, but in “terraform” it was a hack

Erik Osterman

Yea I have heard of similar hacks… basically around code generation

Erik Osterman

Slippery slope

Erik Osterman

I think within a closed ecosystem of a corporate environment that might fly, but it makes it very difficult to write portable code for open source

Erik Osterman
antonbabenko/terrible

Let’s orchestrate Terraform configuration files with Ansible! Terrible! - antonbabenko/terrible

Yeah indeed, im not so keen on it anymore. Good thing TF 0.12 is around the corner . This was a long time ago

joshmyers

I wouldn’t touch 0.12 for a while…

Erik Osterman

I just don’t see a way around it that’s practical to solve (unless your HashiCorp and have 100mil in fresh green)

Yeah true

I might give it 2nd chance, at least it will avoid those fix PRs for a stupid terraform value problem

BTW, its great to have a place to ping-pong this ideas

Erik Osterman

Yea totally!! That what this place is for

Yep, thanks !

2018-11-07

In a similar tone to my previous question, anyone has a working workflow for using terraform plan from PR build or similar? Because while its the recommended way for terraform, I have yet to see a working workflow for it

Erik Osterman

I think the “interactive pull requests” model is ideally suited for terraform

Erik Osterman

the slides linked below show a number of companies (recognizable brands) who use atlantis

Erik Osterman

our fork of atlantis addresses the immediate shortcomings until they are fixed upstream

aknysh

@ if you are asking about how to trigger terraform plan/apply from an open PR, we recently used atlantis for a GitOps workflow. See #atlantis channel. Also @Erik Osterman recently held a meetup during #connectweek in Pasadena (CA) where he gave a live demo using Atlantis with Terraform to provision AWS user accounts using only Pull Requests

aknysh
DevOps Mastermind Group (Pasadena, CA)

This group is targeted to those interested in cloud automation & management, with a specific emphasis on Kubernetes/Docker, Helm, Prometheus, CI/CD, Microservices, etc. All skill levels are welcome. I

aknysh
How to use Terraform with Teams using Atlantis (#GitOps)

GitOps is where everything, including infrastructure, is maintained in Git and controlled via a combination of Pull Requests and CI/CD pipelines. Reduce the learning curve for new devs by providing a familiar, repeatable process. Use Code Reviews to catch bugs and increase operational competency. Pr

2018-11-06

Howdy releasers

I wanted to discuss one topic that is spinning in my head for a while:

  • Reusing docker or other artifacts from PRs or not

Basically, so speed up the build/release cycle, I was thinking on reusing the tested/built artifact in a PR when we deploy to the QA environment and ofcourse subsequently

Erik Osterman

yes, reusing the docker image is ideal for deployment to production and preproduction

Erik Osterman

we tend to treat preproduction and staging slightly different

Erik Osterman

staging ~ master

Erik Osterman

preproduction is a tagged release

Erik Osterman

that tagged release gets promoted to production repos

but do you build it on PR as well? if you dont, do you rerun all tests on the merged commit?

Erik Osterman

yes, we rerun on merge

Erik Osterman

so not quite what you say i guess

Erik Osterman

the key for us is to reuse the image on deploy to production

Erik Osterman

but not necessarily for all other steps

Yep, that part I have “pinned” i was wondering about the ohter part

Erik Osterman

we have a pipeline step to promote images and helm charts to production registry

We have basically 2 “stable” branches

  • stage (staging/preprod)
  • master (prod)

and I wanted to reuse the artifact from the PR to staging, to save some more time

Erik Osterman

gotcha - yea, we don’t have any thing for that. we also tend to do squash+merge, so the commit sha wouldn’t be the same

Yep, you dont even get a parent commit that way do you?

TBH, the core of the issue is not even on the pipeline, but mostly that some steps are just too slow (thanks Java/SBT dependancies)

Erik Osterman

yea, i can see why you’d want to do that

So to clean up:

  • feature/xyz -> staging (PR): build,test
  • staging (commit from merge): build, test, release, deploy

Is that your workflow?

(maybe you do direct to master and use tags or some other CD workflow for prod)

Erik Osterman
  • feature/1234/xyz -> staging (pr), build image, run compose integration tests, push image to registry, deploy helm chart to pr-namespace in kubernetes cluster (e.g. pr-1234)
  • master (squash merge): build, test, release, deploy
  • tag release - build, test, release, deploy to preproduction
  • production deploy - promote artifacts to production, deploy
Erik Osterman

honestly, every customer engagement is slightly different

Erik Osterman

but it’s more or less like that

Yeah, indeed, im just trying to get some ideas

and most of the examples from “blogs” are simple stuff that would not work/fly in production

Erik Osterman

yea… i can show you some examples if you want to zoom sometime

Erik Osterman

what are you using for cicd? are you deploying to kubernetes?

We are still on ECS, looking at EKS right now as we reached that point were ECS is just… annoying and we have enough people to support an EKS

At the moment? Travis, but forget about it as we are migrating to either GitlabCI or Buildkite before EoY hopefully, as we as well outgrew travis

Erik Osterman

What I don’t like about GitLabCI is it only supports one pipeline per repo (last i checked)

Erik Osterman

have you looked at Codefresh?

Yeah, tbh im right now leaning to BuildKite as you can even dynamically load “pipeline”

I did, but found it a bit less flexible

Maybe I just did not understood it completly

Erik Osterman

dynamically load pipeline? what’s that?

Lets say you have a pipeline like:

- build:
  - run x
- if build == commit:
  - loadpipeline pecigonzalo/this/.pipeline.yml

let me find the docs

Defining Your Pipeline Steps | Buildkite Documentation

Automate your team’s software development processes, from testing through to delivery, no matter the language, environment or toolchain.

Trigger Step | Buildkite Documentation

Automate your team’s software development processes, from testing through to delivery, no matter the language, environment or toolchain.

In many things is like a nice buildbot (https://buildbot.net/) but pre done

Buildbot

Buildbot - The Continuous Integration Framework

Now, this is not an easy challenge as its “hard” to identify the built artifact

since we enforce “up to date branches” im testing using git rev-parse --short HEAD^2 on the merge\|commit job to identify the artifact by the tag (we tag with the commit) of the branch that did the PR that was merged and created that commit.

*   2341145 (HEAD -> master, tag: 0.2.1, origin/master, workingset) Merge pull request #15 from thithat/staging
\|\  
\| *   d1bd417 Merge pull request #14 from thithat/feature/this-that
\| \|\  
\| \| * 43ba953 (feature/this-that) Test Version Flow release
* \| \|   16c043a Merge pull request #13 from thithat/staging
❯ git rev-parse --short HEAD^2
d1bd417

❯ git rev-parse --short HEAD^1
16c043a

thoughts? ideas? what is your current CI workflow? What do you build on PR and on merge?

Erik Osterman

so our workflow is slightly different

Erik Osterman

we have “unlimited staging environments” which correspond to PRs - one per PR; those just build images pinned to 0.0.0-gitsha

entire environments or just images?

2018-11-01

Erik Osterman

@dustinvb just joined! he dreams in pipelines

1
dustinvb

Ah bah, no <img src="/assets/images/custom_emojis/superfresh.png" class="em em-superfresh"> emoji.

Erik Osterman

haha

Erik Osterman

we can add that

Erik Osterman

for those that don’t know dustin, he is an awesome support engineer at Codefresh, which is what we use for all of our CI/CD

aknysh

hey @dustinvb

dustinvb

Hi Andriy.

dustinvb

Happy to help out in anyway I can.

aknysh

thanks

aknysh

we all love Codefresh

dustinvb

I do as well. Was a customer for a year before joining.

2018-10-31

@Gabe if you use it, Jira can generate a changelog once you’ve made a release on your project

2018-10-30

do you guys use any tools to automatically generate change logs?

Erik Osterman

explored it briefly, but didn’t end up doing anything

Erik Osterman

maybe it would be a nice github action

1

have you guys been added to the beta yet? we signed up but are still in the queue waiting for them to activate

i’ve found some js tools but was curious if you had some that use go/python

1
aknysh

still waiting for GitHub actions

2018-10-29

Dombo

If you could bring your own PaaS to that - ECS etc. then it would be very compelling. I feel lots of companies underestimate how hard it is to manage the orchestration of a high volume container platform with a varied array of tasks being run on top of it. Look at the CircleCI status page for an example.

An outage to a build pipeline is a blocker that wastes quite a bit of money if a company is pushing at a reasonable velocity.

ndobbs

The Michael DeHaan (ansible creator) released a new CI tool today called vespene: https://vespene.io/

Vespene.io

Signup for release announcements

aknysh

hi @Dombo

aknysh

not an orchestrator, but easy to use for deployments

Dombo

Interesting - have you found much overhead compare to using a managed CI service? Also I suppose you lose support for the pipelines as code pattern with this? i.e. no build manifest.yml/json ?

Erik Osterman

yea, the fargate cli not a complete e2e alternative

Dombo

Good to see alternatives exist

Dombo

Well - good to hear honest people talking about using alternatives

Dombo

Have had a pretty good experience with Circle - only a few limitations bug me

Erik Osterman

the story is that that the fargate cli was written by an engineer at AWS to show the ECS team what kind of cli tool users want

Erik Osterman

to make it as easy as using heroku

Erik Osterman

so far, aws hasn’t taken him up on the suggestion!

Dombo

Ouch

2018-10-16

execute actions in containers in GitHub based on any event trigger

aknysh

so nice!

looks really interesting. official page and beta sign up: https://github.com/features/actions

Features • GitHub Actions

Easily build, package, release, update, and deploy your project in any language—on GitHub or any external system—without having to run code yourself.

Erik Osterman
06:29:32 PM
Erik Osterman

Interesting! Didn’t see GitHub going in this direction

Erik Osterman

Guess it’s playing catch-up with GitLab

2

Looks very interesting

Erik Osterman

Yea, signed up for the beta. Hope they release invites quickly.

Erik Osterman

Wonder what all the GitHub partners think about this

Erik Osterman

CI/CD is the biggest commercial vertical in the GitHub ecosystem

aknysh

interesting how they are going to manage secrets

Erik Osterman

interesting or wonder?

aknysh

wonder

Erik Osterman

Erik Osterman

yea

2018-09-21

Erik Osterman

I created #chatops for those interested

2018-09-18

Erik Osterman

@ one of our top priorities is locking down our strategy for terraform CI/CD

Erik Osterman

interested in atlantis, but some concerns

Trying to figure it out for my current customer

have few ideas but nothing that I would recommend yet

open to sharing current best thinking and hearing your ideas

heard about atlantis but didn’t tests it out yet

can brief you during the call on friday and then share here if there is any value

Erik Osterman

ok, that’s awesome. let’s discuss. spent a lot of time thinking about this over the past month. some small PoCs.

2018-08-19

Erik Osterman


but kinda bothers me that chart version would rev with appVersion (they don’t have to match of course).

Erik Osterman

@ this strategy is designed for repos where we cotenant the chart with the app itself. if the chart is not hosted in app repo, then it’s a much more difficult problem to solve.

Erik Osterman

I don’t see the benefit of treating an app as a “monorepo” just because it also contains a chart. that is, treating the “chart” as one app, and the service itself as another “app”. If the chart and app are in the same repo, then they are the same version IMO.

Erik Osterman

that said, we’re investing now in a monochart.

Erik Osterman

this is one chart to rule them all.

Erik Osterman

it’s still a WIP with lots of comments. @alebabai is working on it.

Erik Osterman

The idea is to get out of the business of writing charts and just defining values.yaml

Erik Osterman

…to stop allowing every service to be a snowflake and instead take the Heroku approach where you write a service that conforms to certain principals and then the monochart just works.

Erik Osterman

Plus, the monochart is composable, so you can create a service with multiple components just by creating an umbrella chart

alebabai
06:03:46 PM

@alebabai has joined the channel

Erik Osterman

@ any recommendations for alternatives to “bats”?

nothing comes to mind, but looks like you are picking bats-core

Erik Osterman

haha

Erik Osterman

yea - just needed to move forward to something. still in the prototyping phase. it can be replaced with something else.

Erik Osterman

seems like it’s no longer maintained

Erik Osterman
sstephenson/bats

bats - Bash Automated Testing System

Erik Osterman

though looks like bats-core is maintained

Erik Osterman
bats-core/bats-core

bats-core - Bash Automated Testing System

Erik Osterman

Travis is using bats

Erik Osterman

@Sebastian Nemeth heads up, i’ve spent some time this weekend working on a prototype for testing #geodesic

Sebastian Nemeth
06:50:31 PM

@Sebastian Nemeth has joined the channel

Erik Osterman

Submitted some PRs against geodesic to make it easier

Erik Osterman

For now, I’m going to be using Codefresh because it’s easier to iterate than using codebuild/codepipeline

Erik Osterman

as it relates to infrastructure, I see it boils down to a few different use-cases

Erik Osterman
  1. testing terraform modules (including terraform-root-modules)
Erik Osterman
  1. testing helm/helmfiles
Erik Osterman
  1. testing account-level infrastructure
Erik Osterman

for our purposes, I’ve ruled out atlantis and test kitchen, at least at this phase. I want something that lets us test more holistically, since what we need to test is the integration of lots of systems across languages, tools, and binaries.

Erik Osterman

for this reason, I am leaning towards using bats (bats-core) - or a tool like it

Erik Osterman

bats-core feels very lightweight, which I like.

Erik Osterman

bash as a “top most level” language for integration also feels right. it doesn’t preclude us using other purpose-built testing frameworks for individual languages or apps.

2018-08-18

On this topic - maybe it’s my misunderstanding of helm? but kinda bothers me that chart version would rev with appVersion (they don’t have to match of course). Or is it OK to keep these decoupled in say, dev, and only revision chart version on promoting to staging/production? Don’t use appVersion in charts at all? What’s the right strategy?

(this is loosely related to the PR above)

2018-08-17

antonbabenko
10:36:34 AM

@antonbabenko has joined the channel

2018-08-16

Erik Osterman
Promote Helm Charts and Docker Images Across Repositories by goruha · Pull Request #85 · cloudposse/build-harness

What Use helm package to change version and app Version of chart Use convention that default image tag is based on app Version Create promote targets that allow to promote chat to required versio…

Erik Osterman

This adds the ability to promote images and charts between repositories

Erik Osterman

We suggest adopting this as your production strategy

Erik Osterman

There are some minor breaking changes which @Igor Rodionov will document

Erik Osterman

(related to renamed ENVs)

Max Moon

2018-08-09

dat.le
07:01:48 AM

@dat.le has joined the channel

2018-08-08

Michael Pereira
03:31:51 PM

@Michael Pereira has joined the channel

Michael Pereira

@ +1 for hadolint as I believe it’s based on the excellent shellcheck, which makes sense since Dockerfile are mostly just a collection of shell commands

Michael Pereira

so it benefits from the existing shellcheck rules

1

2018-08-05

jylee
04:28:41 PM

@jylee has joined the channel

2018-08-01

Phil
08:13:26 AM

@Phil has joined the channel

my-janala
02:23:57 PM

@my-janala has joined the channel

Ziad Hilal
05:34:48 PM

@Ziad Hilal has joined the channel

2018-07-30

Looked at 4-5 docker linters I could find and ran our one of our Dockerfile through them. The two that I liked the most are https://github.com/RedCoolBeans/dockerlint and https://github.com/hadolint/hadolint - the only 2 that complained about the non-array-style CMD or ENTRYPOINT

https://github.com/hadolint/hadolint seems well maintained (last updated 8 days ago) has most stars and forks

I also like the configuration and per-line excludes.

RedCoolBeans/dockerlint

dockerlint - Linting tool for Dockerfiles

hadolint/hadolint

hadolint - Dockerfile linter, validate inline bash, written in Haskell

2018-07-29

Anyone using dockerfile linter they can recommend?

Erik Osterman

A ways back, I had looked into it and didn’t find one that I liked, but that was partially because they were all in either python or ruby.

Erik Osterman

If you find one you like, lmk! Would love to add it to our own build harness. I might concede on the ruby/python now :-)

I don’t think twistlock can do the level of linting we would prefer, so looking for something. I did not realize there are so many (simple google search shows 4-5 right away)

Erik Osterman

Also do a GitHub repo search

Erik Osterman

@ have any linting suggestions

2018-07-27

Erik Osterman

@ not sure if this is too complicated of a PR for you take an interest in

Erik Osterman
Promote Helm Charts and Docker Images Across Repositories by goruha · Pull Request #85 · cloudposse/build-harness

What Use helm package to change version and app Version of chart Use convention that default image tag is based on app Version Create promote targets that allow to promote chat to required versio…

Erik Osterman

but we are working on the ability to easily/cleanly promote images and charts between repositories

Erik Osterman

The goal is to be able to do something like

make release/promote \
    CHART_NAME=api-app \
    SOURCE_VERSION=0.1.0 \
    TARGET_VERSION=0.2.0 \
    SOURCE_IMAGE=api-app:0.1.0-sha.a989ads8 \
    TARGET_IMAGE=api-app:0.2.0
Erik Osterman

(interface not yet formalized)

we’re not there yet but the more I think about it - this seems to be the way to go

Erik Osterman

are you guys using multiple codefresh accounts?

we are not

Erik Osterman

that also seems like a must have

elaborate?

Erik Osterman

it’s the only way to RBAC production kubernetes integration from staging clusters kubernetes integration

on what you posted - in our flow we may do this for releases, for “master” CI we would autoversion

RBAC - waiting for CF to implement, on our “must-have” list, last I heard it is in progress

Erik Osterman

multi-account is available now I thought

Erik Osterman

so you setup all produciton pipelines in the production account

Erik Osterman

all staging pipelines in the other account

Erik Osterman

that way staging pipelines can never accidentally modify production

yeah makes sense

Max Moon
Codefresh adds unified support for multi-git provider accounts - Codefresh

Not so long ago, hosting your code on the cloud almost always meant that you used Github. Teams were fairly standardized in their choice of git provider. This is no longer true. Several other solutions are now challenging Github, including Bitbucket and Gitlab. Moreover, several companies have chosen to actually host their code on-premises creating …

1

2018-07-26

Erik Osterman
Codefresh vs. GitlabCI - Which one should you use

Gitlab is one of the supported GIT providers in Codefresh. In this article, we will look at the advantages of Codefresh compared to the GitlabCI platform.

Yoann
07:00:32 PM

@Yoann has joined the channel

Max Moon

just a heads up, if anyone else had the “CodeFresh” status check as a passing requirement for PRs, it is no longer. Now your PR will get a status update that is pipeline specific. If you have repos that require the CodeFresh status, disable it in favor of Pipeline status

Erik Osterman

thanks @Max Moon! didn’t know that.

Erik Osterman

@ @

Max Moon

NP!

09:06:32 PM

@ has joined the channel

1

2018-07-25

Arkadiy
02:50:15 PM

@Arkadiy has joined the channel

2018-07-24

Erik Osterman
07:15:28 PM

@Erik Osterman has joined the channel

Erik Osterman
07:15:28 PM

@Erik Osterman set the channel purpose: CI/CD Discussions

07:15:29 PM

@ has joined the channel

Igor Rodionov
07:15:29 PM

@Igor Rodionov has joined the channel

Jeremy Grodberg
07:15:29 PM

@Jeremy Grodberg has joined the channel

Max Moon
07:15:29 PM

@Max Moon has joined the channel

07:15:29 PM

@ has joined the channel

07:15:29 PM

@ has joined the channel

aknysh
07:15:29 PM

@aknysh has joined the channel

07:15:29 PM

@ has joined the channel

Erik Osterman

@
how to use semantic versioning, I actually couldn’t figure out how to start from a specific version. We have created charts in a repo under path charts/. I have write a make step to detect the charts which are changed. I want to automatically increment their semantic version. Is this something which can be achieve by this module. What I found is it generates semantic version as 0.0.0-sha(commit-id)

Erik Osterman

We follow the convention of tightly coupling charts with the micro service. We stick them in the the charts sub folder. The calculus of knowing which charts work with with docker images is a lost cause. The official chart repo by Kubernetes is optimizing a different use case, which is why they have all charts in one repo. The semantic version of our charts is derived from the nearest git tag in the tree. So if no previous tags, then you get 0.0.0.

Erik Osterman
Erik Osterman

(with codefresh)

Erik Osterman
07:19:08 PM

@Erik Osterman set the channel topic: All things CI/CD. Specific emphasis on Codefresh and CodeBuild with CodePipeline.

Erik Osterman

@Igor Rodionov is working on a way to promote charts and images between repos (for @Jeremy Grodberg)

the rational of defining charts in a separate repos are

  1. Encapsulate ops from devs (both code and structure wise)
  2. Within http://dev.niki.ai we have the required service/helmfile with image tags, which makes sense to us to syncup whole infra from one repo
  3. The infra-changes (including services) can be monitored in one Pull request
  4. Even we can move chartmuseum to gitlab pages in future, or if we got a chance to free chart hosting service, it can be synced up using 1 repo only
Erik Osterman


Encapsulate ops from devs (both code and structure wise)

What I was looking up from SEMVER is a way to incrementally update the semantic version only

Erik Osterman

isn’t that antithetical from devops?

i think restriction is unethical, encapsulation is not

Erik Osterman

i want the developers to write the charts to deploy their apps

Erik Osterman

i’ll write one as an example, they write the rest

Erik Osterman

when their app architecture changes, I won’t know

Erik Osterman

they should update the chart respectively

I never mentioned that dev’s shouldn’t commit on other repos, but this won’t be as frequent as the code commits

Erik Osterman

i think the semver stuff will work though for your use case anyways

Erik Osterman

but versioning your charts will be manual

even that is also one point,

  1. Seperate pipelines for charts and code
Erik Osterman

the same way they are versioned in kubernetes/charts

Erik Osterman

but the pipeline will generate the semvers for the docker containers

Erik Osterman

and will pass that to the chart as the image tag to deploy

Erik Osterman

our strategy has been to pin charts to containers

Erik Osterman

that way for every single version on an app, there’s a chart that will deploy it

Erik Osterman

makes it very easy to maintain and understand what’s going on. departing from that will introduces new challenges.

in place of pinning to charts, i think this it should be pinned to helmfile

version of charts is entirely separate thing than version of code

Erik Osterman

but there’s no artifact storage for helmfiles

it can still be managed in same way, using yq

Erik Osterman
  - name: '{{ env "RELEASE_NAME" }}'
    labels:
      chart: "somechart"
      component: "app"
    version: '{{ env "CHART_VERSION" }}'
    chart: 'chart-repo/{{ env "CHART_NAME" }}'
    namespace: '{{ env "NAMESPACE" }}'
    values:
      - '{{ env "RELEASE_NAME" }}.yaml'
Erik Osterman

so that’s the helmfile from a microservice that says how to deploy it

Erik Osterman

the envs come from the pipeline

you could add image tag also as part of set

Erik Osterman

yep

Erik Osterman

so that’s the way to do it if you want to decouple the charts from the service repo

the point is that, charts represent how a service should run infrastructure wise, but not what a service is running

what a service is, its being defined by container

Erik Osterman

something similar could be said for the Dockerfile, no?

Erik Osterman

OS ~ Dockerfile ~ service

not at all, its all about how we package the service

Erik Osterman

cluster ~ chart ~ docker

service ~ Dockerfile | cluster ~ chart | release ~ version

services are tightly coupled with dockerfile, we can’t use one service’s dockerfile for another

Erik Osterman

if using a monochart (declarative helm chart) for multiple services, i think it makes sense to move it out

Erik Osterman

but if a chart is 1:1 to a service, there’s no overwhelming reason to separate them

to be frank we are not using monochart but the same replica of chart for each service

just in case we need to modify something

its like we have 3 charts

external, internal, job

all external services had just required external chart, nothing else

all internal services require internal chart

Erik Osterman

but our differences aside, I think doing what you said with passing the image tag in the helmfile from an ENV

and same for jobs

Erik Osterman

will accomplish what you want in the end, no?

kind of, its more of doing a git commit on service updates within helmfile

Erik Osterman

(i want to move to a monochart for new services so they all follow a similar architecutre)

yeah but everything apart => I doubt we have a function which do this, Input => 1.0.0 Output => 1.0.1

Erik Osterman

yea, we don’t do that

Erik Osterman

knowing when to bump versions I think requires a human

Erik Osterman
04:17:19 AM
Erik Osterman

our semantic versioning takes the x.y.z from the most recent tag

Erik Osterman

and computes metadata based on branch information (for staging environments)

Erik Osterman

(so what we’re doing is really not that magical at all)

    keyboard_arrow_up