#gitops

Discuss continuous delivery of infrastructure Archive: https://archive.sweetops.com/gitops/

2019-06-04

2019-05-01

@Erik Osterman do you have a trigger based off of a release or just a tag in codefresh?

Erik Osterman

release only

thats nice

i cant seem to find that in circle

Erik Osterman

i guess tag will work well enough

like the webhook will fire off from github, but circle doesnt listen for that

yeah im using tag, which circle does listen to

but obviously would have liked it to be release instead

Erik Osterman

(codefresh even listens for github comments)

Erik Osterman

“ok to deploy”

Erik Osterman

im going to work with the free version of codefresh on my own time so i can make a pitch to move

do you find your team going through alot of builds though?

Erik Osterman

codefresh is moving to a new model based on concurrent steps

Erik Osterman

that’ll make it a lot easier

Erik Osterman

unlimited builds

pricing will not be per build?

ooh

that’d be nice not to have to worry about optimizing the types of builds youre making

Erik Osterman

yea, the realized the pricing model before was prohibitive/restrictive

Erik Osterman

when in fact, you want people to use it as much as possible

yeah

do you know what their new pricing model will look like?

Erik Osterman

jump into #codefresh and inquire about it

Erik Osterman

someone will reach out to you

thanks

@Erik Osterman are you doing release/tag permission restrictions in github? those dont seem available

Locking/auditing for tags · Issue #1091 · isaacs/github

Right now, it's basically impossible to have any protection around tags: Anyone with write access to a repo can push any tags. There's no auditing/logging of tag changes. Any number of thin…

Erik Osterman

unfortunately, github doesn’t support that. there are a few workarounds.

Erik Osterman

Option 1. Codefresh will soon be adding permissions around “approval” steps; (right now anyone can approve). You can wait for this. Option 2. Embed the deployment permissions into the deploy pipeline by looking at the code committer. This can run as just a separate step in the pipeline.

Erik Osterman

Option 2 is my pref

Erik Osterman

we don’t have an example of this right now

yeah

thats what im going to do in circle

unfortunately its hardcoded into the code

Erik Osterman

i recommend just checking github team membership

thats not provided out of the box by circle, are you hitting the gh api to grab that?

i only get the username of whoever triggered the build

yeah i will just hit the GH api

Erik Osterman

yea, you’ll need to hit the github API

Erik Osterman
cloudposse/github-authorized-keys

Use GitHub teams to manage system user accounts and authorized_keys - cloudposse/github-authorized-keys

Erik Osterman

but I would create a new cli tool that simple does this one thing

yeah that’s what I am planning on doing

python + argparse

1

2019-04-30

So im trying to deploy to my k8s cluster w/ helm via my ci tool and I’d like for those helm deploys to kick off based off merges into special branches. For example, merging a PR into develop would kick off a helm upgrade to the dev cluster and merging a release PR into master would kick off a helm upgrade to the prod cluster. How are those of you following this pattern handling the fact that helm doesn’t redeploy pods using images on a specific tag (develop) even if there is a more recent version of that image?

Erik Osterman

we’ve used a number of strategies

Erik Osterman

for develop you can use helm upgrade --recreate-pods option

Erik Osterman

or just always tag yur docker images also with a git ref

Erik Osterman

then use the git ref for the image tag

Erik Osterman

--recreate-pods will cause a brief outage

so your repo has a tag per commit?

Erik Osterman

we tag every docker image with a git ref

ah

i misread that

Erik Osterman

that way there is a 1:1 relationship between git and docker

so when you merge into master, that will re-build the docker image meaning that you would be technically deploying with a different image (although should be identical)?

Erik Osterman

yes, but our process is different

Erik Osterman

we merge to master, we build

Erik Osterman

especially since we usually squash merge

Erik Osterman

we use release tags to deploy to environments

Erik Osterman

release tags therefore only retag a docker image. no rebuilding required.

Erik Osterman

master is continuously delivered to staging

Erik Osterman

release tags go to production.

Erik Osterman

1.2.3-foobar goes to the foobar environment

Erik Osterman

e.g. 1.2.3-prod

Erik Osterman

1.2.3-preprod

Erik Osterman

1.2.3 goes no where

Erik Osterman

then for PRs we deploy each one into a new namespace on the staging cluster

Erik Osterman

e.g. pr1234 of the example repo gets deployed to pr1234-example namespace

1

so for your prod deploys, they are not being triggered via git

Erik Osterman

they totally are

Erik Osterman

100% git driven

ohh

youre saying

Erik Osterman

by tagging a commit, you are pushing to specific environments?

so when you push that tag, it triggers the deploy

Erik Osterman

yes

1
Erik Osterman
cloudposse/example-app

Example application for CI/CD demonstrations of Codefresh - cloudposse/example-app

Erik Osterman

retag the image

Erik Osterman
cloudposse/example-app

Example application for CI/CD demonstrations of Codefresh - cloudposse/example-app

thank you, i believe thats the step i was missing

Erik Osterman

deploy it

Erik Osterman

in our case, we use github

Erik Osterman

so we use the UI for releases

i do to

Erik Osterman

but essentially it’s tags

Erik Osterman

releases also have webhook events

so you create a release, which triggers a deploy in codefresh

Erik Osterman

we register those with codefresh

Erik Osterman

yup

nice thats what i’ll do

how do you like codefresh?

Erik Osterman

and then you can throw in an approval step

i believe you guys are big fans

Erik Osterman

so it queues it up for deployment, but someone higher up has to click approve

Erik Osterman

yes, we’re big time fans of #codefresh

approval step is a codefresh thing or github?

codefresh correct

Erik Osterman

approval step is a codefresh step

Erik Osterman
ask_for_permission:
    type: pending-approval
    title: Deploy release?

nice nice

we’re on circle right now, which isnt my favorite

Erik Osterman

yea, circle definitely has the marketshare

but i believe i can perform all of those still

with circle

Erik Osterman

and the two have been converging on functionality

Erik Osterman

but codefresh has tighter k8s integration and native helm support

Erik Osterman

plus kanban boards so youo can view your releases and what stage they are in

yeah ive built my own helm deploy script to run from circle

thats why i was interested in http://drone.io but right now its more affordable to pay for circle than run our own 3 node k8s cluster for drone

Erik Osterman

yea, i dn’t like hosting CI/CD

Erik Osterman

it’s foundational infrastructure, that when everything else is done right (IaC) it’s just easier to treat the CI/CD as an axiom; it just exists and we don’t need to worry about how.

same, but drone does run jobs as native k8s jobs which removes the need for managing agents which is nice

Erik Osterman

that’s nice

but yeah, thinking about our CI/CD platform having uptime issues during a client-facing outage would suck

Erik Osterman

i feel like we got enough to manage…. kiam, cert-manager, prometheus, grafana, teleport, kibana, fluentd, external-dns, keycloak, etc. . and for those there is no hosted option.

Erik Osterman

but for CI/CD there is.

Erik Osterman

haha yeah

sysdig datadog

but i get it

Erik Osterman

yea, true

Erik Osterman

sumologic, splunk

Erik Osterman

but daaaaaaang the prices $$$

yeah they are not cheap

thanks for the help, answered with the exact solution i was looking for @Erik Osterman

Erik Osterman

thanks @btai! glad I could help

2019-04-11

Erik Osterman
GitOps with Terraform on Codefresh (Webinar)

Infrastructure as code, pipelines as code, and now we even have code as code! =P In this talk, we show you how we build and deploy applications with Terraform using GitOps with Codefresh. Cloud Posse is a power user of Terraform and have written over 140 Terraform modules. We’ll share how we handl

1

Just read all your slide decks. Very nice demos.

GitOps with Terraform on Codefresh (Webinar)

Infrastructure as code, pipelines as code, and now we even have code as code! =P In this talk, we show you how we build and deploy applications with Terraform using GitOps with Codefresh. Cloud Posse is a power user of Terraform and have written over 140 Terraform modules. We’ll share how we handl

1
Erik Osterman

Thanks @!

1
Erik Osterman

Had a lot of fun making them

1
Erik Osterman

(video will be posted later)

2019-04-10

2019-04-02

2019-03-20

sarkis

Doing anything cool with github actions yet?

Erik Osterman

No! I just emailed them again last week asking when we can use GitHub Actions on public repos

1
Erik Osterman

99% of our repos are public

Erik Osterman

and 100% of the ones I’d like to use it on are public

Erik Osterman

so GitHub Actions to me

Tim Malone

Erik Osterman

They gave the standard generic reply that they’ve passed our feedback on to the product team and have no ETA

Tim Malone

https://developer.github.com/actions/ doesn’t seem to imply it only works on private repos, though?

Tim Malone

only that its in beta

Erik Osterman

it’s more like a scale/security thing

Erik Osterman

guessing it’s how they can control the size of the beta

1
Erik Osterman

welcome @Tim Malone btw

1
Erik Osterman

are you practicing gitops today?

sarkis

Also this looks interestinggggggg https://github.com/tektoncd/pipeline. Kind of applies to this channel

tektoncd/pipeline

A K8s-native Pipeline resource. Contribute to tektoncd/pipeline development by creating an account on GitHub.

Erik Osterman

hah, just saw that promoted today in my github feed

Erik Osterman

didn’t look closer yet though

Erik Osterman

btw, this is the kind of cool stuff we’re starting to track here: https://sweetops.com/

SweetOps - DevOps Community

DevOps Community

1
Erik Osterman

have a backlog of hundreds of things I need to add

sarkis

Nice!!!

Erik Osterman
03:03:51 AM
Tim Malone

nice! will you be recording?

Tim Malone

(i can rarely make webinars but regularly add the recordings to my watch list)

Erik Osterman

yep! it will be recorded

Mohamed.Naseer

please post recording/links - ty

2019-02-14

Erik Osterman
05:19:39 AM

@Erik Osterman set the channel purpose: Discuss continuous delivery of infrastructure Archive: https://archive.sweetops.com/gitops/

2019-01-17

Nikola Velkovski

Morning people! Does anyone know if github actions will be free after the beta ends ? It doesn’t make sense for it to be free and currently it feels like a trap . If we ditch our CIs and move to github actions, and they set some crazy pricing on it after the beta is done it won’t be fun at all.

No idea, haven’t found anything about GitHub Actions pricing. Speaking about actions, found that tommorow: https://github.com/nektos/act I guess it was in the trending list but I feel the need for sharing it.

nektos/act

Run your GitHub Actions locally. Contribute to nektos/act development by creating an account on GitHub.

antonbabenko

We use both at the same time, because there is not enough flexibility in github actions as an executor, but more as a trigger. We use github action to do basic actions, and delegate complex tasks to CircleCI (already existing and working) .

Nikola Velkovski

Thanks for the info ! I’ll try to get some information about the pricing because no one seems to be talking about it

Nikola Velkovski
sdras/awesome-actions

A curated list of awesome actions to use on GitHub - sdras/awesome-actions

antonbabenko

Currently triggers are different for private and public repos, which adds some extra challenges working on private repos before making things public.

2019-01-10

Erik Osterman
4

2018-10-24

aknysh

they will have a service to add encrypted secrets

aknysh

you could use ENV vars

aknysh

and you’ll be able to create actions programmatically

aknysh
workflow "New workflow" {
  on = "push"
  resolves = ["Deploy to Azure"]
}

action "Deploy to Azure" {
  uses = "./.github/azdeploy"
  secrets = ["SERVICE_PASS"]
  env = {
    SERVICE_PRINCIPAL="<http://sdrasApp>",
    TENANT_ID="72f988bf-86f1-41af-91ab-2d7cd011db47",
    APPID="sdrasMoonshine"
  }
}
aknysh

looks like Terraform HCL

Erik Osterman

Yea that’s 100% :-)

2018-10-22

Erik Osterman

I added a #atlantis channel since we’re doing a lot more with it these days (related to http://runatlantis.io).

2018-10-14

10:28:47 PM

@ has joined the channel

2018-10-13

Gaurav Ubnare
11:47:30 AM

@Gaurav Ubnare has joined the channel

2018-10-11

08:32:50 PM

@ has joined the channel

2018-10-10

antonbabenko

Send a email to Luke to get more names.

Erik Osterman

Talking 10/25

Erik Osterman

I also pinged Luke in their new slack community

1

2018-10-09

stephen
03:42:47 PM

@stephen has joined the channel

Adam
04:17:00 PM

@Adam has joined the channel

Yoann
05:52:15 PM

@Yoann has joined the channel

Erik Osterman

@antonbabenko @ any companies you know of using atlantis that I can reference in my slides?

antonbabenko

I don’t remember names. They were pretty small and unknown.

Erik Osterman

Thanks anyways :)

http://Blinkist.com , need to ask the other two for OK

Erik Osterman

ok, great, I’ll add them - thanks!

When is it due?

2018-10-08

antonbabenko
07:30:58 AM

@antonbabenko has joined the channel

08:52:00 AM

@ has joined the channel

Erik Osterman
05:39:37 PM

ah let me remove article with jpg

replace*

05:42:30 PM

AirBNB response to my question about having secrets coupled into Docker. Having secrets maintained away from the application makes it impossible to do an actual rollback to point an application to the previous config set which is a point.

I’m thinking of a way to make SSM/KMS/Chamber compatible with the mindset of that issue at stake.

SSM

/service/db_host  = 'hostA'
/service/db_user  = 'userA'
/service/db_host_migration = 'hostB'
/service/db_user_migration = 'userB'

On the Docker with Chamber /etc/chamber_rewrites ( or defined ENV ) defines how certain env vars can be rewritten with other created env vars.

db_host = db_host_migration
db_user = db_user_migration

This way it is possible to work with a default config set, yet it is still possible to fix config values to a certain deployment. Problem here is transparency, it is not visible from the outside which docker is using what.

Erik Osterman

re:

/service/db_host  = 'hostA'
/service/db_user  = 'userA'
/service/db_host_migration = 'hostB'
/service/db_user_migration = 'userB'
Erik Osterman

why not use different service namespaces

Erik Osterman

e.g.

Erik Osterman
/service/db_host  = 'hostA'
/service/db_user  = 'userA'
/service/migation/db_host = 'hostB'
/service/migration/db_user = 'userB'

TBH, I did think of that too, but I’m afraid that would kill their parser a bit.

Erik Osterman

their = chamber?

Erik Osterman

hrm

Erik Osterman

true, i think their import/export only handles one service namespace at a time

Erik Osterman

checking

i mean, db_host is attribute here, migation/db_host is a nested attribute but has a / in it. I don’t think / are allowed

inside the name

and a full real namespace change, as in, changing the prefix of everything SSM, or a different service name. That of course can work. BUT. you would need to modify iam policies for the service to allow access to it, and I think that is just too much change for something trivial.

Erik Osterman
05:51:41 PM
Erik Osterman

anyways, what i was actually hoping was that the export format supported the service namespace

Erik Osterman

but it doesn’t

Erik Osterman

so to your point

I mean, by changing the full service path, and making it a level deeper more or less, you can make it to work but you would need to duplicate everything to that level deeper

Erik Osterman

i think a cool/needed tool would be a chamber ETL tool with KMS (PKE) support

Erik Osterman

that maybe supports some custom mapping

Erik Osterman

(since i don’t think the guys at chamber are too amenable to changes of this nature)

Maybe yes, but it’s an interesting quest to drop and see what comes out.

Erik Osterman

despite my previous reservations, I think I am leaning still towards your approach

Better ( but impossible to get ) is a new style SSM with versioned change-sets which can be referred to.

2018-10-07

Erik Osterman

for those interested and want to follow along, we’re working on some enhancements for atlantis

Erik Osterman
cloudposse/atlantis

Terraform For Teams. Contribute to cloudposse/atlantis development by creating an account on GitHub.

Erik Osterman

we’ve added atlantis to our alpine repository:

Erik Osterman
curl -sSL <https://apk.cloudposse.com/install.sh> \| sudo bash
Erik Osterman

And the work for integrating it into geodesic: https://github.com/cloudposse/geodesic/pull/298/files

Add support for Atlantis by osterman · Pull Request #298 · cloudposse/geodesic

what Add support for atlantis why GitOps style continuous delivery for infrastructure

2018-10-04

09:51:01 AM

@aknysh thanks.. I responded to the article as this is a step away from factor12. But with their java background env variables were probably something they have never used to begin with.

Erik Osterman

@ insightful comment

Erik Osterman
04:51:04 PM
Erik Osterman

this is a good point

Erik Osterman

what is your approach to this in what you’re doing?

Erik Osterman

e.g. if you have the “widget” app and you encrypt the “prod” mailgun key and commit to source control in the widget app repo, is this blurring the line?

05:15:17 PM

@ has joined the channel

2

2018-10-03

09:28:20 AM

Hi everyone, I wonder how most of you are doing secrets management. In a stack I’m working on the applications their env vars (ssm) are populated through terraform. Secrets like API keys are also stored in git but have been encrypted by KMS. Those secrets will be decrypted by terraform using aws_kms_secrets. My belief is that this is secure as envelope encryption would make the encrypted hash useless without access to KMS, next to that it makes it possible to version everything related to the application and removes manually entering passwords to SSM. @Erik Osterman pointed out to me that git is about transparency and that storing anything else than readable text is a conflict. For me this is less of a problem when the alternative is to manually enter passwords. Happy to collect your thoughts.

09:30:56 AM

@ has joined the channel

jonboulle
09:30:56 AM

@jonboulle has joined the channel

09:30:57 AM

@ has joined the channel

jonboulle

git isn’t just about transparency in terms of content, it’s about transparency in terms of an audit log of changes, which this still provides. of course you can get that with other systems too, but isn’t the point of gitops to have all of that centralised in git?

Steven
11:51:12 AM

@Steven has joined the channel

Erik Osterman

@jonboulle in principle I agree with you. but stoking debate, where is the line drawn? what about binary artifacts from build processes or packages?

Erik Osterman

for example, a common pattern I dislike, but which has emerged with terraform+lambda, is including a zip of the lambda and all dependencies (E.g. npm packages) and sticking it in git. this obscures what is there and precludes effective code reviews.

Erik Osterman

@ in the end, I don’t have a better solution than what you proposed. i still dislike binary assets in git, but like with images it’s a necessary tradeoff sometimes. i think it’s technically sound strategy and quite elegant if you get it working end-to-end.

jonboulle

I don’t think it’s hard to draw the line; artifacts from build processes belong in blob stores (e.g. package repository or image repository). ideally the store is a CAS. then git can house a reference to them as necessary

1
jonboulle

you’ll find no disagreement from me on storing a lambda zip being an antipattern

jonboulle

but to push back on “storing anything else than readable text is a conflict” (not sure if you directly said this or it’s a paraphrase - so apologies for the potential strawman!) - there are plenty of cases of these kinds of content-addressable references (e.g. docker image SHA, Gemfile.lock, whatever) where they should definitely be stored in git but they can hardly qualify as “readable text” in themselves

1

2018-10-02

mallen
07:26:02 PM

@mallen has joined the channel

2018-10-01

jarv
01:56:53 AM

@jarv has joined the channel

2018-09-26

J-Man
06:14:22 PM

@J-Man has joined the channel

2018-09-25

Erik Osterman
06:32:39 PM

@Erik Osterman has joined the channel

Erik Osterman
06:32:39 PM

@Erik Osterman set the channel purpose: Discuss continuous delivery of infrastructure

06:32:40 PM

@ has joined the channel

Max Moon
06:32:40 PM

@Max Moon has joined the channel

aknysh
06:32:40 PM

@aknysh has joined the channel

Huray

Max Moon

huzzah!

#gitops done right

#gitops your stack!

We #gitops your stack!

We do #gitops!

We are #gitops!

1

@Erik Osterman could you add twitter integration to this channel, I have no experience with it, but may be we can lurk twitter to search for #gitops and notify to this channel.

Erik Osterman

Yea that might be a good way to seed some discussion

Erik Osterman
weaveworks/flux

The GitOps Kubernetes operator. Contribute to weaveworks/flux development by creating an account on GitHub.

Erik Osterman

They coined the term :)

Well http://gitops.com 301s to http://www.greenlightgroup.com/ didn’t expect that..

GreenLight Group IT Support Services - Home

GreenLight Group, a software consulting and systems integration services firm specializing in Enterprise Monitoring Software and IT Services.

Erik Osterman

Yea thought that was interesting too.. but I think it means something different for them

Erik Osterman

G for green light

Erik Osterman

IT for the support services

Erik Osterman

And Ops because it sounded good

Read that paragraph twice, still not sure what I’ve read.

aknysh

you mean this

aknysh
GreenLight IT Operations Services (GITOpS) allow customers to be able to leverage the expertise of GreenLight Group to run and manage their IT Operations Management solution(s) for day-to-day operations, taking the technology constraints out of the equation. By leveraging creative licensing programs in combination with our cloud infrastructure partnerships, GreenLight Group can offer customers subscription based licensing for core ITOM capabilities hosted in secure cloud infrastructure at very competitive prices. GITOpS is a unique offering that only GreenLight's expertise can bring.
aknysh

(sounds like an AI bot generated )

hihi, although it could also be written by me as a GH issue * hiding in corner *

Erik Osterman

Sounds like they are a software licensing aggregator

    keyboard_arrow_up