#gitops (2018-10)

Discuss continuous delivery of infrastructure Archive: https://archive.sweetops.com/gitops/

2018-10-24

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

they will have a service to add encrypted secrets

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

you could use ENV vars

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

and you’ll be able to create actions programmatically

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
workflow "New workflow" {
  on = "push"
  resolves = ["Deploy to Azure"]
}

action "Deploy to Azure" {
  uses = "./.github/azdeploy"
  secrets = ["SERVICE_PASS"]
  env = {
    SERVICE_PRINCIPAL="<http://sdrasApp>",
    TENANT_ID="72f988bf-86f1-41af-91ab-2d7cd011db47",
    APPID="sdrasMoonshine"
  }
}
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

looks like Terraform HCL

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yea that’s 100% :-)

2018-10-22

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I added a #atlantis channel since we’re doing a lot more with it these days (related to runatlantis.io).

2018-10-14

fdrescher avatar
fdrescher
10:28:47 PM

@fdrescher has joined the channel

2018-10-13

Gaurav avatar
Gaurav
11:47:30 AM

@Gaurav has joined the channel

2018-10-11

ben avatar
ben
08:32:50 PM

@ben has joined the channel

2018-10-10

antonbabenko avatar
antonbabenko

Send a email to Luke to get more names.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Talking 10/25

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I also pinged Luke in their new slack community

:--1:1

2018-10-09

stephen avatar
stephen
03:42:47 PM

@stephen has joined the channel

Adam avatar
Adam
04:17:00 PM

@Adam has joined the channel

Yoann avatar
Yoann
05:52:15 PM

@Yoann has joined the channel

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@antonbabenko @maarten any companies you know of using atlantis that I can reference in my slides?

antonbabenko avatar
antonbabenko

I don’t remember names. They were pretty small and unknown.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Thanks anyways :)

maarten avatar
maarten

Blinkist.com , need to ask the other two for OK

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ok, great, I’ll add them - thanks!

maarten avatar
maarten

When is it due?

2018-10-08

antonbabenko avatar
antonbabenko
07:30:58 AM

@antonbabenko has joined the channel

rohit.verma avatar
rohit.verma
08:52:00 AM

@rohit.verma has joined the channel

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
05:39:37 PM
maarten avatar
maarten

ah let me remove article with jpg

maarten avatar
maarten

replace*

maarten avatar
maarten
05:42:30 PM

AirBNB response to my question about having secrets coupled into Docker. Having secrets maintained away from the application makes it impossible to do an actual rollback to point an application to the previous config set which is a point.

I’m thinking of a way to make SSM/KMS/Chamber compatible with the mindset of that issue at stake.

SSM

/service/db_host  = 'hostA'
/service/db_user  = 'userA'
/service/db_host_migration = 'hostB'
/service/db_user_migration = 'userB'

On the Docker with Chamber /etc/chamber_rewrites ( or defined ENV ) defines how certain env vars can be rewritten with other created env vars.

db_host = db_host_migration
db_user = db_user_migration

This way it is possible to work with a default config set, yet it is still possible to fix config values to a certain deployment. Problem here is transparency, it is not visible from the outside which docker is using what.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

re:

/service/db_host  = 'hostA'
/service/db_user  = 'userA'
/service/db_host_migration = 'hostB'
/service/db_user_migration = 'userB'
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

why not use different service namespaces

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

e.g.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
/service/db_host  = 'hostA'
/service/db_user  = 'userA'
/service/migation/db_host = 'hostB'
/service/migration/db_user = 'userB'
maarten avatar
maarten

TBH, I did think of that too, but I’m afraid that would kill their parser a bit.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

their = chamber?

maarten avatar
maarten

yes

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

hrm

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

true, i think their import/export only handles one service namespace at a time

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

checking

maarten avatar
maarten

i mean, db_host is attribute here, migation/db_host is a nested attribute but has a / in it. I don’t think / are allowed

maarten avatar
maarten

inside the name

maarten avatar
maarten

and a full real namespace change, as in, changing the prefix of everything SSM, or a different service name. That of course can work. BUT. you would need to modify iam policies for the service to allow access to it, and I think that is just too much change for something trivial.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
05:51:41 PM
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

anyways, what i was actually hoping was that the export format supported the service namespace

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but it doesn’t

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so to your point

maarten avatar
maarten

I mean, by changing the full service path, and making it a level deeper more or less, you can make it to work but you would need to duplicate everything to that level deeper

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i think a cool/needed tool would be a chamber ETL tool with KMS (PKE) support

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

that maybe supports some custom mapping

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(since i don’t think the guys at chamber are too amenable to changes of this nature)

maarten avatar
maarten

Maybe yes, but it’s an interesting quest to drop and see what comes out.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

despite my previous reservations, I think I am leaning still towards your approach

maarten avatar
maarten

Better ( but impossible to get ) is a new style SSM with versioned change-sets which can be referred to.

2018-10-07

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

for those interested and want to follow along, we’re working on some enhancements for atlantis

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/atlantis

Terraform For Teams. Contribute to cloudposse/atlantis development by creating an account on GitHub.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we’ve added atlantis to our alpine repository:

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
curl -sSL <https://apk.cloudposse.com/install.sh> | sudo bash
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

And the work for integrating it into geodesic: https://github.com/cloudposse/geodesic/pull/298/files

Add support for Atlantis by osterman · Pull Request #298 · cloudposse/geodesic

what Add support for atlantis why GitOps style continuous delivery for infrastructure

2018-10-04

maarten avatar
maarten
09:51:01 AM

@Andriy Knysh (Cloud Posse) thanks.. I responded to the article as this is a step away from factor12. But with their java background env variables were probably something they have never used to begin with.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@maarten insightful comment

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
04:51:04 PM
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

this is a good point

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

what is your approach to this in what you’re doing?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

e.g. if you have the “widget” app and you encrypt the “prod” mailgun key and commit to source control in the widget app repo, is this blurring the line?

daveyu avatar
daveyu
05:15:17 PM

@daveyu has joined the channel

2

2018-10-03

maarten avatar
maarten
09:28:20 AM

Hi everyone, I wonder how most of you are doing secrets management. In a stack I’m working on the applications their env vars (ssm) are populated through terraform. Secrets like API keys are also stored in git but have been encrypted by KMS. Those secrets will be decrypted by terraform using aws_kms_secrets. My belief is that this is secure as envelope encryption would make the encrypted hash useless without access to KMS, next to that it makes it possible to version everything related to the application and removes manually entering passwords to SSM. @Erik Osterman (Cloud Posse) pointed out to me that git is about transparency and that storing anything else than readable text is a conflict. For me this is less of a problem when the alternative is to manually enter passwords. Happy to collect your thoughts.

pecigonzalo avatar
pecigonzalo
09:30:56 AM

@pecigonzalo has joined the channel

jonboulle avatar
jonboulle
09:30:56 AM

@jonboulle has joined the channel

jamie avatar
jamie
09:30:57 AM

@jamie has joined the channel

jonboulle avatar
jonboulle

git isn’t just about transparency in terms of content, it’s about transparency in terms of an audit log of changes, which this still provides. of course you can get that with other systems too, but isn’t the point of gitops to have all of that centralised in git?

Steven avatar
Steven
11:51:12 AM

@Steven has joined the channel

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@jonboulle in principle I agree with you. but stoking debate, where is the line drawn? what about binary artifacts from build processes or packages?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

for example, a common pattern I dislike, but which has emerged with terraform+lambda, is including a zip of the lambda and all dependencies (E.g. npm packages) and sticking it in git. this obscures what is there and precludes effective code reviews.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@maarten in the end, I don’t have a better solution than what you proposed. i still dislike binary assets in git, but like with images it’s a necessary tradeoff sometimes. i think it’s technically sound strategy and quite elegant if you get it working end-to-end.

jonboulle avatar
jonboulle

I don’t think it’s hard to draw the line; artifacts from build processes belong in blob stores (e.g. package repository or image repository). ideally the store is a CAS. then git can house a reference to them as necessary

:--1:1
jonboulle avatar
jonboulle

you’ll find no disagreement from me on storing a lambda zip being an antipattern

jonboulle avatar
jonboulle

but to push back on “storing anything else than readable text is a conflict” (not sure if you directly said this or it’s a paraphrase - so apologies for the potential strawman!) - there are plenty of cases of these kinds of content-addressable references (e.g. docker image SHA, Gemfile.lock, whatever) where they should definitely be stored in git but they can hardly qualify as “readable text” in themselves

:--1:1

2018-10-02

mallen avatar
mallen
07:26:02 PM

@mallen has joined the channel

2018-10-01

jarv avatar
jarv
01:56:53 AM

@jarv has joined the channel

    keyboard_arrow_up