#gitops (2018-10)

@jarv has joined the channel

@mallen has joined the channel

Hi everyone, I wonder how most of you are doing secrets management. In a stack I’m working on the applications their env vars (ssm) are populated through terraform. Secrets like API keys are also stored in git but have been encrypted by KMS. Those secrets will be decrypted by terraform using aws_kms_secrets. My belief is that this is secure as envelope encryption would make the encrypted hash useless without access to KMS, next to that it makes it possible to version everything related to the application and removes manually entering passwords to SSM. @Erik Osterman (Cloud Posse) pointed out to me that git is about transparency and that storing anything else than readable text is a conflict. For me this is less of a problem when the alternative is to manually enter passwords. Happy to collect your thoughts.

@pecigonzalo has joined the channel

@jonboulle has joined the channel

@jamie has joined the channel

git isn’t just about transparency in terms of content, it’s about transparency in terms of an audit log of changes, which this still provides. of course you can get that with other systems too, but isn’t the point of gitops to have all of that centralised in git?

@Steven has joined the channel

@jonboulle in principle I agree with you. but stoking debate, where is the line drawn? what about binary artifacts from build processes or packages?

@maarten in the end, I don’t have a better solution than what you proposed. i still dislike binary assets in git, but like with images it’s a necessary tradeoff sometimes. i think it’s technically sound strategy and quite elegant if you get it working end-to-end.

https://medium.com/airbnb-engineering/production-secret-management-at-airbnb-ad230e1bc0f6

I don’t think it’s hard to draw the line; artifacts from build processes belong in blob stores (e.g. package repository or image repository). ideally the store is a CAS. then git can house a reference to them as necessary

you’ll find no disagreement from me on storing a lambda zip being an antipattern

but to push back on “storing anything else than readable text is a conflict” (not sure if you directly said this or it’s a paraphrase - so apologies for the potential strawman!) - there are plenty of cases of these kinds of content-addressable references (e.g. docker image SHA, Gemfile.lock, whatever) where they should definitely be stored in git but they can hardly qualify as “readable text” in themselves

@Andriy Knysh (Cloud Posse) thanks.. I responded to the article as this is a step away from factor12. But with their java background env variables were probably something they have never used to begin with.

@maarten insightful comment

this is a good point

what is your approach to this in what you’re doing?

e.g. if you have the “widget” app and you encrypt the “prod” mailgun key and commit to source control in the widget app repo, is this blurring the line?

@daveyu has joined the channel

for those interested and want to follow along, we’re working on some enhancements for atlantis

https://github.com/cloudposse/atlantis/issues

we’ve added atlantis to our alpine repository:

And the work for integrating it into geodesic: https://github.com/cloudposse/geodesic/pull/298/files

@antonbabenko has joined the channel

@rohit.verma has joined the channel

AirBNB response to my question about having secrets coupled into Docker. Having secrets maintained away from the application makes it impossible to do an actual rollback to point an application to the previous config set which is a point.

I’m thinking of a way to make SSM/KMS/Chamber compatible with the mindset of that issue at stake.

SSM

/service/db_host  = 'hostA'
/service/db_user  = 'userA'
/service/db_host_migration = 'hostB'
/service/db_user_migration = 'userB'

On the Docker with Chamber /etc/chamber_rewrites ( or defined ENV ) defines how certain env vars can be rewritten with other created env vars.

db_host = db_host_migration
db_user = db_user_migration

This way it is possible to work with a default config set, yet it is still possible to fix config values to a certain deployment. Problem here is transparency, it is not visible from the outside which docker is using what.

re:

/service/db_host  = 'hostA'
/service/db_user  = 'userA'
/service/db_host_migration = 'hostB'
/service/db_user_migration = 'userB'

why not use different service namespaces

e.g.

TBH, I did think of that too, but I’m afraid that would kill their parser a bit.

their = chamber?

yes

hrm

true, i think their import/export only handles one service namespace at a time

checking

i mean, db_host is attribute here, migation/db_host is a nested attribute but has a / in it. I don’t think / are allowed

inside the name

and a full real namespace change, as in, changing the prefix of everything SSM, or a different service name. That of course can work. BUT. you would need to modify iam policies for the service to allow access to it, and I think that is just too much change for something trivial.

anyways, what i was actually hoping was that the export format supported the service namespace

but it doesn’t

so to your point

I mean, by changing the full service path, and making it a level deeper more or less, you can make it to work but you would need to duplicate everything to that level deeper

i think a cool/needed tool would be a chamber ETL tool with KMS (PKE) support

that maybe supports some custom mapping

(since i don’t think the guys at chamber are too amenable to changes of this nature)

Maybe yes, but it’s an interesting quest to drop and see what comes out.

despite my previous reservations, I think I am leaning still towards your approach

Better ( but impossible to get ) is a new style SSM with versioned change-sets which can be referred to.

@stephen has joined the channel

@Adam has joined the channel

@Yoann has joined the channel

@antonbabenko @maarten any companies you know of using atlantis that I can reference in my slides?

I don’t remember names. They were pretty small and unknown.

Thanks anyways :)

Blinkist.com , need to ask the other two for OK

ok, great, I’ll add them - thanks!

When is it due?

https://www.reddit.com/r/devops/comments/9mpp0g/terraform_plugins_for_privileged_access/

Send a email to Luke to get more names.

Talking 10/25

I also pinged Luke in their new slack community

@ben has joined the channel

@Gaurav has joined the channel

@fdrescher has joined the channel

I added a #atlantis channel since we’re doing a lot more with it these days (related to runatlantis.io).

continuing this discussion https://sweetops.slack.com/archives/CBW0HJDS8/p1539711018000100

nice article https://css-tricks.com/introducing-github-actions/

they will have a service to add encrypted secrets

you could use ENV vars

and you’ll be able to create actions programmatically

looks like Terraform HCL

Yea that’s 100% :-)