#announcements (2018-09)

Is there an enabled flag?

Enabled by default

Ok, and what happens when you set to false? :)

Nothing gets provisioned

As we already do in terraform-root-modules

Ok

There is 1 event this week

~Actually it will be at 11 am pst~Unfortunately, I have a conflict at 11am PST this time. I’m going to stick with 9:00 am. We’ll schedule the next one at 11am which was the most popular time according to the poll.

Hi Gang,

I’ve just started using your vpc_peering terraform module and have run into an issue during the plan stage. I’m getting

* module.vpc_peering.data.aws_route_table.requestor: data.aws_route_table.requestor: value of 'count' cannot be computed

I checked out the FAQ at https://github.com/cloudposse/docs/blob/master/content/faq/terraform-value-of-count-cannot-be-computed.md and it doesn’t seem to be the same issue.

I am getting it during the plan stage when the requestor_vpc_id is coming from the output of a vpc module, however that vpc hasn’t yet been created and the id is going to be computed at this stage.

Something you’ve seen before and if so is this a supported scenario? Can provide more info if needed.

"${var.mc_vpc_id}" is a hard coded id of an existing vpc in the same account.

hi @Toby I’ll get to your question soon (we’ve seen that too)

let’s move the conversation to #terraform

Thanks. Want me to link or copy/paste there?

yes please

np

@Jeff what’s your status on terraform-aws-rds-cluster?

let’s move to #terraform as well

@Andriy Knysh (Cloud Posse) I wasn’t able to get admin_password to take a variable. I ended up using sed.

we improved the examples for the module, did you try this https://github.com/cloudposse/terraform-aws-rds-cluster/blob/master/examples/basic/main.tf ?

was tested as it is and was working

@Jeff ^

I saw that the terraform-rds-cluster module supports DNS friendly names for both master and replica’s. This may cause issues when SSL connections to the database are being used. Perhaps, it’s wise to document this somewhere? And according to the markdown, the zone id is required - which seems a bit odd to me.

@siert thanks, if you want the zone to be optional (and I see why), please open an issue in the repo and we’ll implement it

https://github.com/opencredo/terrahelp

Awesome chat! Thanks everyone for inputs!

thanks guys! @antonbabenko great talking points. will summarize what I remember.

What you will be missing I will try to reflect in my work document

Just don’t summarize my last sentence, that was quite unsummarizable.

Hey friends, trying to lock down to a specific CP release of terraform-aws-rds-cluster?

having issues where i can build a db, which works fine, and then if i simply run the same tf code again, instead of saying ‘ no changes’ it rips it down and rebuilds : (

hi @h20melonman

hello !

let’s move to #terraform

ok

https://nanobrain.io/ - give it a try if you have strange bug you don’t even know where to look into. I tried it on Terraform errors during apply and it gave me some tips (I knew them very well before, because I could search on github, but still)

Any of you guys configuring your services through SSM/ETCD/etc dynamically (automatically updating after run). There are several examples online of using somthing like confd or similar to dynamically update your service from some K/V store, I was thinking of the failure domain for this scenarios because lets say someone/something updates a k/v for that service, the service reloads and fails now (all containers, since it has no way of green/blue) now your entire service is down. I have not seen any blog/article talk about this either, but it seems really risky.

thoughts?

Yes, I have used etcd/confd in the past. Any system that automatically reloads all services when the value changes is inherently fragile. That’s why the kubernetes approach is the best we can do: update secrets or configmaps, then trigger rolling updates which will abort if health checks fail.

We use chamber, but it does not automatically reload. So it is compatible with the rolling update strategy.

I personally swear by SSM/Consul KV stores (not tried etcd). Couple of thoughts there:

1. there is no reason your configs shouldn’t be subject to the same dev/test/prod flow that everything else is. Keep your SSM config in cloudformation/teraform/ansible/whatever all codified with a deployment pipeline. If a bad setting crashes in test it never makes it into prod. It also means during development it’s super easy to replicate running configs if needed, as well as better unit test potential as you have “real world configs” you can test containers/functions against.
2. for more dynamic settings (that maybe an ops team is applying to a live system to tune things), simply don’t grant direct access to the KV. Instead there should be an api abstraction so your devs can “protect” fields from getting bad settings. Ops don’t really know the ugly details of that var after all, so wrap it up to prevent the unpredictable “oops i used a string instead of a list” drama.
3. SSM is inherently region based. If you blue/green across regions settings don’t “spill”. Consul has a “datacenter” concept which works similarly but is a bit more flexible because it’s not a baked in service. Both also have version history so it’s easy to undo a bad change … and sometimes I’ve even seen companies nest versions into the kv/path, though that’s not something I’ve personally done.
4. It’s just a good idea to keep your configs separate from code. This fits very naturally into that paradigm, and I’m not rebuilding code just because I want to change a minor setting.
5. KV stores don’t automatically reload anything. That’s up to the dev and how he wrote his code. If he only polls the setting on boot then you have to restart the containers. Lambdas obviously will poll every run because of their nature. The fragility of the services is in the hands of the devs at the end of the day, or confd/consul-template if you automated that … but honestly that’s still you making the stability decision.

the best real-world example I could point to would be Amazons RDS service. I’m sure you noticed but every run-time configurable setting is stored in SSM for those database servers.

I’d also add it’s inevitable if you go serverless, which is honestly how I learned to love it. I was forced to take the journey then on the end of it I went “huh, why didn’t I do this sooner”?

I know KV stores dont automatically reload everything, much to the contrary I saw many people advocating/writing/blogs/etc about auto reload, and I was concerned about how they handled failure domains

tbh we only use KV for secrets, and I think they are a good usecase for dynamic values created from other services but not for the core settings of the service itself.

Hi @pecigonzalo What kind of dynamic values would see as a possible fit ? And what about SSM rate limiting ?

No particularly in SSM, I was thinking more about Consul/etcd/etc

something like Service A uses some details from Service B, service B keeps them up to date when running, but Service A still runs without those

like how you do service discovery

What would be the advantage of this over using json http endpoints, as in the way microservices would normally talk to each other. I’d like to understand.

With this I mean, if service A needs some of those params from B, A would connect to B’s json endpoint, retrieves params, and service A stores it in redis with some expiry information.

Im not saying it is better. Its a different way I guess. Just to clarify, im not talking about using it to do queries through Consul, mostly discovering parameters of the other services or similar

in a mesh/service discovery pattern, you will query a KV store like consul/etcd to get the ips/ports of ServiceB and then route to them (be it through itsio/etc or directly from your service A) but I have seen many patterns where people also store Service config or dynamic service options for other services in the KV.

EG: https://labs.magnet.me/nerds/2015/10/26/consultant-configuration-management-with-consul.html

another example of what I was talking about: https://github.com/breser/git2consul

There are no events this week

Redis and Consul aren’t really designed to solve the same problems though. Heck you can find blogs of people using one to help configure the other: https://www.joyent.com/blog/redis-on-autopilot

is this related to my comments? as I never mentioned redis

Can someone advise me good literature on implementing “Continuous Delivery”, preferably something readable for non-devops ?

Hrmmm so are you looking for something that sells the merits of Continuous Delivery for non techies? E.g. why we should practice it rather than how to do it with some specific technologies

Correct.. I found two articles which look helpful. Docker Orchestration is nice and all but the goal is to boost development. I’m currently helping a company which has not really implemented CI let alone CD, and a lot of manual QA..

https://harness.io/2018/04/10-signs-you-dont-do-continuous-delivery/ & https://semaphoreci.com/blog/2017/07/27/what-is-the-difference-between-continuous-integration-continuous-deployment-and-continuous-delivery.html

if you find a resource that convinces this company, i’d love to hear about it

Well.. these were already a good start it seems, so they want to start with the lesser important services first. Challenge is that this works with good test coverage. So I stressed the importance that they start writing proper tests for their new features or this idea will fail and it will go back to manual QA with 2 weekly press on the button releases

@pecigonzalo sorry about that minor derail, @maarten mentioned redis so I was trying to keep both going in one reply. I clearly failed

Hello, I’m new here, any chance we can add custom_error_response = [”${var.custom_error_response}”] to the https://github.com/cloudposse/terraform-aws-cloudfront-s3-cdn module? I have not really contributed to github projects before and thought I’d ask here. Could I use the non S3 cdn module with an existing S3 origen?

@eric_garza hello

let’s move this conversation to #terraform

Hi, just stumbled here by accident. Interesting stuff. Reading into it

hi @ff, welcome to the community

Hi @ff! Most of us are here by accident, so welcome to the club

amazing stuff to explore. more night shifts to come

Hello everyone

welcome @GC @ff!

not a bad place to stumble into

what are you guys working on?

I miss you guys.

I’m doing long days at the moment but I remember our time together fondly

not sure if you all saw this yet https://justi.cz/security/2018/09/13/alpine-apk-rce.html

I know a few security companies that won’t use alpine for compatibility reasons but that’s not good.

Heh, and we just discovered it doesn’t use TLS by default on repos

https://sweetops.slack.com/archives/CBXSAR45B/p1536702692000100

Write everything in Go and use FROM scratch Docker containers in Multi-Stage builds. Problem solved

Morning. Is this the cloudposse general slack hangout?

Setting up a new multi-account setup in aws for small/medium company (not large) .. In the following scenario … would there be any reason to NOT have IAM users in the root account

[Root]                  (Consolidated Billing / Org Root +  IAM users here)
[Security]              (Central logs, audits)
[Prod], [Dev], [Local]  (Environment application specific accounts)

or

[Root]                  (Consolidated Billing / Org Root)
[Security]              (Central logs, audits)
[Identity]              (IAM here?)
[Prod], [Dev], [Local]  (Environment application specific accounts)

So to clarify – are there advantages having a separate Identity / Hub / Gateway account for IAM? or since it’s a somewhat smaller deployment … can I just store IAM in ROOT account? I’ve seen setups both ways .. but looking for a reason FOR or AGAINST

currently we have IAM users/roles in the root account, but we’ve been discussing to move it to a new identity account

the reason might be that nobody should access the root (billing) account

and also, in some companies they don’t even allow to do it

but for a small setup, both ways should work, depends on if you want to give access to the billing account to somebody who needs to manage IAM

@Erik Osterman (Cloud Posse) has more insights into this

but in general, yes, having a separate identity is more secure and will work for more companies/use-cases

Yeah my thoughts similarly … the only thing I could think of if somehow the IAM users are misconfigured could give potentially elevated priv in root account ; while they could be misconfigured in the identity as well .. but slightly harder to cross-account jump into root to elevate

I’m very determined to fix the elasticbeanstalk terraform provider , I miss it saying “nothing changed” haha… Any helpful links for doing Go/Terraform development?

@OGProgrammer thanks, finally somebody would fix that

we have this issue here https://github.com/cloudposse/terraform-aws-elastic-beanstalk-environment/issues/43

we think the problem is that TF has one set of params while EB maintains a diff (smaller) set of params (relevant to your env)

if the assumption is correct, then what needs to be done is to determine the diff b/w only those params that EB returns from the API (not all params that TF has in the module)

although it could be more complicated than that

the code should be here https://github.com/terraform-providers/terraform-provider-aws

https://aws.amazon.com/blogs/aws/new-session-manager/

Wow, gitter room for Terraform is so active. It is nice to see so many familiar faces there Notifications in gitter are weird, I have just received notifications from people who mentioned my name from July…

Haha, job queue is too big :)

Yeah, they definitely have to improve the way they handle queues

There is 1 event this week

CORRECTION this event has moved to 11am PST (two hours later) per popular demand

Ping me if you want to be added to the calendar invite

also, if you have suggestions for managing events like this using som SaaS <—- i want to know

I will try to be on this call but can’t promise today.

Hello to you people of SweetOps! I’m Andrey, solo consultant with focus on CI/CD, Jenkins, Docker, K8S, HashiCorp tools. Good to be here!

nice to meet you @andrey.a.devyatkin

@andrey.a.devyatkin great to meet you! got your email too. Would love to give you a tour of everything going on as it relates to what you’re working on. If that sounds interesting, drop something on my calendar here: https://calendly.com/cloudposse

(the same invitation goes to anyone else here! if anyone wants a tour of everything going on here at cloudposse as it relates to our Open Source initiatives, feel free to use the calendly link above to schedule a time)

Have you guys ever considered the name Insane Cloud Posse?

haha we own the domain

but not branding around it…

that’s awesome

Hi @andrey.a.devyatkin! Great to see you here

I am running a tad late. Got locked out of my office

#townhall has some meeting links

hi all!

I wanted to say “Thank you!” for a LOT of amazing insights & ideas here (and in github repos - that’s how I got here). Though I processed only a bit, I’d already borrowed some.

Hi @tolstikov! Welcome to us

hey @tolstikov

what are you working on?

@Andriy Knysh (Cloud Posse) some CICD, IaC, automation, stuff like that mostly

what CI/CD are you using?

Jenkins pipelines, Gitlab CI, Travis, etc but now I’m working on some idea of making universal pipelines that are not locked into specific CICD solution and could be launched from any CICD or manually (to avoid a vendor lock and well-known complications and limits of available systems)

though to be honest, I don’t consider Gitlab CI & Travis as a fully blown CICD, but rather CI with some functions of CD

@tolstikov we have similar ambitions for the CI/CD - reducing vendor lock-in

we use our https://github.com/cloudposse/build-harness to achieve this to some degree

I believe you already have a good foundation to do it soon, as I could see from repos

we use it with travis and codefresh

in an ideal world, every step of the build pipeline is a container. then the build-harness is a perfect compliment.

yep, I spent some time digging build-harness already

haha yea - it’s pretty massive

TBH though we’re focused mostly on Codefresh these days

I have to say that it changed my “world vision” a bit

so many CICD systems, so little time

seems like Makefile is the almost perfect fit for this goal (at least I sticked to this thought these days). And with some additional wrappers and builders it could do the job perfectly.

do you have any particular reason to use Codefresh?

my favorite quote

I believe that I saw in this slack something like “I can iterate with it faster”, but I never had a chance to check it by myself.

my (3) arguments for codefresh:

1) unlimited pipelines per repo; every step is a container. no need for complicated manifests (E.g. circle). stick all complexity in the container.

2) tight integration with kubernetes and helm. easy 1-click deployments and rollbacks.

3) native support for docker/docker compose. need to spin up 15 containers for CI? no problem. they’ll do it.

4) integrated chartmuseum?

sounds good… But why do you need smth like Atlantis for Terraform CICD?

hah - that’s a good question

i’m teetering back and forth on atlantis

atlantis pros:

• purpose built CI/CD tool for terraform; I like tools that specialize

• chatops style interaction

• run atlantis inside AWS account and use instance profiles for IAM

atlantis cons:

• no easy way to do even basic acls like restricting who can run plan or apply.

• currently no way to have multiple pipelines per repo

yep, I saw the proposal about multiple pipelines from you…

yea - haven’t checked if there was any update on that

for now, we’re going to move forward with Codefresh

downside with codefresh is running build agent inside of AWS account (atlantis style) requires enterprise agreement

since most of our customers have enterprise agreements, this is not a problem

anyway, I believe, that behind each “apply” on PROD environment should be a git tag, and not just some pull request approved

i agree with you in an ideal world

and didn’t see anything like that in Atlantis, but I just did a very quick check

for staging, however, that should not be a requirement

as it might not work first time around. <— hence i like the chat ops

https://github.com/cloudposse/github-commenter

our intent is to mimick some of the atlantis behavior using this

whereby we comment back on PRs

yep, I can see what you mean

also, it’s unclear to me how well atlantis works in a large terraform monorepo with many projects <— we want to only plan/apply projects that have changes

good point

also, I believe that it is better to have separated “build” and “apply” stages for the pipeline itself too:

so, in “build” stage you apply some template processing, etc (e.g. with gomplate, etc) and create the “pipeline artifact” (e.g. commit/tag in some other repo or branch) with the final pipeline and all deps

and in “apply” stage you don’t perform any processing, but only execute command(s) on top of “pipeline artifact”

sounds like over-engineering, but I believe this is the way

@Erik Osterman (Cloud Posse) What is unclear ? The way it works now is that it can autoplan for projects with changes. This could mean that 3 out of 10 projects would be autoplanned for which you after can apply, and then also apply-per-project.

big problem is the one huge yaml

ok that’s cool

Also if you have modules in the same mono-repo, it would automatically autoplan all projects the moment you change the modules. I’m not sure if it actually calculates dependencies.

@maarten so in my own words, if a monorepo has aws/vpc and aws/ecs, and aws/dns folders, and I only make changes to aws/dns, it will detect that and only run plan (and eventually apply) on that

correct

nice

it’s actually working really nice. pre-merge-apply’s are confusing but it never leads to problems. The plan locking works well.

Not having streaming output is more of an issue. I’d like to see what’s happening in real time.

what about the acl issues?

wdym ?

no way to restrict who can run atlantis plan or atlantis apply

other than restricting access to repo altogether

ah that can be, hasn’t been an issue for us

as an active user, what are your thoughts on this? https://github.com/runatlantis/atlantis/issues/249#issuecomment-422240279

We don’t have that structure, but for the way cloudposse is doing stuff it would make it compatible. I also don’t see another way how to do it otherwise,

thanks for the first-hand account

This shouldn’t involve a lot of code change in Atlantis I think, have you taken a look already @Erik Osterman (Cloud Posse)?

I asked @alebabai to take a brief look at it

but haven’t checked in to hear what the scope-of-work would entail.

in the end it’s always a lot, with added testing and what not.

hah, yea

hey folks. didn’t know this existed.

I’m hitting an issue with the VPC peering module. Is this the right room to discuss?

Let’s move to #terraform

thx

Such a great description of GitOps by Flux. https://github.com/weaveworks/flux

so, we have GitOps for Terraform (Atlantis), and GitOps for k8s (flux), but does some universal GitOps solution exist?

Jenkins?

j/k

lulz

ohh, yeahh, but still needs some setup

So, I don’t know of anything off the shelf

also, I believe Jenkins have some great plans for the future now

What I like about Codefresh, is that every step of the pipeline is a container. So if we can containerize the Business Logic, we can do it.

personally i kind of think of pretty much any set of tools that executes a workflow based on PRs, merges, and tags can be considered GitOps

Yes - i don’t think flux or atlantis is technically needed. they are good purpose built tools.

I’d like to see some general framework for any type of “command to launch on PR approval”

instead of kubectl/terraform apply launch hardcoded

here’s what i created to piece things together between codecommit and codebuild, https://github.com/plus3it/terraform-aws-codecommit-flow-ci

probably Cloud Posse will have something at some point

a pull request triggers a job, a merge triggers another job, and a tag triggers a job. you define what each job-type does via your buildspec(s)

yep, but vendor-lock again…

not really

i write the jobs in a Makefile, and the buildspec just executes make targets

easy enough to port to gitlab-ci or travis or whatever

also, this summarizes nicely what i think of vendor lock-in, https://bravenewgeek.com/multi-cloud-is-a-trap/

I’m working on our requirements here: https://cloudposse.quip.com/Mb7cA97IuIm8

The mental gymnastics I see companies go through to avoid vendor lock-in and “reasons” for multi-cloud always astound me. It’s baffling the amount of money companies are willing to spend on things that do not differentiate them in any way whatsoever and, in fact, forces them to divert resources from business-differentiating things.

so many good quotes


When you abstract away the differentiating features to avoid lock-in, you also abstract away the value. You end up with vendor “lock-out,” which basically means you aren’t leveraging the full value of services.

@loren yes and no, I believe there should be “the golden mean” between totally avoiding vendor-lock and total locking, in other words, you should have some way to easy move to other provider

but, with kubernetes the issue is mitigated a lot I believe (especially with k8s-as-service)

i just don’t see it being that big a deal, people get way bent out of shape over the smallest thing

spaces or tabs! vi or emacs! blech. just get the job done, stick with whatever the project already does.

looks like I (almost) always agree with you @loren

but better to be DRY, and don’t repeat yourself with a next job or at least be able to re-use most parts of the previous work

i agree. though i always try to remember also that premature optimization is the root of all evil, and that copy-paste is often better than a bad abstraction

that’s true

https://en.wikipedia.org/wiki/Worse_is_better

“new jersey style”

I started #chatops for those interested

I’ve started an agenda for the next Town Hall on 2018-10-03

https://cloudposse.quip.com/VHF9A1Qrp3eR

I will be on vacation during next Town Hall call, so talk without me.

Ok!

There are no events this week

Anyone know if TF supports disabling/enabling STS regions? I’ve been googling and can’t find anything …

Also – Anyone have link to some cool global AWS organization TF stuff for small/mid/medium sized ORGS in terms of things being applied everywhere (e.g. cloudtrail, config, some global roles, etc.)

Hey @mark

Yo

not sure if it’s cool but here is what we usually do https://docs.cloudposse.com/reference-architectures/

https://github.com/cloudposse/terraform-root-modules

awesome

https://github.com/cloudposse/root.cloudposse.co

https://github.com/cloudposse/testing.cloudposse.co

https://github.com/cloudposse/staging.cloudposse.co

terraform-root-modules is our collection of TF modules that we usually deploy

Looking at that one now

root, testing, staging, prod etc. are our reference architectures for the stages, they don’t have any code, they are just identities, we deploy them do different AWS accounts

gotcha checking it now

hi all, any recommendation on vpn setting, which allow resolving private route53 dns records locally. I have tried this https://github.com/kylemanna/docker-openvpn, along with bind instructions from here https://www.youtube.com/watch?v=xeIWpyst0P4. On my local use tunnelblick to connect. Couldn’t mac the setup working though, got connected to vpn and can use private ips, but dns resolution still not workable. Tried adding nameserver also manually, no luck.

@Erik Osterman (Cloud Posse) @Andriy Knysh (Cloud Posse) Any leads on above

have you seen our bad ass chart for this?

https://github.com/cloudposse/charts/tree/master/incubator/openvpn

this supports GitHub SSO + shortlived open vpn keys

It could use some dusting off - but here to help if you want to

this is required for an old environment, there is no kubernetes there

this image cloudposse/openvpn, can you share code?

it’spublic https://github.com/cloudposse/openvpn

I don’t see any usage of bind, are you sure client can resolve route53 entries, e.g. from my local if I try private.internal.api ( internal.api=private hosted zone), will it work?

so the traffic is NAT’d from internal ec2 instance

so it has the internal-view for route53

it also resolves kube-dns

you mean from my local I can call svc directly, e.g curl <http://alertmanager-operated.monitoring.svc.cluster.local>

but how your local nameservers are updated? which openvpn client you used?

that’s a DHCP setting

https://github.com/cloudposse/charts/blob/master/incubator/openvpn/templates/vpn.deployment.yaml#L61

https://github.com/cloudposse/charts/blob/master/incubator/openvpn/templates/vpn.deployment.yaml#L54

-n $(cat /etc/resolv.conf | grep -i nameserver|head -n1|cut -d ' ' -f2)

as argument to ovpn_genconfig

@rohit.verma

Humbly begging for a plus https://github.com/segmentio/chamber/issues/143

Suggest breaking it apart into what and why

Reading the description it’s not clear what it does does, but more why it does it

What ~ like how it works

good idea, thanks

Ping me when you update it and I’ll Re review :)

@channel hey everyone give a warm welcome to @jonboulle! Good to have you here

Hi @jonboulle!

hi @jonboulle

whoa lots of welcomes

o/

@jonboulle What brings you around?

maarten

Jon replaced me at Blinkist and also works on the ecs airship modules, although he actually is more of an K8S guy.. @jonboulle maybe you can share some more info’s.

Haha ok, well you are amongst friends :) we do lots of k8s and terraform

Hopefully you can join our next town hall zoom call which is next Wednesday 11am PST

@markmutti created a new channel #aws. Join if this sounds interesting!

hey everyone give a warm welcome to @Andy! Good to have you here

Hey @Andy! let me know if there’s anything we can help out with.

hey everyone give a warm welcome to @Daz! Good to have you here

Hi everybody! It’s pleasure to hang out here. @rohit.verma Hey, there’s an easy “openvpn” way with pritunl, https://pritunl.com/ , users can be configured via web if it suits you. Also making a k8s deployment/chart for pritunl should be very easy, it uses mongo which is the only which will require a pvc…

all the dns, dhcp you get out-of-the-box

@dennybaa Thanks for suggesttion, this is really nice. I tried and it was working as expected.

@dennybaa that is a great suggestion. I forgot about it.

hey everyone give a warm welcome to @vitaly.markov! Good to have you here

hey everyone give a warm welcome to @praveen! Good to have you here

hey @vitaly.markov and @praveen, what are you working on?

@Erik Osterman (Cloud Posse) created a new channel #events. Join if this sounds interesting!