#announcements (2018-07)

Second training repo. No step by step instruction yet. If you have ideas, or suggestions please submit issues. https://github.com/firmstep-public/trainingdaytwo

The example module in demo_one is a working module for a web server in an asg. Which I figured was a common desire for the first thing to template for a company.

@rohit.verma I was able to reproduce that s3 path issue you were experiencing when mounting the s3 bucket: https://github.com/cloudposse/geodesic/pull/167 - once we have it CR’ed and tested will have a new release with this fix in it. Will keep you posted.

@mcrowe (and possibly @tamsky) also ran into it

just wondering why the mount command is not honoring the PATH

Hi Erik, we had a quick chat on Friday regarding ECS! I would love to know how you do it. The end-goal for me is to be able to use 3rd party deployment software which most likely pushes a docker image based on a GIT-SHA, meanwhile Terraform should be able to run and take the last active task definition ( this part is simple ). Harder part is when I want Terraform to be able to modify the environment variables while keeping the image of the last task definition.

The idea I now have is by taking the image variable length to decide if it’s bootstrapping or keep_image:

# Initial bootstrapping with ecr/repo:tag module “ecs_app” { image = “ecr/repo:tag” }

# after that, a change to keep_image module “ecs_app” { image = “” ( default) }

when image==””, i create datasources to get the current image definition, use that as input for the updated task definition resource.

Not sure if that would all work, and happy to hear your input on that.

roughly it breaks down like this:

1) deploy ecs service task with container definition. default the container to our default-backend (~ a 404 page). 2) use codebuild/pipeline to CI/CD all image deployments. 3) define one public ALB per cluster 4) use ingress rules (targets) to route traffic based on host/paths to backend services

the inspiration for the architecture comes from our experience with kubernetes and trying to make it feel more like it.

what I describe above is captured in this module:

https://github.com/cloudposse/terraform-aws-ecs-web-app

the CI/CD is here: https://github.com/cloudposse/terraform-aws-ecs-codepipeline (which swaps out the image & tag)

we programmatically generate the container definition JSON in this module: https://github.com/cloudposse/terraform-aws-ecs-container-definition

and our ALB modules are in: https://github.com/cloudposse/terraform-aws-alb https://github.com/cloudposse/terraform-aws-alb-ingress

oh, and our “default-backend”

https://github.com/cloudposse/default-backend

Does Codebuild allow cross account ECR repos by now ?

we have 1:1 codebuild/codepipeline/ecr/web app

so a different build image for your staging , as for prod ?

aha, for now, we have not orchestrated that.

in our case, we would promote an image to the prod repo

but we have not done that yet

(we’re pretty early in our ECS journey as most of what we use is k8s)

let me correct that

we currently would rebuild it

but the way I would want to solve it eventually is to promote the image between ECR repos

still reading through it..

I see this working when having an imagename like repo:latest , but not for repo:unique_id or am I missing something

@sarkis where is an example of our build spec?

we do set a tag

so for example, it’s possible to only deploy tagged releases

then it would be repo:tag

https://github.com/sarkis/docker-php-poc/blob/master/buildspec.yml

(we never pin to latest)

printf '[{"name":"%s","imageUri":"%s"}]' $CONTAINER_NAME $REPO_URI:$IMAGE_TAG > imagedefinitions.json

@maarten I thought you were doing something similar to this

so it all comes down to how IMAGE_TAG is computed

yes, but after the building, how is terraform aware of the new IMAGE_TAG, I’m not seeing that.

oh, the lifecycle of the image:tag is not the job of terraform

this is our concession

terraform is strictly responsible for deploying the infrastructure that powers the service

monitoring

autoscaling

iam permissions, etc

so i think we ignore changes, right @sarkis?

I agree there, but for me ENV VARS are a sort of grey area

SSM

+chamber

Yeah at Blinkist we’re using SSM

(it doesn’t resolve how to “rollback” envs, but we’ve also conceded that we won’t solve for that)

But another customer doesn’t want SSM or doesn’t want a wrapper inside his Docker for that..

So I thought, maybe I can find a way to deal with that using container def. datasources

i agree that I don’t like the wrapper inside the container as the entrypoint, but it’s become the necessary evil to reduce complexity with terraform.

wrapper = chamber

would be nice to have have ENV VARS defined for the ecs service instead, problem solved

something else, which is beautiful, happens if you use SSM though…

https://github.com/cloudposse/terraform-aws-ssm-parameter-store

you can provision those SSM parameters from outputs of your other modules

users, passwords, hosts, etc

yeah I’ve seen that, super cool and will use it

what’s the customer’s counter argument?

…we’ve even started using chamber with kubernetes in place of configmaps and secrets. it makes it much, much easier to manage ACLS+IAM using IAM roles

Another thing in the chain they don’t know.. I was probably not convincing enough.. first customer after my current employer ..

haha, yea, understood - in the end, if you overwhelm them with all the pros, I think the cons are very minimal.

the ECS envs are also not encrypted

ok, outside environment variables you still have CPU and MEMORY definitions

that is an actual terraform argument I think

maybe not for Fargate

that is an actual terraform argument I think

can you elaborate

inside the task definition you define the cpu and memory for a task

Of course you can set these vars during deployment

but is this something any developer should do in some conditions, or rather have memory/cpu centrally orchestrated

aha, yes, but i don’t think this solution precludes that

we set some defaults

but the buildspec.yaml can also override them

printf '[{"name":"%s","imageUri":"%s"}]' $CONTAINER_NAME $REPO_URI:$IMAGE_TAG > imagedefinitions.json

just add memory to that

no?

you’re right. I’m thinking from a perspective where the CI is circleci and not managed by terraform..

aha

yes, i can see that some stuff gets more complicated that way

But still, circleci can invoke codebuild

i guess

or a lambda which does nothing but deployment, and terraform manages the parameters for the lambda.

yea, lambda is the ultimate escape hatch. can probably accomplish it that way.

That also saves me distributing access keys which can stop services.

Do you a tool for remote command execution, fixing a failed migration etc ?

Thanks anyway, I didn’t find the answer to my solution, but it made it clear that when I control the deployment of ecs I can still control the container definition, be it codebuild or not.

yep! no problem. i think we went down a similar path until we resovled that it wasn’t feasible at this point in time with terraform.

hrmm… the only migrations we’ve done so far happen in k8s and then we’re able to exec into the container to perform manual remediations.

@jamie might have some tips

ok because the new guy replacing me is working on my old tool which does this:

• Takes current running task, properties, security groups
• Creates keypair
• Starts EC2 ECS Instance with keypair
• Starts task [same iam etc]
• SSH into EC2 instance , creates a socket forward for /var/run/docker.sock ( this is so cool)
• docker exec into task

that sounds pretty cool

on-demand ssh

i know this is a pattern promoted by Teleport SSH (gravitational)

but haven’t seen it in practice yet with ECS

https://github.com/blinkist/skipper it’s dormant now, also because my golang skills are .. but I’m sure he’ll be able to make something nice of it

Anyway, I’ll keep you updated on it, for now it’s focus is on regular EC2, with already SSH access.

For conditions without VPN we can maybe also add a network load balancer to allow outside access to internal ssh

yea, what I think could be neat is to have something like this:

https://github.com/cloudposse/bastion

that is deployed on demand into the cluster for triaging

e.g. fixing failed migrations

Well it would be nice to be able to completely log it to an S3 bucket, with the private key EC2 generation for the ec2 instance for the specific user (for which it needs MFA ) the extra MFA is probably overkill.

but is ec2 instance access necessary?

what i like about using containers is that it’s still isolated

I like that too, but when things brake someone wants to have access I suppose..

One question, how quick is codebuild now w/r booting up ?

@sarkis can maybe answer this

The idea I have now is this

• Have CircleCI test & build , a lot of startups here are using Circleci
• After build and push to ECR, push textfile to S3 in dev/uat/prod environment with image:tag
• Codebuild just pushes to ECS
• CircleCI loops&polls codebuild result, finishes

Yep, that sounds like a good solution to a common use case

@Erik Osterman (Cloud Posse) https://github.com/cloudposse/geodesic/pull/168

One question regarding terraform-aws-ecs-web-app Have you ever had issues with the ecs_service being created before the listener_rule was added to the target_group ? I don’t see this dependency being forced in terraform-aws-ecs-web-app and I had this quite a lot and caused me to trick around.

@sarkis

@maarten we do have that issue - since it’s a one time problem (cold boot) - we are just running terraform apply twice for the time being. I’d like to at some point dig into the provider and see if there is something to be done there, before trying to hack this with depends_on statements

Clarify what you mean by twice

I think what you mean is two phases

Not twice as retrying after failure :)

well it fails then

depends on definition of fail.

But the hack to mitigate it is .. kind of ugly..

How did you get it working?

https://github.com/blinkist/terraform-aws-airship-ecs-service/blob/master/modules/ecs_service/main.tf

so the listener rules of the alb handling are outputted

inputted after into the ecs service module

then a null resource .. doing nothing

and aws_ecs_service with a depends_on

clever

but maybe the null_resource could be removed, create a local, and add the local to the count = inside the ecs_service I think now

might be a computed count issue

well, we don’t have to count it, just evaluate it in someway

hm

so @maarten looking over your ecs_service/ in more depth - it looks like you need 2 tf runs as well right?

nope

where ?

i guess i’m not seeing where lb_attached gets changed

oh nvm i see how you make it wait with null_resource

lb_attached is just input for if it is a worker or a web service

The ugliest hack you will find here : https://github.com/blinkist/terraform-aws-airship-ecs-service/blob/master/modules/ecs_task_definition/main.tf search for my_random Module has as input key-value pairs which afterwards are turned into Name: Value: pairs for the environment variables of the task definition.. I found out that when “true” runs through a null_resource it will be casted to a 1 ..

I have no idea how I could fix this otherwise, input very much welcome

yea typecast hell is real in TF

we did something similar for converting TF -> json

i mean not similar but a similar typecast issue

let me find it - it might make you feel better about your hack

I think the ecs_service can be done without null_resource, add another empty item to the list, grab that item from the list, call it empty_string , and use that empty_string somewhere in ecs_service

https://github.com/cloudposse/terraform-aws-ecs-container-definition/blob/master/outputs.tf#L9

lol

i call that “cost of doing business with terraform”

price we have to pay

sadly - i was reading through HCL2 didn’t see anything specifically about the typecasting

there was a vague mention that types work better or something

Do you know if your modules will break yes/no ?

that’s also very vague rigth now - i think they are waiting for the community to do the dirty work

Wonder what is smart then, just create a new one calling it module-hc2 and go from there

i was thinking a new branch to start out

for example in the ecs_service part i now have 4 x ecs_service with conditionals.. with hcl2 this can be compacted to just one

but that might be too optimistic

ah yea i hear you though - it’s going to assume you can use the new tf version everywhere too

can prob fix this with tags though… go to a new major release for HCL2 (in your modules)

and then a note in the readme

sounds like they are going to support both languages initially, no?

so that provides an upgrade path

so modules can lay claim to the legacy provider until upgraded

hmm how would that work? oh just depends on what provider version you lock?

but i’m definitely nervously biting my nails right now hoping that it won’t be too painful

we have something like 70 modules

yea and i’m certain we do some interesting workarounds / hacks that are going to be fixed/deprecated in the future

Night ttyl

goodnight!

Good morning… new to the channel and not sure where I should / if I should be asking questions here… A while back I saw you guys do a presentation along side Codefresh…I remember being very impressed with the deployment pipeline you guys had set up and I am trying to get something of my own set up. I am curious how you guys connect git tags and pull them through to your dockerhub registry

I just found this page …. seems like I am on the right track https://docs.cloudposse.com/release-engineering/cicd-process/semantic-versioning/

@cbravo I can share more details a little later today

Currently afk

Yes that’s a good place to start

@Erik Osterman (Cloud Posse) Thank you very much…. I am also scoping out this repo (https://github.com/cloudposse-demo/demo-catalogue) and the build harness repo

we’ve also iterated a lot since that demo

would love to get you to try out the new stuff

all of our new stuff uses helmfile

are you familiar with that?

I am not

it simplifies a lot of stuff around working with helm

we are just getting our feet wet with helm but the particular project I am currently focused on is just a deployment image (it has aws cli in it and some other tools we use…) and I am trying to come up with a way to keep the git tag inline with the tags in docker hub without having to do a bunch of automated steps

codefresh gives you the git hub short revision but no access to the git tags

aha, ok - that is a simpler use-case

we are slowly ramping up our knowledge of help and kubernetes but we aren’t there yet

https://github.com/cloudposse/terraform-root-modules/blob/master/codefresh.yml

here’s a more simple codefresh pipeline

this builds a docker image and tags it with the semantic version generated in a previous step

it’s also an example of pushing the image to multiple registries

Any Datadog users here ?

i think we all have a bit of experience (cloudposse org)

I’ve not seen a good timeboard module yet, and with locals the population of multiple graphs can be fixed .. at least it looks like it

i personally have less than others here with datadog - my use case was limited to monitoring kafka

@Erik Osterman (Cloud Posse) so is the build harness something that the public should be using? like should I consider using your build harness? or is it something that could change without warning

or really something I should just consider taking pieces of and rolling my own solution based off it

we recommend to all of our customers to use the build-harness

it’s well maintained and we tag all releases. so if it suits your needs, by all means, leverage it.

especially with codefresh, the build-harness makes a lot of sense since every step runs in a container

@cbravo let me know if it would be helpful to take a look at what you have

and if we are not currently customers?

only share what’s not subject to an NDA

you can DM me

I am 200 messages behind this chat. Sorry I’ve been afk guys.

you snooze you lose

Totally!

welcome back

If anyone wants to take a look at the basic structure of a datadog timeboard module, feel free to comment.

https://github.com/maartenvanderhoef/terraform-datadog-timeboard/blob/master/examples/main.tf

https://www.terraform.io/docs/providers/datadog/r/timeboard.html

Problem of the datadog_timeboard was that it’s set-up like the cloudfront resource.. many blocks after another inside one resource.. but now with locals it can be modularized a bit. I wanted to be able to create graphs seperatedly from creating the actualy timeboard by creating 2 modules and this seems to be working.

@dave.yu @Daren maybe something interesting for you guys

cool will take a look

anyone know how to get the version of a terraform module programmatically? e.g. ${module.version}

use-case: I want to download artifacts from github release corresponding to the version of the terraform module

(e.g. a lambda zip file)

this appears to work:

outputs:

Hey @Erik Osterman (Cloud Posse) – been looking at aws-cloudfront-s3-cdn. Do you guys have a strategy for a javascript bundled webapp deployed to dev/test/prod via codepipeline?

I’m wondering: 1) Github -> hook -> CodePipeline (builds app via webpack) and pushes to dev 2) Q/A approves, time to move to test/uat 3) (?????) push dev artifacts to uat 4) Customer approves live 5) (?????) pushes uat artifacts to prod

Maybe I’m over-thinking it. Maybe each one is a codepipeline task off a branch

So what you describe is a pretty common pattern. A few of our customers do exactly that. We haven’t packaged that up as a terraform module. Most of them use Codefresh for CI/CD, and then use

https://github.com/cloudposse/terraform-aws-cloudfront-cdn

https://github.com/cloudposse/terraform-aws-s3-website

Here’s a reference implementation https://github.com/cloudposse/terraform-root-modules/tree/master/aws/docs

hey folks, quick question on the prometheus-cloudwatch exporter. When i try to follow the kubernetes instructions (https://github.com/cloudposse/prometheus-to-cloudwatch) the prometheus-to-cloudwatch pod fails to deploy, saying the image is unavailable. I’m running helm install . from within the charts subdirectory so I’m not sure what to do next.

@tom thanks for the report. I will check into this a bit later today.

Hello @tom - do you have some more info you can share, namely helm/tiller version while we look into reproducing the issue?

@Andriy Knysh (Cloud Posse) are you are around to take a look?

@sarkis probably just wrong image tag

It should correspond to the latest release in the repo

ah yea

i see an old version here: https://github.com/cloudposse/prometheus-to-cloudwatch/blob/master/chart/values.yaml#L21

i’ll check in after i push through the few tasks i have today - if you haven’t looked yet I can dig into it..

@tom

For our use case, it’s so we can download a zip artifact for a lambda function from the module GitHub release page.

https://github.com/cloudposse/terraform-aws-elastic-beanstalk-environment/pull/37

@jamie any idea how we can change this to only apply Name and Namespace on first apply?

apparently, all applys after the first one fail if these are set

Yo

just a moment ill look

thx

well, yeah

lifecycle { ignore_changes = [“tags”]}

might do it

ah, yea, I think that’s a better fix

I’m not set up to test that it will fix it though

are you able to?

i’ll ask that they try it out

if not… then its possible to do their fix… or strip “Name” and “Namespace” tags back out

but i think their solution is .. tidier then removing two map items from a map

in hcl 2, i expect you could workaround this

by having a data lookup the existing tags on the elb

and only apply the extra tags

ok, my response: https://github.com/cloudposse/terraform-aws-elastic-beanstalk-environment/pull/37#pullrequestreview-134777928

since you can reference resources directly

so you could do { if lookup(this.resource.tags, “Name”, “NAME_NOT_EXISTS”) != “NAME_NOT_EXISTS”} add full list of tags { else } add subset of tags {end}

^psudocode

Thanks for thinking of me

i’m gonna go back to my show now

yea, that will simplify so many things in complex terraform modules

thanks!

https://www.hashicorp.com/blog/terraform-0-12-preview-first-class-expressions

i should start constructing a sed expression to replace “${var…}” to var…. and post to the cloudposse blog

no need i think

they will have a conversion method

oh like a HCL 1->2?

so all the standard stuff will convert as easily as something like terraform fmt .

yea

nice

thats true this should be a fmt change

(also, I’m not sure if fmt is the action they will use, but I read that they will provide a means of automatically upgrading code)

https://github.com/kubernetes/kops/issues/2537 (via @Daren)

@Max Moon @dave.yu heads up

there’s an issue with kops 1.9 where it has trouble detecting failed pod evictions

Good looking out, thank you!

We hit it on every node with nginx-ingress-controller

Dang, I upgraded worker node size yesterday and fortunately it went smoothly

@dave.yu @Max Moon heads up: https://github.com/cloudposse/geodesic/pull/172/

we’re planning on merging this soon.

it uses inline values.yaml to make it easier to maintain

Thanks @Erik Osterman (Cloud Posse)

Hey i have a tip for you guys when dealing with ecs

in many cases the service, the containers, and the metrics all want the cluster name, not the arn

but the resource doesn’t provide a name, just the arn

but you can do this:locals { cluster_name = "${basename(aws_ecs_cluster.default.arn)}"}

to get the name

that’s a good one! like the use of basename(...) for this

I was doing

locals { cluster_name = "${element(split("/",(aws_ecs_cluster.default.arn), 1)}"}

Before.

really nice tip - thanks @jamie i was trying to figure out last week how to get the name with just the ARN (was looking at data source, but some sources don’t have name available :()

I thought it would be up your alley

I have grown to dislike “terraform_remote_state” data provider

hmmm - not good in practice?

how come?

It requires a lot of details to collect details from a remote state

It has meant that when writing code that depends on other modules

you have to pass in “workspace”, “s3 bucket name”, “state path”, and use shit like:

### For looking up info from the other Terraform States
variable "state_bucket" {
  description = "The bucket name where the chared Terraform state is kept"
}

variable "state_region" {
  description = "The region for the Terraform state bucket"
}

variable "env" {
  description = "The terraform workspace name."
}

locals {
  state_path = "${var.env == "default" ? "" : "env:/${var.env}/" }"
}

### Look up remote state info

data "terraform_remote_state" "vpc" {
  backend = "s3"

  config {
    bucket = "${var.state_bucket}"
    key    = "${local.state_path}vpc/state.tfstate"
    region = "${var.state_region}"
  }
}

That locals hack to get around the non-conformance that it has with workspaces

hrmm… yes… fwiw, here’s what we’ve recently done

we have a module that keep track of a lot of constants

that can be reused

i think just stay clear of workspaces

we don’t love it

Workspaces were so good when I discovered them

yea because you’d think it would help DRY

With one command I could switch from the ecs container task that has jenkins deploying, to the nginx container, to the node container

all the same code, but dif vars

Yeah, but when it comes to state management and workspaces

there is a lot to be desired

If only terraform_remote_state wasn’t the only data source with a default

I’d be using parameter_store, or an api call.

\/whinge

One last gripe….!

I would prefer that blocks that only have one entry in them were put all in one line when you formatted.

haha devops therapy sessions

how does that make you feel?

IMO, I don’t like that. I agree it’s concise, but in most language frameworks I’ve dealt with, they are strict about how braces are used. Typically, they enforce one of:

if (...) {
....
}

or

but never

if (...) { ... }

Its like some asshole just likes brackets and whitespace

Where’s the Terraform yaml option!

Okay… gripe done ;-D

heh, would be interesting to see what TF would look like in YAML

AKA

and

can’t help but feel like it’s “cloudformation” when seeing it in YAML format

it feels like they just “borrowed” the golang approach to { }

https://twitter.com/mitchellh/status/1016799930684354560

https://twitter.com/mitchellh/status/1016475256498761728

now thats a nice positive rant

i’ve noticed he does these multiple tweets a lot lol

that’s cool

I’m mid module writing for handling ecs instance draining for spot instances, but I want it to handle ASG standard istances as well…

so now it looks like im making another module for my module

how are you implementing the draining?

i assume https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_UpdateContainerInstancesState.html#API_UpdateContainerInstancesState_RequestSyntax

Good question

boto3

on asg instance termination sns alert

i get the instance id

And currently i grab the list of all ecs clusters, then in each cluster, i list all container_instances, and then i compare my instance id to the container instance id.

Once I have that i change state to draining

I don’t know how heavy those requests are when there are 100 clusters with 10000 instances on each

but it works for mine

I expect a faster lookup would be: tag each instance on creation with its cluster name

then, use the tags to get the cluster name from the id.

But then it involves an extra step on creation.

Whereas this way, i don’t need access to the ASG resource, just the asg name

I am still in POC stage with it really

I can see many ways to smooth it out, break it in to smaller reusable chunks and so on

but … maybe once I get the functions and permission sets all working

I can split the modulerisation of it with Sarkis or something

Well, it works

https://github.com/cloudposse/terraform-aws-ecs-launch-template/tree/init

It now also handles draining on scale in

Next it likely needs the lambda function rewritten to trim all of the junk off it, and set up a step function, so that a 5 or 10 second delay can be added between checking if the tasks have drained.

at the moment it just loops with no delay, between sns -> lambda -> sns ->lambda etc

until it finishes draining the tasks

@jamie the upstream tf PR was merged to support the launch templates?

Not for spot fleets.

So while waiting for that, I made the template module handle both spot fleets, asg on demand, and asg spot.

That’s just the launch template module

The other module which I’m making with you in mind is the auto scaling spot fleet one.

And that requires the merge first

https://techcrunch.com/2018/07/12/datadog-launches-watchdog-to-help-you-monitor-your-cloud-apps/

I like writing modules because it makes me do less work in the long term.

Question for the ones who worked with K8S before, I have not really. How does it compare to ECS for you. I see small startups offering jobs to devops for their to-be-configured k8s cluster ( on AWS) and I really think, why not just ecs ? It’s so much hype.

feature velocity of k8s is insane. huge ecosystem of tools, much lager than for ECS. IMO, it’s easier to deploy complex apps on kubernetes than on ECS. ECS needs something like Helm. Not sure if “terraform” is the answer for that. ECS has evolved leaps and bounds from when I first looked at it (e.g. DNS service discovery), but it still feels rather primitive coming from an kubernetes background. I think one big thing ECS has going for it is that it is simpler, but in being simpler lacks a lot of the cool features of kubernetes.

https://github.com/kubernetes/charts/tree/master/stable

There’s no signficant library of apps for ECS like there is for kubernetes. That to me is a little bit of a redflag.

(that I’m aware of)

True, but tbh, how many of the cool features do you really need. And will some cool features like persistent storage let people make decisions which are contrary to the reason for moving to AWS, like, having AWS taking care of your stateful services.

The idea is to represent as much of the infrastructure as possible so that i can be easily deployed without involving ops

with ECS, that’s not really possible.

with k8s, you can almost do everything but IAM roles/policies

volumes, load balancers, ingress, namespaces, secrets, configmaps, replicasets, deployments, pods, etc.

automatic TLS

automatic DNS

so basically a single helm chart is capable of provisioning a web app with a dynamic TLS certificate, automatic public DNS registration, pulling secrets and exposing them as envs, mounting configmaps as files on the filesystem, provisioning EBS volumes for scratch storage, and more…

we use these features all the time.

CI/CD of this is very easy with Kubernetes

few orgs actually do CI/CD of terraform

ok, but when connected to AWS in ways of networking, iam, .. then you do get terraform after all

persistent storage doesn’t have to be “persistent”

but you need it for big data applications like cassandara or HDFS

and for staging environments, we run apps like postgres

so having attached storage is necessary since the host machine is limited

so kubernetes cannot exist without terraform

that’s why it’s so central still to everything we do visible in our github

but kubernetes is headed in the right direction

I’d love to see a kind: Terraform in kubernetes

then it would truly allow everything

for what resources then ?

IAM roles, policies

RDS databases

elasicache instances

EFS filesystems

etc.

…so fully managed services

Here’s a POC presented at HashiConf: https://github.com/kris-nova/terraformctl

The other thing about k8s, is it’s more like like a framework (like “rails”) for how to do cloud automation. It provides the interfaces, scheduling, state, service discovery. Then makes it extensible to add anything else on top of it. So for more complex applications, e.g. “postgres”, in order to run it in a containerized environment, you need an “application aware” controller. Something that knows instinctively how to manage postgres. How to do updates, upgrades, rollbacks, etc.

So people are developing operators like this: https://github.com/CrunchyData/postgres-operator

https://github.com/coreos/etcd-operator

for running complex applications that aren’t just stateless webapps

Sure, but having seen a few meetups with pains of guys running stateful inside k8s..

Didn’t we move to AWS to not have state anymore

i agree that not dealing with state is the “ideal” situation

and we encourage our customers to push that off as long as possible

but I would rather have a platform capable of handling that in addition to all the stateless apps

at somepoint, someone needs to manage the state.

aws doesn’t provide state management for every application

thus some apps do need to handle that.

true, although, there are so many providers who offer services through vpc peering now

mongo atlas for example

yea, i would love to see more of that

i think that’s a cool direction

also, we’re still in the wee early days of k8s, but look how far & fast it’s come?

totally, in the end this is about not having 24/7 shifts, and not needing a 100% devops

With ECS I think this is possible, with self managed k8s, maybe less

yea, so back to perhaps your original question

but I don’t know k8s enough

to really judge there

companies who don’t have any dedicated devops, i would recommend considering ECS

smaller shops, with 1-4 people, ECS is probably better/simpler.

also with fargate then.. just great

i came from 80, but also only devops .. so ECS is perfect then

i think we’re coming at this from 2 differnet backgrounds. i spent the last 3 years working with k8s and 2 months with ECS.

so I don’t yet fully appreciate perhaps ECS.

you will, let’s just wait for that one k8s update hehe

haha

i really want to start working on our TF modules for EKS.

having deployed both - i do prefer k8s - something about it just appeals to me - i think it’s mostly the fact that i can do everything out of the box via command line - i.e. kubectl

def possible for ecs i bet but aws-cli is meh

for example - so easy to cat out the logs for ingress and describe a pod etc… for ECS i still find myself in the AWS web console - i know I am doing it wrong - but aws doesn’t make it a no brainer task like k8s for cli equivalents

so in a 20 developer situation how many people have kubectl and can do damage ?

yea good point - not ideal in production

and imo kubectl is the emergency hatch - so whoever is dealt the devops card in the 20 dev group

ok

i.e. knows what they are doing

hehe

https://www.hashicorp.com/blog/hashicorp-terraform-0-12-preview-for-and-for-each

@Erik Osterman (Cloud Posse) uploaded a file: Pasted image at 2018-07-12, 6:49 PM

This is awesome, but I find the formatting awkward without additional indention

https://developers.redhat.com/blog/2018/06/28/why-kubernetes-is-the-new-application-server/

@Erik Osterman (Cloud Posse) /@sarkis Within geodesic wrapper we are publishing geodesic port and binding kubernetes_api_port to it, can you tell why we are doing this

also how can I proxy something out of container to host

say for example kubernetes-dashboard

Yep! This is for kubectl proxy

So you can do exactly what you want to do

Also, for dashboard you can do something else

But I am on my way to bed. Can demo our portal for you tomorrow or next week

It uses bitly oauth2 proxy

kubectl proxy –port=0.0.0.0:8080

its seems not working as expected, so if I do kubectl proxy –port=$GEODESIC_PORT

From inside geodesic

No

yes thats what i am doing

okay

You are not binding to 0.0.0.0

By default it is 127.0.0.1

Docker port forwarding does not work to local host

Actually, arg is diff

kubectl proxy --port=0.0.0.0:8080 gives invalid port syntax exception

—addresss=0.0.0.0

okay

sorry, on phone so hard to type

thats working

thanks

https://stackoverflow.com/questions/42095142/kubectl-proxy-unauthorized-when-accessing-from-another-machine/42095931

Might need to add this too

I will document this too

It’s a good question

Anyways thanks a lot, it was an instant resolution, cheers

Haha welcome! :)

Hi, someone with spare time and wants to help me out with something. I’m passing a list with maps to a resource. This works as long as there is no interpolation happening with a variable from an external source. When I do it fails and the resource complains certain keys are missing from the map.

But when I output that structure, the structure is the same, just in a different order, I can’t figure out what is wrong with it.

https://gist.github.com/maartenvanderhoef/83047f578486dce8f5995d3c728b99d3

Can you share the precise error

Error: datadog_timeboard.this: “graph.0.request”: required field is not set

Error: datadog_timeboard.this: “graph.0.title”: required field is not set

Error: datadog_timeboard.this: “graph.0.viz”: required field is not set

This sounds familiar. @Andriy Knysh (Cloud Posse) I think ran into this in one of our other modules, but I don’t remember which one

Have you tried not using a local?

For the data structure

Have you tried removing the brackets here:

Then it does work, but that wouldn’t work for my module ..

graph = [”${local.not_working_graph}”]

let me try

the thing is, i’m passing a list of maps there normally, not just one, so it’s an actual list..

but let me try just a single one.

Actually, I misread your local

Thought it was already in a list

What if you put the local in a list

I don’t have any ideas other than to try all kinds of permutations of what you are attempting to do

(On my phone)

first attempt “datadog_timeboard.not_working: graph: should be a list”

the datadog_timeboard can have multiple graph { } blocks, so it must be a list.

haha, thanks , i’ll try the other option

2nd option, same problem as initial error. When outputted I have this:

not_working = {
  request = [map[style:map[type:solid width:normal palette:dog_classic] q:avg:aws.applicationelb.target_response_time.p95{targetgroup:targetgroup/qa-web-backend-web/123} aggregator:avg type:line]]
  title = not_working
  viz = timeseries
}
working = {
  request = [map[q:avg:aws.applicationelb.target_response_time.p95{targetgroup:targetgroup/qa-web-backend-web/123} aggregator:avg type:line style:map[palette:dog_classic type:solid width:normal]]]
  title = working
  viz = timeseries
}

I’ll wait for the new terraform I think.

@Erik Osterman (Cloud Posse) I think there is some issue with git::<https://github.com/cloudposse/terraform-aws-rds-cluster.git?ref=master>

What’s the problem?

I have create

module "rds_mysql" {
  source             = "git::<https://github.com/cloudposse/terraform-aws-rds-cluster.git?ref=master>"
  engine             = "aurora-mysql"
  cluster_size       = "${var.MYSQL_CLUSTER_SIZE}"
  cluster_family     = "aurora-mysql5.7"
  namespace          = "${var.namespace}"
  stage              = "${var.stage}"
  name               = "${var.MYSQL_DB_NAME}"
  admin_user         = "${var.MYSQL_ADMIN_NAME}"
  admin_password     = "${var.MYSQL_ADMIN_PASSWORD}"
  db_name            = "${var.MYSQL_DB_NAME}"
  instance_type      = "${var.MYSQL_INSTANCE_TYPE}"
  vpc_id             = "${module.vpc.vpc_id}"
  availability_zones = ["us-west-2b", "us-west-2c"]
  security_groups    = ["${aws_security_group.store_pv.id}"]
  subnets            = ["${module.subnets.private_subnet_ids}"]
  zone_id            = "${var.zone_id}"

  cluster_parameters = [
    {
      name  = "character_set_client"
      value = "utf8"
    },
    {
      name  = "character_set_connection"
      value = "utf8"
    },
    {
      name  = "character_set_database"
      value = "utf8"
    },
    {
      name  = "character_set_results"
      value = "utf8"
    },
    {
      name  = "character_set_server"
      value = "utf8"
    },
    {
      name         = "lower_case_table_names"
      value        = "1"
      apply_method = "pending-reboot"
    },
    {
      name         = "skip-character-set-client-handshake"
      value        = "1"
      apply_method = "pending-reboot"
    },
  ]
}

but if I run terraform apply , 2 nd time it recreates the instance

Probably something making it not idempotent

I cannot look at it now though - on my way out

The problem looks like your AZ map is not static

no problem, initially i though it can be due to either you are calculation azs or subnets counts somewhere

Consider hardcodifnit

Hard coding it

Or at the very least sorting it

Already did that ` availability_zones = [“us-west-2b”, “us-west-2c”]`

Hrmm I see

That is the line of investigation I would pursue

We used this module for multiple enagagemebrs

Probably a regression caused by a newer version of terraform

Show me your subnet invocation

hardcoded here as well

Hrmmm yea was going to be my other suggestion

You can try upgrading / downgrading the AWS provider

Hrmmm can you check the status of your 2a az?

AWS takes zones out of commission

Though unlikely in us-west

Also, not all services are available in all zones

Try a different az selection and see if it makes a difference

Also try reducing to just 2, for example

And don’t include the 2a

okay

But their weird thing is it’s saying you are going from 3 => 2

As an outsider, it looks like you previously provisioned the cluster in 3 az and now want to shrink it

That will destroy the cluster

Terraform is not a good tool for that kind of automation

it never actually happened and i verified that

Hrm odd indeed

i have actually destroy the module and recreated also

and the aws console shows 2 azs only

this looks more of an issue on terraform

Ya…

@mcrowe are you using the RDS cluster module?

For my info, what is the reason to both specify vpc subnets and ec2 availability zones ?

If you leave out availability zones it will work out most likely. The subnet group defines the azs.

availability_zones - (Optional) A list of EC2 Availability Zones that instances in the DB cluster can be created in

Yea that’s a good suggestion @maarten

@rohit.verma

Has there been a discussion about whether having security groups defined inside modules along the resource of purpose is OK or not, like with the rds module ? I personally dislike it a lot as I used ot create modules which did exactly that. It makes migrations extremely complex in some cases. Next to that the AWS/Terraform security group implementation is poor enough on itself, so it’s something I’m extremely careful with. Having vpc_security_group_ids as list variable for a module is a lot simpler and safer I think.

@rohit.verma Can you show us the output of terraform plan?

@maarten yes/no, but to your point, this module does not do it correctly

@Erik Osterman (Cloud Posse) uploaded a file: image.png

is bad practice. We should use security group rules for stability/interoperability with other modules

i think it’s ok, so long as the module returns the security group, so that other modules or consumers can add rules.

i am open to discussion around this. @Igor Rodionov @jamie @mcrowe and @Andriy Knysh (Cloud Posse) probably have more thoughts

let’s decide on something.

@Erik Osterman (Cloud Posse) what module is that?

https://github.com/cloudposse/terraform-aws-rds-cluster

@maarten @Erik Osterman (Cloud Posse) and @mcrowe, thanks for your suggestions. For the time being i have actually added all 3 availability zones, anyways as mentioned the instances are created in provided subnets only, so it don’t impact anything

hi team, did any one tried kube2iam with eks and got it working?

Use Kiam

kiam has master agent configuration

Crap

i don’t understood how to schedule master

eks have no master nodes

You might need to have a pseudo master tier

On phone

Kube2iam has a lot of serious issues

sorry didn’t got that, what is pseudo master tier on phone ?

Any interest to collaborate on our EKS modules?

sure i will

I am on my phone - hard to type :-)

okay, no problem

Bagel in other hand

ah ha

take your time

also to support eks, you have to make changes to your subnet module

i will send pr, small change

Ok many thanks

just when you got time, tell me about psuedo master,

if i promote any worker to act as master, i can’t run anything on it which need role assumption

Yes, want to review the EKS modules I am working on with you

@rohit.verma are you around?

i can share now

(or later today - ping me)

Just dropped a PR for elasticache-redis, need to be able to pass the encrypt at rest / enable TLS flags. Should be backwards compatible with previous releases (e.g. defaults to false on both). No idea if this is how you prefer I contribute, but let me know: https://github.com/cloudposse/terraform-aws-elasticache-redis/pull/15

thanks @jonathan.olson

@evan @Max Moon @Daren @chris might be interested in this enhancement for encryption at rest and TLS

@dave.yu also a heads up, we might need to do this: https://github.com/cloudposse/geodesic/issues/180

(reported by @Daren)

they are seeing memory leaks in kiam

(and possibly some excess network traffic)

We just released Cloud Posse reference architectures:

https://github.com/cloudposse/terraform-root-modules - Collection of Terraform root module invocations for provisioning reference architectures https://github.com/cloudposse/root.cloudposse.co - Terraform Reference Architecture of a Geodesic Module for a Parent (“Root”) Organization in AWS https://github.com/cloudposse/prod.cloudposse.co - Terraform Reference Architecture of a Geodesic Module for a Production Organization in AWS https://github.com/cloudposse/staging.cloudposse.co - Terraform Reference Architecture of a Geodesic Module for a Staging Organization in AWS https://github.com/cloudposse/dev.cloudposse.co - Terraform Reference Architecture of a Geodesic Module for a Development Sandbox Organization in AWS https://github.com/cloudposse/audit.cloudposse.co - Terraform Reference Architecture of a Geodesic Module for an Audit Logs Organization in AWS https://github.com/cloudposse/testing.cloudposse.co - Terraform Reference Architecture that implements a Geodesic Module for an Automated Testing Organization in AWS

They show how we provision AWS accounts and what Terraform modules we use. Complete description is here https://docs.cloudposse.com/reference-architectures

Thanks everybody for your contributions. We will be improving the repos and the docs. Your input and PRs are very welcome.

Awesome team, thanks for open sourcing

https://kubernetes.io/blog/2018/07/18/11-ways-not-to-get-hacked/ (via @Daren)

Hi Team, I’m writing a naming/tagging strategy document. I would love another set of eyes on it and suggestions. As far as I can I’m using the cloudposse terraform naming convention, but I needed to extend it to cover a larger number of scenarios.

https://docs.google.com/document/d/1tkzy-eMxaSnrse_uUQ5JqIILkamEe4Sy6gplp53VYpY/edit?usp=sharing

Sorry, just fixed the access

try and click again now

Permission to comment in the doc is enabled.

wow, great document

very detailed

I think you have everything covered there

Hah, well, thats kind of you. But critical feedback will help me improve it

nice doc @jamie

here is some feedback

1) we usually use namespace as a company name or abbreviation so we can distinguish the company’s resources from other companies resources (but it’s your choice to expand its usage as you described in the doc)

2) the third part in our naming (label) pattern is name, not role. The difference is what it does vs. what it is. Those could be just small differences, but we usually try not to use the resource types in resource names, e.g. we don’t use cp-prod-subnet-xxx-yyy-zzz but instead cp-prod-app1-xxx-yyy-zzz to name a subnet. But I think your doc has described both cases

I wanted to disambiguate the Name input from the Name output (id) that the label module does, but as a convention document.

Which is why i changed from name to role.

Well, I can tell you from years of field experience that most people don’t even have the document for the tags, but plenty of tags they want to use

So you end up with a giant cluster f*$! of mismatched tags with very little direction, and plenty of weird dependancies and gotchas baked into the system.

So, touch somethign and watch the dumpster start burning

Having a document like that to start with is a solid way of moving, good footing from the start is a powerful position for sure

what it is vs what it does is a very valid point.

Haha, well thats good news @krogebry - its for a transformation project one of my clients have. They have a monolith of aws EC2 instances, and are moving to serverless and docker microservices. And they have hired new teams and such, so before the big work starts on the transform i’m wanting to get a a few standards in place for consistancy

nice

yeah, that’s solid, I’ve done that work to transform, doing that work now

if you use role as a name, it should be role from the business point of view, not resource types point of view. Although in some cases it’s difficult to assign names (yes, naming is hard)

i'm wanting to get a a few standards in place for consistency - very valid point @jamie

Whats that saying you have about naming?

https://martinfowler.com/bliki/TwoHardThings.html

Love that

I may put that as a footnote.

Jamie this is awesome. I will take a closer look later today. At first glance love the selection of tags.

Access controls via tags is only sometimes supported

Ha, thanks Erik. Yeah, I have the list

Maybe add a note in regards to that. It should be used as a last resort IMO

For stage segregation, tags are not well suited since there are resources which do not even support tags

I’m gonna do a policy that means that anyone in the ‘developer’ group that wants to create new resources from that list, must apply tags at creation. As well as to ‘start’ a resource.

That way the greenfields stuff has to have tags, even if they are not awesome.

That is cool. Enforcing tags at some level would be a nice account level module!

Maybe i can see if I can just make a cloudtrail metric alert that looks for missing tags on creation, and notifys slack, or a dashboard via sns

Btw love the font in your doc

Oh thanks

More of a ‘soft’ way to enforce it

and implement the hard enforcement if required.

Can I offer an idea on that front?

I just did something with the tags on this client wrt missing tags

It’s an idea that Intuit implemented, but geared for tags.

Sure!

Alright, so Intuit started this with an “up and to the right” progress over perfection initiative with regards to the overall security of any given account that ran a service ( mobile stuff, mint, etc… ). I’ve implemented the same idea with various different things including pager duty noise levels.

Start with a simple code base ( lambda could work, or ruby+docker+jenkins, whatever ), analyze tags, then create a grading metric. So, A for >90% compliance, B for compliance of >80% >90%, C,D,F etc.

Or just go with the numeric, but i think there’s almost a cognitive hook on the grades.

I usually end up representing these as graphs in Jira

I was thinking I would use https://www.terraform.io/docs/providers/aws/r/config_config_rule.html

A config rule, to do the checks

@krogebry So this is in terms of bringing an account into compliance?

@krogebry uploaded a file: Screen Shot 2018-07-19 at 11.13.49 AM.png

Yeah, so forcing compliance from resource creation can work, but does get in the way.

However, a stance of “progress over perfection” usually works better in the transformation world of things.

Like, the reality is that you’re probably not going to get people to conform to tags right away, and in some cases you can’t enforce things because enforcing the standard might actually break things.

So the idea is the Riot sort of mindset where we’re just trying to move things up and to the right, or progress over time.

Usually more effictive when dealing with people who are maybe a little timid around big changes.

</rant cents=”2”>

@krogebry agree. But how do you do it from the automation point of view? E.g. we create and provision TF resources without tags (or with some tags), and then use the process to validate the tags and then update all the resources from TF?

@Andriy Knysh (Cloud Posse) with aws config rules you can have it provide a compliance report based on your terraform resources https://docs.aws.amazon.com/config/latest/developerguide/evaluate-config_manage-rules.html

nice

So instead of hard enforicing it, it can just be dashboard

Okay, I can tell you a tale of woe as to how not to do it

and its something that could be a TF module easily

Yeah, so you’re on the right track, basically you have two good options: either enforce it from the creation of things, or enforce it later with some kind of dashboard/report’y thingie

If you enforce at creation the only real risk is the timing with things, so in many cases the tag creation is a secondary action after the resource is created

So you just have to be aware of that, if you do something like “kill instance if tags are missing” with an ASG, you’re going to have a bad time because of the timing stuff.

yea, that’s why naming is hard

It would be neat to have some kind of way to have TF actually enforce the naming conventions with if conditionals

Might be an option in 0.12?

Its an option now

using my nifty hack

ohh, with config rules?

so I think it’s good to combine the two strategies: 1) enforce some rules at creation; 2) check it after creation and alert/dashboard