#announcements (2018-06)

@Erik Osterman (Cloud Posse) has joined the channel

@Bot has joined the channel

@chris has joined the channel

@Andriy Knysh (Cloud Posse) has joined the channel

@wookasz has joined the channel

set the channel topic: Cloud Posse Open Source Community

@tamsky has joined the channel

anyone else have trouble in their terminal with the selected glyphs in the prompt?

• https://github.com/cloudposse/geodesic/blob/master/rootfs/etc/profile.d/prompt.sh#L32-L35

yea, it’s unfortunately a bug we haven’t solved

it’s a combination of usign UTF8, prompts and terminals

I’ve tried quite a few fonts in xterm

i think most of us use iTerm2, but there are still some problems

(OSX thing)

what are the goals behind the glyphs ?

@Erik Osterman (Cloud Posse) uploaded a file: image.png

to visually communicate information like the powerline shell for zsh/bash

I suppose we could create a non-fancy mode

currently, the non-printing glyphs disturb readline, so history editing gets confused

ok, I’ll offer a non-unicode set of glyphs in a PR soon

that would be awesome!

I know a few who would appreciate it

wow, I think you might have helped me find the answer though

I think it’s related to this: https://superuser.com/questions/301353/escape-non-printing-characters-in-a-function-for-a-bash-prompt

we already use \[,\] in the PS1 , but maybe we need to throw in some \001, and \002 in there too

@mcrowe has joined the channel

welcome @mcrowe!

yeah, but for you, they’re printing, and for me they’re not, so use of \[ won’t help

hrm

can you send me a screenshot?

@tamsky uploaded a file: image.png

oh wow, yea, that’s bad

@Erik Osterman (Cloud Posse) uploaded a file: image.png

I understand, but my guess is, iTerm is actually choosing a different font for rendering unicode glyphs, and the font in use for your terminal doesn’t actually include those codepoints.

which isn’t something that most terminal programs can do (least among them: xterm)

I’ll admit it works in iterm, I tried it… I just can’t use it.

i agree we should offer a mode that does require it.

ok, great, I’ll do the work

my prompt, btw: https://github.com/tamsky/dotfiles/blob/master/.bash_prompt#L177-L188

you got inputrc, readline and bash questions, I got answers.

redirecting file descriptors in bash is still a head scratcher for me though

yea, i’ll never remember the syntax for that

  [ -n "${UNICODE_PROMPT_DISABLED}" ] && {
      WHITE_HEAVY_CHECK_MARK=$'\u2705 ' ;
      BLACK_RIGHTWARDS_ARROWHEAD=$'\u27A4 ' ;
      TWO_JOINED_SQUARES=$'\u29C9 ' ;
      CROSS_MARK=$'\u274C ' ;
  } || {
      # these use 8859-1 codepoints:                                                                      
      WHITE_HEAVY_CHECK_MARK=$'X ' ;
      BLACK_RIGHTWARDS_ARROWHEAD=$'-> ' ;
      TWO_JOINED_SQUARES=$'¤ ' ;
      CROSS_MARK=$'× ' ;
  }

?

Let’s use something more generic. For example, PROMPT_STYLE=plain

As for the bash logic, please use standard if/then rather than pipeline notation

Feels like we’re introducing technical debt if we repurpose the current variables.

I can take a look at it later tonight. Currently afk.

@tamsky
redirecting file descriptors in bash is still a head scratcher for me though This one still doesn’t click for me: exec 1> >(logger -t process-psv) 2>&1

I get it in principle, but I always have to lookup how to do it

@sarkis has joined the channel

@mike.ballou has joined the channel

@Robert Tisdale has joined the channel

@alex has joined the channel

@alebabai has joined the channel

@dave.yu has joined the channel

@yurchenko has joined the channel

@evan has joined the channel

@tamsky I had a long chat with @mcrowe today. lots of great feedback. one of the things that came up was using a dotfile manager. You recently shared your dotfiles. is this something you would use?

I’d need to understand what the “dotfile manager” would do?

w.r.t. geodesic and friends

e.g. https://github.com/TheLocehiliosan/yadm

so there’s a convention/interface for adding shell customizations (E.g. aliases, prompts, etc)

https://github.com/dotphiles/dotphiles

conventions would be awesome. the lack of .inputrc in the current geodesic has me adding ADD <https://raw.githubusercontent.com/play-with-docker/play-with-docker/master/dockerfiles/pwm/.inputrc> /conf/.inputrc to many sub-Dockerfiles

I think the interface is the important part

i’m actually not familiar with .intputrc, which is probably why it’s not there

I’m trying to figure out how I’d set some default for PROMPT_STYLE in my OSX shell that would get honored in geodesic shells

consider passing --env-file?

I know I heard once here in slack that someone wanted $PWD mapped into their geodesic shell.

https://github.com/cloudposse/geodesic/blob/master/rootfs/templates/wrapper#L73

could also introduce something like a ~/.geodesic/env file

sure, have the docker run wrapper test for the file ~/.geodesic-env and add the flag --env-file ~/.geodesic-env if it exists

@mcrowe added this to track it: https://github.com/cloudposse/geodesic/issues/148

also @mcrowe if you don’t mind making an issue around the region issue from yesterday… I do want to take a look but it’s looking more like this weekend the earliest

sorry for delay on that

@sarkis – actually, I needed to ping @Andriy Knysh (Cloud Posse) instead of you. It was his repo

ah ok - i still don’t mind looking into it… if he can’t get to it

Ah, @Andriy Knysh (Cloud Posse) replied in the other channel with some feedback. I’ll get it setup and submit a PR with his feedback

@mcrowe @sarkis you talking about this https://github.com/mike-zipit/terraform-aws-acm-request-certificate/blob/5ac54c38a5ca2cea715e5c2cee1218c508a0f888/main.tf or something else?

to be clear i was referring to: >@sarkis Ran into an interesting situation today. Using Cognito, it needs certificates in us-east-1 in order to be used for the auth domain. We’re in us-east-2, so I’m going to have to: …

right

ah, @Jamie replied to that

in the other channel

ah yea reading over that code - looks like thats it @Andriy Knysh (Cloud Posse) - yep @mcrowe just referenced that - going to get us a PR soon

yea it’s a nice way to do it, specify two providers with alias

provider "aws" {
  region = "us-east-2"
}
provider "aws" {
  alias  = "east"
  region = "us-east-1"
}

that’s how we do it https://github.com/cloudposse/terraform-root-modules/blob/master/aws/acm-cloudfront/main.tf

we use us-west-2 too for all other stuff

For @tamsky and @Erik Osterman (Cloud Posse) – .inputrc, prompt and all this fall under this rubric of “dotfile” management. IMO, the way to approach this is:

• Move any files that are “user breakable” from /etc/profile.d into a cloudeposse/geodesic-dotfiles (which has the default configuration for environment, prompt, anything shell related)
• Dockerfile for geodesic has something like:

  ENV USER_HOME=<https://github.com/cloudposse/geodesic-dotfiles>
  RUN curl -fLo /usr/local/bin/yadm <https://github.com/TheLocehiliosan/yadm/raw/master/yadm> && chmod a+x /usr/local/bin/yadm
  RUN cd /conf && yadm clone $USER_HOME
  

• This would put the user in control of their bash setup – they could clone the “breakable” dotfiles from cloudposse/geodesic-dotfiles and then add their custom settings

though I’m sure once I try to implement this I’ll find some holes in that logic

yea, something along those lines…

could you add that to the issue? @mcrowe

Yeah

personally I don’t think we need a dotfile manager

I put it under the rubric of “homedir” management

what I don’t like about homedir management is consistency

I see what you’re getting at

it’s like an escape hatch

if I don’t want to make my homedir git repo public, then here be demons

should only be used when knowing what you’re doing.

yea, true

cloning git private repos in docker no fun

especially using some 3rd party “manager”

I’m pretty certain alias commands will want to enter the mix as well at some point.

yadm = git (but the .git directory is offset from your home directory).
yadm clone [repo] yadm add .inputrc yadm commit yadm push etc

I already have a git repo

But, I’m not planning on putting my home directory in geodesic. But my aliases, maybe some other helpers, yea

i guess my Q would be - when would we want to do something more than just terraform plan/apply or like chamber cli commands in geodesic.. since it mounts your homedir to /localhost - i can do most things outside of the container

Here’s a flavor:

 $ yadm ls
Commit: 4d4ee021adb12c3c147f69dc0979f00094456c08 (HEAD -> master)
Author: Mike Crowe <[email protected]>
Date:   2018-04-06 22:20:14 -0400 (10 weeks ago)

Latest changes

 .aws/config                                  |  11 ++++++
 .bash-custom/bash-it.sublime-project         |  11 ++++++
 .bash-custom/bash-it.sublime-workspace       | 339 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 .bash-custom/custom.aliases.bash             |   2 +-
 .bash-custom/environ.bash                    |   4 +-
 .bash-custom/functions.bash                  |   7 ----
 .bash-custom/history.bash                    |   2 +-
 .bash-custom/nvm.bash                        |   2 +-
 .bash-custom/rerun.bash                      |   2 +-

what I don’t like about homedir management is consistency vs
it’s like an escape hatch
should only be used when knowing what you’re doing.

I’m pretty sure we can’t have both consistency and escape hatches

I’ll rejoin the conversation in a couple hours. Good luck to us

So, for my local development, I have a pretty extensive .bashrc that configures a bunch of stuff. However, there’s a case to be made for those guest systems (like geodesic, remote ssh logins, etc), maybe that needs to be a public gist or something different

ENV USER_HOME=<https://github.com/cloudposse/geodesic-dotfiles>
RUN curl -fLo /usr/local/bin/yadm <https://github.com/TheLocehiliosan/yadm/raw/master/yadm> && chmod a+x /usr/local/bin/yadm
RUN cd /conf && yadm clone $USER_HOME

@Erik Osterman (Cloud Posse) I guess this reminds me that I’ve been wondering how/what you expect geodesic users to handle geodesic – as a template that individuals and organizations modify and then use internally, a stepping off point, so to speak…. or more as a common central starting point for all things in the CP ecosystem?

I’ve seen both. I think inheriting from our “base image” is a great start, but anticipate that as a company’s usage increases, that they’ll eventually “hard fork” and have their own base.

organizations with a sizable ops team may prefer to manage their own.

@Jeremy G (Cloud Posse) has joined the channel

@jamie has joined the channel

@Erik Osterman (Cloud Posse) you May have noticed the two new alerts repos I put in.

They are 90% done codewise. Needs readme done.

I’m going to rename all the repos so that the alert subject is last. So that they sort nicely.

And in the latest repos I’ve added ‘additional endpoints’ variables

So instead of only having an sns topic, you can pass in autoscaling groups as endpoints, or lambda functions as endpoints.

So I’ll add that feature to the others too

thanks sweet!

@mcrowe was just telling me that tackling cloudwatch alerts of the type you’re creating are next up on his list of things todo

thanks @jamie! do you want us to review anything yet?

I’m doing a demo of them to my client next week, so I should have the code to 100 by Tuesday. I’ll do my pull requests then. :)

@mcrowe if you put yours off till next week I’ll have the template sorted. And we can both continue making alerts modules

The alerts modules are for: Awareness - such as slack. Automation- such as asg scaling Deployments- such as rolling back a codedeploy if the alert is triggered on a new version

I’ll do a push to my branches in a moment so you can see what’s been done

@LeoGmad has joined the channel

@Erik Osterman (Cloud Posse) and @mcrowe and the rest, I’m wanting to do a training day for my clients engineering team

https://github.com/firmstep-public/trainingdayone

I’ve just written this up today

@Purva has joined the channel

If anyone wants to use it you’re welcome, and if anyone has any good inputs for it, please let me know.

that’s cool

we’re starting to do more training. we’ll need to develop more materials like this.

@Purva, @chris have you guys tried/started doing any terraform training?

@jamie looks like a good start. Inevitably, a big project you’re taking on to teach this stuff.

Totally big job. I need to get their team of 14 introduced to what Terraform can do.

I don’t need to train to pro level. But I do need to get them thinking about what they can accomplish with Terraform

Day one is being filled out, and will likely need more READMEs

Day two is taking a more real world example, and working through that.

In this case it will be creating a CI/CD pipeline for Serverless deployments

With updating a lambda function, and API Gateway, and doing a rolling deploy with monitored events which would cause roll back.

Once training week is done, i’ll be back to the modules…

would love to incorporate something like this eventually into docs.cloudposse.com

@mcrowe https://github.com/cloudposse/terraform-aws-lb-s3-bucket/

One more fargate tip: no more than 20 tasks running per account per region

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service_limits.html

which can be changed

have you tried requesting a limit increase, or is that already on your path forward?

Hasn’t affected me yet. Eric and @sarkis have been working on an ecs fargate module for a client, so I was hoping to point it out!

@jamie thanks!

i bet they’d raise the limit like any other - just depends if you have biz level support on how quickly they get around to it

highly recommend this for clients that have business level+ support btw: https://aws.amazon.com/premiumsupport/trustedadvisor/ - one of the things it does is track if you get close to service limits

Other useful info includes reserved instance allocation and idle ebs volumes

Hey guys, I’m finding a need to document a geodesic module for internal use. For example, the inputs needed, but also I need to export the outputs from the modules included. For example:

module "ecs_alb_service_task" {
  source                    = "git::<https://github.com/cloudposse/terraform-aws-ecs-alb-service-task.git?ref=0.1.0>"
...

However, I typically want the outputs of this alb in my state, so I need to re-output the CP module’s output like this:

output "app_service_name" {
    value = "module.ecs_alb_service_task.service_name"
    description = "ECS Service name."
}

Check this out: https://gist.github.com/mike-zipit/fa936438c1edbb5f26ba68eec1cffaf8 This is WIP, but would like to validate if this is a need others see

Looks in line with some of things we are doing for readme generation

@sarkis do you have a minute? I’m trying to use terraform-aws-ecs-alb-service-task but having a weird problem

sure whats up

So, using terraform-aws-ecs-alb-service-task, the nginx container deploys and comes up (I see the default page). However, when my custom container deploys, it ends up dying

anything in the ECS service logs/container output?

what’s strange is I think it’s a health check

@mcrowe uploaded a file: image.png

I’ve ssh’d into the container (fixed various environment/permission issues I had), and I can run the container manually on the ECS host with docker run …..

does it go into a loop of starting/stopping the container?

yep

i think you are right in that it sounds like the scheduler is giving up after failed healthcheck threshold and just keeps trying to start a “healthy” container

I’ve bumped up the timeout on the alb:

  health_check_timeout  = 20
  health_check_interval = 30

Plus, my target groups look like this in the EC2 console:

@mcrowe uploaded a file: image.png

(the healthy one is the nginx controller – the others are failed deployments of my container)

do you have your target group tf resource handy?

i ran into something similar when i wasn’t pointing my target resources to the right healthcheck endpoint

endpoint/port

@mcrowe uploaded a file: image.png

and the above unhealthy status seems to reaffirm this is where the issue is…

oh, traffic port – is this supposed to be ip?

what is on that custom container?

i dont think there is any http server on your custom container*

a microservice. if you GET /, it returns a JSON document of the status

ah ok… i think i know something else to look into.. does this container take a bit to load?

yeah, it does

so there is a grace period

before ECS will start considering failed healthchecks as unhealthy

ah, good point.

let me find it

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html#container_definition_healthcheck

startPeriod The optional grace period within which to provide containers time to bootstrap before failed health checks count towards the maximum number of retries. You may specify between 0 and 300 seconds. The startPeriod is disabled by default.

that’s my guess right now - is that the container is taking a bit to start.. healthchecks failing and getting into a loop there

if you used our module - can modify that here in this map: https://github.com/cloudposse/terraform-aws-ecs-container-definition/blob/master/variables.tf#L34

which reminds me to fix the README on that ^

well, i don’t think that’s in the terraform-aws-ecs-alb-service-task module yet, is it?

this should be used on it’s own… so an example: https://github.com/cloudposse/terraform-aws-ecs-alb-service-task/blob/master/example/main.tf#L62-L89

i’m in the process of cleaning this all up and things may change :disappointed: - but all that module does is allow you to create a json doc out of terraform code - it’s not required to use it … you can just modify your .json for the container_definition_json

btw, did you see the issue I submitted about the environment variables? I had to just use pure .json files because I couldn’t pass environment variables in

did not, until now.. I just assigned to myself and will reproduce and see if we can workaround that

lmk if the startPeriod fixes your issue - now I’m about it

testing it now

BTW, another thin I ran into but didn’t report yet is this: we need the ability to customize the task security policy (we needed to grant permissions to SSM with a specific arn). I hacked it like this in the module (it’s not ideal, but I think you get the idea):

data "aws_iam_policy_document" "ecs_execution_role" {
  source_json = "${var.custom_policy_document}"

  statement {
    sid       = "vLMpjauEwiCGsAJ9tJKsbSgn"
    effect    = "Allow"
    resources = ["*"]

    actions = [
      "ecr:GetAuthorizationToken",
      "ecr:BatchCheckLayerAvailability",
      "ecr:GetDownloadUrlForLayer",
      "ecr:BatchGetImage",
      "logs:CreateLogStream",
      "logs:PutLogEvents",
    ]
  }
}

hmm yea that makes sense and should be an option to pass in our own existing or managed security policy (if not provided then generate)

would you mind opening an issue @mcrowe if not, I can

I’ll do it later today if that’s ok

that works - i don’t know if i can get to anything today - so would be tomorrow at the earliest after client work

that may have done it – not getting an OOM error (so further along in the chain)

btw, do you know how to kill an ecs codepipeline deploy when it fails?

@sarkis Adding:

health_check_grace_period_seconds  = 30

to my local copy of terraform-aws-ecs-alb-service-task fixed the issue

@antonbabenko has joined the channel

@mcrowe https://github.com/cloudposse/terraform-aws-ecs-web-app

this implements the common pattern of provisioning an ECR integrated with CI/CD to deploy containers to an ECS task and adding an ALB ingress

That’s pretty sweet – needed that 2w ago

but it’s also just a prime example of using all other ecs modules

https://github.com/cloudposse/terraform-aws-ecs-web-app/blob/master/main.tf

(realize it’s a bit late)

also, this uses our “readme generator” to convert a README.yaml (using gomplate) to a README.md

https://github.com/cloudposse/terraform-aws-ecs-web-app/blob/master/README.yaml

as part of this, it uses terraform-docs to generate the wellformed markdown

I scheduled a quick call with you tomorrow about this. I think what I did has a different application

ok, look forward to talking at 10:30

@Erik Osterman (Cloud Posse) uploaded a file: image.png

being able to easily identify out-of-date modules and automatically generate all the output definitions from nested modules is something I’ve wanted

Did any of the modules i made help with the ecs web app?

I think we used your ecs codepipline changes

@sarkis would know more

i can walk you through what we have this week sometime

we’re showing the client how to use it right now

and i think sarkis also used your ecs task module as an initial starting off point

i’d be really interested in your feedback on our ecs-web-app module <— from an interface perspective

does it make sense?

we’ll probably create an ecs-job-app soon <— which wouldn’t depend on an ALB

I read through it earlier very briefly and I thought it was well made from what I saw. I’ll give it a test later on - :)

yea the codepipeline and codebuild code you contributed - thanks a ton Jamie!

I’ll make sure you get credit in the READMEs for those projects

In my init-terraform scripts, I am seeing mount -a failing to mount s3 to local

I was looking at previous chats, seems like this is common

what version of build-harness?

and geodesic

i am using a fork of geodesic,

Also I am not sure where we are putting build-harness version, I think its always master

<https://raw.githubusercontent.com/cloudposse/build-harness/master/templates/Makefile.build-harness>

My geodesic is mostly in sync with master

@rohit.verma what does which s3fs return? (from inside geodesic container)

i am on goofys v0.19.0

looking at the s3fs on my geodesic container…

cat /usr/local/bin/s3fs
#!/bin/bash
# This script is used to mount filesystems from `/etc/fstab`
export DEVICE="$(eval echo ${1})"
export MOUNT_POINT="$(eval echo ${2})"
export BUCKET_REGION="${BUCKET_REGION:-${AWS_REGION}}"

# Evaluate the bash envs passed in via args and then execute goofys
exec goofys --file-mode=0600 --dir-mode=0700 --region=${BUCKET_REGION} --sse --acl=private ${DEVICE##s3://} ${MOUNT_POINT}

probably an issue with one of the last 2 args there ^ not being set

not sure yet why - tried to reproduce to no avail…

in my dockerfile I have added just RUN s3 fstab '${TF_BUCKET}' '/' '/secrets/tf'

@rohit.verma do you have ENV TF_BUCKET set before somewhere in the Dockerfile?

yes

running s3 fstab '${TF_BUCKET}' '/' '/secrets/tf' directly in container gives no error

I think ideally it should mount the terraform bucket to /secrets/tf

but that is still empty

@sarkis Next week I’ll have an ECS treat for you

https://github.com/cloudposse/terraform-aws-ecs-ecs-spot-fleet/tree/init

Its pending an update to the SpotFleet for allowing Launch Templates

A fraction of the cost of Fargate.

Looks really interesting Jamie - are you going to have a % of spot instances or I guess spot fleet handles that? I’m thinking what happens when prices spike…

Yep looks like it:
The Spot Fleet also attempts to maintain its target capacity fleet if your Spot Instances are interrupted due to a change in Spot prices or available capacity.

well, the fleet will be made of a mix of instance types

yea

so, if one instance type goes out of price, it boots another

last i tried anything with spot - there was no fleet

or it was super new i forget why we didn’t look into it

yeah, its new.

and they have released autoscaling for spot fleet recently too

it was pretty much unusable for anything mission critical for this reason

so, you can create a spot fleet, and scale it

really cool - looking forward to seeing the finished product!

when combined with a watcher script that runs on the instance (which you will see if you look in the user-data dir)

that starts ‘draining’ the instance as soon as it gets a spot instance termination notice

and setting the ‘draining lifetime’ of tasks to 1 min 45 seconds

it means that all of the containers running on the interupted instance get moved to another server before its removed

so you can keep your workload stable

its good for web sites, slightly less good for things that need to gaurentee connections for longer than 2 mins

i have been running spot instances for years, and they have only rotated out a few times

Very cool! I’d like to demo this for a company currently using spotinst.com

I think their spend is like 800K/mo on AWS :-)

Jamie, re: ssm param store policy docs PR, put in a comment in there - lmk if you have any Qs about the process for that stuff… I’m working on moving more of our repos to that README.yaml format

Thank you @sarkis

btw, added #releases where we’ll announce releases of our projects

@Erik Osterman (Cloud Posse) uploaded a file: image.png

Jamie - I know it’s getting late there so not doing the @ pings , do you have any prior art or tips for ECS Fargate CloudWatch monitoring - we obviously don’t have/care about the ECS cluster health, but starting to look into what we can monitor in the containers themselves - basic mem, cpu ,etc.

@sarkis Yo, keep the pings coming

I’ve done the README update that you suggested

ah great -

Okay so…

https://github.com/bitflight-public/terraform-aws-ecs-events

Thats a minor one…

but when do you need it by? I’ll make a fuller one

that follows the format of https://github.com/cloudposse/terraform-aws-cloudwatch-alarms-lambda/tree/init

@jamie I reviewed your terraform-aws-ssm-parameter-store-policy-documents - just a few nit picks

Also just curious Jamie - what’s your thoughts on the README generation process

It’s annoying. A form would be good ;)

what part of it could be improved?

A form would be good

in otherwords, an “init” process?

I’ll elaborate shortly. Just walking somewhere

Hi, I’m back.

So in regards to the README.yaml it isn’t very DRY

which parts?

hello all

Can’t remember if I said hello

Don’t want to derail

Holy paste batman

sorry Robert!

hey @Robert Tisdale!

bad syncronisity

LOL, no problem I’m just yanking your chain @jamie

Hey @Erik Osterman (Cloud Posse)

can you propose (in pseudo code) how the YAML should look instead?

@Erik Osterman (Cloud Posse) the terraform-aws-ssm-parameter-store-policy-documents (project name) gets used over and over

and in the intro areas especially, it would help if it was a variable

unfortunately, don’t see a way around that without recursion.

ah… it might turn into a case of mutliple levels of interpolation though since we already render this - thinking something like terraforms $$

-rw-r--r--    1 erik  staff  11291 Jun 20 12:50 README.md
-rw-r--r--    1 erik  staff   5429 Jun 20 12:50 README.yaml

but it’s at least 50% more DRY

yeah

whoa didn’t bother to look at that - nice

but there’s a stronger, more overwhelming reason we cannot do that

to me the biggest bonus is to only ever have to update the legal jargon in one spot

the purpose for the .yaml format is so we can include it in our docs portal (docs.cloudposse.com)

if the .yaml was itself templated, we would then need to create multiple parsers

one for readmes and one for hugo

Why use YAML for that instead of just markdown?

to enforce consistency

so basically, in hugo (our docs CMS) we can do this:

this allows us to pull documentation in from individual repos

the hardest part about writing documentation is keeping it up to date

by allowing individual repositories to manage the general usage/examples/inputs/outputs/etc, we can reference that in our main documentation portal via YAML and ensure it’s up to date in both places

“documentation as code”

I have started to look into doing TDD with terraform.

My current opinion is not to do it

but to do validations as part of a test process

I would like to do a compromise

i think writing full on tests for everything is overkill

but running plan/apply/destroy 80% of the battle

i want to use our [testing.cloudposse.co](http://testing.cloudposse.co) account for that

its also fucking horrid, terraform is not at the stage to validate that beyond plan

and use codebuild with codepipeline to automate that

@jamie something i’ve been pondering, more than TDD is RDD (Readme Driven Development)

yeah, well… Ive been reading on awspec, kitchen-terraform, goss, and inspec for the last 2 weks

weeks

@sarkis im in to that

@antonbabenko will probably have some more insights on it <– i know he uses kitchen-terraform

but @jamie what do you think about the plan/apply/destroy part?

I think that there is a portion that could be done, and i think there could be a ‘minimum’ that can be tested for at ‘unit test’ time

@Erik Osterman (Cloud Posse) i think the plan/apply/destroy part could take too long as a ‘unit-test’ but as a further along test it works

you think so? i really feel like integration tests are going to be the bulk of the tests if and when we get to it

i dont know how useful unit tests are is what im tying to say

I agree. Its what I was attempting to articulate too.

I am not from a recent dev background. The unit test part allows for testing as you create quickly.

But Integration tests can be done… quite well with terraform

our modules are breaking frequently due to terraform changes outside of our control

although they take longer to run, and can be a lot of work to write

anyone check out terratest yet?

I read about terratest yesterday… did you read about it on reddit?

they had a big announcement a couple of months ago

it’s by Grunt Works (terragrunt)

i had talked to Yevgeniy about it as I helped bug squash a bit on terragrunt… he was telling me they had it ready to release but hashicorp kept telling him to hold off because they have something they want to come out with (surprise, they are all over the place :P)

so yea officially publicly available a couple months ago as erik said ^

Like, Terraform’s ‘issues’ list is huge. But i don’t like them any less.

They are the best of the crowd

oh yea - i just mean they should just let the community run with some things - it might help them focus on the core

but i can see the other side of that coin too

nice to have control (enforce quality)

I normally don’t use kitchen-terraform and will not use terratest because it is very hard to make meaningful tests. hcl/tf files are not code, so I don’t think we need just-developer-friendly-tool. I think we need something what is more like DSL (in a form of BDD/Gherkin). Also, there will be a validation of values before apply, and a lot more…

Remember this - https://www.terraform.io/docs/providers/aws/events/gardening-summer-2018.html

Room in Gitter is - https://gitter.im/hashicorp-terraform/gardening

something like serverspec but for aws resources lol

awspec https://github.com/k1LoW/awspec

I wish something more abstract.

Well, if there was a terraform for testing that was multi provider enabled…

For example, when I was looking for something like inspec for google provider a year and more ago I could not, and I don’t know if there is awspec for google now.

true

From what I see, there is 70% of testing things for AWS, 30% for Azure, and 0% for Google.

Google have a test system… I found it when researching this week..

let me find it

Its not quite right but this goes towards it https://github.com/inspec/inspec-gcp

Are you guys using any pentest bots for testing code and infrastructure as part of the deploy pipeline?

https://github.com/guardicore/monkey <– I’ve been watching this repo

but haven’t used it yet

And it looks very thorough

I actually expect very few clients to pass muster

hmm that is nice - thanks for sharing it @jamie - starred to dig into on my free time

the only thing that worries me with this stuff is getting an aws account suspended - i’ve heard some horror stories

not specifically about this but just in general

Well, yeah… if you are doing anything that is DOS related

you have to inform AWS

right

its on their TOS

like i know security firms need to have a special deal with AWS and follow some standards, etc

So, if you are running this bot and doing any DDOS or DOS features you have to fill in the AWS Support request first

and say that is what you are doing

But this bot, doesn’t just brute force

So, its likely it will be outside of the AWS security remit

https://www.guardicore.com/infectionmonkey/

yea it looks more like discovery

https://aws.amazon.com/marketplace/pp/B07B3J7K6D

aws marketplace even allows it

lol

Thanks @jamie for sharing links. inspec-gcp is just 3 months old project.

Have any of you made a metadata index for s3 in terraform? like this? https://github.com/aws-samples/aws-big-data-blog/tree/master/aws-blog-s3-index-with-lambda-ddb

If no one has, @Erik Osterman (Cloud Posse) can you add it to me on your project board?

Also: An aws alarm terraform module for detecting root account logins.

And: An aws alarm terraform module for alerting on unauthorized aws api access.

These are part of the aws CIS best practices.

And should be small and easy to add

And a module to deploy this task to a cluster. https://github.com/ExpediaDotCom/c3vis/blob/master/README.md

I will track those in the project later today

https://github.com/orgs/cloudposse/projects/3

@Daren anything you can share for the root account logins?

Thanks!

No rush I just don’t want to forget them

@Erik Osterman (Cloud Posse) Sure. Id prefer to touch it up before open sourcing it, it was hacked together to meet the pci deadline

ok, understandable

It does the following:

• alert when a public is made public
• alert on root login
• alert on modification of cloudtrails
• auto re-enable disabled cloudtrails

a bucket is made public?

@Daren would really appreciate you publishing it, i’m working through this doc https://d0.awsstatic.com/whitepapers/compliance/AWS_CIS_Foundations_Benchmark.pdf

(we could also probably help with “clean up” if you want - PM)

@Daren I did most of what you have just mentioned. https://github.com/TeliaSoneraNorge/telia-terraform-modules/blob/master/cloudtrail-forwarder/cloudwatch-metric-filters.tf - maybe it can help. And this - https://github.com/TeliaSoneraNorge/telia-terraform-modules/blob/master/cloudtrail-forwarder/cloudwatch-metric-alarms.tf

those are great anton

may I use them?

@antonbabenko i just spotted something that may cause a headache for you…

At the bottom of https://github.com/TeliaSoneraNorge/telia-terraform-modules/blob/master/cloudtrail-forwarder/main.tf

You use aws_iam_policy_attachment

instead of aws_iam_role_policy_attachment

Incase it wasn’t intentional

Absolutely, use anything from there. I quit from that company today

no way!

what are you going to focus on now?

there are some modules which I want to move to open-source under terraform-aws-modules umbrella

long vacation first, and then find something related to Terraform & AWS. Have nothing settled yet

Thanks for pointing to aws_iam_policy_attachment - I didn’t know that

Guys who are from Russia or Europe, will you consider going to https://devoops.ru/en/ ? I am going there and will be talking about Terraform there.

man I wanna goto St Petersburg if it wasn’t a 20+ hr flight i’d be buying a ticket lol

Yeah, it is far from you. Will anyone go to HashiConf?

If I was looking for work I would

Alarm module made, with working example…. with some README

https://github.com/cloudposse/terraform-aws-cloudtrail-cloudwatch-alarms/tree/init

I added a thanks, a reference, and the MIT licence+copyright under the Apache2 license.

https://github.com/cloudposse/terraform-aws-cloudtrail-cloudwatch-alarms/tree/init

Is that correct?

Looks good, but I would not use locals (in alarms.tf), but rather create a bunch of resources aws_cloudwatch_metric_alarm manually. This way, you can easily delete something in the middle of the list without recreating the rest of the list. This will be fixed in HCL2 and Terraform 0.12, but for now it is still an issue with ordered lists.

@antonbabenko I’ve updated it to use the pretty readme, added dashboards, and added screenshots

Although I’m going to keep the lists in locals because I find the compact layout easier to read. And the lists are static.

I’m looking forward to hcl2 and getting ForExpressions https://github.com/hashicorp/hcl2/blob/master/hcl/hclsyntax/spec.md#for-expressions

Hey Joost!

Is there a way sending cloudtrail logs (from several customer aws accounts – all regions) to a specific cloudwatch log group at my security aws account?

I want to use the module terraform-aws-cloudtrail-cloudwatch-alarms at my security aws account

Hrmm.. we do something very similar, but ship them to an encrypted s3 bucket. What you propose sounds nice though.

I think @jamie might have recently done what you want

Cool. When you ship them to s3, what do you use for notifications? S3 events to lambda?

With the s3 bucket, we are not doing any no notifications. :( just storing logs.

This is something I’d need to defer to @jamie who has been working on all the alarm modules and probably has done what you are trying to achieve.

This a an area we just started investing in - so it will be a little bit before we have everything fully baked.

Ok, I understand. Thx!

https://sweetops.slack.com/archives/CB3579ZM3/p1529695427000484

Jamie is working on implementing modules to address as much of this as possible.

Aws KB has published the Iam policies, guidelines and such for centralizing the cloudwatch logs. I can pull the policies into the module as an option. And put the guide in the readme. So you can use the alerts on one account, but have it track and alert for all accounts.

That would be awesome

@joost another good item for you to read through this [WIP] [ETA: 30.6.2018] Terraform best practices https://github.com/antonbabenko/terraform-best-practices by @antonbabenko it links to examples of organizing code for multiple aws accounts such as you may need

“WIP” will have to be remaining for that repo until I am back from my vacation. I want to make it complete in September before I speak about it at conferences.

Thx @antonbabenko for the great link

@Daren did you guys get kiam working? We started on this: https://github.com/cloudposse/geodesic/pull/163

ran into some issues

I wish i liked writing READMEs

New module WIP https://github.com/cloudposse/terraform-aws-s3-logs-athena-query/tree/init

It works. But is pending readme creation

Ok - we can help with that

Really not fond of heredocs. Can we move that to template files?

Perhaps named query should be a separate module that reads queries from a template file

Whats wrong with heredocs?

Legibility of code

Can not properly indent

okay

I can split it into its own sql folder

That would be perfect

@jamie @Andriy Knysh (Cloud Posse) @Igor Rodionov - can we schedule some time to talk about standardizing how we use IAM permissions?

@Igor Rodionov has some problems with one of our older modules on ecr and before he makes any changes, we thought it a good idea to come up with a standard

@jamie - I know you’re working a lot with locking down IAM permissions these days

I can provide an example of my vision on ecr module

yea, that would be a good start

we can CR that <— as a proposal

and bring it to the table for meeting

Sure. In brief: the iam_policy_documents can be overwritten, or appended to depending on if the statement has a SID. Combining that, with the concept of a per project ‘policy module’ such as https://github.com/cloudposse/terraform-aws-ssm-parameter-store-policy-documents Or an integrated policy module, so that as part of the current module, it outputs the json policy documents json, and sids, so they can be seen. And includes an option for including json documents that can be passed through for appending.

It will be more work, but allows the lowest possible permissions to be added by the developer. But still allows additional permissions to be included in the module for custom needs.

https://www.terraform.io/docs/providers/aws/d/iam_policy_document.html

@jamie @mcrowe https://github.com/cloudposse/terraform-null-label/pull/25

I added an example and such to the terraform-aws-ecs-service-cloudwatch-sns-alarms init

@jamie you are on a roll today - i need to know what brand/bean of coffee you having

thank you

Did you see the null label one?

yes

hehe

It was very heavily refactored

just to get consistency on the data types, and then outputs them in the old output style

yea, that was a clever trick. hadn’t considered it, but i think it’s a necessary evil.

https://www.hashicorp.com/blog/terraform-0-1-2-preview

https://www.terraform.io/upgrade-guides/0-12.html

Interesting

so i was doing the whole compact concat dance before.. but i think this works in tf now…

alarm_actions       = ["${local.sns_topic_arn}", "${var.additional_notify_arns}"]
ok_actions          = ["${local.sns_topic_arn}", "${var.additional_notify_arns}"]

where local.sns_topic_arn is a string and var.additional_notify_arns is a list

hmm yea just tested it and it does work as expected… reference: https://github.com/hashicorp/terraform/issues/5682#issuecomment-278483475

I saw that it was working

like magic

so much nicer than the old way

@sarkis https://ember.com/products/ceramic-mug

is that for you?

(via: @Max Moon)

@Erik Osterman (Cloud Posse) uploaded a file: image.png

have you guys seen or worked with https://github.com/blinkist

some very thorough modules built by him (almost to @sarkis completeness grade)

https://github.com/blinkist/terraform-aws-airship-ecs-service

hmmm looks interesting - thanks for sharing @jamie

at the very least - I need some diagrams for the ecs-web-app module … it’s getting huge

Diagrams are annoying and always appreciated

Hi fellas, I’m the Blinkist guy

Welcome :)

Few small bugs still here and there, but slowly getting there. I was discussing earlier with Jamie I think if it’s possible to have Terraform manage the ECS container definitions except for the image parameter.. Where Terraform runs would extract the image:tag from aws_ecs_container_definition. Only solution I can think of is by creating a bootstrap=true for the module which would turn on/off a datasource for the first run, and other logic on the second run.. Would need to play around with that..

Guys, I invited that guy from the links I shared

Thanks for the invite

Welcome to the channel!

Nice work on those modules

I am afk, but will share how we are doing it

@maarten it’s Friday night for like most of the crew, and it’s quieter here over the weekend. But core group of @sarkis @Erik Osterman (Cloud Posse) sand @Andriy Knysh (Cloud Posse) are keeping an eye on it

As seen

Yeah sure.. Normally I’m not around on a Friday either … I’ll introduce myself quickly.

Yes, things are pretty quiet now as it’s the end of the week

I’m the founder of the Terraform User Group in Berlin, started it when I didn’t know where to start with Terraform roughly 2 years ago. Worked for 2.5 years at Blinkist where we went for AWS with ECS, coming from Heroku. Was always a dream to be self-employed, Blinkist fully supported that and made me a contractor with as first assignment open sourcing the ECS stuff I did earlier. Anothing thing I made is a golang tool called skipper which does some magic regarding task execution through a docker tunnel etc.etc. but that project is on hold until I get a bit of golang help.

@Erik Osterman (Cloud Posse) Yeah, happy to see some of that Have a good weekend.

Welcome! Its 11:30pm for me too so I’m going offline as well. But we will chat soon.

Sure, again, thanks for the invite.

Yea let’s sync up on Monday !

Also, maybe it would be interesting to have a weekly standing “round table” where we get together in a zoom room

Welcome Maarten

https://github.com/cloudposse/terraform-aws-elasticache-redis/pull/12 if anyone has time for a one line code review

Thanks in advance!

@Max Moon thanks for the PR. Looks like the variables.tf entry was there but just wasn’t being passed?

aha

I’ll wait for 1 more approval and can merge. Is this blocking you right now?

if i missed that then no big deal on this PR. No it’s not blocking me, i just referenced my branch.

After reviewing variables.tf I think that despite being referenced in [variables.tf](http://variables.tf) it wasn’t being picked up because the reference wasn’t made in the module, but i could be wrong

Yea - I think CI would catch this (maybe) either way put a to-do for myself to get that PRed up

it certainly was not working this morning, at least. So I think the PR for this is still relevant, pending review from others

Oh absolutely your PR is necessary. Sorry should have gave context. I was confused why you didn’t have a variables.tf change - made sense when I looked at master. It has the unref variable on there.

ah yes okay, i gotchya now

So one last thing. Now I’m thinking if we default to 4.0

i was thinking the same.

Since that was the previous behavior it seems from your testing

Let’s do that so it’s less of a backwards incompatible risk

If you don’t mind pushing that to your PR I’ll grab my laptop so I can merge and release

yeah sure one sec

already done and done

im headed out for the afternoon but updated default family and engine version to latest available via AWS documentation (family: redis4.0 and engine_version: 4.0.10) in [variables.tf](http://variables.tf) on the PR. Thanks again @sarkis

thanks @Max Moon! pushing out 0.5.0 now

thanks!!!

https://github.com/cloudposse/terraform-aws-elasticache-redis/releases/tag/0.5.0

i thought it was a change that warranted a minor version++

agreed, thanks again!