#announcements (2018-06)

cloudposse Cloud Posse Open Source Community #geodesic #terraform #release-engineering #random #releases #docs

This channel is for workspace-wide communication and announcements. All members are in this channel.

Archive: https://archive.sweetops.com

2018-06-06

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
08:51:02 PM

@Erik Osterman (Cloud Posse) has joined the channel

2018-06-13

Bot avatar
Bot
11:56:50 PM

@Bot has joined the channel

chris avatar
chris
12:29:27 AM

@chris has joined the channel

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
12:30:26 AM

@Andriy Knysh (Cloud Posse) has joined the channel

wookasz avatar
wookasz
12:41:45 AM

@wookasz has joined the channel

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
12:44:55 AM

set the channel topic: Cloud Posse Open Source Community

tamsky avatar
tamsky
12:52:49 AM

@tamsky has joined the channel

tamsky avatar

anyone else have trouble in their terminal with the selected glyphs in the prompt?

tamsky avatar
  WHITE_HEAVY_CHECK_MARK=$'\u2705 '
  BLACK_RIGHTWARDS_ARROWHEAD=$'\u27A4 '
  TWO_JOINED_SQUARES=$'\u29C9 '
  CROSS_MARK=$'\u274C '
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, it’s unfortunately a bug we haven’t solved

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

it’s a combination of usign UTF8, prompts and terminals

tamsky avatar

I’ve tried quite a few fonts in xterm

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i think most of us use iTerm2, but there are still some problems

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(OSX thing)

tamsky avatar

what are the goals behind the glyphs ?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
01:02:30 AM

@Erik Osterman (Cloud Posse) uploaded a file: image.png

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

to visually communicate information like the powerline shell for zsh/bash

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I suppose we could create a non-fancy mode

tamsky avatar

currently, the non-printing glyphs disturb readline, so history editing gets confused

tamsky avatar

ok, I’ll offer a non-unicode set of glyphs in a PR soon

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

that would be awesome!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I know a few who would appreciate it

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

wow, I think you might have helped me find the answer though

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Escape non-printing characters in a function for a Bash prompt

In a Bash Prompt (PS1 variable), I’m calling a function to potentially add text to the prompt: export PS1=”\u@\h $(my_function) $ “ However, the function in the prompt contains ANSI color codes …

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we already use \[,\] in the PS1 , but maybe we need to throw in some \001, and \002 in there too

mcrowe avatar
mcrowe
01:14:41 AM

@mcrowe has joined the channel

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

welcome @mcrowe!

tamsky avatar

yeah, but for you, they’re printing, and for me they’re not, so use of \[ won’t help

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

hrm

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

can you send me a screenshot?

tamsky avatar
tamsky
01:18:02 AM

@tamsky uploaded a file: image.png

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

oh wow, yea, that’s bad

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
01:18:52 AM

@Erik Osterman (Cloud Posse) uploaded a file: image.png

tamsky avatar

I understand, but my guess is, iTerm is actually choosing a different font for rendering unicode glyphs, and the font in use for your terminal doesn’t actually include those codepoints.

tamsky avatar

which isn’t something that most terminal programs can do (least among them: xterm)

tamsky avatar

I’ll admit it works in iterm, I tried it… I just can’t use it.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i agree we should offer a mode that does require it.

tamsky avatar

ok, great, I’ll do the work

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

tamsky avatar
tamsky/dotfiles

dotfiles - .files, including ~/.osx — sensible hacker defaults for OS X

1
tamsky avatar

you got inputrc, readline and bash questions, I got answers.

tamsky avatar

redirecting file descriptors in bash is still a head scratcher for me though

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, i’ll never remember the syntax for that

tamsky avatar
  [ -n "${UNICODE_PROMPT_DISABLED}" ] && {
      WHITE_HEAVY_CHECK_MARK=$'\u2705 ' ;
      BLACK_RIGHTWARDS_ARROWHEAD=$'\u27A4 ' ;
      TWO_JOINED_SQUARES=$'\u29C9 ' ;
      CROSS_MARK=$'\u274C ' ;
  } || {
      # these use 8859-1 codepoints:                                                                      
      WHITE_HEAVY_CHECK_MARK=$'X ' ;
      BLACK_RIGHTWARDS_ARROWHEAD=$'-> ' ;
      TWO_JOINED_SQUARES=$'¤ ' ;
      CROSS_MARK=$'× ' ;
  }

?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Let’s use something more generic. For example, PROMPT_STYLE=plain

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

As for the bash logic, please use standard if/then rather than pipeline notation

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Feels like we’re introducing technical debt if we repurpose the current variables.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I can take a look at it later tonight. Currently afk.

mcrowe avatar

@tamsky
redirecting file descriptors in bash is still a head scratcher for me though
This one still doesn’t click for me: exec 1> >(logger -t process-psv) 2>&1

1
mcrowe avatar

I get it in principle, but I always have to lookup how to do it

sarkis avatar
sarkis
02:54:46 AM

@sarkis has joined the channel

mike.ballou avatar
mike.ballou
02:58:11 AM

@mike.ballou has joined the channel

2018-06-14

Robert Tisdale avatar
Robert Tisdale
10:06:59 AM

@Robert Tisdale has joined the channel

alex avatar
alex
02:39:54 PM

@alex has joined the channel

alebabai avatar
alebabai
03:12:31 PM

@alebabai has joined the channel

dave.yu avatar
dave.yu
05:19:40 PM

@dave.yu has joined the channel

yurchenko avatar
yurchenko
05:31:46 PM

@yurchenko has joined the channel

evan avatar
evan
06:24:17 PM

@evan has joined the channel

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@tamsky I had a long chat with @mcrowe today. lots of great feedback. one of the things that came up was using a dotfile manager. You recently shared your dotfiles. is this something you would use?

tamsky avatar

I’d need to understand what the “dotfile manager” would do?

tamsky avatar

w.r.t. geodesic and friends

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
TheLocehiliosan/yadm

yadm - Yet Another Dotfiles Manager

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so there’s a convention/interface for adding shell customizations (E.g. aliases, prompts, etc)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
dotphiles/dotphiles

dotphiles - A community driven framework of dotfiles.

tamsky avatar

conventions would be awesome. the lack of .inputrc in the current geodesic has me adding ADD <https://raw.githubusercontent.com/play-with-docker/play-with-docker/master/dockerfiles/pwm/.inputrc> /conf/.inputrc to many sub-Dockerfiles

tamsky avatar

I think the interface is the important part

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i’m actually not familiar with .intputrc, which is probably why it’s not there

tamsky avatar

I’m trying to figure out how I’d set some default for PROMPT_STYLE in my OSX shell that would get honored in geodesic shells

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

consider passing --env-file?

tamsky avatar

I know I heard once here in slack that someone wanted $PWD mapped into their geodesic shell.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/geodesic

Geodesic is the fastest way to get up and running with a rock solid, production grade cloud platform https://docs.cloudposse.com/geodesic/

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

could also introduce something like a ~/.geodesic/env file

tamsky avatar

sure, have the docker run wrapper test for the file ~/.geodesic-env and add the flag --env-file ~/.geodesic-env if it exists

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Add Support for Custom Prompts · Issue #150 · cloudposse/geodesic

what Add support for PROMPT_STYLE environment variable that alters the behavior of the shell prompt why The current utf8 heavy prompt breaks some terminals Users have mentioned they&#39;d prefer a …

Add Support for Automatic Env File · Issue #149 · cloudposse/geodesic

what Look for a ~/.geodesic/env file and pass it as –env-file, if found why Provide a way to set some defaults for all geodesic shells.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Add Support for a Dotfile Manager · Issue #148 · cloudposse/geodesic

what Provide a standard interface for shell extensions Re-use some existing standard why Allow users to define their own aliases, prompts, inputrcs, etc references https://github.com/dotphiles/dotp

tamsky avatar

Erik, does this flag and logic that we just imagined we need, already exist:: https://github.com/cloudposse/geodesic/blob/master/rootfs/templates/wrapper#L72-L74

  if [ -n "${ENV_FILE}" ]; then
    DOCKER_ARGS+=(--env-file ${ENV_FILE})
  fi
cloudposse/geodesic

Geodesic is the fastest way to get up and running with a rock solid, production grade cloud platform https://docs.cloudposse.com/geodesic/

tamsky avatar

# ENV_FILE=~/.geodesic/env geodesic looks like it works

tamsky avatar

do we want to also test for a default file and add it to DOCKER_ARGS[@]?

tamsky avatar
  local gedesic_env_file="~/.geodesic/env"
  # use our default env file if it exists                                                                 
  if [ -f ${geodesic_env_file} ]; then
      DOCKER_ARGS+=(--env-file=${geodesic_env_file})
  fi
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yea, so I think checking for a default is what we could add

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I’d make this: GEODESIC_DEFAULT_ENV_FILE or something. so even that can be overridden

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

e..g for different projects

sarkis avatar

also @mcrowe if you don’t mind making an issue around the region issue from yesterday… I do want to take a look but it’s looking more like this weekend the earliest

sarkis avatar

sorry for delay on that

mcrowe avatar

@sarkis – actually, I needed to ping @Andriy Knysh (Cloud Posse) instead of you. It was his repo

sarkis avatar

ah ok - i still don’t mind looking into it… if he can’t get to it

mcrowe avatar

Ah, @Andriy Knysh (Cloud Posse) replied in the other channel with some feedback. I’ll get it setup and submit a PR with his feedback

1
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
mike-zipit/terraform-aws-acm-request-certificate

terraform-aws-acm-request-certificate - Terraform module to request an ACM certificate for a domain name and create a CNAME record in the DNS zone to complete certificate validation

sarkis avatar

to be clear i was referring to: >@sarkis Ran into an interesting situation today. Using Cognito, it needs certificates in us-east-1 in order to be used for the auth domain. We’re in us-east-2, so I’m going to have to: …

mcrowe avatar

right

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

ah, @Jamie replied to that

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

in the other channel

sarkis avatar

ah yea reading over that code - looks like thats it @Andriy Knysh (Cloud Posse) - yep @mcrowe just referenced that - going to get us a PR soon

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

yea it’s a nice way to do it, specify two providers with alias

provider "aws" {
  region = "us-east-2"
}
provider "aws" {
  alias  = "east"
  region = "us-east-1"
}
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
cloudposse/terraform-root-modules

terraform-root-modules - Collection of Terraform root module invocations for provisioning reference architectures

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

we use us-west-2 too for all other stuff

mcrowe avatar

For @tamsky and @Erik Osterman (Cloud Posse) – .inputrc, prompt and all this fall under this rubric of “dotfile” management. IMO, the way to approach this is:

  • Move any files that are “user breakable” from /etc/profile.d into a cloudeposse/geodesic-dotfiles (which has the default configuration for environment, prompt, anything shell related)
  • Dockerfile for geodesic has something like:
    ENV USER_HOME=<https://github.com/cloudposse/geodesic-dotfiles>
    RUN curl -fLo /usr/local/bin/yadm <https://github.com/TheLocehiliosan/yadm/raw/master/yadm> && chmod a+x /usr/local/bin/yadm
    RUN cd /conf && yadm clone $USER_HOME
    
  • This would put the user in control of their bash setup – they could clone the “breakable” dotfiles from cloudposse/geodesic-dotfiles and then add their custom settings
mcrowe avatar

though I’m sure once I try to implement this I’ll find some holes in that logic

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, something along those lines…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

could you add that to the issue? @mcrowe

mcrowe avatar

Yeah

tamsky avatar

personally I don’t think we need a dotfile manager

tamsky avatar

I put it under the rubric of “homedir” management

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

what I don’t like about homedir management is consistency

tamsky avatar

I see what you’re getting at

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

it’s like an escape hatch

tamsky avatar

if I don’t want to make my homedir git repo public, then here be demons

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

should only be used when knowing what you’re doing.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, true

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

cloning git private repos in docker no fun

tamsky avatar

especially using some 3rd party “manager”

tamsky avatar

I’m pretty certain alias commands will want to enter the mix as well at some point.

mcrowe avatar

yadm = git (but the .git directory is offset from your home directory).
yadm clone [repo] yadm add .inputrc yadm commit yadm push etc

tamsky avatar

I already have a git repo

mcrowe avatar

But, I’m not planning on putting my home directory in geodesic. But my aliases, maybe some other helpers, yea

sarkis avatar

i guess my Q would be - when would we want to do something more than just terraform plan/apply or like chamber cli commands in geodesic.. since it mounts your homedir to /localhost - i can do most things outside of the container

mcrowe avatar

Here’s a flavor:

 $ yadm ls
Commit: 4d4ee021adb12c3c147f69dc0979f00094456c08 (HEAD -> master)
Author: Mike Crowe <[email protected]>
Date:   2018-04-06 22:20:14 -0400 (10 weeks ago)

Latest changes

 .aws/config                                  |  11 ++++++
 .bash-custom/bash-it.sublime-project         |  11 ++++++
 .bash-custom/bash-it.sublime-workspace       | 339 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 .bash-custom/custom.aliases.bash             |   2 +-
 .bash-custom/environ.bash                    |   4 +-
 .bash-custom/functions.bash                  |   7 ----
 .bash-custom/history.bash                    |   2 +-
 .bash-custom/nvm.bash                        |   2 +-
 .bash-custom/rerun.bash                      |   2 +-
tamsky avatar


what I don’t like about homedir management is consistency
vs
it’s like an escape hatch
should only be used when knowing what you’re doing.

I’m pretty sure we can’t have both consistency and escape hatches

1
tamsky avatar

I’ll rejoin the conversation in a couple hours. Good luck to us

mcrowe avatar

So, for my local development, I have a pretty extensive .bashrc that configures a bunch of stuff. However, there’s a case to be made for those guest systems (like geodesic, remote ssh logins, etc), maybe that needs to be a public gist or something different

tamsky avatar
ENV USER_HOME=<https://github.com/cloudposse/geodesic-dotfiles>
RUN curl -fLo /usr/local/bin/yadm <https://github.com/TheLocehiliosan/yadm/raw/master/yadm> && chmod a+x /usr/local/bin/yadm
RUN cd /conf && yadm clone $USER_HOME

@Erik Osterman (Cloud Posse) I guess this reminds me that I’ve been wondering how/what you expect geodesic users to handle geodesic – as a template that individuals and organizations modify and then use internally, a stepping off point, so to speak…. or more as a common central starting point for all things in the CP ecosystem?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I’ve seen both. I think inheriting from our “base image” is a great start, but anticipate that as a company’s usage increases, that they’ll eventually “hard fork” and have their own base.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

organizations with a sizable ops team may prefer to manage their own.

Jeremy G (Cloud Posse) avatar
Jeremy G (Cloud Posse)
09:29:24 PM

@Jeremy G (Cloud Posse) has joined the channel

2
jamie avatar
jamie
10:01:36 PM

@jamie has joined the channel

2018-06-15

jamie avatar

@Erik Osterman (Cloud Posse) you May have noticed the two new alerts repos I put in.

jamie avatar

They are 90% done codewise. Needs readme done.

jamie avatar

I’m going to rename all the repos so that the alert subject is last. So that they sort nicely.

1
jamie avatar

And in the latest repos I’ve added ‘additional endpoints’ variables

jamie avatar

So instead of only having an sns topic, you can pass in autoscaling groups as endpoints, or lambda functions as endpoints.

1
jamie avatar

So I’ll add that feature to the others too

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

thanks sweet!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@mcrowe was just telling me that tackling cloudwatch alerts of the type you’re creating are next up on his list of things todo

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

thanks @jamie! do you want us to review anything yet?

jamie avatar

I’m doing a demo of them to my client next week, so I should have the code to 100 by Tuesday. I’ll do my pull requests then. :)

jamie avatar

@mcrowe if you put yours off till next week I’ll have the template sorted. And we can both continue making alerts modules

jamie avatar

The alerts modules are for: Awareness - such as slack. Automation- such as asg scaling Deployments- such as rolling back a codedeploy if the alert is triggered on a new version

jamie avatar

I’ll do a push to my branches in a moment so you can see what’s been done

LeoGmad avatar
LeoGmad
05:00:03 PM

@LeoGmad has joined the channel

wave1
jamie avatar

@Erik Osterman (Cloud Posse) and @mcrowe and the rest, I’m wanting to do a training day for my clients engineering team

jamie avatar
firmstep-public/trainingdayone

trainingdayone - The repo containing the training Terraform configurations

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@mike.ballou @dave.yu might be something for joany

firmstep-public/trainingdayone

trainingdayone - The repo containing the training Terraform configurations

jamie avatar

I’ve just written this up today

Purva avatar
Purva
06:12:28 PM

@Purva has joined the channel

wave2
jamie avatar

If anyone wants to use it you’re welcome, and if anyone has any good inputs for it, please let me know.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

that’s cool

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we’re starting to do more training. we’ll need to develop more materials like this.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Purva, @chris have you guys tried/started doing any terraform training?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@jamie looks like a good start. Inevitably, a big project you’re taking on to teach this stuff.

jamie avatar

Totally big job. I need to get their team of 14 introduced to what Terraform can do.

jamie avatar

I don’t need to train to pro level. But I do need to get them thinking about what they can accomplish with Terraform

jamie avatar

Day one is being filled out, and will likely need more READMEs

jamie avatar

Day two is taking a more real world example, and working through that.

jamie avatar

In this case it will be creating a CI/CD pipeline for Serverless deployments

jamie avatar

With updating a lambda function, and API Gateway, and doing a rolling deploy with monitored events which would cause roll back.

jamie avatar

Once training week is done, i’ll be back to the modules…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

would love to incorporate something like this eventually into docs.cloudposse.com

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Incorporate Terraform Training Materials · Issue #140 · cloudposse/docs

what Add materials for holding trainings why Needed for many customer engagements references https://github.com/firmstep-public/trainingdayone

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

jamie avatar

Well great!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/terraform-aws-lb-s3-bucket

terraform-aws-lb-s3-bucket - Terraform module to provision an S3 bucket with built in IAM policy to allow AWS Load Balancers to ship access logs

jamie avatar

One more fargate tip: no more than 20 tasks running per account per region

jamie avatar
Amazon ECS Service Limits - Amazon Elastic Container Service

The following table provides the default limits for Amazon ECS for an AWS account which can be changed. For more information on the service limits for other AWS services that you can use with Amazon ECS, such as Elastic Load Balancing and Auto Scaling, see

tamsky avatar


which can be changed

tamsky avatar

have you tried requesting a limit increase, or is that already on your path forward?

jamie avatar

Hasn’t affected me yet. Eric and @sarkis have been working on an ecs fargate module for a client, so I was hoping to point it out!

1
sarkis avatar

@jamie thanks!

sarkis avatar

i bet they’d raise the limit like any other - just depends if you have biz level support on how quickly they get around to it

sarkis avatar

highly recommend this for clients that have business level+ support btw: https://aws.amazon.com/premiumsupport/trustedadvisor/ - one of the things it does is track if you get close to service limits

AWS Support – Trusted Advisor

Learn about the cost savings, security, and performance features and benefits of AWS Trusted Advisor.

1
sarkis avatar

Other useful info includes reserved instance allocation and idle ebs volumes

2018-06-16

mcrowe avatar

Hey guys, I’m finding a need to document a geodesic module for internal use. For example, the inputs needed, but also I need to export the outputs from the modules included. For example:

module "ecs_alb_service_task" {
  source                    = "git::<https://github.com/cloudposse/terraform-aws-ecs-alb-service-task.git?ref=0.1.0>"
...

However, I typically want the outputs of this alb in my state, so I need to re-output the CP module’s output like this:

output "app_service_name" {
    value = "module.ecs_alb_service_task.service_name"
    description = "ECS Service name."
}

Check this out: https://gist.github.com/mike-zipit/fa936438c1edbb5f26ba68eec1cffaf8 This is WIP, but would like to validate if this is a need others see

2
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

This is definitely a need as the number of module deps grow. I think @Andriy Knysh (Cloud Posse) and @jamie will dig it.

1
jamie avatar

Thank you for helping to remove the tedium

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Support for HCL (Terraform Modules and Providers) · Issue #118 · dependabot/feedback

what Open PRs against terraform repos when terraform modules have new releases Open PRs against terraform repos when terraform providers have new releases why It&#39;s an extremely diverse ecosyste…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Looks in line with some of things we are doing for readme generation

2018-06-17

mcrowe avatar

@sarkis do you have a minute? I’m trying to use terraform-aws-ecs-alb-service-task but having a weird problem

sarkis avatar

sure whats up

mcrowe avatar

So, using terraform-aws-ecs-alb-service-task, the nginx container deploys and comes up (I see the default page). However, when my custom container deploys, it ends up dying

sarkis avatar

anything in the ECS service logs/container output?

mcrowe avatar

what’s strange is I think it’s a health check

mcrowe avatar
mcrowe
04:31:35 PM

@mcrowe uploaded a file: image.png

mcrowe avatar

I’ve ssh’d into the container (fixed various environment/permission issues I had), and I can run the container manually on the ECS host with docker run …..

sarkis avatar

does it go into a loop of starting/stopping the container?

mcrowe avatar

yep

sarkis avatar

i think you are right in that it sounds like the scheduler is giving up after failed healthcheck threshold and just keeps trying to start a “healthy” container

mcrowe avatar

I’ve bumped up the timeout on the alb:

  health_check_timeout  = 20
  health_check_interval = 30
mcrowe avatar

Plus, my target groups look like this in the EC2 console:

mcrowe avatar
mcrowe
04:35:14 PM

@mcrowe uploaded a file: image.png

mcrowe avatar

(the healthy one is the nginx controller – the others are failed deployments of my container)

sarkis avatar

do you have your target group tf resource handy?

sarkis avatar

i ran into something similar when i wasn’t pointing my target resources to the right healthcheck endpoint

sarkis avatar

endpoint/port

mcrowe avatar
mcrowe
04:38:10 PM

@mcrowe uploaded a file: image.png

sarkis avatar

and the above unhealthy status seems to reaffirm this is where the issue is…

mcrowe avatar

oh, traffic port – is this supposed to be ip?

sarkis avatar

what is on that custom container?

sarkis avatar

i dont think there is any http server on your custom container*

mcrowe avatar

a microservice. if you GET /, it returns a JSON document of the status

sarkis avatar

ah ok… i think i know something else to look into.. does this container take a bit to load?

mcrowe avatar

yeah, it does

sarkis avatar

so there is a grace period

sarkis avatar

before ECS will start considering failed healthchecks as unhealthy

mcrowe avatar

ah, good point.

sarkis avatar

let me find it

sarkis avatar
Task Definition Parameters - Amazon Elastic Container Service

Task definitions are split into separate parts: the task family, the IAM task role, the network mode, container definitions, volumes, task placement constraints, and launch types. The family is the name of the task, and each family can have multiple revisions. The IAM task role specifies the permissions that containers in the task should have. The network mode determines how the networking is configured for your containers. Container definitions specify which image to use, how much CPU and memory the container are allocated, and many more options. Volumes allow you to share data between containers and even persist the data on the container instance when the containers are no longer running. The task placement constraints customize how your tasks are placed within the infrastructure. The launch type determines which infrastructure your tasks use.

sarkis avatar

startPeriod The optional grace period within which to provide containers time to bootstrap before failed health checks count towards the maximum number of retries. You may specify between 0 and 300 seconds. The startPeriod is disabled by default.

sarkis avatar

that’s my guess right now - is that the container is taking a bit to start.. healthchecks failing and getting into a loop there

sarkis avatar
cloudposse/terraform-aws-ecs-container-definition

terraform-aws-ecs-container-definition - A Terraform module to generate well-formed JSON documents (container definitions) that are passed to the aws_ecs_task_definition Terraform resource

sarkis avatar

which reminds me to fix the README on that ^

mcrowe avatar

well, i don’t think that’s in the terraform-aws-ecs-alb-service-task module yet, is it?

sarkis avatar
cloudposse/terraform-aws-ecs-alb-service-task

terraform-aws-ecs-alb-service-task - A Terraform module which implements an ECS service which exposes a web service via ALB.

sarkis avatar

i’m in the process of cleaning this all up and things may change :disappointed: - but all that module does is allow you to create a json doc out of terraform code - it’s not required to use it … you can just modify your .json for the container_definition_json

mcrowe avatar

btw, did you see the issue I submitted about the environment variables? I had to just use pure .json files because I couldn’t pass environment variables in

sarkis avatar

did not, until now.. I just assigned to myself and will reproduce and see if we can workaround that

sarkis avatar

lmk if the startPeriod fixes your issue - now I’m about it

mcrowe avatar

testing it now

mcrowe avatar

BTW, another thin I ran into but didn’t report yet is this: we need the ability to customize the task security policy (we needed to grant permissions to SSM with a specific arn). I hacked it like this in the module (it’s not ideal, but I think you get the idea):

data "aws_iam_policy_document" "ecs_execution_role" {
  source_json = "${var.custom_policy_document}"

  statement {
    sid       = "vLMpjauEwiCGsAJ9tJKsbSgn"
    effect    = "Allow"
    resources = ["*"]

    actions = [
      "ecr:GetAuthorizationToken",
      "ecr:BatchCheckLayerAvailability",
      "ecr:GetDownloadUrlForLayer",
      "ecr:BatchGetImage",
      "logs:CreateLogStream",
      "logs:PutLogEvents",
    ]
  }
}
sarkis avatar

hmm yea that makes sense and should be an option to pass in our own existing or managed security policy (if not provided then generate)

sarkis avatar

would you mind opening an issue @mcrowe if not, I can

mcrowe avatar

I’ll do it later today if that’s ok

sarkis avatar

that works - i don’t know if i can get to anything today - so would be tomorrow at the earliest after client work

mcrowe avatar

that may have done it – not getting an OOM error (so further along in the chain)

mcrowe avatar

btw, do you know how to kill an ecs codepipeline deploy when it fails?

mcrowe avatar

@sarkis Adding:

health_check_grace_period_seconds  = 30

to my local copy of terraform-aws-ecs-alb-service-task fixed the issue

3
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

PR?

antonbabenko avatar
antonbabenko
06:46:40 PM

@antonbabenko has joined the channel

wave2

2018-06-18

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/terraform-aws-ecs-web-app

terraform-aws-ecs-web-app - A Terraform module which implements a web app on ECS and supporting AWS resources.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

this implements the common pattern of provisioning an ECR integrated with CI/CD to deploy containers to an ECS task and adding an ALB ingress

mcrowe avatar

That’s pretty sweet – needed that 2w ago

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but it’s also just a prime example of using all other ecs modules

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/terraform-aws-ecs-web-app

terraform-aws-ecs-web-app - A Terraform module which implements a web app on ECS and supporting AWS resources.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(realize it’s a bit late)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

also, this uses our “readme generator” to convert a README.yaml (using gomplate) to a README.md

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/terraform-aws-ecs-web-app

terraform-aws-ecs-web-app - A Terraform module which implements a web app on ECS and supporting AWS resources.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

as part of this, it uses terraform-docs to generate the wellformed markdown

mcrowe avatar

I scheduled a quick call with you tomorrow about this. I think what I did has a different application

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ok, look forward to talking at 10:30

2018-06-19

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
07:47:03 PM

@Erik Osterman (Cloud Posse) uploaded a file: image.png

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

being able to easily identify out-of-date modules and automatically generate all the output definitions from nested modules is something I’ve wanted

jamie avatar

Did any of the modules i made help with the ecs web app?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think we used your ecs codepipline changes

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@sarkis would know more

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i can walk you through what we have this week sometime

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we’re showing the client how to use it right now

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

and i think sarkis also used your ecs task module as an initial starting off point

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i’d be really interested in your feedback on our ecs-web-app module <— from an interface perspective

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

does it make sense?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we’ll probably create an ecs-job-app soon <— which wouldn’t depend on an ALB

jamie avatar

I read through it earlier very briefly and I thought it was well made from what I saw. I’ll give it a test later on - :)

sarkis avatar

yea the codepipeline and codebuild code you contributed - thanks a ton Jamie!

sarkis avatar

I’ll make sure you get credit in the READMEs for those projects

rohit.verma avatar
rohit.verma

In my init-terraform scripts, I am seeing mount -a failing to mount s3 to local

rohit.verma avatar
rohit.verma
(dev-admin) backing-services ➤ init-terraform 
/bin/sh: s3fs: not found
rohit.verma avatar
rohit.verma

I was looking at previous chats, seems like this is common

sarkis avatar

what version of build-harness?

sarkis avatar

and geodesic

rohit.verma avatar
rohit.verma

i am using a fork of geodesic,

rohit.verma avatar
rohit.verma

Also I am not sure where we are putting build-harness version, I think its always master

rohit.verma avatar
rohit.verma

<https://raw.githubusercontent.com/cloudposse/build-harness/master/templates/Makefile.build-harness>

rohit.verma avatar
rohit.verma

My geodesic is mostly in sync with master

sarkis avatar

@rohit.verma what does which s3fs return? (from inside geodesic container)

rohit.verma avatar
rohit.verma
 s3fs
Error: goofys takes exactly two arguments.
rohit.verma avatar
rohit.verma

i am on goofys v0.19.0

sarkis avatar

looking at the s3fs on my geodesic container…

cat /usr/local/bin/s3fs
#!/bin/bash
# This script is used to mount filesystems from `/etc/fstab`
export DEVICE="$(eval echo ${1})"
export MOUNT_POINT="$(eval echo ${2})"
export BUCKET_REGION="${BUCKET_REGION:-${AWS_REGION}}"

# Evaluate the bash envs passed in via args and then execute goofys
exec goofys --file-mode=0600 --dir-mode=0700 --region=${BUCKET_REGION} --sse --acl=private ${DEVICE##s3://} ${MOUNT_POINT}
sarkis avatar

probably an issue with one of the last 2 args there ^ not being set

sarkis avatar

not sure yet why - tried to reproduce to no avail…

rohit.verma avatar
rohit.verma

in my dockerfile I have added just RUN s3 fstab '${TF_BUCKET}' '/' '/secrets/tf'

sarkis avatar

@rohit.verma do you have ENV TF_BUCKET set before somewhere in the Dockerfile?

rohit.verma avatar
rohit.verma

yes

rohit.verma avatar
rohit.verma

running s3 fstab '${TF_BUCKET}' '/' '/secrets/tf' directly in container gives no error

rohit.verma avatar
rohit.verma

I think ideally it should mount the terraform bucket to /secrets/tf

rohit.verma avatar
rohit.verma

but that is still empty

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We can look into this together

2018-06-20

jamie avatar

@sarkis Next week I’ll have an ECS treat for you

jamie avatar
cloudposse/terraform-aws-ecs-ecs-spot-fleet

terraform-aws-ecs-ecs-spot-fleet - Terraform module to create a diversified spot fleet for ECS clusters

jamie avatar

Its pending an update to the SpotFleet for allowing Launch Templates

jamie avatar

A fraction of the cost of Fargate.

sarkis avatar

Looks really interesting Jamie - are you going to have a % of spot instances or I guess spot fleet handles that? I’m thinking what happens when prices spike…

sarkis avatar

Yep looks like it:
The Spot Fleet also attempts to maintain its target capacity fleet if your Spot Instances are interrupted due to a change in Spot prices or available capacity.

jamie avatar

well, the fleet will be made of a mix of instance types

sarkis avatar

yea

jamie avatar

so, if one instance type goes out of price, it boots another

sarkis avatar

last i tried anything with spot - there was no fleet

sarkis avatar

or it was super new i forget why we didn’t look into it

jamie avatar

yeah, its new.

jamie avatar

and they have released autoscaling for spot fleet recently too

sarkis avatar

it was pretty much unusable for anything mission critical for this reason

jamie avatar

so, you can create a spot fleet, and scale it

sarkis avatar

really cool - looking forward to seeing the finished product!

jamie avatar

when combined with a watcher script that runs on the instance (which you will see if you look in the user-data dir)

jamie avatar

that starts ‘draining’ the instance as soon as it gets a spot instance termination notice

jamie avatar

and setting the ‘draining lifetime’ of tasks to 1 min 45 seconds

jamie avatar

it means that all of the containers running on the interupted instance get moved to another server before its removed

jamie avatar

so you can keep your workload stable

jamie avatar

its good for web sites, slightly less good for things that need to gaurentee connections for longer than 2 mins

jamie avatar

i have been running spot instances for years, and they have only rotated out a few times

2
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Very cool! I’d like to demo this for a company currently using spotinst.com

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think their spend is like 800K/mo on AWS :-)

sarkis avatar

Jamie, re: ssm param store policy docs PR, put in a comment in there - lmk if you have any Qs about the process for that stuff… I’m working on moving more of our repos to that README.yaml format

jamie avatar

Thank you @sarkis

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

btw, added #releases where we’ll announce releases of our projects

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
05:38:03 PM

@Erik Osterman (Cloud Posse) uploaded a file: image.png

sarkis avatar

Jamie - I know it’s getting late there so not doing the @ pings , do you have any prior art or tips for ECS Fargate CloudWatch monitoring - we obviously don’t have/care about the ECS cluster health, but starting to look into what we can monitor in the containers themselves - basic mem, cpu ,etc.

jamie avatar

@sarkis Yo, keep the pings coming

jamie avatar

I’ve done the README update that you suggested

sarkis avatar

ah great -

jamie avatar

Okay so…

jamie avatar
bitflight-public/terraform-aws-ecs-events

terraform-aws-ecs-events - Add on an SNS topic for capturing ECS events

sarkis avatar

I think this is great… gives me a nudge on where to look and I can just borrow this as a base (and give you credit of course), if you don’t mind

bitflight-public/terraform-aws-ecs-events

terraform-aws-ecs-events - Add on an SNS topic for capturing ECS events

jamie avatar

Can you use the format of the lambda one I posted? So that all the new alarm modules can work the same?

sarkis avatar

Yea - I’m reviewing that one now as well

sarkis avatar
cloudposse/terraform-aws-cloudwatch-alarms-lambda

terraform-aws-cloudwatch-alarms-lambda - Terraform module for creating a set of Lambda alarms and outputting to an endpoint

jamie avatar

Exactly. I’ve made the repo now..

jamie avatar

Thats a minor one…

jamie avatar

but when do you need it by? I’ll make a fuller one

jamie avatar
cloudposse/terraform-aws-cloudwatch-alarms-lambda

terraform-aws-cloudwatch-alarms-lambda - Terraform module for creating a set of Lambda alarms and outputting to an endpoint

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@jamie I reviewed your terraform-aws-ssm-parameter-store-policy-documents - just a few nit picks

1
sarkis avatar

Also just curious Jamie - what’s your thoughts on the README generation process

jamie avatar

It’s annoying. A form would be good ;)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

what part of it could be improved?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)


A form would be good

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

in otherwords, an “init” process?

jamie avatar

I’ll elaborate shortly. Just walking somewhere

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

jamie avatar

Hi, I’m back.

jamie avatar

So in regards to the README.yaml it isn’t very DRY

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

which parts?

Robert Tisdale avatar
Robert Tisdale

hello all

Robert Tisdale avatar
Robert Tisdale

Can’t remember if I said hello

jamie avatar
# Name of this project
name: terraform-aws-ssm-parameter-store-policy-documents

# Logo for this project
#logo: docs/logo.png

# License of this project
license: "APACHE2"

# Canonical GitHub repo
github_repo: cloudposse/terraform-aws-ssm-parameter-store-policy-documents

# Badges to display
badges:
  - name: "Build Status"
    image: "<https://travis-ci.org/cloudposse/terraform-aws-ssm-parameter-store-policy-documents.svg?branch=master>"
    url: "<https://travis-ci.org/cloudposse/terraform-aws-ssm-parameter-store-policy-documents>"
  - name: "Latest Release"
    image: "<https://img.shields.io/github/release/cloudposse/terraform-aws-ssm-parameter-store-policy-documents.svg>"
    url: "<https://github.com/cloudposse/terraform-aws-ssm-parameter-store-policy-documents/releases>"
  - name: "Slack Community"
    image: "<https://slack.cloudposse.com/badge.svg>"
    url: "<https://slack.cloudposse.com>"

related:
  - name: "terraform-aws-ssm-parameter-store"
    description: "AWS SSM Parameter Store module"
    url: "<https://github.com/cloudposse/terraform-aws-ssm-parameter-store>"



# Short description of this project
description: |-
  This module generates JSON documents for restricted permission sets for AWS SSM Parameter Store access.
  Helpful when combined with [terraform-aws-ssm-parameter-store](<https://github.com/cloudposse/terraform-aws-ssm-parameter-store>)
# How to use this project

examples: |-
  Create a policy that allows access to write all parameters
hcl
  module "ps_policy" {
    source = "git::<https://github.com/cloudposse/terraform-aws-ssm-parameter-store-policy-documents.git?ref=master>"
  }
Robert Tisdale avatar
Robert Tisdale

Don’t want to derail

Robert Tisdale avatar
Robert Tisdale

Holy paste batman

jamie avatar

sorry Robert!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

hey @Robert Tisdale!

jamie avatar

bad syncronisity

1
Robert Tisdale avatar
Robert Tisdale

LOL, no problem I’m just yanking your chain @jamie

Robert Tisdale avatar
Robert Tisdale

Hey @Erik Osterman (Cloud Posse)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

can you propose (in pseudo code) how the YAML should look instead?

jamie avatar

@Erik Osterman (Cloud Posse) the terraform-aws-ssm-parameter-store-policy-documents (project name) gets used over and over

jamie avatar

and in the intro areas especially, it would help if it was a variable

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

unfortunately, don’t see a way around that without recursion.

sarkis avatar

ah… it might turn into a case of mutliple levels of interpolation though since we already render this - thinking something like terraforms $$

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
-rw-r--r--    1 erik  staff  11291 Jun 20 12:50 README.md
-rw-r--r--    1 erik  staff   5429 Jun 20 12:50 README.yaml

but it’s at least 50% more DRY

jamie avatar

yeah

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

anyways, thanks for the feedback. I think we can take over some of the “cleanup” burden to keep you focused on coding

sarkis avatar

whoa didn’t bother to look at that - nice

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but there’s a stronger, more overwhelming reason we cannot do that

sarkis avatar

to me the biggest bonus is to only ever have to update the legal jargon in one spot

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

the purpose for the .yaml format is so we can include it in our docs portal (docs.cloudposse.com)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

if the .yaml was itself templated, we would then need to create multiple parsers

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

one for readmes and one for hugo

Robert Tisdale avatar
Robert Tisdale

Why use YAML for that instead of just markdown?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

to enforce consistency

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so basically, in hugo (our docs CMS) we can do this:

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
{{% include-github org="cloudposse" repo="terraform-root-modules" ref="0.1.5" file="codefresh.yml" title="Example codefresh.yml" language="yaml" type="code-block" %}}
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

this allows us to pull documentation in from individual repos

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

the hardest part about writing documentation is keeping it up to date

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

by allowing individual repositories to manage the general usage/examples/inputs/outputs/etc, we can reference that in our main documentation portal via YAML and ensure it’s up to date in both places

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

“documentation as code”

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

jamie avatar

I have started to look into doing TDD with terraform.

jamie avatar

My current opinion is not to do it

jamie avatar

but to do validations as part of a test process

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I would like to do a compromise

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i think writing full on tests for everything is overkill

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but running plan/apply/destroy 80% of the battle

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i want to use our [testing.cloudposse.co](http://testing.cloudposse.co) account for that

jamie avatar

its also fucking horrid, terraform is not at the stage to validate that beyond plan

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

and use codebuild with codepipeline to automate that

sarkis avatar

@jamie something i’ve been pondering, more than TDD is RDD (Readme Driven Development)

jamie avatar

yeah, well… Ive been reading on awspec, kitchen-terraform, goss, and inspec for the last 2 weks

jamie avatar

weeks

jamie avatar

@sarkis im in to that

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@antonbabenko will probably have some more insights on it <– i know he uses kitchen-terraform

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but @jamie what do you think about the plan/apply/destroy part?

jamie avatar

I think that there is a portion that could be done, and i think there could be a ‘minimum’ that can be tested for at ‘unit test’ time

jamie avatar

@Erik Osterman (Cloud Posse) i think the plan/apply/destroy part could take too long as a ‘unit-test’ but as a further along test it works

sarkis avatar

you think so? i really feel like integration tests are going to be the bulk of the tests if and when we get to it

sarkis avatar

i dont know how useful unit tests are is what im tying to say

jamie avatar

I agree. Its what I was attempting to articulate too.

jamie avatar

I am not from a recent dev background. The unit test part allows for testing as you create quickly.

jamie avatar

But Integration tests can be done… quite well with terraform

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

our modules are breaking frequently due to terraform changes outside of our control

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

this is why I want plan/apply/destroy

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

the time to run the tests isn’t a concern

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

it’s very frequent in software projects that tests taken 30-45 minutes or even more

jamie avatar

I think we can get that bit…at least

jamie avatar

although they take longer to run, and can be a lot of work to write

sarkis avatar

anyone check out terratest yet?

jamie avatar

I read about terratest yesterday… did you read about it on reddit?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

they had a big announcement a couple of months ago

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

it’s by Grunt Works (terragrunt)

sarkis avatar

i had talked to Yevgeniy about it as I helped bug squash a bit on terragrunt… he was telling me they had it ready to release but hashicorp kept telling him to hold off because they have something they want to come out with (surprise, they are all over the place :P)

sarkis avatar

so yea officially publicly available a couple months ago as erik said ^

jamie avatar

Like, Terraform’s ‘issues’ list is huge. But i don’t like them any less.

jamie avatar

They are the best of the crowd

sarkis avatar

oh yea - i just mean they should just let the community run with some things - it might help them focus on the core

sarkis avatar

but i can see the other side of that coin too

sarkis avatar

nice to have control (enforce quality)

antonbabenko avatar
antonbabenko

I normally don’t use kitchen-terraform and will not use terratest because it is very hard to make meaningful tests. hcl/tf files are not code, so I don’t think we need just-developer-friendly-tool. I think we need something what is more like DSL (in a form of BDD/Gherkin). Also, there will be a validation of values before apply, and a lot more…

antonbabenko avatar
antonbabenko
hashicorp-terraform/gardening

Where developers come to talk.

sarkis avatar

something like serverspec but for aws resources lol

antonbabenko avatar
antonbabenko

I wish something more abstract.

jamie avatar

Well, if there was a terraform for testing that was multi provider enabled…

antonbabenko avatar
antonbabenko

For example, when I was looking for something like inspec for google provider a year and more ago I could not, and I don’t know if there is awspec for google now.

sarkis avatar

true

antonbabenko avatar
antonbabenko

From what I see, there is 70% of testing things for AWS, 30% for Azure, and 0% for Google.

jamie avatar

Google have a test system… I found it when researching this week..

jamie avatar

let me find it

jamie avatar

Its not quite right but this goes towards it https://github.com/inspec/inspec-gcp

inspec/inspec-gcp

inspec-gcp - InSpec GCP (Google Cloud Platform) Resource Pack

jamie avatar

Are you guys using any pentest bots for testing code and infrastructure as part of the deploy pipeline?

jamie avatar

https://github.com/guardicore/monkey <– I’ve been watching this repo

guardicore/monkey

monkey - Infection Monkey - An automated pentest tool

jamie avatar

but haven’t used it yet

jamie avatar

And it looks very thorough

jamie avatar

I actually expect very few clients to pass muster

sarkis avatar

hmm that is nice - thanks for sharing it @jamie - starred to dig into on my free time

sarkis avatar

the only thing that worries me with this stuff is getting an aws account suspended - i’ve heard some horror stories

sarkis avatar

not specifically about this but just in general

jamie avatar

Well, yeah… if you are doing anything that is DOS related

jamie avatar

you have to inform AWS

sarkis avatar

right

jamie avatar

its on their TOS

sarkis avatar

like i know security firms need to have a special deal with AWS and follow some standards, etc

jamie avatar

So, if you are running this bot and doing any DDOS or DOS features you have to fill in the AWS Support request first

jamie avatar

and say that is what you are doing

jamie avatar

But this bot, doesn’t just brute force

jamie avatar

So, its likely it will be outside of the AWS security remit

jamie avatar
Infection Monkeyattachment image

Infection Monkey is an open source breach and attack simulation tool to evaluate the security posture of your network

sarkis avatar

yea it looks more like discovery

jamie avatar

aws marketplace even allows it

1
jamie avatar

lol

2018-06-21

antonbabenko avatar
antonbabenko

Thanks @jamie for sharing links. inspec-gcp is just 3 months old project.

2018-06-22

jamie avatar

Have any of you made a metadata index for s3 in terraform? like this? https://github.com/aws-samples/aws-big-data-blog/tree/master/aws-blog-s3-index-with-lambda-ddb

aws-samples/aws-big-data-blog

Contribute to aws-big-data-blog development by creating an account on GitHub.

jamie avatar

If no one has, @Erik Osterman (Cloud Posse) can you add it to me on your project board?

jamie avatar

Also: An aws alarm terraform module for detecting root account logins.

jamie avatar

And: An aws alarm terraform module for alerting on unauthorized aws api access.

jamie avatar

These are part of the aws CIS best practices.

jamie avatar

And should be small and easy to add

jamie avatar

And a module to deploy this task to a cluster. https://github.com/ExpediaDotCom/c3vis/blob/master/README.md

ExpediaDotCom/c3vis

c3vis - Visualize the resource utilisation of Amazon ECS clusters

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I will track those in the project later today

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Build software better, together

GitHub is where people build software. More than 28 million people use GitHub to discover, fork, and contribute to over 85 million projects.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Daren anything you can share for the root account logins?

jamie avatar

Thanks!

jamie avatar

No rush I just don’t want to forget them

Daren avatar

@Erik Osterman (Cloud Posse) Sure. Id prefer to touch it up before open sourcing it, it was hacked together to meet the pci deadline

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ok, understandable

Daren avatar

It does the following:

  • alert when a public is made public
  • alert on root login
  • alert on modification of cloudtrails
  • auto re-enable disabled cloudtrails
jamie avatar

a bucket is made public?

jamie avatar

@Daren would really appreciate you publishing it, i’m working through this doc https://d0.awsstatic.com/whitepapers/compliance/AWS_CIS_Foundations_Benchmark.pdf

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(we could also probably help with “clean up” if you want - PM)

antonbabenko avatar
antonbabenko
Daren avatar

Thanks this looks promising. We tackled our using cloudwatch event rule -> lambda -> pagerduty/slack/…

jamie avatar

those are great anton

jamie avatar

may I use them?

jamie avatar

@antonbabenko i just spotted something that may cause a headache for you…

jamie avatar

You use aws_iam_policy_attachment

jamie avatar

instead of aws_iam_role_policy_attachment

jamie avatar

Incase it wasn’t intentional

antonbabenko avatar
antonbabenko

Absolutely, use anything from there. I quit from that company today

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

no way!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

what are you going to focus on now?

antonbabenko avatar
antonbabenko

there are some modules which I want to move to open-source under terraform-aws-modules umbrella

antonbabenko avatar
antonbabenko

long vacation first, and then find something related to Terraform & AWS. Have nothing settled yet

antonbabenko avatar
antonbabenko

Thanks for pointing to aws_iam_policy_attachment - I didn’t know that

antonbabenko avatar
antonbabenko

Guys who are from Russia or Europe, will you consider going to https://devoops.ru/en/ ? I am going there and will be talking about Terraform there.

DevOps conference. October 14, Saint Petersburg. More than 400 participants, 20 technical talks.attachment image

DevOps conference. October 14, Saint Petersburg. More than 400 participants, 20 technical talks.

1
sarkis avatar

man I wanna goto St Petersburg if it wasn’t a 20+ hr flight i’d be buying a ticket lol

antonbabenko avatar
antonbabenko

Yeah, it is far from you. Will anyone go to HashiConf?

jamie avatar

If I was looking for work I would

1
jamie avatar

Alarm module made, with working example…. with some README

jamie avatar
cloudposse/terraform-aws-cloudtrail-cloudwatch-alarms

terraform-aws-cloudtrail-cloudwatch-alarms - Terraform module that notifies an endpoint such as SNS when cloudtrail logs unauthorised API access

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

make sure to add all proper attributions

cloudposse/terraform-aws-cloudtrail-cloudwatch-alarms

terraform-aws-cloudtrail-cloudwatch-alarms - Terraform module that notifies an endpoint such as SNS when cloudtrail logs unauthorised API access

jamie avatar

will do

2018-06-23

jamie avatar

I added a thanks, a reference, and the MIT licence+copyright under the Apache2 license.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Looks like there are some extra brackets around the image. See GitHub rendering

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Missing period before “With many thanks”

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I would write [Anton Babenko](…link to github…]

jamie avatar
cloudposse/terraform-aws-cloudtrail-cloudwatch-alarms

terraform-aws-cloudtrail-cloudwatch-alarms - Terraform module for creating alarms for tracking important changes and occurances from cloudtrail.

jamie avatar

Is that correct?

antonbabenko avatar
antonbabenko

Looks good, but I would not use locals (in alarms.tf), but rather create a bunch of resources aws_cloudwatch_metric_alarm manually. This way, you can easily delete something in the middle of the list without recreating the rest of the list. This will be fixed in HCL2 and Terraform 0.12, but for now it is still an issue with ordered lists.

jamie avatar

@antonbabenko I’ve updated it to use the pretty readme, added dashboards, and added screenshots

jamie avatar

Although I’m going to keep the lists in locals because I find the compact layout easier to read. And the lists are static.

1
jamie avatar

I’m looking forward to hcl2 and getting ForExpressions https://github.com/hashicorp/hcl2/blob/master/hcl/hclsyntax/spec.md#for-expressions

hashicorp/hcl2

hcl2 - Temporary home for experimental new version of HCL

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Hey Joost!

joost avatar

Is there a way sending cloudtrail logs (from several customer aws accounts – all regions) to a specific cloudwatch log group at my security aws account?

joost avatar

I want to use the module terraform-aws-cloudtrail-cloudwatch-alarms at my security aws account

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Hrmm.. we do something very similar, but ship them to an encrypted s3 bucket. What you propose sounds nice though.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think @jamie might have recently done what you want

joost avatar

Cool. When you ship them to s3, what do you use for notifications? S3 events to lambda?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

With the s3 bucket, we are not doing any no notifications. :( just storing logs.

This is something I’d need to defer to @jamie who has been working on all the alarm modules and probably has done what you are trying to achieve.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

This a an area we just started investing in - so it will be a little bit before we have everything fully baked.

joost avatar

Ok, I understand. Thx!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Daren would really appreciate you publishing it, i’m working through this doc https://d0.awsstatic.com/whitepapers/compliance/AWS_CIS_Foundations_Benchmark.pdf

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Jamie is working on implementing modules to address as much of this as possible.

1
jamie avatar

Aws KB has published the Iam policies, guidelines and such for centralizing the cloudwatch logs. I can pull the policies into the module as an option. And put the guide in the readme. So you can use the alerts on one account, but have it track and alert for all accounts.

joost avatar

That would be awesome

2018-06-24

jamie avatar

@joost another good item for you to read through this [WIP] [ETA: 30.6.2018] Terraform best practices https://github.com/antonbabenko/terraform-best-practices by @antonbabenko it links to examples of organizing code for multiple aws accounts such as you may need

antonbabenko/terraform-best-practices

terraform-best-practices - [WIP] [ETA: 30.6.2018] Terraform best practices

antonbabenko avatar
antonbabenko

“WIP” will have to be remaining for that repo until I am back from my vacation. I want to make it complete in September before I speak about it at conferences.

1
joost avatar

Thx @antonbabenko for the great link

2018-06-25

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Daren did you guys get kiam working? We started on this: https://github.com/cloudposse/geodesic/pull/163

Daren avatar

Actually started on it this week

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Our first stab was giving us 403 to the AWS metadata api

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Will get back to it today

Daren avatar

Are you deploying the agent to masters?

Daren avatar

also why no iptables or rbac?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We haven’t enabled rbac yet in our kops manifest

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Iptables false was just an attempt to get it working

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(It also happens to be the default)

Daren avatar

hm, I thought it was required. but the doc is a little vague

Daren avatar


An DNAT iptables rule is required to intercept the traffic. The agent is capable of adding and removing the required rule for you through use of the –iptables flag.

Daren avatar
uswitch/kiam

kiam - Integrate AWS IAM with Kubernetes

kubernetes/charts

charts - Curated applications for Kubernetes

Daren avatar

Wont this lead to the server being blocked by the agent

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

also, doesn’t [kubernetes.io/role](http://kubernetes.io/role): master require a toleration? they don’t have one

Daren avatar

hmm

Daren avatar
failed to load system roots and no roots provided - TLS error · Issue #36 · uswitch/kiam

{&quot;generation.metadata&quot;&quot;error&quot;,&quot;msg&quot;:&quot;error warming credentials: RequestError: send request failed\ncaused by: Post https://sts.amazonaws.com/

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

did it fix it for you?

Daren avatar

Going to try shortly

Daren avatar

Unfortunately we still have a mix of coreos + debian hosts. Not sure if the path is consistent

Daren avatar

Well I got the server up and running fine. However the agents are reporting this error:
error creating server gateway: error waiting for address being available in the balancer: context deadline exceeded

Daren avatar
Kiam server readiness/liveness probe issues (v2.8) · Issue #94 · uswitch/kiam

I have Kiam setup on a 3 master cluster - the server runs on two of the three masters with some readiness probes failing but the 3rd node is in a crash loop backoff with the liveness probe failing….

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ran into some issues

2018-06-26

jamie avatar

I wish i liked writing READMEs

jamie avatar
cloudposse/terraform-aws-s3-logs-athena-query

terraform-aws-s3-logs-athena-query - A Terraform module that creates an Athena Database and Structure for querying S3 access logs

jamie avatar

It works. But is pending readme creation

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Ok - we can help with that

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Really not fond of heredocs. Can we move that to template files?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Perhaps named query should be a separate module that reads queries from a template file

jamie avatar

Whats wrong with heredocs?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Legibility of code

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Can not properly indent

jamie avatar

okay

jamie avatar

I can split it into its own sql folder

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

That would be perfect

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@jamie @Andriy Knysh (Cloud Posse) @Igor Rodionov - can we schedule some time to talk about standardizing how we use IAM permissions?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Igor Rodionov has some problems with one of our older modules on ecr and before he makes any changes, we thought it a good idea to come up with a standard

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@jamie - I know you’re working a lot with locking down IAM permissions these days

Igor Rodionov avatar
Igor Rodionov

I can provide an example of my vision on ecr module

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, that would be a good start

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we can CR that <— as a proposal

Igor Rodionov avatar
Igor Rodionov

and bring it to the table for meeting

jamie avatar

Sure. In brief: the iam_policy_documents can be overwritten, or appended to depending on if the statement has a SID. Combining that, with the concept of a per project ‘policy module’ such as https://github.com/cloudposse/terraform-aws-ssm-parameter-store-policy-documents Or an integrated policy module, so that as part of the current module, it outputs the json policy documents json, and sids, so they can be seen. And includes an option for including json documents that can be passed through for appending.

cloudposse/terraform-aws-ssm-parameter-store-policy-documents

terraform-aws-ssm-parameter-store-policy-documents - A Terraform module that generates JSON documents for access for common AWS SSM Parameter Store policies

jamie avatar

It will be more work, but allows the lowest possible permissions to be added by the developer. But still allows additional permissions to be included in the module for custom needs.

jamie avatar
source_json (Optional) - An IAM policy document to import as a base for the current policy document. Statements with non-blank sids in the current policy document will overwrite statements with the same sid in the source json. Statements without an sid cannot be overwritten.
override_json (Optional) - An IAM policy document to import and override the current policy document. Statements with non-blank sids in the override document will overwrite statements with the same sid in the current document. Statements without an sid cannot be overwritten.

2018-06-27

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Add context by osterman · Pull Request #25 · cloudposse/terraform-null-label

– THIS IS CURRENTLY A POC/RFC – what Support passing a label&#39;s context between label modules why DRY demo module &quot;label1&quot; { source = &quot;../../&quot; namespace = &quo…

2018-06-28

jamie avatar

I added an example and such to the terraform-aws-ecs-service-cloudwatch-sns-alarms init

sarkis avatar

@jamie you are on a roll today - i need to know what brand/bean of coffee you having

jamie avatar

thank you

jamie avatar

Did you see the null label one?

sarkis avatar

yes

jamie avatar

hehe

jamie avatar

It was very heavily refactored

jamie avatar

just to get consistency on the data types, and then outputs them in the old output style

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, that was a clever trick. hadn’t considered it, but i think it’s a necessary evil.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
HashiCorp Terraform 0.12 Preview

Terraform 0.12 focuses on major Terraform language improvements and will be released later this summer. We are communicating about Terraform 0.12 prior to release to highlight the …

2018-06-29

jamie avatar

Interesting

sarkis avatar

so i was doing the whole compact concat dance before.. but i think this works in tf now…

alarm_actions       = ["${local.sns_topic_arn}", "${var.additional_notify_arns}"]
ok_actions          = ["${local.sns_topic_arn}", "${var.additional_notify_arns}"]
sarkis avatar

where local.sns_topic_arn is a string and var.additional_notify_arns is a list

sarkis avatar

hmm yea just tested it and it does work as expected… reference: https://github.com/hashicorp/terraform/issues/5682#issuecomment-278483475

Appending a value to an array · Issue #5682 · hashicorp/terraform

It seems to be tricky to append a value to an array. I thought that something like this should work: &quot;${compact(concat(split(&quot;,&quot;, var.security_groups), var.common_security_group))}&q…

jamie avatar

I saw that it was working

jamie avatar

like magic

sarkis avatar

so much nicer than the old way

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Ember® Ceramic Mug

Beautifully designed to be used in your home or at your desk, the Ember® Ceramic Mug keeps your beverages at the perfect temperature from the first sip to the last drop. Buy the World’s First Temperature Control Mug™ at Ember.com.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

is that for you?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(via: @Max Moon)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
07:04:19 PM

@Erik Osterman (Cloud Posse) uploaded a file: image.png

jamie avatar

have you guys seen or worked with https://github.com/blinkist

Blinkist

GitHub is where people build software. More than 28 million people use GitHub to discover, fork, and contribute to over 85 million projects.

jamie avatar

some very thorough modules built by him (almost to @sarkis completeness grade)

jamie avatar
blinkist/terraform-aws-airship-ecs-service

terraform-aws-airship-ecs-service - Terraform module which creates an ECS Service, IAM roles, Scaling, ALB listener rules.. Fargate & AWSVPC compatible

sarkis avatar

hmmm looks interesting - thanks for sharing @jamie

sarkis avatar

at the very least - I need some diagrams for the ecs-web-app module … it’s getting huge

jamie avatar

Diagrams are annoying and always appreciated

maarten avatar
maarten

Hi fellas, I’m the Blinkist guy

jamie avatar

Welcome :)

maarten avatar
maarten

Few small bugs still here and there, but slowly getting there. I was discussing earlier with Jamie I think if it’s possible to have Terraform manage the ECS container definitions except for the image parameter.. Where Terraform runs would extract the image:tag from aws_ecs_container_definition. Only solution I can think of is by creating a bootstrap=true for the module which would turn on/off a datasource for the first run, and other logic on the second run.. Would need to play around with that..

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@maarten so I want to share our strategy for this. it’s not original and I’ve seen it mentioned in a few places.

jamie avatar

Guys, I invited that guy from the links I shared

maarten avatar
maarten

Thanks for the invite

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Welcome to the channel!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Nice work on those modules

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I am afk, but will share how we are doing it

jamie avatar

@maarten it’s Friday night for like most of the crew, and it’s quieter here over the weekend. But core group of @sarkis @Erik Osterman (Cloud Posse) sand @Andriy Knysh (Cloud Posse) are keeping an eye on it

jamie avatar

As seen

maarten avatar
maarten

Yeah sure.. Normally I’m not around on a Friday either … I’ll introduce myself quickly.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yes, things are pretty quiet now as it’s the end of the week

maarten avatar
maarten

I’m the founder of the Terraform User Group in Berlin, started it when I didn’t know where to start with Terraform roughly 2 years ago. Worked for 2.5 years at Blinkist where we went for AWS with ECS, coming from Heroku. Was always a dream to be self-employed, Blinkist fully supported that and made me a contractor with as first assignment open sourcing the ECS stuff I did earlier. Anothing thing I made is a golang tool called skipper which does some magic regarding task execution through a docker tunnel etc.etc. but that project is on hold until I get a bit of golang help.

maarten avatar
maarten

@Erik Osterman (Cloud Posse) Yeah, happy to see some of that Have a good weekend.

jamie avatar

Welcome! Its 11:30pm for me too so I’m going offline as well. But we will chat soon.

maarten avatar
maarten

Sure, again, thanks for the invite.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yea let’s sync up on Monday !

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Also, maybe it would be interesting to have a weekly standing “round table” where we get together in a zoom room

2
sarkis avatar

It might even be something to record and distribute podcast style…

sarkis avatar

Welcome Maarten wave

2
maarten avatar
maarten

Thanks

maarten avatar
maarten

Can’t have enough slack channels

sarkis avatar

I need all the ones I have open plus more for terraform/hcl workarounds

maarten avatar
maarten

Where is the hcl2 countdown channel ? ;)

2018-06-30

Max Moon avatar
Max Moon

https://github.com/cloudposse/terraform-aws-elasticache-redis/pull/12 if anyone has time for a one line code review

Adding engine_version to elasticache replication group arguments by MoonMoon1919 · Pull Request #12 · cloudposse/terraform-aws-elasticache-redis

What it is Adds engine_version argument to the aws_elasticache_replication_group module. Why Previously when referencing this module in an external terraform template any attempt to override the en…

1
Max Moon avatar
Max Moon

Thanks in advance!

sarkis avatar

@Max Moon thanks for the PR. Looks like the variables.tf entry was there but just wasn’t being passed?

Max Moon avatar
Max Moon

aha

sarkis avatar

I’ll wait for 1 more approval and can merge. Is this blocking you right now?

Max Moon avatar
Max Moon

if i missed that then no big deal on this PR. No it’s not blocking me, i just referenced my branch.

Max Moon avatar
Max Moon

After reviewing variables.tf I think that despite being referenced in [variables.tf](http://variables.tf) it wasn’t being picked up because the reference wasn’t made in the module, but i could be wrong

sarkis avatar

Yea - I think CI would catch this (maybe) either way put a to-do for myself to get that PRed up

Max Moon avatar
Max Moon

it certainly was not working this morning, at least. So I think the PR for this is still relevant, pending review from others

sarkis avatar

Oh absolutely your PR is necessary. Sorry should have gave context. I was confused why you didn’t have a variables.tf change - made sense when I looked at master. It has the unref variable on there.

Max Moon avatar
Max Moon

ah yes okay, i gotchya now

sarkis avatar

So one last thing. Now I’m thinking if we default to 4.0

Max Moon avatar
Max Moon

i was thinking the same.

sarkis avatar

Since that was the previous behavior it seems from your testing

sarkis avatar

Let’s do that so it’s less of a backwards incompatible risk

sarkis avatar

If you don’t mind pushing that to your PR I’ll grab my laptop so I can merge and release

Max Moon avatar
Max Moon

yeah sure one sec

Max Moon avatar
Max Moon

already done and done

Max Moon avatar
Max Moon

im headed out for the afternoon but updated default family and engine version to latest available via AWS documentation (family: redis4.0 and engine_version: 4.0.10) in [variables.tf](http://variables.tf) on the PR. Thanks again @sarkis

sarkis avatar

thanks @Max Moon! pushing out 0.5.0 now

Max Moon avatar
Max Moon

thanks!!!

sarkis avatar
cloudposse/terraform-aws-elasticache-redis

terraform-aws-elasticache-redis - Terraform module to provision an ElastiCache Redis Cluster

sarkis avatar

i thought it was a change that warranted a minor version++

Max Moon avatar
Max Moon

agreed, thanks again!

    keyboard_arrow_up