#announcements (2018-06)

cloudposse Cloud Posse Open Source Community <#CB84E9V54 geodesic> <#CB6GHNLG0 terraform> <#CBW0HJDS8 release-engineering> <#CB2PXUHLL random> <#CB9N1MMFV releases> <#CB7CA7X0D docs>

This channel is for workspace-wide communication and announcements. All members are in this channel. Archive: https://archive.sweetops.com

2018-06-30

Max Moon avatar
Max Moon

https://github.com/cloudposse/terraform-aws-elasticache-redis/pull/12 if anyone has time for a one line code review

Adding engine_version to elasticache replication group arguments by MoonMoon1919 · Pull Request #12 · cloudposse/terraform-aws-elasticache-redis

What it is Adds engine_version argument to the aws_elasticache_replication_group module. Why Previously when referencing this module in an external terraform template any attempt to override the en…

1
Max Moon avatar
Max Moon

Thanks in advance!

sarkis avatar
sarkis

@Max Moon thanks for the PR. Looks like the variables.tf entry was there but just wasn’t being passed?

Max Moon avatar
Max Moon

aha

sarkis avatar
sarkis

I’ll wait for 1 more approval and can merge. Is this blocking you right now?

Max Moon avatar
Max Moon

if i missed that then no big deal on this PR. No it’s not blocking me, i just referenced my branch.

Max Moon avatar
Max Moon

After reviewing [variables.tf> I think that despite being referenced in <http://variables.tf|variables.tf](http://variables\.tf) it wasn’t being picked up because the reference wasn’t made in the module, but i could be wrong

sarkis avatar
sarkis

Yea - I think CI would catch this (maybe) either way put a to-do for myself to get that PRed up

Max Moon avatar
Max Moon

it certainly was not working this morning, at least. So I think the PR for this is still relevant, pending review from others

sarkis avatar
sarkis

Oh absolutely your PR is necessary. Sorry should have gave context. I was confused why you didn’t have a variables.tf change - made sense when I looked at master. It has the unref variable on there.

Max Moon avatar
Max Moon

ah yes okay, i gotchya now

sarkis avatar
sarkis

So one last thing. Now I’m thinking if we default to 4.0

Max Moon avatar
Max Moon

i was thinking the same.

sarkis avatar
sarkis

Since that was the previous behavior it seems from your testing

sarkis avatar
sarkis

Let’s do that so it’s less of a backwards incompatible risk

sarkis avatar
sarkis

If you don’t mind pushing that to your PR I’ll grab my laptop so I can merge and release

Max Moon avatar
Max Moon

yeah sure one sec

Max Moon avatar
Max Moon

already done and done

Max Moon avatar
Max Moon

im headed out for the afternoon but updated default family and engine version to latest available via AWS documentation (family: redis4.0 and engine_version: 4.0.10) in [variables.tf> on the PR. Thanks again @sarkis</strong](http://variables.tf)

sarkis avatar
sarkis

thanks @Max Moon! pushing out 0.5.0 now

Max Moon avatar
Max Moon

thanks!!!

sarkis avatar
sarkis
cloudposse/terraform-aws-elasticache-redis

terraform-aws-elasticache-redis - Terraform module to provision an ElastiCache Redis Cluster

sarkis avatar
sarkis

i thought it was a change that warranted a minor version++

Max Moon avatar
Max Moon

agreed, thanks again!

2018-06-29

jamie avatar
jamie

Interesting

sarkis avatar
sarkis

so i was doing the whole compact concat dance before.. but i think this works in tf now…

alarm_actions       = ["${local.sns_topic_arn}", "${var.additional_notify_arns}"]
ok_actions          = ["${local.sns_topic_arn}", "${var.additional_notify_arns}"]
sarkis avatar
sarkis

where local.sns_topic_arn is a string and var.additional_notify_arns is a list

sarkis avatar
sarkis

hmm yea just tested it and it does work as expected… reference: https://github.com/hashicorp/terraform/issues/5682#issuecomment-278483475

Appending a value to an array · Issue #5682 · hashicorp/terraform

It seems to be tricky to append a value to an array. I thought that something like this should work: &quot;${compact(concat(split(&quot;,&quot;, var.security_groups), var.common_security_group))}&q…

jamie avatar
jamie

I saw that it was working

jamie avatar
jamie

like magic

sarkis avatar
sarkis

so much nicer than the old way

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Ember® Ceramic Mug

Beautifully designed to be used in your home or at your desk, the Ember® Ceramic Mug keeps your beverages at the perfect temperature from the first sip to the last drop. Buy the World’s First Temperature Control Mug™ at Ember.com.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

is that for you?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(via: @Max Moon)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
07:04:19 PM
jamie avatar
jamie

have you guys seen or worked with https://github.com/blinkist

Blinkist

GitHub is where people build software. More than 28 million people use GitHub to discover, fork, and contribute to over 85 million projects.

jamie avatar
jamie

some very thorough modules built by him (almost to @sarkis completeness grade)

jamie avatar
jamie
blinkist/terraform-aws-airship-ecs-service

terraform-aws-airship-ecs-service - Terraform module which creates an ECS Service, IAM roles, Scaling, ALB listener rules.. Fargate & AWSVPC compatible

sarkis avatar
sarkis

hmmm looks interesting - thanks for sharing @jamie

sarkis avatar
sarkis

at the very least - I need some diagrams for the ecs-web-app module … it’s getting huge

jamie avatar
jamie

Diagrams are annoying and always appreciated

maarten avatar
maarten

Hi fellas, I’m the Blinkist guy

jamie avatar
jamie

Welcome :)

maarten avatar
maarten

Few small bugs still here and there, but slowly getting there. I was discussing earlier with Jamie I think if it’s possible to have Terraform manage the ECS container definitions except for the image parameter.. Where Terraform runs would extract the image:tag from aws_ecs_container_definition. Only solution I can think of is by creating a bootstrap=true for the module which would turn on/off a datasource for the first run, and other logic on the second run.. Would need to play around with that..

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@maarten so I want to share our strategy for this. it’s not original and I’ve seen it mentioned in a few places.

jamie avatar
jamie

Guys, I invited that guy from the links I shared

maarten avatar
maarten

Thanks for the invite

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Welcome to the channel!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Nice work on those modules

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I am afk, but will share how we are doing it

jamie avatar
jamie

@maarten it’s Friday night for like most of the crew, and it’s quieter here over the weekend. But core group of @sarkis @Erik Osterman (Cloud Posse) sand @Andriy Knysh (Cloud Posse) are keeping an eye on it

jamie avatar
jamie

As seen

maarten avatar
maarten

Yeah sure.. Normally I’m not around on a Friday either … I’ll introduce myself quickly.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yes, things are pretty quiet now as it’s the end of the week

maarten avatar
maarten

I’m the founder of the Terraform User Group in Berlin, started it when I didn’t know where to start with Terraform roughly 2 years ago. Worked for 2.5 years at Blinkist where we went for AWS with ECS, coming from Heroku. Was always a dream to be self-employed, Blinkist fully supported that and made me a contractor with as first assignment open sourcing the ECS stuff I did earlier. Anothing thing I made is a golang tool called skipper which does some magic regarding task execution through a docker tunnel etc.etc. but that project is on hold until I get a bit of golang help.

jamie avatar
jamie

maarten avatar
maarten

@Erik Osterman (Cloud Posse) Yeah, happy to see some of that Have a good weekend.

jamie avatar
jamie

Welcome! Its 11:30pm for me too so I’m going offline as well. But we will chat soon.

maarten avatar
maarten

Sure, again, thanks for the invite.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yea let’s sync up on Monday !

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Also, maybe it would be interesting to have a weekly standing “round table” where we get together in a zoom room

:--1:2
sarkis avatar
sarkis

It might even be something to record and distribute podcast style…

sarkis avatar
sarkis

Welcome Maarten

2
maarten avatar
maarten

Thanks

maarten avatar
maarten

Can’t have enough slack channels

sarkis avatar
sarkis

I need all the ones I have open plus more for terraform/hcl workarounds

maarten avatar
maarten

Where is the hcl2 countdown channel ? ;)

2018-06-28

jamie avatar
jamie

I added an example and such to the terraform-aws-ecs-service-cloudwatch-sns-alarms init

sarkis avatar
sarkis

@jamie you are on a roll today - i need to know what brand/bean of coffee you having

jamie avatar
jamie

thank you

jamie avatar
jamie

Did you see the null label one?

sarkis avatar
sarkis

yes

jamie avatar
jamie

hehe

jamie avatar
jamie

It was very heavily refactored

jamie avatar
jamie

just to get consistency on the data types, and then outputs them in the old output style

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, that was a clever trick. hadn’t considered it, but i think it’s a necessary evil.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
HashiCorp Terraform 0.12 Preview

Terraform 0.12 focuses on major Terraform language improvements and will be released later this summer. We are communicating about Terraform 0.12 prior to release to highlight the …

2018-06-27

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Add context by osterman · Pull Request #25 · cloudposse/terraform-null-label

– THIS IS CURRENTLY A POC/RFC – what Support passing a label&#39;s context between label modules why DRY demo module &quot;label1&quot; { source = &quot;../../&quot; namespace = &quo…

2018-06-26

jamie avatar
jamie

I wish i liked writing READMEs

jamie avatar
jamie
cloudposse/terraform-aws-s3-logs-athena-query

terraform-aws-s3-logs-athena-query - A Terraform module that creates an Athena Database and Structure for querying S3 access logs

jamie avatar
jamie

It works. But is pending readme creation

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Ok - we can help with that

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Really not fond of heredocs. Can we move that to template files?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Perhaps named query should be a separate module that reads queries from a template file

jamie avatar
jamie

Whats wrong with heredocs?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Legibility of code

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Can not properly indent

jamie avatar
jamie

okay

jamie avatar
jamie

I can split it into its own sql folder

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

That would be perfect

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@jamie @Andriy Knysh (Cloud Posse) @Igor Rodionov - can we schedule some time to talk about standardizing how we use IAM permissions?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Igor Rodionov has some problems with one of our older modules on ecr and before he makes any changes, we thought it a good idea to come up with a standard

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@jamie - I know you’re working a lot with locking down IAM permissions these days

Igor Rodionov avatar
Igor Rodionov

I can provide an example of my vision on ecr module

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, that would be a good start

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we can CR that <— as a proposal

Igor Rodionov avatar
Igor Rodionov

and bring it to the table for meeting

jamie avatar
jamie

Sure. In brief: the iam_policy_documents can be overwritten, or appended to depending on if the statement has a SID. Combining that, with the concept of a per project ‘policy module’ such as https://github.com/cloudposse/terraform-aws-ssm-parameter-store-policy-documents Or an integrated policy module, so that as part of the current module, it outputs the json policy documents json, and sids, so they can be seen. And includes an option for including json documents that can be passed through for appending.

cloudposse/terraform-aws-ssm-parameter-store-policy-documents

terraform-aws-ssm-parameter-store-policy-documents - A Terraform module that generates JSON documents for access for common AWS SSM Parameter Store policies

jamie avatar
jamie

It will be more work, but allows the lowest possible permissions to be added by the developer. But still allows additional permissions to be included in the module for custom needs.

jamie avatar
jamie
source_json (Optional) - An IAM policy document to import as a base for the current policy document. Statements with non-blank sids in the current policy document will overwrite statements with the same sid in the source json. Statements without an sid cannot be overwritten.
override_json (Optional) - An IAM policy document to import and override the current policy document. Statements with non-blank sids in the override document will overwrite statements with the same sid in the current document. Statements without an sid cannot be overwritten.

2018-06-25

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Daren did you guys get kiam working? We started on this: https://github.com/cloudposse/geodesic/pull/163

Daren avatar
Daren

Actually started on it this week

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Our first stab was giving us 403 to the AWS metadata api

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Will get back to it today

Daren avatar
Daren

Are you deploying the agent to masters?

Daren avatar
Daren

also why no iptables or rbac?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We haven’t enabled rbac yet in our kops manifest

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Iptables false was just an attempt to get it working

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(It also happens to be the default)

Daren avatar
Daren

hm, I thought it was required. but the doc is a little vague

Daren avatar
Daren


An DNAT iptables rule is required to intercept the traffic. The agent is capable of adding and removing the required rule for you through use of the –iptables flag.

Daren avatar
Daren
uswitch/kiam

kiam - Integrate AWS IAM with Kubernetes

kubernetes/charts

charts - Curated applications for Kubernetes

Daren avatar
Daren

Wont this lead to the server being blocked by the agent

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

also, doesn’t [kubernetes.io/role](http://kubernetes\.io/role): master require a toleration? they don’t have one

Daren avatar
Daren

hmm

Daren avatar
Daren
failed to load system roots and no roots provided - TLS error · Issue #36 · uswitch/kiam

{&quot;generation.metadata&quot;:0,&quot;level&quot;&quot;error warming credentials: RequestError: send request failed\ncaused by: Post https://sts.amazonaws.com/

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

did it fix it for you?

Daren avatar
Daren

Going to try shortly

Daren avatar
Daren

Unfortunately we still have a mix of coreos + debian hosts. Not sure if the path is consistent

Daren avatar
Daren

Well I got the server up and running fine. However the agents are reporting this error:
error creating server gateway: error waiting for address being available in the balancer: context deadline exceeded

Daren avatar
Daren
Kiam server readiness/liveness probe issues (v2.8) · Issue #94 · uswitch/kiam

I have Kiam setup on a 3 master cluster - the server runs on two of the three masters with some readiness probes failing but the 3rd node is in a crash loop backoff with the liveness probe failing….

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ran into some issues

2018-06-24

jamie avatar
jamie

@joost another good item for you to read through this [WIP] [ETA: 30.6.2018] Terraform best practices https://github.com/antonbabenko/terraform-best-practices by @antonbabenko it links to examples of organizing code for multiple aws accounts such as you may need

antonbabenko/terraform-best-practices

terraform-best-practices - [WIP] [ETA: 30.6.2018] Terraform best practices

antonbabenko avatar
antonbabenko

“WIP” will have to be remaining for that repo until I am back from my vacation. I want to make it complete in September before I speak about it at conferences.

:--1:1
joost avatar
joost

Thx @antonbabenko for the great link

2018-06-23

jamie avatar
jamie

I added a thanks, a reference, and the MIT licence+copyright under the Apache2 license.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Looks like there are some extra brackets around the image. See GitHub rendering

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Missing period before “With many thanks”

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I would write [Anton Babenko](…link to github…]

jamie avatar
jamie
cloudposse/terraform-aws-cloudtrail-cloudwatch-alarms

terraform-aws-cloudtrail-cloudwatch-alarms - Terraform module for creating alarms for tracking important changes and occurances from cloudtrail.

jamie avatar
jamie

Is that correct?

antonbabenko avatar
antonbabenko

Looks good, but I would not use locals (in alarms.tf), but rather create a bunch of resources aws_cloudwatch_metric_alarm manually. This way, you can easily delete something in the middle of the list without recreating the rest of the list. This will be fixed in HCL2 and Terraform 0.12, but for now it is still an issue with ordered lists.

jamie avatar
jamie

@antonbabenko I’ve updated it to use the pretty readme, added dashboards, and added screenshots

jamie avatar
jamie

Although I’m going to keep the lists in locals because I find the compact layout easier to read. And the lists are static.

:--1:1
jamie avatar
jamie

I’m looking forward to hcl2 and getting ForExpressions https://github.com/hashicorp/hcl2/blob/master/hcl/hclsyntax/spec.md#for-expressions

hashicorp/hcl2

hcl2 - Temporary home for experimental new version of HCL

joost avatar
joost

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Hey Joost!

joost avatar
joost

Is there a way sending cloudtrail logs (from several customer aws accounts – all regions) to a specific cloudwatch log group at my security aws account?

joost avatar
joost

I want to use the module terraform-aws-cloudtrail-cloudwatch-alarms at my security aws account

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Hrmm.. we do something very similar, but ship them to an encrypted s3 bucket. What you propose sounds nice though.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think @jamie might have recently done what you want

joost avatar
joost

Cool. When you ship them to s3, what do you use for notifications? S3 events to lambda?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

With the s3 bucket, we are not doing any no notifications. :( just storing logs.

This is something I’d need to defer to @jamie who has been working on all the alarm modules and probably has done what you are trying to achieve.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

This a an area we just started investing in - so it will be a little bit before we have everything fully baked.

joost avatar
joost

Ok, I understand. Thx!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Daren would really appreciate you publishing it, i’m working through this doc https://d0.awsstatic.com/whitepapers/compliance/AWS_CIS_Foundations_Benchmark.pdf

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Jamie is working on implementing modules to address as much of this as possible.

:--1:1
jamie avatar
jamie

Aws KB has published the Iam policies, guidelines and such for centralizing the cloudwatch logs. I can pull the policies into the module as an option. And put the guide in the readme. So you can use the alerts on one account, but have it track and alert for all accounts.

joost avatar
joost

That would be awesome

2018-06-22

jamie avatar
jamie

Have any of you made a metadata index for s3 in terraform? like this? https://github.com/aws-samples/aws-big-data-blog/tree/master/aws-blog-s3-index-with-lambda-ddb

aws-samples/aws-big-data-blog

Contribute to aws-big-data-blog development by creating an account on GitHub.

jamie avatar
jamie

If no one has, @Erik Osterman (Cloud Posse) can you add it to me on your project board?

jamie avatar
jamie

Also: An aws alarm terraform module for detecting root account logins.

jamie avatar
jamie

And: An aws alarm terraform module for alerting on unauthorized aws api access.

jamie avatar
jamie

These are part of the aws CIS best practices.

jamie avatar
jamie

And should be small and easy to add

jamie avatar
jamie

And a module to deploy this task to a cluster. https://github.com/ExpediaDotCom/c3vis/blob/master/README.md

ExpediaDotCom/c3vis

c3vis - Visualize the resource utilisation of Amazon ECS clusters

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I will track those in the project later today

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Build software better, together

GitHub is where people build software. More than 28 million people use GitHub to discover, fork, and contribute to over 85 million projects.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Daren anything you can share for the root account logins?

jamie avatar
jamie

Thanks!

jamie avatar
jamie

No rush I just don’t want to forget them

Daren avatar
Daren

@Erik Osterman (Cloud Posse) Sure. Id prefer to touch it up before open sourcing it, it was hacked together to meet the pci deadline

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ok, understandable

Daren avatar
Daren

It does the following:

  • alert when a public is made public
  • alert on root login
  • alert on modification of cloudtrails
  • auto re-enable disabled cloudtrails
jamie avatar
jamie

a bucket is made public?

jamie avatar
jamie

@Daren would really appreciate you publishing it, i’m working through this doc https://d0.awsstatic.com/whitepapers/compliance/AWS_CIS_Foundations_Benchmark.pdf

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(we could also probably help with “clean up” if you want - PM)

antonbabenko avatar
antonbabenko
Daren avatar
Daren

Thanks this looks promising. We tackled our using cloudwatch event rule -> lambda -> pagerduty/slack/…

jamie avatar
jamie

those are great anton

jamie avatar
jamie

may I use them?

jamie avatar
jamie

@antonbabenko i just spotted something that may cause a headache for you…

jamie avatar
jamie
TeliaSoneraNorge/telia-terraform-modules

telia-terraform-modules - Terraform modules for creating cloud infrastucture

jamie avatar
jamie

You use aws_iam_policy_attachment

jamie avatar
jamie

instead of aws_iam_role_policy_attachment

jamie avatar
jamie

Incase it wasn’t intentional

antonbabenko avatar
antonbabenko

Absolutely, use anything from there. I quit from that company today

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

no way!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

what are you going to focus on now?

antonbabenko avatar
antonbabenko

there are some modules which I want to move to open-source under terraform-aws-modules umbrella

antonbabenko avatar
antonbabenko

long vacation first, and then find something related to Terraform & AWS. Have nothing settled yet

antonbabenko avatar
antonbabenko

Thanks for pointing to aws_iam_policy_attachment - I didn’t know that

antonbabenko avatar
antonbabenko

Guys who are from Russia or Europe, will you consider going to https://devoops.ru/en/ ? I am going there and will be talking about Terraform there.

DevOps conference. October 14, Saint Petersburg. More than 400 participants, 20 technical talks. attachment image

DevOps conference. October 14, Saint Petersburg. More than 400 participants, 20 technical talks.

:--1:1
sarkis avatar
sarkis

man I wanna goto St Petersburg if it wasn’t a 20+ hr flight i’d be buying a ticket lol

antonbabenko avatar
antonbabenko

Yeah, it is far from you. Will anyone go to HashiConf?

jamie avatar
jamie

If I was looking for work I would

:--1:1
jamie avatar
jamie

Alarm module made, with working example…. with some README

jamie avatar
jamie
cloudposse/terraform-aws-cloudtrail-cloudwatch-alarms

terraform-aws-cloudtrail-cloudwatch-alarms - Terraform module that notifies an endpoint such as SNS when cloudtrail logs unauthorised API access

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

make sure to add all proper attributions

cloudposse/terraform-aws-cloudtrail-cloudwatch-alarms

terraform-aws-cloudtrail-cloudwatch-alarms - Terraform module that notifies an endpoint such as SNS when cloudtrail logs unauthorised API access

jamie avatar
jamie

:–1: will do

2018-06-21

antonbabenko avatar
antonbabenko

Thanks @jamie for sharing links. inspec-gcp is just 3 months old project.

jamie avatar
jamie

2018-06-20

jamie avatar
jamie

@sarkis Next week I’ll have an ECS treat for you

jamie avatar
jamie
cloudposse/terraform-aws-ecs-ecs-spot-fleet

terraform-aws-ecs-ecs-spot-fleet - Terraform module to create a diversified spot fleet for ECS clusters

jamie avatar
jamie

Its pending an update to the SpotFleet for allowing Launch Templates

jamie avatar
jamie

A fraction of the cost of Fargate.

sarkis avatar
sarkis

Looks really interesting Jamie - are you going to have a % of spot instances or I guess spot fleet handles that? I’m thinking what happens when prices spike…

sarkis avatar
sarkis

Yep looks like it:
The Spot Fleet also attempts to maintain its target capacity fleet if your Spot Instances are interrupted due to a change in Spot prices or available capacity.

jamie avatar
jamie

well, the fleet will be made of a mix of instance types

sarkis avatar
sarkis

yea

jamie avatar
jamie

so, if one instance type goes out of price, it boots another

sarkis avatar
sarkis

last i tried anything with spot - there was no fleet

sarkis avatar
sarkis

or it was super new i forget why we didn’t look into it

jamie avatar
jamie

yeah, its new.

jamie avatar
jamie

and they have released autoscaling for spot fleet recently too

sarkis avatar
sarkis

it was pretty much unusable for anything mission critical for this reason

jamie avatar
jamie

so, you can create a spot fleet, and scale it

sarkis avatar
sarkis

really cool - looking forward to seeing the finished product!

jamie avatar
jamie

when combined with a watcher script that runs on the instance (which you will see if you look in the user-data dir)

jamie avatar
jamie

that starts ‘draining’ the instance as soon as it gets a spot instance termination notice

jamie avatar
jamie

and setting the ‘draining lifetime’ of tasks to 1 min 45 seconds

jamie avatar
jamie

it means that all of the containers running on the interupted instance get moved to another server before its removed

jamie avatar
jamie

so you can keep your workload stable

jamie avatar
jamie

its good for web sites, slightly less good for things that need to gaurentee connections for longer than 2 mins

jamie avatar
jamie

i have been running spot instances for years, and they have only rotated out a few times

:--1:2
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Very cool! I’d like to demo this for a company currently using spotinst.com

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think their spend is like 800K/mo on AWS :-)

sarkis avatar
sarkis

Jamie, re: ssm param store policy docs PR, put in a comment in there - lmk if you have any Qs about the process for that stuff… I’m working on moving more of our repos to that README.yaml format

jamie avatar
jamie

Thank you @sarkis

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

btw, added #releases where we’ll announce releases of our projects

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
05:38:03 PM
sarkis avatar
sarkis

Jamie - I know it’s getting late there so not doing the @ pings , do you have any prior art or tips for ECS Fargate CloudWatch monitoring - we obviously don’t have/care about the ECS cluster health, but starting to look into what we can monitor in the containers themselves - basic mem, cpu ,etc.

jamie avatar
jamie

@sarkis Yo, keep the pings coming

jamie avatar
jamie

I’ve done the README update that you suggested

sarkis avatar
sarkis

ah great -

jamie avatar
jamie

Okay so…

jamie avatar
jamie
bitflight-public/terraform-aws-ecs-events

terraform-aws-ecs-events - Add on an SNS topic for capturing ECS events

sarkis avatar
sarkis

I think this is great… gives me a nudge on where to look and I can just borrow this as a base (and give you credit of course), if you don’t mind

bitflight-public/terraform-aws-ecs-events

terraform-aws-ecs-events - Add on an SNS topic for capturing ECS events

jamie avatar
jamie

Can you use the format of the lambda one I posted? So that all the new alarm modules can work the same?

sarkis avatar
sarkis

Yea - I’m reviewing that one now as well

sarkis avatar
sarkis
cloudposse/terraform-aws-cloudwatch-alarms-lambda

terraform-aws-cloudwatch-alarms-lambda - Terraform module for creating a set of Lambda alarms and outputting to an endpoint

jamie avatar
jamie

Exactly. I’ve made the repo now..

jamie avatar
jamie

Thats a minor one…

jamie avatar
jamie

but when do you need it by? I’ll make a fuller one

jamie avatar
jamie
cloudposse/terraform-aws-cloudwatch-alarms-lambda

terraform-aws-cloudwatch-alarms-lambda - Terraform module for creating a set of Lambda alarms and outputting to an endpoint

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@jamie I reviewed your terraform-aws-ssm-parameter-store-policy-documents - just a few nit picks

:--1:1
sarkis avatar
sarkis

Also just curious Jamie - what’s your thoughts on the README generation process

jamie avatar
jamie

It’s annoying. A form would be good ;)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

what part of it could be improved?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)


A form would be good

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

in otherwords, an “init” process?

jamie avatar
jamie

I’ll elaborate shortly. Just walking somewhere

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

:–1:

jamie avatar
jamie

Hi, I’m back.

jamie avatar
jamie

So in regards to the README.yaml it isn’t very DRY

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

which parts?

Robert Tisdale avatar
Robert Tisdale

hello all

Robert Tisdale avatar
Robert Tisdale

Can’t remember if I said hello

jamie avatar
jamie

\# Name of this project
name: terraform-aws-ssm-parameter-store-policy-documents


\# Logo for this project

\#logo: docs/logo.png


\# License of this project
license: "APACHE2"


\# Canonical GitHub repo
github_repo: cloudposse/terraform-aws-ssm-parameter-store-policy-documents


\# Badges to display
badges:
  - name: "Build Status"
    image: "<https://travis-ci.org/cloudposse/terraform-aws-ssm-parameter-store-policy-documents.svg?branch=master>"
    url: "<https://travis-ci.org/cloudposse/terraform-aws-ssm-parameter-store-policy-documents>"
  - name: "Latest Release"
    image: "<https://img.shields.io/github/release/cloudposse/terraform-aws-ssm-parameter-store-policy-documents.svg>"
    url: "<https://github.com/cloudposse/terraform-aws-ssm-parameter-store-policy-documents/releases>"
  - name: "Slack Community"
    image: "<https://slack.cloudposse.com/badge.svg>"
    url: "<https://slack.cloudposse.com>"

related:
  - name: "terraform-aws-ssm-parameter-store"
    description: "AWS SSM Parameter Store module"
    url: "<https://github.com/cloudposse/terraform-aws-ssm-parameter-store>"




\# Short description of this project
description: |-
  This module generates JSON documents for restricted permission sets for AWS SSM Parameter Store access.
  Helpful when combined with [terraform-aws-ssm-parameter-store](<https://github.com/cloudposse/terraform-aws-ssm-parameter-store>)

\# How to use this project

examples: |-
  Create a policy that allows access to write all parameters
hcl
  module "ps_policy" {
    source = "git::<https://github.com/cloudposse/terraform-aws-ssm-parameter-store-policy-documents.git?ref=master>"
  }
Robert Tisdale avatar
Robert Tisdale

Don’t want to derail

Robert Tisdale avatar
Robert Tisdale

Holy paste batman

jamie avatar
jamie

sorry Robert!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

hey @Robert Tisdale!

jamie avatar
jamie

bad syncronisity

1
Robert Tisdale avatar
Robert Tisdale

LOL, no problem I’m just yanking your chain @jamie

Robert Tisdale avatar
Robert Tisdale

Hey @Erik Osterman (Cloud Posse)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

can you propose (in pseudo code) how the YAML should look instead?

jamie avatar
jamie

@Erik Osterman (Cloud Posse) the terraform-aws-ssm-parameter-store-policy-documents (project name) gets used over and over

jamie avatar
jamie

and in the intro areas especially, it would help if it was a variable

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

unfortunately, don’t see a way around that without recursion.

sarkis avatar
sarkis

ah… it might turn into a case of mutliple levels of interpolation though since we already render this - thinking something like terraforms $$

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
-rw-r--r--    1 erik  staff  11291 Jun 20 12:50 README.md
-rw-r--r--    1 erik  staff   5429 Jun 20 12:50 README.yaml

but it’s at least 50% more DRY

jamie avatar
jamie

yeah

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

anyways, thanks for the feedback. I think we can take over some of the “cleanup” burden to keep you focused on coding

sarkis avatar
sarkis

whoa didn’t bother to look at that - nice

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but there’s a stronger, more overwhelming reason we cannot do that

sarkis avatar
sarkis

to me the biggest bonus is to only ever have to update the legal jargon in one spot

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

the purpose for the .yaml format is so we can include it in our docs portal (docs.cloudposse.com)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

if the .yaml was itself templated, we would then need to create multiple parsers

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

one for readmes and one for hugo

Robert Tisdale avatar
Robert Tisdale

Why use YAML for that instead of just markdown?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

to enforce consistency

:--1:1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so basically, in hugo (our docs CMS) we can do this:

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
{{% include-github org="cloudposse" repo="terraform-root-modules" ref="0.1.5" file="codefresh.yml" title="Example codefresh.yml" language="yaml" type="code-block" %}}
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

this allows us to pull documentation in from individual repos

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

the hardest part about writing documentation is keeping it up to date

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

by allowing individual repositories to manage the general usage/examples/inputs/outputs/etc, we can reference that in our main documentation portal via YAML and ensure it’s up to date in both places

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

“documentation as code”

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

jamie avatar
jamie

I have started to look into doing TDD with terraform.

jamie avatar
jamie

My current opinion is not to do it

jamie avatar
jamie

but to do validations as part of a test process

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I would like to do a compromise

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i think writing full on tests for everything is overkill

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but running plan/apply/destroy 80% of the battle

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i want to use our [testing.cloudposse.co](http://testing\.cloudposse\.co) account for that

jamie avatar
jamie

its also fucking horrid, terraform is not at the stage to validate that beyond plan

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

and use codebuild with codepipeline to automate that

sarkis avatar
sarkis

@jamie something i’ve been pondering, more than TDD is RDD (Readme Driven Development)

jamie avatar
jamie

yeah, well… Ive been reading on awspec, kitchen-terraform, goss, and inspec for the last 2 weks

jamie avatar
jamie

weeks

jamie avatar
jamie

@sarkis im in to that

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@antonbabenko will probably have some more insights on it <– i know he uses kitchen-terraform

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but @jamie what do you think about the plan/apply/destroy part?

jamie avatar
jamie

I think that there is a portion that could be done, and i think there could be a ‘minimum’ that can be tested for at ‘unit test’ time

jamie avatar
jamie

@Erik Osterman (Cloud Posse) i think the plan/apply/destroy part could take too long as a ‘unit-test’ but as a further along test it works

sarkis avatar
sarkis

you think so? i really feel like integration tests are going to be the bulk of the tests if and when we get to it

sarkis avatar
sarkis

i dont know how useful unit tests are is what im tying to say

jamie avatar
jamie

I agree. Its what I was attempting to articulate too.

jamie avatar
jamie

I am not from a recent dev background. The unit test part allows for testing as you create quickly.

jamie avatar
jamie

But Integration tests can be done… quite well with terraform

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

our modules are breaking frequently due to terraform changes outside of our control

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

this is why I want plan/apply/destroy

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

the time to run the tests isn’t a concern

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

it’s very frequent in software projects that tests taken 30-45 minutes or even more

jamie avatar
jamie

I think we can get that bit…at least

jamie avatar
jamie

although they take longer to run, and can be a lot of work to write

sarkis avatar
sarkis

anyone check out terratest yet?

jamie avatar
jamie

I read about terratest yesterday… did you read about it on reddit?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

they had a big announcement a couple of months ago

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

it’s by Grunt Works (terragrunt)

sarkis avatar
sarkis

i had talked to Yevgeniy about it as I helped bug squash a bit on terragrunt… he was telling me they had it ready to release but hashicorp kept telling him to hold off because they have something they want to come out with (surprise, they are all over the place :P)

sarkis avatar
sarkis

so yea officially publicly available a couple months ago as erik said ^

jamie avatar
jamie

Like, Terraform’s ‘issues’ list is huge. But i don’t like them any less.

jamie avatar
jamie

They are the best of the crowd

sarkis avatar
sarkis

oh yea - i just mean they should just let the community run with some things - it might help them focus on the core

sarkis avatar
sarkis

but i can see the other side of that coin too

sarkis avatar
sarkis

nice to have control (enforce quality)

antonbabenko avatar
antonbabenko

I normally don’t use kitchen-terraform and will not use terratest because it is very hard to make meaningful tests. hcl/tf files are not code, so I don’t think we need just-developer-friendly-tool. I think we need something what is more like DSL (in a form of BDD/Gherkin). Also, there will be a validation of values before apply, and a lot more…

antonbabenko avatar
antonbabenko
hashicorp-terraform/gardening

Where developers come to talk.

sarkis avatar
sarkis

something like serverspec but for aws resources lol

jamie avatar
jamie
:point_up_2:1
antonbabenko avatar
antonbabenko

I wish something more abstract.

jamie avatar
jamie

Well, if there was a terraform for testing that was multi provider enabled…

antonbabenko avatar
antonbabenko

For example, when I was looking for something like inspec for google provider a year and more ago I could not, and I don’t know if there is awspec for google now.

sarkis avatar
sarkis

true

antonbabenko avatar
antonbabenko

From what I see, there is 70% of testing things for AWS, 30% for Azure, and 0% for Google.

jamie avatar
jamie

Google have a test system… I found it when researching this week..

jamie avatar
jamie

let me find it

jamie avatar
jamie

Its not quite right but this goes towards it https://github.com/inspec/inspec-gcp

inspec/inspec-gcp

inspec-gcp - InSpec GCP (Google Cloud Platform) Resource Pack

jamie avatar
jamie

Are you guys using any pentest bots for testing code and infrastructure as part of the deploy pipeline?

jamie avatar
jamie

https://github.com/guardicore/monkey <– I’ve been watching this repo

guardicore/monkey

monkey - Infection Monkey - An automated pentest tool

jamie avatar
jamie

but haven’t used it yet

jamie avatar
jamie

And it looks very thorough

jamie avatar
jamie

I actually expect very few clients to pass muster

sarkis avatar
sarkis

hmm that is nice - thanks for sharing it @jamie - starred to dig into on my free time

sarkis avatar
sarkis

the only thing that worries me with this stuff is getting an aws account suspended - i’ve heard some horror stories

sarkis avatar
sarkis

not specifically about this but just in general

jamie avatar
jamie

Well, yeah… if you are doing anything that is DOS related

jamie avatar
jamie

you have to inform AWS

sarkis avatar
sarkis

right

jamie avatar
jamie

its on their TOS

sarkis avatar
sarkis

like i know security firms need to have a special deal with AWS and follow some standards, etc

jamie avatar
jamie

So, if you are running this bot and doing any DDOS or DOS features you have to fill in the AWS Support request first

jamie avatar
jamie

and say that is what you are doing

jamie avatar
jamie

But this bot, doesn’t just brute force

jamie avatar
jamie

So, its likely it will be outside of the AWS security remit

jamie avatar
jamie
Infection Monkey attachment image

Infection Monkey is an open source breach and attack simulation tool to evaluate the security posture of your network

sarkis avatar
sarkis

yea it looks more like discovery

jamie avatar
jamie

aws marketplace even allows it

:--1:1
jamie avatar
jamie

lol

2018-06-19

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
07:47:03 PM
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

being able to easily identify out-of-date modules and automatically generate all the output definitions from nested modules is something I’ve wanted

jamie avatar
jamie

Did any of the modules i made help with the ecs web app?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think we used your ecs codepipline changes

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@sarkis would know more

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i can walk you through what we have this week sometime

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we’re showing the client how to use it right now

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

and i think sarkis also used your ecs task module as an initial starting off point

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i’d be really interested in your feedback on our ecs-web-app module <— from an interface perspective

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

does it make sense?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we’ll probably create an ecs-job-app soon <— which wouldn’t depend on an ALB

jamie avatar
jamie

I read through it earlier very briefly and I thought it was well made from what I saw. I’ll give it a test later on - :)

sarkis avatar
sarkis

yea the codepipeline and codebuild code you contributed - thanks a ton Jamie!

sarkis avatar
sarkis

I’ll make sure you get credit in the READMEs for those projects

rohit.verma avatar
rohit.verma

In my init-terraform scripts, I am seeing mount -a failing to mount s3 to local

rohit.verma avatar
rohit.verma
(dev-admin) backing-services ➤ init-terraform 
/bin/sh: s3fs: not found
rohit.verma avatar
rohit.verma

I was looking at previous chats, seems like this is common

sarkis avatar
sarkis

what version of build-harness?

sarkis avatar
sarkis

and geodesic

rohit.verma avatar
rohit.verma

i am using a fork of geodesic,

rohit.verma avatar
rohit.verma

Also I am not sure where we are putting build-harness version, I think its always master

rohit.verma avatar
rohit.verma

<https://raw.githubusercontent.com/cloudposse/build-harness/master/templates/Makefile.build-harness>

rohit.verma avatar
rohit.verma

My geodesic is mostly in sync with master

sarkis avatar
sarkis

@rohit.verma what does which s3fs return? (from inside geodesic container)

rohit.verma avatar
rohit.verma
 s3fs
Error: goofys takes exactly two arguments.
rohit.verma avatar
rohit.verma

i am on goofys v0.19.0

sarkis avatar
sarkis

looking at the s3fs on my geodesic container…

cat /usr/local/bin/s3fs

\#!/bin/bash

\# This script is used to mount filesystems from `/etc/fstab`
export DEVICE="$(eval echo ${1})"
export MOUNT_POINT="$(eval echo ${2})"
export BUCKET_REGION="${BUCKET_REGION:-${AWS_REGION}}"


\# Evaluate the bash envs passed in via args and then execute goofys
exec goofys --file-mode=0600 --dir-mode=0700 --region=${BUCKET_REGION} --sse --acl=private ${DEVICE##s3://} ${MOUNT_POINT}
sarkis avatar
sarkis

probably an issue with one of the last 2 args there ^ not being set

sarkis avatar
sarkis

not sure yet why - tried to reproduce to no avail…

rohit.verma avatar
rohit.verma

in my dockerfile I have added just RUN s3 fstab '${TF_BUCKET}' '/' '/secrets/tf'

sarkis avatar
sarkis

@rohit.verma do you have ENV TF_BUCKET set before somewhere in the Dockerfile?

rohit.verma avatar
rohit.verma

yes

rohit.verma avatar
rohit.verma

running s3 fstab '${TF_BUCKET}' '/' '/secrets/tf' directly in container gives no error

rohit.verma avatar
rohit.verma

I think ideally it should mount the terraform bucket to /secrets/tf

rohit.verma avatar
rohit.verma

but that is still empty

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We can look into this together

2018-06-18

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/terraform-aws-ecs-web-app

terraform-aws-ecs-web-app - A Terraform module which implements a web app on ECS and supporting AWS resources.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

this implements the common pattern of provisioning an ECR integrated with CI/CD to deploy containers to an ECS task and adding an ALB ingress

mcrowe avatar
mcrowe

That’s pretty sweet – needed that 2w ago

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but it’s also just a prime example of using all other ecs modules

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/terraform-aws-ecs-web-app

terraform-aws-ecs-web-app - A Terraform module which implements a web app on ECS and supporting AWS resources.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(realize it’s a bit late)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

also, this uses our “readme generator” to convert a README.yaml (using gomplate) to a README.md

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/terraform-aws-ecs-web-app

terraform-aws-ecs-web-app - A Terraform module which implements a web app on ECS and supporting AWS resources.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

as part of this, it uses terraform-docs to generate the wellformed markdown

mcrowe avatar
mcrowe

I scheduled a quick call with you tomorrow about this. I think what I did has a different application

:--1:1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ok, look forward to talking at 10:30

2018-06-17

mcrowe avatar
mcrowe

@sarkis do you have a minute? I’m trying to use terraform-aws-ecs-alb-service-task but having a weird problem

sarkis avatar
sarkis

sure whats up

mcrowe avatar
mcrowe

So, using terraform-aws-ecs-alb-service-task, the nginx container deploys and comes up (I see the default page). However, when my custom container deploys, it ends up dying

sarkis avatar
sarkis

anything in the ECS service logs/container output?

mcrowe avatar
mcrowe

what’s strange is I think it’s a health check

mcrowe avatar
mcrowe
04:31:35 PM
mcrowe avatar
mcrowe

I’ve ssh’d into the container (fixed various environment/permission issues I had), and I can run the container manually on the ECS host with docker run …..

sarkis avatar
sarkis

does it go into a loop of starting/stopping the container?

mcrowe avatar
mcrowe

yep

sarkis avatar
sarkis

i think you are right in that it sounds like the scheduler is giving up after failed healthcheck threshold and just keeps trying to start a “healthy” container

mcrowe avatar
mcrowe

I’ve bumped up the timeout on the alb:

  health_check_timeout  = 20
  health_check_interval = 30
mcrowe avatar
mcrowe

Plus, my target groups look like this in the EC2 console:

mcrowe avatar
mcrowe
04:35:14 PM
mcrowe avatar
mcrowe

(the healthy one is the nginx controller – the others are failed deployments of my container)

sarkis avatar
sarkis

do you have your target group tf resource handy?

sarkis avatar
sarkis

i ran into something similar when i wasn’t pointing my target resources to the right healthcheck endpoint

sarkis avatar
sarkis

endpoint/port

mcrowe avatar
mcrowe
04:38:10 PM
sarkis avatar
sarkis

and the above unhealthy status seems to reaffirm this is where the issue is…

mcrowe avatar
mcrowe

oh, traffic port – is this supposed to be ip?

sarkis avatar
sarkis

what is on that custom container?

sarkis avatar
sarkis

i dont think there is any http server on your custom container*

mcrowe avatar
mcrowe

a microservice. if you GET /, it returns a JSON document of the status

sarkis avatar
sarkis

ah ok… i think i know something else to look into.. does this container take a bit to load?

mcrowe avatar
mcrowe

yeah, it does

sarkis avatar
sarkis

so there is a grace period

sarkis avatar
sarkis

before ECS will start considering failed healthchecks as unhealthy

mcrowe avatar
mcrowe

ah, good point.

sarkis avatar
sarkis

let me find it

sarkis avatar
sarkis
Task Definition Parameters - Amazon Elastic Container Service

Task definitions are split into separate parts: the task family, the IAM task role, the network mode, container definitions, volumes, task placement constraints, and launch types. The family is the name of the task, and each family can have multiple revisions. The IAM task role specifies the permissions that containers in the task should have. The network mode determines how the networking is configured for your containers. Container definitions specify which image to use, how much CPU and memory the container are allocated, and many more options. Volumes allow you to share data between containers and even persist the data on the container instance when the containers are no longer running. The task placement constraints customize how your tasks are placed within the infrastructure. The launch type determines which infrastructure your tasks use.

sarkis avatar
sarkis

startPeriod The optional grace period within which to provide containers time to bootstrap before failed health checks count towards the maximum number of retries. You may specify between 0 and 300 seconds. The startPeriod is disabled by default.

sarkis avatar
sarkis

that’s my guess right now - is that the container is taking a bit to start.. healthchecks failing and getting into a loop there

sarkis avatar
sarkis
cloudposse/terraform-aws-ecs-container-definition

terraform-aws-ecs-container-definition - A Terraform module to generate well-formed JSON documents (container definitions) that are passed to the aws_ecs_task_definition Terraform resource

sarkis avatar
sarkis

which reminds me to fix the README on that ^

mcrowe avatar
mcrowe

well, i don’t think that’s in the terraform-aws-ecs-alb-service-task module yet, is it?

sarkis avatar
sarkis
cloudposse/terraform-aws-ecs-alb-service-task

terraform-aws-ecs-alb-service-task - A Terraform module which implements an ECS service which exposes a web service via ALB.

sarkis avatar
sarkis

i’m in the process of cleaning this all up and things may change :disappointed: - but all that module does is allow you to create a json doc out of terraform code - it’s not required to use it … you can just modify your .json for the container_definition_json

mcrowe avatar
mcrowe

btw, did you see the issue I submitted about the environment variables? I had to just use pure .json files because I couldn’t pass environment variables in

sarkis avatar
sarkis

did not, until now.. I just assigned to myself and will reproduce and see if we can workaround that

sarkis avatar
sarkis

lmk if the startPeriod fixes your issue - now I’m about it

mcrowe avatar
mcrowe

testing it now

mcrowe avatar
mcrowe

BTW, another thin I ran into but didn’t report yet is this: we need the ability to customize the task security policy (we needed to grant permissions to SSM with a specific arn). I hacked it like this in the module (it’s not ideal, but I think you get the idea):

data "aws_iam_policy_document" "ecs_execution_role" {
  source_json = "${var.custom_policy_document}"

  statement {
    sid       = "vLMpjauEwiCGsAJ9tJKsbSgn"
    effect    = "Allow"
    resources = ["*"]

    actions = [
      "ecr:GetAuthorizationToken",
      "ecr:BatchCheckLayerAvailability",
      "ecr:GetDownloadUrlForLayer",
      "ecr:BatchGetImage",
      "logs:CreateLogStream",
      "logs:PutLogEvents",
    ]
  }
}
sarkis avatar
sarkis

hmm yea that makes sense and should be an option to pass in our own existing or managed security policy (if not provided then generate)

sarkis avatar
sarkis

would you mind opening an issue @mcrowe if not, I can

mcrowe avatar
mcrowe

I’ll do it later today if that’s ok

sarkis avatar
sarkis

that works - i don’t know if i can get to anything today - so would be tomorrow at the earliest after client work

mcrowe avatar
mcrowe

that may have done it – not getting an OOM error (so further along in the chain)

mcrowe avatar
mcrowe

btw, do you know how to kill an ecs codepipeline deploy when it fails?

mcrowe avatar
mcrowe

@sarkis Adding:

health_check_grace_period_seconds  = 30

to my local copy of terraform-aws-ecs-alb-service-task fixed the issue

3
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

PR?

antonbabenko avatar
antonbabenko
06:46:40 PM

@antonbabenko has joined the channel

2

2018-06-16

mcrowe avatar
mcrowe

Hey guys, I’m finding a need to document a geodesic module for internal use. For example, the inputs needed, but also I need to export the outputs from the modules included. For example:

module "ecs_alb_service_task" {
  source                    = "git::<https://github.com/cloudposse/terraform-aws-ecs-alb-service-task.git?ref=0.1.0>"
...

However, I typically want the outputs of this alb in my state, so I need to re-output the CP module’s output like this:

output "app_service_name" {
    value = "module.ecs_alb_service_task.service_name"
    description = "ECS Service name."
}

Check this out: https://gist.github.com/mike-zipit/fa936438c1edbb5f26ba68eec1cffaf8 This is WIP, but would like to validate if this is a need others see

:--1:2
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

This is definitely a need as the number of module deps grow. I think @Andriy Knysh (Cloud Posse) and @jamie will dig it.

1
jamie avatar
jamie

Thank you for helping to remove the tedium

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Support for HCL (Terraform Modules and Providers) · Issue #118 · dependabot/feedback

what Open PRs against terraform repos when terraform modules have new releases Open PRs against terraform repos when terraform providers have new releases why It&#39;s an extremely diverse ecosyste…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Looks in line with some of things we are doing for readme generation

2018-06-15

jamie avatar
jamie

@Erik Osterman (Cloud Posse) you May have noticed the two new alerts repos I put in.

jamie avatar
jamie

They are 90% done codewise. Needs readme done.

jamie avatar
jamie

I’m going to rename all the repos so that the alert subject is last. So that they sort nicely.

:--1:1
jamie avatar
jamie

And in the latest repos I’ve added ‘additional endpoints’ variables

jamie avatar
jamie

So instead of only having an sns topic, you can pass in autoscaling groups as endpoints, or lambda functions as endpoints.

:--1:1
jamie avatar
jamie

So I’ll add that feature to the others too

:--1:1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

thanks sweet!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@mcrowe was just telling me that tackling cloudwatch alerts of the type you’re creating are next up on his list of things todo

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

thanks @jamie! do you want us to review anything yet?

jamie avatar
jamie

I’m doing a demo of them to my client next week, so I should have the code to 100 by Tuesday. I’ll do my pull requests then. :)

jamie avatar
jamie

@mcrowe if you put yours off till next week I’ll have the template sorted. And we can both continue making alerts modules

jamie avatar
jamie

The alerts modules are for: Awareness - such as slack. Automation- such as asg scaling Deployments- such as rolling back a codedeploy if the alert is triggered on a new version

jamie avatar
jamie

I’ll do a push to my branches in a moment so you can see what’s been done

LeoGmad avatar
LeoGmad
05:00:03 PM

@LeoGmad has joined the channel

1
jamie avatar
jamie

@Erik Osterman (Cloud Posse) and @mcrowe and the rest, I’m wanting to do a training day for my clients engineering team

jamie avatar
jamie
firmstep-public/trainingdayone

trainingdayone - The repo containing the training Terraform configurations

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@mike.ballou @dave.yu might be something for joany

firmstep-public/trainingdayone

trainingdayone - The repo containing the training Terraform configurations

jamie avatar
jamie

I’ve just written this up today

Purva avatar
Purva
06:12:28 PM

@Purva has joined the channel

2
jamie avatar
jamie

If anyone wants to use it you’re welcome, and if anyone has any good inputs for it, please let me know.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

that’s cool

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we’re starting to do more training. we’ll need to develop more materials like this.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Purva, @chris have you guys tried/started doing any terraform training?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@jamie looks like a good start. Inevitably, a big project you’re taking on to teach this stuff.

jamie avatar
jamie

Totally big job. I need to get their team of 14 introduced to what Terraform can do.

jamie avatar
jamie

I don’t need to train to pro level. But I do need to get them thinking about what they can accomplish with Terraform

jamie avatar
jamie

Day one is being filled out, and will likely need more READMEs

jamie avatar
jamie

Day two is taking a more real world example, and working through that.

jamie avatar
jamie

In this case it will be creating a CI/CD pipeline for Serverless deployments

jamie avatar
jamie

With updating a lambda function, and API Gateway, and doing a rolling deploy with monitored events which would cause roll back.

jamie avatar
jamie

Once training week is done, i’ll be back to the modules…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

would love to incorporate something like this eventually into docs.cloudposse.com

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Incorporate Terraform Training Materials · Issue #140 · cloudposse/docs

what Add materials for holding trainings why Needed for many customer engagements references https://github.com/firmstep-public/trainingdayone

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

jamie avatar
jamie

Well great!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/terraform-aws-lb-s3-bucket

terraform-aws-lb-s3-bucket - Terraform module to provision an S3 bucket with built in IAM policy to allow AWS Load Balancers to ship access logs

jamie avatar
jamie

One more fargate tip: no more than 20 tasks running per account per region

jamie avatar
jamie
Amazon ECS Service Limits - Amazon Elastic Container Service

The following table provides the default limits for Amazon ECS for an AWS account which can be changed. For more information on the service limits for other AWS services that you can use with Amazon ECS, such as Elastic Load Balancing and Auto Scaling, see

tamsky avatar
tamsky


which can be changed

tamsky avatar
tamsky

have you tried requesting a limit increase, or is that already on your path forward?

jamie avatar
jamie

Hasn’t affected me yet. Eric and @sarkis have been working on an ecs fargate module for a client, so I was hoping to point it out!

1
sarkis avatar
sarkis

@jamie thanks!

sarkis avatar
sarkis

i bet they’d raise the limit like any other - just depends if you have biz level support on how quickly they get around to it

sarkis avatar
sarkis

highly recommend this for clients that have business level+ support btw: https://aws.amazon.com/premiumsupport/trustedadvisor/ - one of the things it does is track if you get close to service limits

AWS Support – Trusted Advisor

Learn about the cost savings, security, and performance features and benefits of AWS Trusted Advisor.

1
sarkis avatar
sarkis

Other useful info includes reserved instance allocation and idle ebs volumes

2018-06-14

Robert Tisdale avatar
Robert Tisdale
10:06:59 AM

@Robert Tisdale has joined the channel

alex avatar
alex
02:39:54 PM

@alex has joined the channel

alebabai avatar
alebabai
03:12:31 PM

@alebabai has joined the channel

dave.yu avatar
dave.yu
05:19:40 PM

@dave.yu has joined the channel

yurchenko avatar
yurchenko
05:31:46 PM

@yurchenko has joined the channel

evan avatar
evan
06:24:17 PM

@evan has joined the channel

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@tamsky I had a long chat with @mcrowe today. lots of great feedback. one of the things that came up was using a dotfile manager. You recently shared your dotfiles. is this something you would use?

tamsky avatar
tamsky

I’d need to understand what the “dotfile manager” would do?

tamsky avatar
tamsky

w.r.t. geodesic and friends

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
TheLocehiliosan/yadm

yadm - Yet Another Dotfiles Manager

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so there’s a convention/interface for adding shell customizations (E.g. aliases, prompts, etc)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
dotphiles/dotphiles

dotphiles - A community driven framework of dotfiles.

tamsky avatar
tamsky

conventions would be awesome. the lack of .inputrc in the current geodesic has me adding ADD <https://raw.githubusercontent.com/play-with-docker/play-with-docker/master/dockerfiles/pwm/.inputrc> /conf/.inputrc to many sub-Dockerfiles

tamsky avatar
tamsky

I think the interface is the important part

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i’m actually not familiar with .intputrc, which is probably why it’s not there

tamsky avatar
tamsky

I’m trying to figure out how I’d set some default for PROMPT_STYLE in my OSX shell that would get honored in geodesic shells

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

consider passing --env-file?

tamsky avatar
tamsky

I know I heard once here in slack that someone wanted $PWD mapped into their geodesic shell.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/geodesic

Geodesic is the fastest way to get up and running with a rock solid, production grade cloud platform https://docs.cloudposse.com/geodesic/

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

could also introduce something like a ~/.geodesic/env file

tamsky avatar
tamsky

sure, have the docker run wrapper test for the file ~~/.geodesic-env and add the flag --env-file ~~.geodesic-env if it exists

:--1:1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Add Support for Custom Prompts · Issue #150 · cloudposse/geodesic

what Add support for PROMPT_STYLE environment variable that alters the behavior of the shell prompt why The current utf8 heavy prompt breaks some terminals Users have mentioned they&#39;d prefer a …

Add Support for Automatic Env File · Issue #149 · cloudposse/geodesic

what Look for a ~/.geodesic/env file and pass it as –env-file, if found why Provide a way to set some defaults for all geodesic shells.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@mcrowe added this to track it: https://github.com/cloudposse/geodesic/issues/148

Add Support for a Dotfile Manager · Issue #148 · cloudposse/geodesic

what Provide a standard interface for shell extensions Re-use some existing standard why Allow users to define their own aliases, prompts, inputrcs, etc references https://github.com/dotphiles/dotp

tamsky avatar
tamsky

Erik, does this flag and logic that we just imagined we need, already exist:: https://github.com/cloudposse/geodesic/blob/master/rootfs/templates/wrapper#L72-L74

  if [ -n "${ENV_FILE}" ]; then
    DOCKER_ARGS+=(--env-file ${ENV_FILE})
  fi
cloudposse/geodesic

Geodesic is the fastest way to get up and running with a rock solid, production grade cloud platform https://docs.cloudposse.com/geodesic/

tamsky avatar
tamsky

# ENV_FILE=~/.geodesic/env geodesic looks like it works

tamsky avatar
tamsky

do we want to also test for a default file and add it to DOCKER_ARGS[@]?

tamsky avatar
tamsky
  local gedesic_env_file="~/.geodesic/env"
  # use our default env file if it exists                                                                 
  if [ -f ${geodesic_env_file} ]; then
      DOCKER_ARGS+=(--env-file=${geodesic_env_file})
  fi
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yea, so I think checking for a default is what we could add

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I’d make this: GEODESIC_DEFAULT_ENV_FILE or something. so even that can be overridden

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

e..g for different projects

sarkis avatar
sarkis

also @mcrowe if you don’t mind making an issue around the region issue from yesterday… I do want to take a look but it’s looking more like this weekend the earliest

sarkis avatar
sarkis

sorry for delay on that

mcrowe avatar
mcrowe

@sarkis – actually, I needed to ping @Andriy Knysh (Cloud Posse) instead of you. It was his repo

sarkis avatar
sarkis

ah ok - i still don’t mind looking into it… if he can’t get to it

mcrowe avatar
mcrowe

Ah, @Andriy Knysh (Cloud Posse) replied in the other channel with some feedback. I’ll get it setup and submit a PR with his feedback

:--1:1
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
mike-zipit/terraform-aws-acm-request-certificate

terraform-aws-acm-request-certificate - Terraform module to request an ACM certificate for a domain name and create a CNAME record in the DNS zone to complete certificate validation

sarkis avatar
sarkis

to be clear i was referring to: >@sarkis Ran into an interesting situation today. Using Cognito, it needs certificates in us-east-1 in order to be used for the auth domain. We’re in us-east-2, so I’m going to have to: …

mcrowe avatar
mcrowe

right

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

ah, @Jamie replied to that

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

in the other channel

sarkis avatar
sarkis

ah yea reading over that code - looks like thats it @Andriy Knysh (Cloud Posse) - yep @mcrowe just referenced that - going to get us a PR soon :–1:

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

yea it’s a nice way to do it, specify two providers with alias

provider "aws" {
  region = "us-east-2"
}
provider "aws" {
  alias  = "east"
  region = "us-east-1"
}
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
cloudposse/terraform-root-modules

terraform-root-modules - Collection of Terraform root module invocations for provisioning reference architectures

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

we use us-west-2 too for all other stuff

mcrowe avatar
mcrowe

For @tamsky and @Erik Osterman (Cloud Posse) – .inputrc, prompt and all this fall under this rubric of “dotfile” management. IMO, the way to approach this is:

  • Move any files that are “user breakable” from /etc/profile.d into a cloudeposse/geodesic-dotfiles (which has the default configuration for environment, prompt, anything shell related)
  • Dockerfile for geodesic has something like:
    ENV USER_HOME=<https://github.com/cloudposse/geodesic-dotfiles>
    RUN curl -fLo /usr/local/bin/yadm <https://github.com/TheLocehiliosan/yadm/raw/master/yadm> && chmod a+x /usr/local/bin/yadm
    RUN cd /conf && yadm clone $USER_HOME
    
  • This would put the user in control of their bash setup – they could clone the “breakable” dotfiles from cloudposse/geodesic-dotfiles and then add their custom settings
mcrowe avatar
mcrowe

though I’m sure once I try to implement this I’ll find some holes in that logic

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, something along those lines…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

could you add that to the issue? @mcrowe

mcrowe avatar
mcrowe

Yeah

tamsky avatar
tamsky

personally I don’t think we need a dotfile manager

tamsky avatar
tamsky

I put it under the rubric of “homedir” management

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

what I don’t like about homedir management is consistency

tamsky avatar
tamsky

I see what you’re getting at

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

it’s like an escape hatch

tamsky avatar
tamsky

if I don’t want to make my homedir git repo public, then here be demons

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

should only be used when knowing what you’re doing.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, true

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

cloning git private repos in docker no fun

tamsky avatar
tamsky

especially using some 3rd party “manager”

tamsky avatar
tamsky

I’m pretty certain alias commands will want to enter the mix as well at some point.

mcrowe avatar
mcrowe

yadm = git (but the .git directory is offset from your home directory).
yadm clone [repo] yadm add .inputrc yadm commit yadm push etc

tamsky avatar
tamsky

I already have a git repo

mcrowe avatar
mcrowe

But, I’m not planning on putting my home directory in geodesic. But my aliases, maybe some other helpers, yea

sarkis avatar
sarkis

i guess my Q would be - when would we want to do something more than just terraform plan/apply or like chamber cli commands in geodesic.. since it mounts your homedir to /localhost - i can do most things outside of the container

mcrowe avatar
mcrowe

Here’s a flavor:

 $ yadm ls
Commit: 4d4ee021adb12c3c147f69dc0979f00094456c08 (HEAD -> master)
Author: Mike Crowe <[email protected]>
Date:   2018-04-06 22:20:14 -0400 (10 weeks ago)

Latest changes

 .aws/config                                  |  11 ++++++
 .bash-custom/bash-it.sublime-project         |  11 ++++++
 .bash-custom/bash-it.sublime-workspace       | 339 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 .bash-custom/custom.aliases.bash             |   2 +-
 .bash-custom/environ.bash                    |   4 +-
 .bash-custom/functions.bash                  |   7 ----
 .bash-custom/history.bash                    |   2 +-
 .bash-custom/nvm.bash                        |   2 +-
 .bash-custom/rerun.bash                      |   2 +-
tamsky avatar
tamsky


what I don’t like about homedir management is consistency
vs
it’s like an escape hatch
should only be used when knowing what you’re doing.

I’m pretty sure we can’t have both consistency and escape hatches

1
tamsky avatar
tamsky

I’ll rejoin the conversation in a couple hours. Good luck to us

mcrowe avatar
mcrowe

So, for my local development, I have a pretty extensive .bashrc that configures a bunch of stuff. However, there’s a case to be made for those guest systems (like geodesic, remote ssh logins, etc), maybe that needs to be a public gist or something different

tamsky avatar
tamsky
ENV USER_HOME=<https://github.com/cloudposse/geodesic-dotfiles>
RUN curl -fLo /usr/local/bin/yadm <https://github.com/TheLocehiliosan/yadm/raw/master/yadm> && chmod a+x /usr/local/bin/yadm
RUN cd /conf && yadm clone $USER_HOME

@Erik Osterman (Cloud Posse) I guess this reminds me that I’ve been wondering how/what you expect geodesic users to handle geodesic – as a template that individuals and organizations modify and then use internally, a stepping off point, so to speak…. or more as a common central starting point for all things in the CP ecosystem?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I’ve seen both. I think inheriting from our “base image” is a great start, but anticipate that as a company’s usage increases, that they’ll eventually “hard fork” and have their own base.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

organizations with a sizable ops team may prefer to manage their own.

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)
09:29:24 PM

@Jeremy (Cloud Posse) has joined the channel

2
jamie avatar
jamie
10:01:36 PM

@jamie has joined the channel

2018-06-13

Bot avatar
Bot
11:56:50 PM

@Bot has joined the channel

chris avatar
chris
12:29:27 AM

@chris has joined the channel

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
12:30:26 AM

@Andriy Knysh (Cloud Posse) has joined the channel

wookasz avatar
wookasz
12:41:45 AM

@wookasz has joined the channel

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
12:44:55 AM

@Erik Osterman (Cloud Posse) set the channel topic: Cloud Posse Open Source Community

tamsky avatar
tamsky
12:52:49 AM

@tamsky has joined the channel

tamsky avatar
tamsky

anyone else have trouble in their terminal with the selected glyphs in the prompt?

tamsky avatar
tamsky
  WHITE_HEAVY_CHECK_MARK=$'\u2705 '
  BLACK_RIGHTWARDS_ARROWHEAD=$'\u27A4 '
  TWO_JOINED_SQUARES=$'\u29C9 '
  CROSS_MARK=$'\u274C '
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, it’s unfortunately a bug we haven’t solved

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

it’s a combination of usign UTF8, prompts and terminals

tamsky avatar
tamsky

I’ve tried quite a few fonts in xterm

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i think most of us use iTerm2, but there are still some problems

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(OSX thing)

tamsky avatar
tamsky

what are the goals behind the glyphs ?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
01:02:30 AM
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

to visually communicate information like the powerline shell for zsh/bash

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I suppose we could create a non-fancy mode

tamsky avatar
tamsky

currently, the non-printing glyphs disturb readline, so history editing gets confused

tamsky avatar
tamsky

ok, I’ll offer a non-unicode set of glyphs in a PR soon

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

that would be awesome!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I know a few who would appreciate it

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

wow, I think you might have helped me find the answer though

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Escape non-printing characters in a function for a Bash prompt

In a Bash Prompt (PS1 variable), I’m calling a function to potentially add text to the prompt: export PS1=”\[email protected]\h $(my_function) $ “ However, the function in the prompt contains ANSI color codes …

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we already use \[,\] in the PS1 , but maybe we need to throw in some \001, and \002 in there too

mcrowe avatar
mcrowe
01:14:41 AM

@mcrowe has joined the channel

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

welcome @mcrowe!

tamsky avatar
tamsky

yeah, but for you, they’re printing, and for me they’re not, so use of \[ won’t help

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

hrm

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

can you send me a screenshot?

tamsky avatar
tamsky
01:18:02 AM
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

oh wow, yea, that’s bad

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
01:18:52 AM
tamsky avatar
tamsky

I understand, but my guess is, iTerm is actually choosing a different font for rendering unicode glyphs, and the font in use for your terminal doesn’t actually include those codepoints.

tamsky avatar
tamsky

which isn’t something that most terminal programs can do (least among them: xterm)

tamsky avatar
tamsky

I’ll admit it works in iterm, I tried it… I just can’t use it.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i agree we should offer a mode that does require it.

tamsky avatar
tamsky

ok, great, I’ll do the work

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

tamsky avatar
tamsky
tamsky/dotfiles

dotfiles - .files, including ~/.osx — sensible hacker defaults for OS X

:--1:1
tamsky avatar
tamsky

you got inputrc, readline and bash questions, I got answers.

tamsky avatar
tamsky

redirecting file descriptors in bash is still a head scratcher for me though

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, i’ll never remember the syntax for that

tamsky avatar
tamsky
  [ -n "${UNICODE_PROMPT_DISABLED}" ] && {
      WHITE_HEAVY_CHECK_MARK=$'\u2705 ' ;
      BLACK_RIGHTWARDS_ARROWHEAD=$'\u27A4 ' ;
      TWO_JOINED_SQUARES=$'\u29C9 ' ;
      CROSS_MARK=$'\u274C ' ;
  } || {
      # these use 8859-1 codepoints:                                                                      
      WHITE_HEAVY_CHECK_MARK=$'X ' ;
      BLACK_RIGHTWARDS_ARROWHEAD=$'-> ' ;
      TWO_JOINED_SQUARES=$'¤ ' ;
      CROSS_MARK=$'× ' ;
  }

?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Let’s use something more generic. For example, PROMPT_STYLE=plain

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

As for the bash logic, please use standard if/then rather than pipeline notation

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Feels like we’re introducing technical debt if we repurpose the current variables.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I can take a look at it later tonight. Currently afk.

mcrowe avatar
mcrowe

@tamsky
redirecting file descriptors in bash is still a head scratcher for me though
This one still doesn’t click for me: exec 1> >(logger -t process-psv) 2>&1

1
mcrowe avatar
mcrowe

I get it in principle, but I always have to lookup how to do it

sarkis avatar
sarkis
02:54:46 AM

@sarkis has joined the channel

mike.ballou avatar
mike.ballou
02:58:11 AM

@mike.ballou has joined the channel

2018-06-06

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
08:51:02 PM

@Erik Osterman (Cloud Posse) has joined the channel

    keyboard_arrow_up