#terraform (2018-07)

terraform Discussions related to Terraform or Terraform Modules

Archive: https://archive.sweetops.com/terraform/

2018-07-31

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Okay! I took a look. Looks good, almost exactly what we needed. Let’s add test case for quoted numbers and booleans as well.

:--1:1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@i5okie thanks for joining! Saw your comment on the issue here: https://github.com/cloudposse/terraform-aws-elastic-beanstalk-environment/issues/42

cloudposse/terraform-aws-elastic-beanstalk-environment

terraform-aws-elastic-beanstalk-environment - Terraform module to provision an AWS Elastic Beanstalk Environment

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Let me know if I can be of assistance.

i5okie avatar
i5okie

yeah we had an older release of the module. i just copied the lifestyle block there

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Is it working now with the latest release?

i5okie avatar
i5okie

i didn’t update to latest release (long story short, cant lol) but i copied the fix across. and its working now. thanks

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ok, great!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Consider Tightening Restrictions · Issue #6 · cloudposse/terraform-aws-iam-assumed-roles

what Do not allow MFA device deletion without MFA present why I think this allows an attacker to create a new MFA device without MFA authentication references <https://github.com/cloudposse/terr…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Would like feedback on this. Am I missing something here?

pmuller avatar
pmuller
03:10:00 AM

@pmuller has joined the channel

pmuller avatar
pmuller

one thing i’d like to do here is to restrict aws account usage by source ip address

pmuller avatar
pmuller

if your users all work in the same office

pmuller avatar
pmuller

you could restrict mfa management to your office ip address

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, that could be an enhancement

pmuller avatar
pmuller

that way an attacker should steal api keys or password + be physically close to your office and hack your wifi to abuse your aws account

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

one thing I’ve loved about Duo is the ability to do geofencing

pmuller avatar
pmuller

did not try this yet but i’d like to do this here

pmuller avatar
pmuller

Duo ?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, they are a MFA saas

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(and now also do SSO)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
The Trusted Access Company attachment image

Duo’s Trusted Access platform verifies the identity of your users with two-factor authentication and security health of their devices before they connect to the apps you want them to access.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We have support for this in our bastion container: https://github.com/cloudposse/bastion

cloudposse/bastion

bastion - Secure Bastion implemented as Docker Container running Alpine Linux with Google Authenticator & DUO MFA support

pmuller avatar
pmuller

so basically Duo will store your users secret mfa keys?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yep

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

and offer multiple forms of 2FA

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

you can choose which ones you allow

pmuller avatar
pmuller

interesting ! (i dropped the google authenticator idea precisely because i was afraid of storing those mfa secrets)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

you can also require, for example, that their MFA device is up to date

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

that their OS is current

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

etc..

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so policy management at a higher level

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, GA is a nice contribution by google, but seriously difficult to truly secure.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Also, are you familiar with Teleport for SSH?

pmuller avatar
pmuller

their current os ? is it using TNC ?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

their = ?

pmuller avatar
pmuller
03:20:08 AM

This “their”

pmuller avatar
pmuller

how Duo is checking for OS version ?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

since one factor is mobile push notifications

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

they can get a lot of metadata

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

including gps location

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

OS version

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

authenticator version

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

device os

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

etc

pmuller avatar
pmuller

very interesting

pmuller avatar
pmuller

about teleport, i would like to use it and log everything with it, but haven’t taken time to try it yet

pmuller avatar
pmuller

my current compromise is logging a lot with auditd and pam_audit_tty

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yea, we used to do something similar.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/sudosh

sudosh - Shell wrapper to run a login shell with sudo as the current user for the purpose of audit logging

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

does pam_audit_tty work without ttys?

pmuller avatar
pmuller

nope

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
03:25:49 AM
pmuller avatar
pmuller

that’s from Duo ?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea

pmuller avatar
pmuller

nice!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
03:27:24 AM
pmuller avatar
pmuller

maybe i am a bit too paranoid for the cloud era … i grew up in datacenters, and started using aws in 2016 … but i still have too much of the data center / oss guy thinking: DIY ! so at first i even had a hard time trusting AWS (then I realized how absurd it was..) let alone talking about a 3rd party for authentication

pmuller avatar
pmuller

but yeah, i sometimes feel like i am wasting my time

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

[override.tf](http://override\.tf) appears to have no impact/influence over terraform { ... } block

pmuller avatar
pmuller

too bad

pmuller avatar
pmuller

cannot terragrunt be helpful with your sed hack? (haven’t tried it yet)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i am not well versed in terragrunt

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i know terragrunt can do what our module does out of the box

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but i like that we’re defining the infrastructure for bucket/dynamodb with terraform rather than another tool

pmuller avatar
pmuller

i like ideas terragrunt brings on the table, but i know from experience it’s better to use vanilla tools in the long term … wrappers comes and go … and creates debt!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yes - agree

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i would consider using it for specific cases such as this

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

small in scope

pmuller avatar
pmuller

i’d be happy to hear about your feedback

pmuller avatar
pmuller

btw, do you test your tf modules ? if so, how ? (doing so with all the code i write… except tf, so it’s frustrating)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we’re not quite there yet. right now, we do lint testing only.

pmuller avatar
pmuller

same here

pmuller avatar
pmuller

not even that … we just run “terraform fmt” and check if there is a change in the repo

pmuller avatar
pmuller

i feel poor with just this

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Can we automate geodesic terraform to build and destroy a cluster in a CI env ?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

in the #geodesic channel

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

basically, we package all infrastructure in a container along with terraform

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

thus, it’s an app like everything else

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

our strategy is to use our testing account to run: plan, apply, plan, destroy, plan

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I am less enthused by atlantis and terratest

pmuller avatar
pmuller

haven’t heard about atlantis

pmuller avatar
pmuller

terratest requires to write tests in go, which is rather unpleasant

pmuller avatar
pmuller
03:56:50 AM

wants a nice test runner/framework like pytest ;)

pmuller avatar
pmuller

lunch time, bbl

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

cool, ttyl!

pmuller avatar
pmuller

ior have a good night

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(yea, I’m in Los Angeles)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
wata727/tflint

tflint - TFLint is a Terraform linter for detecting errors that can not be detected by terraform plan

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
akaspin/terraform-provider-generic

Contribute to terraform-provider-generic development by creating an account on GitHub.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

this is an interesting concept. it reminds me of the “porcelain” plugin module used in helm, but applied to terraform providers

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

using this, it’s possible to define a “basic” provider in any language, and then accept targets like apply, destroy, etc

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

the implementation above seems like a POC, but if anyone else has seen one, lmk

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

note, this is very different from the data external provider which does not distinguish between phases

pmuller avatar
pmuller

oh nice!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Related to the earlier question of terraform cicd, this looks interesting

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
mercari/tfnotify

tfnotify - A CLI command to parse Terraform execution result and notify it to GitHub

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

This will take the output of terraform plan/apply and post it back to GitHub or Slack.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Andriy Knysh (Cloud Posse) wrote a generic tool for this, not specialized for terraform called ‘github-commenter’

2018-07-30

johntellsall avatar
johntellsall
04:53:58 PM

@johntellsall has joined the channel

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Adjust output regexp to preserve string type of environment values by fernandosilvacornejo · Pull Request #9 · cloudposse/terraform-aws-ecs-container-definition

What Adjust the regexp used to overcome Terraform&#39;s type conversions for integer and boolean parameters in the JSON container definition. The new regexp preservers the string type for environme…

fernando avatar
fernando

Travis build timed out for some reason, no explanations given

Adjust output regexp to preserve string type of environment values by fernandosilvacornejo · Pull Request #9 · cloudposse/terraform-aws-ecs-container-definition

What Adjust the regexp used to overcome Terraform&#39;s type conversions for integer and boolean parameters in the JSON container definition. The new regexp preservers the string type for environme…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Could be intermittent failure on their end

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Just getting up :) will take a look in a couple of hours.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I’m just about to step out the door for the evening, so will take a look tomorrow.

fernando avatar
fernando

Sure thing, let me know what you think, and enjoy the rest of the evening!

2018-07-27

loren avatar
loren
10:43:54 AM

@loren has joined the channel

Ruslan avatar
Ruslan
10:59:33 AM

@Ruslan has joined the channel

Ruslan avatar
Ruslan

Hi Everyone! First of all - big thanks for Terraform scripts on Github! It was invaluable source of knowledge and easy to use as is too.

Ruslan avatar
Ruslan

Come to talk about terraform-null-label and name sequence

Ruslan avatar
Ruslan


any insights on ordering? namespace-stage-environment or namespace-environment-stage

Ruslan avatar
Ruslan

In my opinion this should be configurable, because on the projects I am on right now we have namespace-app-stage.

Ruslan avatar
Ruslan

I`ve created issue on GitHub for that: https://github.com/cloudposse/terraform-null-label/issues/35

Make it possible to have different naming sequences · Issue #35 · cloudposse/terraform-null-label

Currently, this repo supports only namespace-stage-name sequence. In some organizations, there is a different standard for example namespace-name-stage. Please consider and explore whether it is po…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Ruslan I have been considering this as well

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

To me, it’s most natural to move the environment after the stage

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

But allowing order to be overridden would be also a nice feature

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@jamie

jamie avatar
jamie

No problem

jamie avatar
jamie

I’ll give it a pattern variable

Ruslan avatar
Ruslan

Personally I believe namespace-stage-app is proper way (because stage/env is the isolated space in which all the apps are working). But on the current project it is done as namespace-app-stage.

loren avatar
loren

hey folks! does anyone know of tooling that helps create a terraform state file, based on a given terraform configuration, by importing existing resources?

loren avatar
loren

we have a number of accounts with resources we can’t recreate, but we’re just getting started with terraform (switching from cloudformation). we’ve written terraform modules to manage everything, and it works great for new accounts, and now we’re looking into how we can extend it to prior accounts…

loren avatar
loren

terraform state list gives all the resource names, which seems to work with terraform import. however, it also outputs all the data source names (with no differentiation between resources and data sources), which don’t need to be imported

jamie avatar
jamie

@loren that is what terraforming does, http://terraforming.dtan4.net/

:--1:1
loren avatar
loren

ahh thanks, i looked at that earlier, but it didn’t seem like quite what i needed… i’ll give it a try and see how it goes

loren avatar
loren

i don’t want to import all resources of a particular type, and i don’t want the resources in the root module, which is what the examples look to be doing

loren avatar
loren

but perhaps it has more options

loren avatar
loren

seems not, unfortunately. any other tools?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We had to do this a year ago for another customer, but we went the tedious route of defining all the resources by hand and importing the state. It took a lot of effort to get a clean plan.

loren avatar
loren

that’s what i’m going through now… i have the output of state list, so i’m using that as a known good starting point to define an configuration map of resource: id and a simple wrapper to import all resources with a defined id

loren avatar
loren

then the tedious process of populating the config file for each account

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yea, it sucks. Fortunately I had some help from offshore contractors. I can make an intro if you’d like.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

PM me

loren avatar
loren

other option i suppose would be to take the state file from a known good account and template that

loren avatar
loren

offshore contractor not an option for this customer, unfortunately. we pretty much are the hired help. thanks for the offer though

jamie avatar
jamie

You can use it to import everything, and then just manually edit the state file to only include the ones you want.

jamie avatar
jamie

Its about all we have at the moment

jamie avatar
jamie

@Ruslan @Erik Osterman (Cloud Posse) I’ve created a pull request that adds that feature you were after https://github.com/cloudposse/terraform-null-label/pull/36

Label order by Jamie-BitFlight · Pull Request #36 · cloudposse/terraform-null-label

A new variable that allows for providing a label_order as a list to specify which variables and in what order they go into the id output and Name tag. The list can include all or none of the a vari…

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I will take a look as soon as I get into the office

Ruslan avatar
Ruslan

Yes, it works, Ive tried it. <https://github.com/cloudposse/terraform-null-label/pull/36#issuecomment-408472191> However I am using label through terraform-aws-cicd and terraform-aws-codebuild` projects and need support in them for label_order.

Ruslan avatar
Ruslan

Should I add issue (or even better, PR) to these projects as well?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yes we will accept PRs

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Issue not needed

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Regarding label module, I have an idea I want to test before I comment

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

But currently afk

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

For the label module, it looks now it would be better to just add a few parameters, p1, p2, p3, p4, p5, etc. and use them as we want. Instead of assigning a meaning to all of them and then decide on the order?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

No

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

100% want meaning

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I was just reflecting on this point

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

The thing is we need to standardize nomenclature

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

This is what I love about @jamie document for this

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Even if I disagree with ‘environment’

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

However I have an idea based on what you say @Andriy Knysh (Cloud Posse)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Will whip up prototype to express how this should be working

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think it will satisfy everyone but might belong in a new module

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

Ok

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

Why naming is so hard :)

:100:2
jamie avatar
jamie

@Ruslan once the guys have reviewed the pull request and its good to add, we can start updating the other modules to be compatible with context.

jamie avatar
jamie

which will allow you to pass in a label modules context and use that for formatting.

1

2018-07-25

jamie avatar
jamie

In your terraform modules you can just leave it out

jamie avatar
jamie

It can be either. But other companies may use both when stage is being referred to as the disposable type of stage. Like source, build, deploy. Or extract, transform, load. Or raw, resized, thumbnail.

:--1:1
jamie avatar
jamie
firmstep-public/trainingdayfour

trainingdayfour - Training Day Four: Review how to use Terraform Variables, and look at how to lay out a Terraform Directory Structure.

:--1:1
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@jamie @tamsky I understand the thinking about having stage and environment at the same time. But IMO it just complicates everything. You are saying that stage could refer to source, build, deploy, extract, transform, load. raw, resized, thumbnail. Those seem to be stages in an CI/CD pipeline or some other process of transforming data or building/deploying some artifacts. As I see it, they don’t relate to infrastructure environments that we deploy with Terraform. Are you saying that we need to deploy some TF resources with names like cp-prod-deploy-myapp, cp-prod-load-myapp? Sounds like parts of a pipeline, not a separate set of AWS resources

jamie avatar
jamie

I’m saying that by having both, people who use the module can keep the it close to their own naming system, without breaking the cloudposse naming system.

1
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

what would be a real example of using names like these in a real infrastructure having separate stages/environments like dev, prod?

jamie avatar
jamie

Outside of Cloudposse: If one was making a codepipeline pipeline for example, and you had two parts to it, all using the dockerised codebuild system, one of the codebuild stages might be called stage:build attributes: [“java”], another might be stage:build attributes: [“golang”], another might be stage:deploy attributes: [“s3-assets”]

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

Hmm I see your point. But are those Terraform resources? The label module is used to name TF resources

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

And we already have attributes in the module to add more parts to the names

jamie avatar
jamie

I realise, but when I introduce the label to my company and my clients, they have said they have a naming convention that includes environment. By having Environment optional, it also allows us to name items using the word environment in the same way we have been doing on the cloudposse modules.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

i agree, we can use the module in many ways now

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

just did not want to over-complicate things that could be simple

jamie avatar
jamie

One nice thing though, is in the future of the cloudposse modules, the modules all the modules can contain a “context” map variables too.

:100:2
jamie avatar
jamie

So you just pass context to the modules, instead of all of the other fields

:--1:1
Arkadiy avatar
Arkadiy
02:50:24 PM

@Arkadiy has joined the channel

jamie avatar
jamie
05:41:39 PM
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

thanks @jamie

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

looks like all of that stage vs. environment stuff is used mostly for tagging, not for naming

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

which are diff things

jamie avatar
jamie

ya

2018-07-24

Yoann avatar
Yoann
08:10:41 AM

@Yoann has joined the channel

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
cloudposse/terraform-aws-elasticsearch

Contribute to terraform-aws-elasticsearch development by creating an account on GitHub.

jamie avatar
jamie

@Erik Osterman (Cloud Posse) are we going to release that null-label branch we made together that adds context as a variable?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Add context by osterman · Pull Request #25 · cloudposse/terraform-null-label

what Support passing a label&#39;s context between label modules why DRY demo module &quot;label1&quot; { source = &quot;../../&quot; namespace = &quot;Namespace&quot; stage = &…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, i was just holding off since it was “complicated”

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but we can merge

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@jamie do you want to add environment?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(in a follow up PR)

jamie avatar
jamie

I did yeah :)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Is his change by @dennybaa in that commit not needed?

jamie avatar
jamie

He pushed his change through…. its in now

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

whoot!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so, one more test and then approve?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(i can’t approve, since I opened the PR)

jamie avatar
jamie

I have just pushed the changes

jamie avatar
jamie

I have updated the readme to include the new features too

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

tested and working?

jamie avatar
jamie

yeah, using the example

jamie avatar
jamie

./examples/complete

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

cool

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ok, you can approve, merge, and tag a new minor release

jamie avatar
jamie
 terraform apply                                                    [ruby-2.5.1p57]
data.null_data_source.tags_as_list_of_maps[5]: Refreshing state...
data.null_data_source.tags_as_list_of_maps[1]: Refreshing state...
data.null_data_source.tags_as_list_of_maps[0]: Refreshing state...
data.null_data_source.tags_as_list_of_maps[4]: Refreshing state...
data.null_data_source.tags_as_list_of_maps[3]: Refreshing state...
data.null_data_source.tags_as_list_of_maps[2]: Refreshing state...
data.null_data_source.tags_as_list_of_maps[0]: Refreshing state...
data.null_data_source.tags_as_list_of_maps[4]: Refreshing state...
data.null_data_source.tags_as_list_of_maps[1]: Refreshing state...
data.null_data_source.tags_as_list_of_maps[2]: Refreshing state...
data.null_data_source.tags_as_list_of_maps[3]: Refreshing state...
data.null_data_source.tags_as_list_of_maps[0]: Refreshing state...
data.null_data_source.tags_as_list_of_maps[2]: Refreshing state...
data.null_data_source.tags_as_list_of_maps[1]: Refreshing state...
data.null_data_source.tags_as_list_of_maps[3]: Refreshing state...
data.null_data_source.tags_as_list_of_maps[4]: Refreshing state...

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

label1 = {
  attributes = fire-water-earth-air
  id = cloudposse-uat-build-winstonchurchroom-fire-water-earth-air
  name = winstonchurchroom
  namespace = cloudposse
  stage = build
}
label1_context = {
  attributes = [fire-water-earth-air]
  delimiter = [-]
  environment = [uat]
  name = [winstonchurchroom]
  namespace = [cloudposse]
  stage = [build]
  tags_keys = [City Environment Name Namespace Stage]
  tags_values = [Dublin Private cloudposse-uat-build-winstonchurchroom-fire-water-earth-air cloudposse build]
}
label1_tags = {
  City = Dublin
  Environment = Private
  Name = cloudposse-uat-build-winstonchurchroom-fire-water-earth-air
  Namespace = cloudposse
  Stage = build
}
label2 = {
  attributes = fire-water-earth-air
  id = cloudposse-uat-build-charlie-fire-water-earth-air
  name = charlie
  namespace = cloudposse
  stage = build
}
label2_context = {
  attributes = [fire-water-earth-air]
  delimiter = [-]
  environment = [uat]
  name = [charlie]
  namespace = [cloudposse]
  stage = [build]
  tags_keys = [City Environment Name Namespace Stage]
  tags_values = [London Public cloudposse-uat-build-charlie-fire-water-earth-air cloudposse build]
}
label2_tags = {
  City = London
  Environment = Public
  Name = cloudposse-uat-build-charlie-fire-water-earth-air
  Namespace = cloudposse
  Stage = build
}
label3 = {
  attributes = 
  id = release-starfish
  name = starfish
  namespace = 
  stage = release
}
label3_context = {
  attributes = []
  delimiter = [-]
  environment = []
  name = [starfish]
  namespace = []
  stage = [release]
  tags_keys = [Animal Eat Environment Name Namespace Stage]
  tags_values = [Rabbit Carrot  release-starfish  release]
}
label3_tags = {
  Animal = Rabbit
  Eat = Carrot
  Environment = 
  Name = release-starfish
  Namespace = 
  Stage = release
}
jamie avatar
jamie

Minor?

jamie avatar
jamie

What constitues a major?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

haha, i know…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

just i don’t want to go to 1.0.0 yet

jamie avatar
jamie

thats like… only got the same variables as the original

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so, we say pre 1.0

jamie avatar
jamie

okay

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so we are at 0.3.8 now, this should be 0.4.0

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i am not yet sure what constitutes a 1.0.0 release - we don’t yet have any

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

probably a topic for another day… but some considerations are “stable interface”

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so if we plan on adding a bunch more parameters, feels like the interface is not yet stable

jamie avatar
jamie

You need a new “Definition of done”

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

even terraform is pre 1.0

jamie avatar
jamie
cloudposse/terraform-null-label

terraform-null-label - Terraform Module to define a consistent naming convention by (namespace, stage, name, [attributes])

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

thanks @jamie!

2
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@jamie I missed something maybe in that release… or at least let’s discuss

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

local.namespace, local.environment, local.stage

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Seems to me that stage is more broad than environments

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

So it should come before environment

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

No?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@tamsky have any insights

tamsky avatar
tamsky

I strongly agree with the rationale in the release notes; some organizations have existing mental models and terminology.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Hrmm I guess I can see pros and cons both ways

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

What would be a real-life example of using both at the same time?

sarkis avatar
sarkis
02:13:20 AM

@sarkis has joined the channel

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

the challenge here is what we do with our 100 terraform modules that don’t yet support environment

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@tamsky any insights on ordering? namespace-stage-environment or namespace-environment-stage

2018-07-23

maarten avatar
maarten
11:40:40 AM

@maarten has joined the channel

Cristin avatar
Cristin

@Andriy Knysh (Cloud Posse) I managed to make it work by using the configuration you used in the Jenkins example

Cristin avatar
Cristin

Thank you for you help!

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@Cristin sorry I did not help you yet, just wanted to look at your code in more details (from what I saw briefly, you mixed a lot of concepts)

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

glad the jenkins examples hepled

Cristin avatar
Cristin

@Andriy Knysh (Cloud Posse) I still have one question. How is the best way of connecting the elasticbeanstalk envirment to a RDS database? Just create the database and inject in env vars the database connection details?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

we have https://github.com/cloudposse/terraform-aws-rds to create an RDS database (no Aurora)

cloudposse/terraform-aws-rds

terraform-aws-rds - Terraform module to provision AWS RDS instances

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
cloudposse/terraform-aws-rds-cluster

terraform-aws-rds-cluster - Terraform module to provision an RDS Aurora cluster for MySQL or Postgres

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
cloudposse/terraform-root-modules

terraform-root-modules - Collection of Terraform root module invocations for provisioning reference architectures

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

yes, the best way is to put the RDS credentials (username, password, etc.) in ENV vars, and then in Elastic Beanstalk environment add those ENV vars to the environment. Then read them in your app

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
cloudposse/terraform-aws-jenkins

terraform-aws-jenkins - Terraform module to build Docker image with Jenkins, save it to an ECR repo, and deploy to Elastic Beanstalk running Docker stack

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

(you can add any number of ENVs)

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

we try to follow 12-factor https://12factor.net/

The Twelve-Factor App

A methodology for building modern, scalable, maintainable software-as-a-service apps.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
III. Config
Store config in the environment
Cristin avatar
Cristin

Awesome, thank you for all this details

:--1:1
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

it will output the credentials

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
cloudposse/terraform-aws-elastic-beanstalk-environment

terraform-aws-elastic-beanstalk-environment - Terraform module to provision an AWS Elastic Beanstalk Environment

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

at Cloud Posse, we use chamber to securily work with ENV vars and secrets

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
segmentio/chamber

chamber - CLI for managing secrets

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
:--1:1
Cristin avatar
Cristin

Awesome information, thanks a lot!

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

geodesic supports chamber natively https://github.com/cloudposse/geodesic

cloudposse/geodesic

Geodesic is the fastest way to get up and running with a rock solid, production grade cloud platform built on strictly Open Source tools. https://docs.cloudposse.com/geodesic/

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

when you provision an RDS cluster from geodesic, you add the ENVs to chamber

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
cloudposse/terraform-root-modules

terraform-root-modules - Collection of Terraform root module invocations for provisioning reference architectures

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

then you provision terraform-aws-elastic-beanstalk-environment from geodesic, you read the ENVs from chamber

krogebry avatar
krogebry
05:40:40 PM

@krogebry has joined the channel

Cristin avatar
Cristin

@Andriy Knysh (Cloud Posse) Thank you for all the help! For now I setup a simple RDS postgress db an injected the env variables in a ebs env

Cristin avatar
Cristin

I still have a question, related to setting up multiple envs like “develop” and “production”

Cristin avatar
Cristin

I noticed that I can create one ebs application and create multiple environments for it

Cristin avatar
Cristin

But in this situation I don’t understand what’s the role of the stage attribute for the elastic_beanstalk_application module

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

stage is to separate the resources b/w diff stages - see the latest discussing in #announcements

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

also, by using diff stage, e.g. dev and prod, you can deploy the same environment two times for development and production

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

you need to deploy it two times obviously

Cristin avatar
Cristin

Yes, I know this. But if I have 1 application and 2 environments for it, what’s the role of having stage setup on the application itself?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We generally do not prescribe using a single beanstalk for multiple stages even though it is supported

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

stage is used in all resource names by using the label module. So (almost) all resources will be names like namespace-stage-name

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

so, if you use the same AWS account and the same EB application to deploy two environments into it, then stage could be anything you like - but as @Erik Osterman (Cloud Posse) mentioned, we don’t usually do it

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

we try to completely separate stages

Cristin avatar
Cristin

Ok, I understand. So is a best practive to have 1 app = 1 env

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

even using diff AWS accounts for more security and control

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yes - that way they share nothing so making changes to one can never impact the other

:--1:1
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

and much easier to control IAM

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Also, have you seen our modules for ECS

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

?

Cristin avatar
Cristin

Sorry, which one? I looked over a lot of your modules, they are really helpful

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Since you are starting from scratch, I would consider using ECS rather than Beanstalk.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/terraform-aws-ecs-web-app

terraform-aws-ecs-web-app - Terraform module that implements a web app on ECS and supporting AWS resources.

Cristin avatar
Cristin

Well, the Beanstalk is actually a requirment on the project I’m doing, so I will try ecs on the next one

Cristin avatar
Cristin

Ok, so if I will create seperate resources (security group, vpc, db, app, env) for each stage what would be the best way to run this twice and not having to duplicate the code? Should I create a module around your modules ?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@Cristin we still recommend to separate all resources into at lest two stages (dev and prod) and don’t mix anything b/w/ them

:--1:1
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

take a look how it’s separated here https://github.com/cloudposse/staging.cloudposse.co

cloudposse/staging.cloudposse.co

staging.cloudposse.co - Example Terraform Reference Architecture for Geodesic Module Staging Organization in AWS.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
cloudposse/staging.cloudposse.co

staging.cloudposse.co - Example Terraform Reference Architecture for Geodesic Module Staging Organization in AWS.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

we use ENV vars for stage too

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
cloudposse/prod.cloudposse.co

prod.cloudposse.co - Example Terraform/Kubernetes Reference Infrastructure for Cloud Posse Production Organization in AWS

Cristin avatar
Cristin

Ok, I will look into this. Thanks a lot again for your time and availability to help

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

no problem

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

the container + env vars pattern is our approach to DevOps orchestration and env separation (there could be other ways of doing it)

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

geodesic is that container, with much more tools inside

2018-07-22

Cristin avatar
Cristin
05:44:16 PM
Cristin avatar
Cristin

@Andriy Knysh (Cloud Posse) Here it is

Cristin avatar
Cristin

Thank you!

2018-07-20

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@jamie here’s the TF channel

jamie avatar
jamie
06:00:02 PM

@jamie has joined the channel

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

jamie avatar
jamie

Oh hi! I can keep my terraform troubleshooting to here without disturbing your geodesic users :)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We can try - just sensitive to splintering off eyeballs until we have more critical mass

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but it is nice to keep the threads separated

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, perhaps it’s time. just like we started using #random <— nice

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
08:22:55 PM

@Andriy Knysh (Cloud Posse) has joined the channel

tamsky avatar
tamsky
08:37:07 PM

@tamsky has joined the channel

Cristin avatar
Cristin
08:54:49 PM

@Cristin has joined the channel

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Cristin which module are you having trouble with?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(i’m about to jump on a call - so may be delayed in following up)

:--1:1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Andriy Knysh (Cloud Posse) can also help

Cristin avatar
Cristin

I’m using elastic-beanstalk-environment and I’m getting some error related to aws_iam

Cristin avatar
Cristin
08:58:33 PM
Cristin avatar
Cristin

I took a look over the parameters and the code of the module, but I don’t see a way of passing a aws_iam_role_policy_attachment or profile

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@Cristin i’ll look into the error (little bit later), but for now you can look at this repo where we use elastic-beanstalk-environment: https://github.com/cloudposse/terraform-aws-jenkins/blob/master/main.tf

cloudposse/terraform-aws-jenkins

terraform-aws-jenkins - Terraform module to build Docker image with Jenkins, save it to an ECR repo, and deploy to Elastic Beanstalk running Docker stack

:--1:1
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
cloudposse/terraform-aws-jenkins

terraform-aws-jenkins - Terraform module to build Docker image with Jenkins, save it to an ECR repo, and deploy to Elastic Beanstalk running Docker stack

Cristin avatar
Cristin

I took a look over the example and how you are attaching a policy to the ec2 instances. I tried doing the same thing but I still get the above errors

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@Cristin can you share your code? If not public, you can message me or send email to [email protected], I’ll take a look

2018-07-10

achrstl avatar
achrstl
06:16:37 PM

@achrstl has joined the channel

2018-07-05

jonathan.olson avatar
jonathan.olson
09:01:43 PM

@jonathan.olson has joined the channel

2018-07-02

ag4ve.us avatar
ag4ve.us
12:00:51 PM

@ag4ve.us has joined the channel

zerocoolback avatar
zerocoolback
05:10:28 AM

@zerocoolback has joined the channel

    keyboard_arrow_up