Can someone list their workflow please for deploying a terraform project to dev and then to, say, sandbox. My understanding of CP’s tools is this:

Switch to Dev

• Use TFENV to switch AWS account w/ aws-vault + backend configuration (S3 for state and DynamoDB for lock) && run terraform init every time doing a deploy
• Run tfenv terraform apply

Switch to Sandbox/Staging/Prod

• Use TFENV to switch AWS account w/ aws-vault + backend configuration (S3 for state and DynamoDB for lock) && run terraform init every time doing a deploy
• Run tfenv terraform apply

we don’t use it as a wrapper

though it does support being called as a wrapper

we use it with direnv which has a stdlib

https://github.com/cloudposse/testing.cloudposse.co/blob/master/conf/acm/.envrc

our direnv configuration is here: https://github.com/cloudposse/geodesic/tree/master/rootfs/etc/direnv

And do you have multiple .envrc’s to switch between envs (staging/dev/prod) to set hte bucket name for instance?

Yeh that direnv seems to do that

we have one repo per stage.

that’s a lot of code to maintain

nope

that’s what I’ve been avoiding this whole time

terraform-root-modules is our service catalog

(do read the README.md on that repo though b/c it’s often misunderstood)

So we can call a module with very few lines of code

So when calling terraform init, it will download the remote module and initialize it

We exploit: https://www.terraform.io/docs/commands/environment-variables.html#tf_cli_args-and-tf_cli_args_name

using tfenv, we pass all parameters via the TF_CLI_ARGS_init environment variable

this allows us to call all terraform commands without a wrapper

however, all this stuff is a PIA to implement if you don’t use a preconfigured shell

that’s why we have geodesic

The readme for geodisc is even more confusing and doesn’t really articulate workflows

it’s 1000 things

there’s no one workflow

IMO documentation MUST show how one can use it. It might seem like spoonfeeding but knowledge bases should assume that existing knowledge has gaps etc.

I look at geodisc

see an amazing product

yup, we need to add more docs.

but as with many CP products am left with: 1) Where does this fit into my workflow 2) How do people use it in their workflow w/ CLI commands spelled out in an example etc. 3) How do I start using it

2 and 3 being the most critical, as 1 an engineer can usually work out

if I download geodisc I’d have 0 idea how to start using it

Also I’m seeing a not github link now

to docs

So I’ll review those :]

https://docs.cloudposse.com/geodesic/

it’s far from complete.

https://docs.cloudposse.com/local-dev-environments/quickstart/ When you assume I’m using mac

note it calls out OSX

The whole doc is just OSX

AWS VAULT is a sub header of OSX

How can I do a PR for docs Erik?

https://github.com/cloudposse/docs/issues

we have hundreds of issues no shortage

No PR template

https://github.com/99designs/aws-vault/blob/master/USAGE.md Love it

Ok that’s two PRs for you

Does anyone know of a good way to check whether or not a lot of terraform configurations are up to date? Basically I’d like to run terraform plan on $x amount of services/environments and get some sort of alterting if the plan isn’t empty

Pop it in a CI pipeline?

Or a bash script that iterates

terraform plan -detailed-exitcode would do the trick

as for the process it depends on what you have

e.g. if you have jenkins then you will do a bash scripts that runs in a CI pipeline

The plan was to use atlantis rather than pipelines for terraform, but I guess I could do both (where the pipeline only does a plan)

this is what i do, yes, terraform plan -detailed-exitcode

have scheduled jobs in CI with read-only privs that runs that every day for all tf configs

@Erik Weber both both!

Cheers

how can I show the state of a single resource instance?

 $ terraform state list | grep aws_subnet.public
module.subnets.aws_subnet.public[0]
module.subnets.aws_subnet.public[1]
module.subnets.aws_subnet.public[2]
module.subnets.aws_subnet.public[3]
 
 $ terraform state show module.subnets.aws_subnet.public[1]
Multiple instances found for the given pattern!

This command requires that the pattern match exactly one instance
of a resource. To view the matched instances, use "terraform state list".
Please modify the pattern to match only a single instance.

https://www.terraform.io/docs/commands/state/show.html @daveyu?

That’s weird that doing [1] doesn’t work

any different with 0 or *

thanks. no, it’s the same Multiple instances found for the given pattern!

Not really used state show before, but this looked like a potential cause https://github.com/hashicorp/terraform/issues/8929 if you haven’t seen it already

hmm well terraform show does output the expected state for module.subnets.aws_subnet.public.1

I guess that will work for now

Just wanted to say thanks for the terraform modules, it helps me a lot for my school project

welcome @sytten

Why does reference-architectures insist on using keybase instead of slack? Can this be ignored? What do I miss out on?

i think it is for verification of user identity using pgp keys

something around this feature, no? https://www.terraform.io/docs/providers/aws/r/iam_user_login_profile.html#pgp_key

Keybase is used to encrypt user credentials so that only that user can decrypt the terraform output (or email)

It’s nice as otherwise you need to figure out how to let users know their credentials in non plain text

Thanks both

hi all, I have a quick question. how do you provide estate visibility within dynamic infrastructure?

There is a “portal” that links you to k8s dashboard, grafana, Prometheus, Kibana

I am not referring just to k8s env dashboard but also to multi-cloud deployment of resource provisioned with terraform for example

What kind of visibility are you after?

Ah, of all your Terraform provisioned infra? Like, what have I actually created?

I would like to create a visual single source of truth in which show all the resource provisioned

Terraform graph and feed it to graphiz?

I was referring to something able to put terraform resources information into a kv to be consumed with other tools

statefile -> exporter -> kv

Consumed by anything specific?

not currently

@deftunix Terraboard might be the type of end result you’re looking for https://github.com/camptocamp/terraboard

I will check

yes, it’s exactly what I was looking for.

good starting point. thank you @Steven

welcome

Nice

Does anyone know how to use TF_CLI_ARGS_name in a .tf file?

For instance ` comment = “${var.TF_CLI_ARGS_stage}”`

unknown variable referenced: 'TF_CLI_ARGS_stage'; define it with a 'variable' block .. obvs setting it as a variable doesn’t fix this.

Just wondering how I’m to access TF_CLI_ARGS especialy when they’re coming through geodesic

Ok think I found it https://docs.cloudposse.com/geodesic/module/with-terraform/

not sure about the geodesic part, but any variable in terraform would generally just be var.stage, and terraform will automatically read it from the env TF_VAR_stage if present

https://www.terraform.io/docs/commands/environment-variables.html#tf_var_name

TF_CLI_ARGS is something else entirely

This should help anyone understand how to use GEODESIC vars inside of Terraform

Though those variables are all for the s3 backend in the terraform provider.

Terraform backends do not support interpolation

That’s why we use the TF_CLI_ARGS pattern

../modules/aws_cross_account_role

resource "aws_iam_role_policy_attachment" "policies" {
  count = "${length(var.role_policies)}"
  policy_arn = "${var.role_policies[count.index]}"
  role = "${aws_iam_role.role.id}"
}

another file

module "remote_role" {
  source = "../modules/aws_cross_account_role"

  aws_external_account_id = "${local.aws_account_id}"
  role_name = "remote_role"
  role_policies = [
    "arn:aws:iam::aws:policy/ReadOnlyAccess",
    "${aws_iam_policy.invoke_flow_resources_lambda_access.arn}",
    "${aws_iam_policy.create_function_flow_resources_lambda_access.arn}"
  ]
}

does anyone know what am I missing here? I’m getting error * module.remote_role.aws_iam_role_policy_attachment.policies: aws_iam_role_policy_attachment.policies: value of 'count' cannot be computed

could one of the aws_iam_policy in the latter file be missing when you are trying to execute it?

@Maxim Tishchenko that’s the infamous count error

we have some info here https://docs.cloudposse.com/troubleshooting/terraform-value-of-count-cannot-be-computed/

in your case, TF can’t calculate dynamic counts across modules

(if you put all the resources into one module, it should be ok)

the easiest (but not prettiest) way to fix it in your case would be to add var.role_policies_count variable to the modules/aws_cross_account_role module and provide it from module "remote_role"

Mmmm

I got you, thanks

we do it in some of our modules, like i said not pretty, but works

https://github.com/cloudposse/terraform-aws-eks-cluster/blob/master/main.tf#L64

https://github.com/cloudposse/terraform-aws-eks-cluster/blob/master/variables.tf#L74

hi,somebody have a module for create a cloudwatch event for keep lambda warm?

@Richy de la cuadra found this example, can be put into a module https://github.com/terraform-providers/terraform-provider-aws/issues/3040

this has a working example https://medium.com/build-acl/aws-lambda-deployment-with-terraform-24d36cc86533

very helpfull thanks @Andriy Knysh (Cloud Posse)!

Anyone else experiencing very slow TF Lambda deployments? 30 seconds+ for every single function. This was much faster before. Already upgraded to the newest TF, no changes.

we did not test lambda recently

does it eventually finish successfully ?

can you test the same from the AWS console and compare the times?

@squidfunk are things slow inside a geodesic container?

(he’s not using geodesic yet)

ok, just curious if it was related to TF_CLI_PLAN_PARALLELISM=2 – but looking at that var now, I’m guessing it only affects plan ?

Yea… but you’re right, that does slow things down.

We have that due to running terraform under ECS wher we experienced AWS platform rate limits

Hi, is anyone using Travis with private modules, what’s the best way to get git+ssh working there ?

Hey guys. I was at my girlfriend’s parents for the last days and, well, it turned out it was just the upstream. They have a really bad internet connection. Stupid me Sorry!

@squidfunk AWS WorkSpaces FTW!

of anyone is looking for something to contribute, looks like there’s a bug affecting a lot of users: https://github.com/cloudposse/terraform-aws-cloudfront-s3-cdn/issues/39

has anyone tried terratest ? new toy from gruntwork ?

Yes it’s pretty good

Tests written in go.

I will eventually share my testing setup but that will take around 3 weeks

Combination of: Atlantis Terratest GOSS TFlint

I like the look of terraformcompliance tho

BDD is nice

Was talking to Erik on Monday and turns out my plans for Atlantis weren’t right. It’s meant to be used more as a CD tool than a tester… but all will be written up

https://www.terraform.io/docs/providers/aws/guides/version-2-upgrade.html

geez, so many breaking changes

ran into this: https://github.com/terraform-providers/terraform-provider-aws/issues/7846

(our bad for not pinning provider)

All these deprecation but where’s the new features list yo https://github.com/terraform-providers/terraform-provider-aws/blob/master/CHANGELOG.md

breaking changes are what major releases are for, features go in minor releases

Hi guys, i’m having kind of an issue with the “terraform-aws-iam-user” module. Once I create a user and i get the keybase_password_decrypt_command i’m perfectlly capable of decrypting the user password but then when i try to login to the aws console it says that the login information is incorrect. Just as a quick debug i changed the user password using the root account and i logged in successfully.

Could anyone help me debug this? Thanks in advanced. BTW you have some awsome terraform modules. Greetings from Argentina!

hi all, I am deploying a consul and vault cluster on aws using terraform. do you have any example of pipeline able to manage the rolling upgrade of the cluster? all the nodes are in ASG

Solved/Uderstood it!

the echo command is adding a “%” at the end of the password

for some reason

Anyone running into x509: certificate signed by unknown authority with registry.terraform.io?

running into it when running a terraform init

@sweetops Is your certstore up to date? Does curl connect without issues?

curl connects without issues

It’d be nice if we could configure EKS workers to be spot instances here: https://github.com/cloudposse/terraform-aws-eks-workers

So, I just used this example to get up and running with an EKS cluster: https://github.com/cloudposse/terraform-aws-eks-cluster

But I’m getting the error: no nodes available to schedule pods

I can see that I have one worker node up, but I can’t see that node via kubectl

lol found someone that has the same issue, with no solution: http://www.snowcrash.eu/kubernetes-eks-warning-failedscheduling-default-scheduler-no-nodes-available-to-schedule-pods/

@cabrinha have you seen our example in terraform-root-modules

no, i was working off the examples dir: https://github.com/cloudposse/terraform-aws-eks-cluster/tree/master/examples/complete

Have a look at this too https://github.com/cloudposse/terraform-root-modules/tree/master/aws/eks

@Andriy Knysh (Cloud Posse) has been keeping this up to date

@Erik Osterman (Cloud Posse) thanks for pointing me in the right direction, just wondering why the module doesn’t have a complete working example in it

keep in mind, recent changes in terraform-aws-provider has introduced some issues; pinning to a pre-2.0 module might help or ensuring you’re on 2.1

there are so many moving pieces in terraform between versions of terraform, versions of providers and versions of modules. We cannot test all combinations. We try to keep it all stable and working, but we are not a large company :smiley: and all maintenance is paid for by cloudposse (~$10K/mo!) Despite that, we try to stay on top of our 130+ terraform modules, ~60 tools in cloudposse/packages, 50+ helm charts in cloudposse/charts, 300+ original open source projects on our github, dozens of pull requests every week, code reviews, etc. It’s not easy and pull requests are greatly appreciated.

I understand thanks

@cabrinha the example here https://github.com/cloudposse/terraform-aws-eks-cluster/tree/master/examples/complete is a complete and working example, was tested about a month ago and also deployed by a few other people

did you run this for the worker nodes to join the cluster https://github.com/cloudposse/terraform-aws-eks-cluster/blob/master/examples/complete/kubectl.tf ?

Anyone already established a smart way of getting the VPC ID from another account (data resource from another account) when VPC peering?

# Mgmt VPC
data "aws_vpc" "requester_vpc" {
  tags = {
    Name = "${var.stage}-vpc"
  }
}

# Other sub-account VPCs
data "aws_vpc" "accepter_vpc" {
  tags = {
    Name = "${var.stage}-vpc"
  }
}

module "vpc_peering" {
  source           = "git::<https://github.com/cloudposse/terraform-aws-vpc-peering-multi-account.git?ref=master>"
  namespace        = "he"
  stage            = "${var.stage}"
  name             = "vpn"
  requester_vpc_id = "${data.aws_vpc.requester_vpc.vpc_id}"
  requester_aws_assume_role_arn = "arn:aws:iam::xxx:role/vpc-admin"
  requester_region = "${var.aws_region}"
  accepter_vpc_id  = "${var.vpn-vpc}"
  accepter_aws_assume_role_arn = "arn:aws:iam::xxx:role/vpc-admin"
  accepter_region = "${var.region}"
}

variable?

Would like to avoid hardcoding… edit: or user prompts

But I’ve taken on the tech debt of having it hardcoded in my Geodesic Dockerfile

How is it hard coding if you have it as a variable?

It is either hardcoded (the vpc-id) or it requires input. Both aren’t a typical solution I go for..

you can look it up, but you have to pass the provider to the data source so the credential has permissions to query the other account

https://www.terraform.io/docs/configuration/providers.html#alias-multiple-provider-instances

but that would require some identification to be added eg. the var.stage in the example (+ the provider configuration) so why not just use vpc-id?

personally i agree, i don’t like implicit references like that, but i’m just offering an answer to the question tho

so why not just use vpc-id?

I would 100% much rather have a global variable for a role ARN than a one/two-use variable like VPC-id being hardcoded

Thanks the multi provider link should do the trick.

Here’s how we used the terraform remote state provider to achieve a similar requirement https://github.com/cloudposse/terraform-root-modules/blob/master/aws/root-dns/ns/main.tf#L27-L51

Trying to hunt down the source of a count error.. despite project having no lists or maps

what was the tf tool being used to mask passwords

output "db_password" {
  value       = aws_db_instance.db.password
  description = "The password for logging in to the database."
  sensitive   = true
}

`

sensitive = true

oh really

tfmask was the tool though

thanks dude

https://github.com/cloudposse/tfmask

Hi guys sorry to bother you with a newbie question. I’m not completely sure as to how to use both terraform-aws-iam-user and terraform-aws-iam-assumed-roles combined. Can anyone help?

@AgustínGonzalezNicolini here is an example https://github.com/cloudposse/terraform-aws-iam-user/blob/master/examples/simple/main.tf

you create IAM users and then add them to the groups (admin or readonly)

You should start with https://github.com/cloudposse/terraform-aws-iam-user/blob/master/main.tf

terraform-aws-iam-user help to create a normal IAM user (u can login with password aws_iam_user_login_profile). After that, U need to assign this user to a group.

another example, you create users and then https://github.com/cloudposse/terraform-root-modules/blob/master/aws/root-iam/root.tf#L14

terraform-aws-iam-assumed-roles is a big module, create a group, role, policy ….

Anyone got a favourite way of creating openvpn server? Unlicensed as only need max 2 users.

Current choices are: 1) Use existing AMI on marketplace 2) Use existing AMI on marketplace as base in packer and provision with my stuff 3) New packer project htat installs openvpn with Ansible or Docker (or Ansible installing container…)

@Arvind Few things i would like to understand:

1. How can sententiously multiple developer can work on one Terraform Repo.
   S3 Backend
2. How can I manage multiple .tfsstate file, as I have three environments lab/stage/prod
   Geodesic. Have multiple accounts. Each one has a bucket. You go into a different Geodesic container for each environment (one env per account).
3. What is the different b/w variables.tf and .tfsvar.
   [variables.tf](http://variables.tf) declares your variables and their data structure. A .tfvars file sets the value of the variable neatly, instead of doing it in the same file as say a resource creation. Typical project of mine:

jenkins: ec2.tf elb.tf jenkins_sg.tf outputs.tf r53.tf terraform.tf > declare backend and other common data bits variables.tf

@Arvind has joined the channel

Do we have any sample git repo where we are provisioning the infrastructure as per environments basis ex:stage/lab/prod.And i need to understand how terraform.tf file will read the environments specific variables. ex: in lab i require Redis instance smaller size while in stage and prod i require bigger size Redis instances.

@Arvind you mean like https://github.com/cloudposse/testing.cloudposse.co ?

@joshmyers’s suggestion is spot on

basically, you put all common code in terraform-root-modules (like blueprints)

then you use that code in all of your account environments (E.g. [testing.cloudposse.co](http://testing.cloudposse.co) is ours)

we have root, staging, prod, etc.

that’s where the custom settings go for each environment.

it’s been our experience that we want to run identical code (e.g. modules) in environments like staging and prod, but staging and prod necessarily differ in areas like instance sizes or perhaps the number of clusters

for example, in production for an Enterprise SaaS platform, there might be one cluster per enterprise customer, but you wouldn’t run one cluster per enterprise customer in staging.

Hey, I have a issue with terrarom-aws-iam-user. First i run apply, my user add to group admin, but when i run again, my user is remove to group

  ~ module.assumed_roles.aws_iam_group_membership.admin
      users.#:          "1" => "0"
      users.1070365961: "quang" => ""

  ~ module.quang.aws_iam_user_group_membership.default
      groups.#:          "0" => "1"
      groups.3764225916: "" => "g_ops_users"

@xluffy what are you running?

Anyone had this before when using the ec2 module? https://github.com/cloudposse/terraform-aws-ec2-instance/issues/37

* module.instance_01.output.network_interface_id: Resource 'aws_instance.default' does not have attribute 'network_interface_id' for variable 'aws_instance.default.*.network_interface_id'

is there any possibility of mentioning an EBS resource in launch configuration using interpolation? ( i am trying to abstract out this , to conditionally mention an extra block device in LC)

heads up this is no longer valid, https://github.com/cloudposse/terraform-root-modules/blob/master/aws/account-dns/main.tf#L26-L35

* aws_route53_record.dns_zone_soa: 1 error(s) occurred:

• aws_route53_record.dns_zone_soa: [ERR]: Error building changeset: InvalidChangeBatch: [Tried to create resource record set [name='some.example.com.', type='SOA'] but it already exists]
 

used to work, 27 days ago for sure

but also not needed as SOA gets created by default

@Jan please see https://github.com/terraform-providers/terraform-provider-aws/issues/7846

ah

It’s not that we want to create a new SOA; it is essential to set a low SOA TTL to prevent negative ttl caching in dynamic service discovery environments. E.g. a negative cache hit can cause 15 minutes of downtime for no good reason.

Has this guy popped up in here yet? https://github.com/iann0036/AWSConsoleRecorder

Hah, wow, haven’t seen this before

Holy cow that’s.. an amazing project. I do find aws easier to understand and ‘do’ with TF than when I use the console though tbh

I wonder if it works

I just got it today via meduim and quickly tried it. It’s a bit raw but an awsome tool

Hi everyone. I don’t know which channel to post this question but as I tend to implement with TF so I posted here.

My question is : How did you manage ECR repository on Terraform or alternative tools you using ? The problem I facing is

• i want to create multiple ecr repositories in 1 TF code, so which is the good way to generate those codes ? I writing go template. I think I will use that to generate the TF code from Yaml File My yaml file format like this ``` [name-of-repository]
  • [username-arn-id-a] - pull
  • [username-arn-id-b] - push [name-of-repository2]
  • [username-arn-id-d] - pull
  • [username-arn-id-c] - push ``` Any one has other ideas feel free to tell me once, Appreciate !

@h3in3k3n the way I would do it is to create/find a module that has a proper ecr policy set and just use that one. The resource by itself in terraform is super small but with the policy it can get a bit bigger also you might need to pass the policy per repo or whatever your use case requires

Hi People, any recommendation for an IDE or a vim setting in which I can detect unused variables in terraform ?

I think that one of the jetbrains IDEs has this feature.

Jetbrains IDEA has a very nice terraform support

Can detect missing and unused variables

yeah that’s what I remember as well

but I don’t want to switch my precious VIM

And highlights everything

It can emulate VIM editor so you don’t lose anything :)

hmmm

ok let me check that out

thanks!

It’s a very nice product in my opinion

vscode does this also for terraform

Supports all possible languages and frameworks

ok vscode I’ave installed but I haven’t seen this feature in action , do I need to install a special plugin @loren?

Vscode not bad too

yes, you need a plugin, one sec and i’ll get the name

thanks, I already have the tf plugin installed though

yeah, should just be terraform

you don’t see references like this?

no I get

for every var..

I think it’s the way we use terraform ..

oh, it might not work when the references are in a different file?

yes they are in a different folder..

and then we sprinkle some makefile magic

to make an apply

well, different file, same directory works fine

sorry different folder

so the plugin thinks it’s a different state

ok now I get it

that’s in a variables.tf file, same directory as main.tf where all the references are

yeah , that’s not the case where I currently work at

gotcha

but thanks for the help ! now I know that vscode works as expected

it is hard to do this kind of lint/validation when you have some wrapper/generator around your tf code

well, harder, i guess. need to generate it, then validate the output

yeah, I am not a fan of the wrappers as well.

but there’s a way , the Makefile generates a main.tf file

so I can just use that one

yep

yeah

thanks guys for your help!

Much appreciated

hmm I found out that the current plugin in vsode sets 0 references even if the variable is invoked inside a function

like so "{tf_function(var.something)}"

IDEA can actually see and analyze variables in functions (and even in plain text)

it does not show the ref count as vscode does

but when you click “Find Usages” on anything

it shows all references

oh nice..

I am definitelly trying it out

highlights wrong/missing vars

that moment when you’re getting a product demo from a company and you notice a Cloud Posse browser tab + the label module naming format

Turns out, @Erik Osterman (Cloud Posse) is known from the days at Sumo (by Nick Coffee)

Hahah wow! small freggin world.

Yea, I remember Nick.

He’s at Harness https://harness.io/

if I want to to test terraform 0.12 beta.. how do I get providers built against it?

I haven’t yet had a chance to try this, but it was on my mind.

Using Geodesic across multiple AWS accounts for each stage, I have Route 53 records to create. I have one domain name: acme.co.uk I own [acme.co.uk](http://acme.co.uk). I have [acme.co.uk](http://acme.co.uk) NS pointing to my ROOT account.

Scenario: I have to create r53 records, say [test.acme.co.uk](http://test.acme.co.uk). Naturally I want to create this on my testing account. I want this r53 record to be public. Naturally this means the testing account needs to have an [acme.co.uk](http://acme.co.uk) r53 public zone… but wait… I already have a public zone for this in ROOT with the public NS pointing to ROOT account.

Problem: Is this possible? Or to have public records for my one domain, must I assume a role into my ROOT account and only create public records there?

we do DNS zone delegation. In the root DNS, we provision the parent domain ([acme.co.uk](http://acme.co.uk)) Zone and subdomains (e.g. [test.acme.co.uk](http://test.acme.co.uk)) zones, then we provision the DNS zone [test.acme.co.uk](http://test.acme.co.uk) in the test account, and then add the name server for [test.acme.co.uk](http://test.acme.co.uk) in the parent DNS zone in the root account

https://github.com/cloudposse/terraform-root-modules/tree/master/aws/root-dns

https://github.com/cloudposse/terraform-root-modules/blob/master/aws/root-dns/parent-prod-ns.tf

for root ^

https://github.com/cloudposse/terraform-root-modules/blob/master/aws/account-dns/main.tf

for member account ^

I was wondering the same thing today. I’m debating with myself if it makes sense to split your naming into an infastructure zone like acme.cloud and have a separate one for your public-facing domain, being more picky which namespaces to use.

we recommend having a service discovery domain which is not your branded vanity domain

so we have [cloudposse.org](http://cloudposse.org) as our infra domain

[cloudposse.com](http://cloudposse.com) as our vanity domain

(and [cloudposse.co](http://cloudposse.co) is just for our examples)

Yes so CloudPosse.org has NS records for all our member accounts

Makes total sense now that you say it.

prod.cloudposse.org NS points to prod.cloudposse.org zone

The only issue I thought might be a little bit tricky is with certificates, but I think it would just be the load balancer that needs to have a certificate for the vanity domain applied, so not too bad.

it’s possible now to associate multiple ACM ARNs with an ALB

the alternative is to use SANs

Ah, I somehow missed that. If possible, I try to avoid using SANs on vanity domains, it’s just neater.

Thanks for the clarification.

what we have in the root account cloudposse.co

certificates get provisioned in each account separately https://github.com/cloudposse/terraform-root-modules/blob/master/aws/acm/main.tf

Since we’re on the subject, is it a good idea to create a hosting zone for a customer’s site and ask them to create an NS record, instead of giving them the ALB CNAME? My thoughts are that this way I can change the ALB if ever required without having to request a customer to make a DNS change.

if the service discovery domain will not change, then probably yes, give them the NS records

on the other hand, if you have a service discovery domain, e.g. [prod.mycompany.co](http://prod.mycompany.co) which you control, you can ask them to add the CNAME record from their main (vanity) domain to [prod.mycompany.co](http://prod.mycompany.co) (which should never change as well). No ALBs involved

Is one approach better than the other?

Amazon charges for CNAMEs, if that kind of penny pinching bothers you. Clarification: cnames count towards your queries, but those are stupidly cheap anyways. I’ve had bosses squawk over it though. A ALIAS records do not.

I think the two approaches, it’s mostly just how you want to manage it.

In this case, I think the CNAME would be on the customer side, so no additional cost

Oh, I missed the “customer’s site” - in that case, I might say you want to control your own domain stuff, make them use cnames, but that’s just me

CNAME has more visibility. Someone can access the site using [customer.mycompany.co](http://customer.mycompany.co) instead of [site.customer.co](http://site.customer.co) as intended.

I just tried the NS approach for the first time, and the ask was met with some skepticism (from the customer)

Thanks.. great discussion. Your solution seems v easy @Andriy Knysh (Cloud Posse)

thanks for usage screenshots also

Are you using some of our terraform-modules in your projects? Maybe you could leave us a testimonial! It means a lot to us to hear from people like you.

Could resources be deployed into two different AWS accounts having a single “TF environment” and still use some sort of ENV variables for keys? Is there even support for vault?

we handle that by having “generic” TF modules (that provision resources but all settings come from variables, they have no identity by itself). Then we have Docker containers (geodesic in our case) for each AWS account. We copy the modules into each containers. All those TF vars get populated from ENV vars which can come from a few different places (Dockerfile, CI/CD pipeline, AWS SSM, chamber, https://direnv.net, etc.). When starting a container for a particular account, it already has all the code (TF, Helm, Helfiles, etc.) and all the ENV vars populated, so the container has all the info to provision infra for that account

since all settings are coming from ENV vars (including AWS regions and account IDs), and all TF modules use terraform-null-label module to uniquely name all resources, the same setup could be used with multiple AWS accounts, or just with one if needed (in which case all containers will login/assume role into the same AWS account, but all the resources will have diff environment/stage in their names, so all names are unique anyway)

Does this work cross accounts? Would I just need to create another hosted zone with the same NS?

it is cross-accounts

in the member accounts (prod, staging, dev) you create corresponding DNS zones (e.g. prod.cloudposse.co)

then get the NS servers from them

and update the NS records in the root account

obviously you update the NS in the registrar (if it’s not Route53) to the root DNS zone name servers (first record in the image above)

then, to put it very simple, every request to let’s say prod.cloudposse.co will first hit the root zone for cloudposse.co and the DNS server will return the name servers for prod.cloudposse.co

then all other records for prod.cloudposse.co will be requested from its name servers (which could be in the same AWS account or diff, does not matter)

Anyone else get access to TFE for the free remote state? I’m curious how people feel about it.

Can’t comment on TFE but I’m very happy with my S3 backend. TFE is out of the box and you pay for it. S3 backend is ‘free’ once you get your head around it.

@Erik Osterman (Cloud Posse) perhaps for the first hangout I could demo how to set up one of my aws accounts to work with geodesic, aws vault, and s3 backend

I think @johncblandii meant Terraform Saas which is expected to be affordable for many once it is public. If so, my question is more like this:

1. Will I be able to run Terragrunt there (as a custom docker image with all other deps)?

@Andriy Knysh (Cloud Posse) could you just double check I’m doing this right please?

ROOT account: Has acme.co.uk public zone Has NS record for sandbox.acme.co.uk pointing to the NS addresses that come default with the sandbox.acme.co.uk r53 zone I created in my sandbox account

Sandbox account: Has sandbox.acme.co.uk private zone without default NS records Has test.sandbox.acme.co.uk A record to 127.0.0.1

Expected result:

host [test.sandbox.acme.co.uk](http://test.sandbox.acme.co.uk)
127.0.0.1

Actual result:
Host test.sandbox.acme.co.uk not found: 2(SERVFAIL)

You need a public zone for sandbox.acme.co.uk.

Ty

Private zones only work within the VPC.

Great fab. That’s now working

Man that’s so cool. So much easier. What a great piece of knowledge right there

I wonder if I can go a step further

If you want to see how that works you can use dig to see the delegations happening: dig ns [sandbox.acme.co.uk](http://sandbox.acme.co.uk) +trace +nodnssec

ROOT: acme.co.uk Public

Sandbox: sandbox.acme.co.uk Public NS to sandbox.acme.net

Sandbox: sandbox.acme.net Private

I don’t like the idea of my internal infra being on a public zone

Got a few solutions up my sleeve that I’m going to test out.

As far as I know you can’t scan DNS records easilly

Oh really? A long time ago someone told me “don’t put internal infra on public DNS, it just makes it easier for intruders to map out your kit”

kinda just stuck as it sounded reasonable

We have that split dns setup in Route53. I wouldn’t do it again, starting over

Did it for that same reason, originally

So, not worth it?

Not with the same zone name, for sure

A quick article I just found on this topic is https://hackertarget.com/find-dns-host-records/. The only way to get the records is doing a DNS Zone Transfer, which doesn’t work from the public.

Use a totally different zone name in the private zone, and that’s generally ok

If you have a private zone, I’d use something like .local or .acme as the root zone.

The problem is, if you are working from outside (like you do with a multi-account AWS setup), you will run into more problems than not, if stuff doesn’t work.

Also, we run into problems we with multi account setups, where the accounts cannot resolve each other’s records from the private zones. There are ways of doing it, but it’s kludgey. So if you have internal services that should be resolvable across accounts, private zones end up being painful. This basic NS delegation that works with public zones does not work at all with private zones

My goal was:

VPC peer a VPC with Jenkins + VPN to all sub accounts. Continue having acme.co.uk on our ROOT account where all our infra currently is. Have a sub-account for each stage. Have all apps go through a loadbalancer that terminates SSL on acme.co.uk (we own this + have SSL). Have all internal infra on acme.net (we don’t own this and no need for ssl as terminates on ELBs)

should be resolvable across accounts Yep jenkins would need this big time

I’m hoping the new route53 resolvers will get me out of this mess

I didn’t really understand what this was showing to me. When I searched my public acme.co.uk only two results came up and both had public IPs

What does this tell me?

This was about finding DNS records out of the blue.

But what does it tell me about my current setup that only two of our many public A records in our public acme.co.uk zone appear?

Nothing, I was just backing up my claim.
oscarsullivan [1:00 PM]
Oh really? A long time ago someone told me “don’t put internal infra on public DNS, it just makes it easier for intruders to map out your kit”

Ah, now I understand what you mean. You should only receive all the records that are for acme.co.uk (and not any subdomain).

That site is doing more than just attempting a zone transfer (which doesn’t work with route53). They’re also scraping for site info and guessing common names

Yes, it was more about the text, than the tool itself.

Hey, I checked some modules about VPC (on GitHub). But nobody implements a Network ACL on AWS. What are the reasons why don’t do that?

Hey guys, I’m trying to create a domain zone on AWS Route53 and create a certificate that will be used in an ALB later. However, I need to validate the record first. I’m trying the validation_method="DNS" but is taking ages and nothing happens. My impression is that I need to create the domain, create the ALB, assign a record to the ALB and then try to validate the record. Is that right?

By the way, I’m talking about this: https://www.terraform.io/docs/providers/aws/r/acm_certificate_validation.html

Hi @Valter Henrique so the dns zone you are requesting the certificate for should be accessible von the internet, meaning if the registrar is not Route53, then you need to create the zone in Route53, get the name servers and update the NS records in the registrar

In other words, if AWS can’t access your dns records, the validation will take forever and never finish

Right, thank you very much for that @Andriy Knysh (Cloud Posse)!

I was almost bashing my head against my keyboard hehehe

Yea, we were bitten by that ad well. Certificate Manager adds the validation record to your zone, and you think if it can write it, it should be able to read it as well :)

But no, it tries to read using public internet, not the internal network

Indeed

But of course, the purpose of that is to validate that you own the domain. Only if you bought the domain, you could update its name servers. Otherwise I could create a DNS zone for your domain and request a cert for it

I was looking at the ecs modules and I’m trying to figure out, how it’s supposed to work with atlantis. The terraform-aws-ecs-atlantis module uses the default-backend for both service tasks (404 and atlantis), but shouldn’t it use the container image from ECR for the atlantis instance? (I hope this is the right place).

@mmuehlberger sounds about right. You might notice that we hardcore the image to the default backend. That is by design.

The image should always be updated using codebuild/codepipeline, so if you see the 404 then you know it didn’t work :-)

https://github.com/cloudposse/terraform-root-modules/tree/master/aws/ecs

This is our example implementation

https://github.com/cloudposse/root.cloudposse.co/blob/master/buildspec.yml

That’s the build spec we use

God, I’m an idiot (was a long day already). Of course it get’s updated in the CodePipeline.

Because I used a private helmfiles image, the CodeBuild project failed, and it didn’t update yet. Duh.

The problem is completely on my end. Sometimes I get lost in the dependency chain of Terraform modules.

Ok! Let me know it goes…

It deploys fine now, as it should be (thank you for the pointers). Adding encrypted secrets to the buildspec is kind of a hastle though, because the user is burried deep in modules. Also there is issues with the healthcheck and atlantis shutting down. I need to investigate further tomorrow.

we use chamber to write all secrets to ssm

@mmuehlberger are you using atlantis with #geodesic?

Yes, I’m using geodesic in a slightly more modern form than reference-architectures (like testing). It can read the parameters from SSM just fine, the role just doesn’t have permission to decrypt them.

Cool!

Does anyone knows if it is possible to construct from a map a string.

For example

{
   "key1" = "val1"
   "key2" = "val2"
}

To: "key1, val1, key2, val2"

Currently having a tf module wich accepts as input a map for tagging in AWS. But there is on one place I need to pass the tags as a list. I prefer to keep my work backwards compatible.

Anyone familiar with this?

Looks like the aws_region is not passed through the module itself to the submodules..

Hmm having looked at the jenkins modules and the erroring sub modules it does pass aws_region down

https://www.terraform.io/docs/providers/aws/guides/version-2-upgrade.html#data-source-aws_region

Simply remove current = true from your Terraform configuration. The data source defaults to the current provider region if no other filtering is enabled.

:thinking_face:

Thanks for the link though I don’t understand to which file it refers nor do I recognise the current = true flag

Ah got it

ah man

https://github.com/cloudposse/terraform-aws-cicd/blob/master/main.tf#L4

its everywhere

I’m going to have to use so many forks until this is merged in lol

the argument has been deprecated for a while, just printing warnings instead error and exit non-zero

removing it is backwards compatible for a reasonable number of versions

PRs made for it

any idea how to change to my branch:

  source              = "git::<https://github.com/osulli/terraform-aws-cicd.git?ref=heads/osulli:patch-1>"

instead of tags

I’ve tried heads

Just use the branch as a ref. ref=master for instance

ty

fab that works

Gone ahead and put it in here: https://github.com/cloudposse/terraform-aws-jenkins/pull/39

That’s weird.. Still getting …

Error: module.jenkins.module.cicd.data.aws_region.default: "current": [REMOVED] Defaults to current provider region if no other filtering is enabled

even though..

oscar@infinitum:~/code/terraform-aws-efs-backup$ git checkout patch-1 
Branch 'patch-1' set up to track remote branch 'patch-1' from 'origin'.
Switched to a new branch 'patch-1'
oscar@infinitum:~/code/terraform-aws-efs-backup$ grep -iR "current" ./*
./docs/terraform.md:| noncurrent_version_expiration_days | S3 object versions expiration period (days) | string | `35` | no |
./README.md:  noncurrent_version_expiration_days = "${var.noncurrent_version_expiration_days}"
./README.md:> NOTE on Security Groups and Security Group Rules: Terraform currently provides both a standalone Security Group Rule resource 
./README.md:| noncurrent_version_expiration_days | S3 object versions expiration period (days) | string | `35` | no |
./README.yaml:    noncurrent_version_expiration_days = "${var.noncurrent_version_expiration_days}"
./README.yaml:  > NOTE on Security Groups and Security Group Rules: Terraform currently provides both a standalone Security Group Rule resource 
./s3.tf:    noncurrent_version_expiration {
./s3.tf:      days = "${var.noncurrent_version_expiration_days}"
./variables.tf:variable "noncurrent_version_expiration_days" {

Next…

* module.jenkins.module.efs_backup.output.sns_topic_arn: Resource 'aws_cloudformation_stack.sns' does not have attribute 'outputs.TopicArn' for variable 'aws_cloudformation_stack.sns.outputs.TopicArn'

Has anyone used the Jenkins module lately?

https://github.com/cloudposse/terraform-aws-efs-backup/issues/36

Ok it was looking for sns.TopicArn instead of sns.arn https://www.terraform.io/docs/providers/aws/r/sns_topic.html#arn

arn not valid either. Just going to remove the output.. Hopefully the output isn’t referenced elsewhere? Not sure you can reference an output anyway. That’s just visual.

Well, you can when looking up a remote terraform state.

No but it’s not like in another module ${output.x.x} is a thing

So it should be safe to remove this one output that breaks the whole project

Weird you can’t GET zone_id for r53 zone? https://www.terraform.io/docs/providers/aws/d/route53_zone.html#zone_id

Damn, received this a bunch of times.

module.jenkins.module.efs.aws_efs_mount_target.default[1]: Creation complete after 2m44s (ID: fsmt-xxx)

Error: Error applying plan:

1 error(s) occurred:

* module.jenkins.module.efs.aws_efs_mount_target.default[0]: 1 error(s) occurred:

* aws_efs_mount_target.default.0: MountTargetConflict: mount target already exists in this AZ
	status code: 409, request id: 0df8f8c2-xxx-xxx-xxx-55a525bfd810

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

changed the versions of CP/efs but no luck. It seems to be duplicated to create it twice???

Yet doesn’t appear to be created twice:

oscar@infinitum:/devops/terraform/providers/aws/jenkins/.terraform/modules$ grep -ir "aws_efs_mount_target" ./*
./08d41c3e3037162fadbb4393ce396759/outputs.tf:  value = ["${aws_efs_mount_target.default.*.id}"]
./08d41c3e3037162fadbb4393ce396759/outputs.tf:  value = ["${aws_efs_mount_target.default.*.ip_address}"]
./08d41c3e3037162fadbb4393ce396759/main.tf:resource "aws_efs_mount_target" "default" {
./808f0aa181de2ea4cc344b3503eff684/efs.tf:data "aws_efs_mount_target" "default" {
./808f0aa181de2ea4cc344b3503eff684/cloudformation.tf:    myEFSHost                  = "${var.use_ip_address == "true" ? data.aws_efs_mount_target.default.ip_address : format("%s.efs.%s.amazonaws.com", data.aws_efs_mount_target.default.file_system_id, (signum(length(var.region)) == 1 ? var.region : data.aws_region.default.name))}"
./808f0aa181de2ea4cc344b3503eff684/security_group.tf:  security_group_id        = "${data.aws_efs_mount_target.default.security_groups[0]}"

only one resource for it in the whole of the jenkins project and its modules

wants to create it a second time despite it already existing

Found the cause: terraform-aws-efs

Inside [main.tf](http://main.tf)

resource "aws_efs_mount_target" "default" {
  count           = "${length(var.availability_zones)}"
  file_system_id  = "${aws_efs_file_system.default.id}"
  subnet_id       = "${element(var.subnets, count.index)}"
  security_groups = ["${aws_security_group.default.id}"]
}

The length was what was causing multiple to be created……. so I just only used one availability zone and no longer receiving that dupe error,

Onto the next error

Error: Error refreshing state: 1 error(s) occurred:

* module.jenkins.module.efs_backup.output.datapipeline_ids: At column 48, line 1: map "aws_cloudformation_stack.datapipeline.outputs" does not have any elements so cannot determine type. in:

${aws_cloudformation_stack.datapipeline.outputs["DataPipelineId"]}

Oh. Cool. I can’t terraform destroy either

^ Commented out the output..

And now….

* aws_elastic_beanstalk_environment.default: InvalidParameterValue: No Solution Stack named '64bit Amazon Linux 2017.09 v2.8.4 running Docker 17.09.1-ce' found.
	status code: 400, request id: d7bc0ae2-2278-4bbd-9540-bda532e9cd71

Feel like I’m getting closer

You know what.. I’m going to try the live version and either: 1) Define the AWS provider version 2) Use it how it is + grep for the deprecated argument and just manually remove it

Guys, is there a way to do a split and get the last element of the list ?

or do I need to know the size of the list ?

try [-1] for the index

-1 doesn’t work

I ended up using length function

you can use https://www.terraform.io/docs/configuration-0-11/interpolation.html#slice-list-from-to-

For getting last element in a list, I think can use like that

variable "public_subnet" {
  default = ["10.20.99.0/24" , "10.20.111.0/24", "10.20.222.0/24"]
}

output "last_element" {
  value = "${element(var.public_subnet, length(var.public_subnet) - 1 )}"
}

will return to last_element = 10.20.222.0/24

@oscarsullivan_old https://github.com/cloudposse/terraform-aws-jenkins was tested by us about a year ago (was deployed many times at the time), so prob a lot of things changed since then

I’ve realised @Andriy Knysh (Cloud Posse) trying so hard to use it lmao

I can’t believe no one else (?) has used it recently though.

i think some people used it

just need to find them

We have not personally used it for some time

It’s worrying that there are issues in it that actually prevent me from running terraform destroy?!

#codefresh FTW

Astonished that’s possible

I don’t have everything in containers yet

Hello! is this the place to ask for help about https://github.com/cloudposse/terraform-aws-cloudfront-s3-cdn ?

@oscarsullivan_old what we noticed before, if deployment fails for any reasons, you need to manaully destroy those data pipelines

TF does not detroy them (uses CloudFormation)

and also, all of that should be updated to use https://aws.amazon.com/backup/

Honestly, looking at the architecture for deploying a single, non HA Jenkins on beanstalk is enough for me to say I just don’t think it’s worth running jenkins. Plus, to get HA with Jenkins you have to go enterprise. At that point might as well look at other options.

instead of the efs-backup module which was a hack

lmao data pipelines aren’t even available in London

I wonder where they were created

Checked all the data centers and not there lawwwwwwd

and cloud-nuke won’t solve this either

Ya @Andriy Knysh (Cloud Posse) is correct that pipelines should now be replaced with the backups service

execute me now pls

How long did you wait @Mohamed Lrhazi ?

Any idea what I am missing?

oh.. maybe 10 mins ?

It takes sometimes up to 30 minutes to create a distribution

h.. but it says DEPLOYED as status…

Hrm

10 m is faster than I have seen. Not sure what is wrong based on your example.

and I think its actually been more than 30mins

You can go into the webconsole and poke around

still giving same error…

yes, I looked at the s3 bucket and looks like it did assign the right perms, from what I can guess

Did you use a custom SSL cert?

Nope!

could that be it? docs dont says thats required!

If yes, the cloudfront provided one will not work

In your code I see the cert ARN provided

Oh sorry you’re right!!! I did add that

Also the S3 bucket is not a website

oh.. the module does not do that for me?

You have to access any file by adding its name after the URL

There is another module for that

ok. let me poke around and see if I can make it work… one last question.. is the module supposed to also create route53 records needed?

cause it does not seem like it did in my simple test case.

I just want this jenkins module off of my account now.. Any idea how to get past this:

* module.jenkins.module.efs.module.dns.output.hostname: variable "default" is nil, but no error was reported
* module.jenkins.module.elastic_beanstalk_environment.module.tld.output.hostname: variable "default" is nil, but no error was reported

Have tried commenting that output out and also removing all outputs.tf files

Just need to be able to run terraform destroy

@Mohamed Lrhazi 1 sec

been in a half hour loop trying to purge it but always end up at the same spot

@oscarsullivan_old did you find the pipelines in the AWS console? you need to manually delete them

They don’t exist

I looked through all the data centers

I’m set to eu-west-2 (london) and that’s not even an available data center!

what do you see in CloudFormation? Delete those stacks manually

0 stacks

Also checked the US DCs in case it defaulted to CP’s DC

when you run terraform destroy, what’s the error?

hmm, never seen those

Have tried commenting out those ouputs, have tried removing all outputs.tf files, have attried re-applying then destroying etc

I get something like this erry tim I use a CP module

but this is a complex one and hard to cleanup manually

i guess just go to Route53 and delete those records

(and fix the variables and outputs so it does not complain)

There aren’t any R53 records

oh god

i hope you did that in a test account that you could nuke somehow

Yeh but unsure nuke will work on this

if we’re taling cloud-nuke

had a read through the resources it can od

i mean anything to destroy it, not exactly using the nuke module

jeeeeez

Deprecate that god damn repo, please. This has been really unpleasant

so I think part of the issue was that the pipelines are not supported in the region (and maybe other resources)

but the error reporting is bad

@Mohamed Lrhazi this module https://github.com/cloudposse/terraform-aws-s3-website does create an S3 website

and then you use this https://github.com/cloudposse/terraform-aws-cloudfront-cdn to add CDN for it

(this https://github.com/cloudposse/terraform-aws-cloudfront-s3-cdn creates a regular S3 bucket, not a website, and points a CloudFront distribution to it)

OK. Thanks! I assumed this was a full working example: https://github.com/cloudposse/terraform-aws-cloudfront-s3-cdn/blob/master/example/main.tf

since it’s not a website, you have to access all the files by their names, e.g. https://d18shdqwx0ry07.cloudfront.net/index.html

I tried that, still getting same “Failed to contact the origin.” error!

and since you provided your own SSL cert (I guess you created it already), you have to access the site by the DNS name (not CloudFront) : https://catholic.edu/index.html

Well, according to my state file it is GONE…