#terraform (2020-02)

i get “invalid attribute name” after plan

try to use a dynamic . this is ripped from the Terraform Up & Running book.

sorry that paste sucks coming from PDF. It’s chapter 6 tips and tricks

thanks, ill give it a try

actually i dont think that will work, as that will loop though an element “like tags” within the resource, i want it to loop the entire resource and change the provider (i.e. region)

I heard there was work on getting for_each to work for modules, that is likely the limitation im hitting here as well

I’m not an expert, but I suspect you might be right.

Ha, nor am I, but thanks for the input

check this issue and see if any of the workarounds might help.

https://github.com/hashicorp/terraform/issues/17519

ha, my exact issue: https://github.com/hashicorp/terraform/issues/17519#issuecomment-550003810

awesome

guess ill be writing in triplicate for now, until the module support comes out

Also it depends on what your reasons are for going multi region, but from an HA perspective sharing the same state bucket across regions could be limiting true HA if the failed region happens to be where you store terraform state

do you recommend splitting the state per region typically Erik?

i was looking to set up global tables, based on the tf documentation you create all 3 individual tables, then tie them together with aws_dynamodb_global_table resource

but i do see your point, i am using workspaces for ecs clusters that would be reading from same table, suppose i would stick to my same process, and just keep the global table state to single bucket

this is my first jump into multi-region, so im used to all my statefile eggs in the same basket of us-east-1

until the module support comes out This will be a loooooong wait.

But, you can generate Terraform programmatically in which case you get for-each in modules for free.

Here’s one possible approach - https://github.com/mjuenema/python-terrascript though anything that can generate JSON will do - https://www.terraform.io/docs/configuration/syntax-json.html#json-file-structure

Thanks I’ll give it a look

This will be a loooooong wait. https://twitter.com/mitchellh/status/1157288848654647298

do you recommend splitting the state per region typically Erik? I do, if you can stomach the extra complexity of managing an additional state bucket. It also depends on how mission critical this stuff is and if your organization has the (human) resources to manage it. Also, realize these things trickle down to things like DNS zones and service discovery as well. If you’re managing DNS entries for resources in a specific region with a different state backend, then the zone should also be managed in that region.

so from strictly an architectural POV, I think it’s the right way to go. But when considered in light of the management trade offs, then maybe not worth it.

I will strongly recommend not to use a single state bucket for multi region and will strongly recommend to run terraform for how many regions you have by way of using a variable instead of going trough a loop

So you will end up with state buckets per region that is resilient to full region failure

thank you for the answer

Plus you need to keep in mind naming convention for all resources that global like iam

So add the region to the name of every resource

We just went through all this and we are now multi region and we learned a few lessons

just getting started here, so really appreciate all the knowledge to make my journey more pleasant

It is painful, I can tell you that much

I’ve been in the game over 20 years. Can’t be as painful as a bunch of engineers turning wrenches by hand.

You will see….

Soon enough

thanks @Richy de la cuadra we’ll review it ASAP

@Rich Allen please share your module invocation terraform code

for what it is worth, I now do not think this is an ssl issue. If you turn off redirects, and navigate to http I received a origin access error.

@Rich Allen did you request the certificate just for the parent domain, or for subdomains as well (*.[example.com](http://example.com))?

one of the possible reasons for ERR_SSL_VERSION_OR_CIPHER_MISMATCH is cert name mismatch

just the sub domain, not the bare domain

https://kinsta.com/knowledgebase/err_ssl_version_or_cipher_mismatch/

I will reprovision the cert using the bare domain + san

if you using CNAME, this [ourCFDISTROID.cloudfront.net](http://ourCFDISTROID.cloudfront.net) should be included in SANs as well

make sure the CNAME is included in the aliases for the distribution, like here for example https://github.com/cloudposse/terraform-root-modules/blob/master/aws/docs/main.tf#L106

also, did you provision DNS zone delegation in the child account?

since mysite.example.com is in diff account, you need to have a Route53 zone for it in the child account

and add Name Servers to the child DNS zone pointing to the root DNS zone

I did not, I provisioned mysite.example.com -> cfid.cloudfront.net in the root account

it was my understanding that a hosted zone, was unique to an account, so are you saying I should have a hosted zone in both the root and child account?

yes

and zone delegation

https://serverfault.com/questions/530415/what-is-dns-delegation

otherwise DNS resolution will not work

I mean, you can provision everything (master zone, sun-domain zone) in the root account and it will work

but if you are using child accounts, you prob want to provision everything related to the sub-account in it

might be not your case, just throwing out ideas

so I think you need to check the following:

if you provision the site/CDN in the child account, you need to have the certificate provisioned in the same child account and assigned to the CloudFront distribution

do you have two certificates, in root and child accounts?

then the CNAME must be added to aliases for the distribution

so here is the thing: if you created the SSL cert only in the root account and created the sub-domain DNS record in the root account, then the CloudFront distribution URL ourCFDISTROID.cloudfront.net must be added to the SANs of the certificate

the module will not work cross-account, it will not create alias in the parent zone which is in diff account https://github.com/cloudposse/terraform-aws-cloudfront-s3-cdn/blob/master/main.tf#L254

so you have to set var.parent_zone_id = ""

no I must have misspoke, the ssl cert is only provisioned in the child account. The dns validation record/(ACME) was set on the root account.

FYI appreciate the help here, I’m working through a few of these just running a bit behind with your advice haha!

ok

anyway, you need to add the distribution URL to the SANs

and CNAME must be added to aliases for the distribution

Okay so for now, I don’t have multi account dns resolution set up, and I think I would have to authorize and test that change a bit more as affects my scope here. Knowing that is staying the same right now, it seems like I need to do the following: add a SANS record in our certificate for the cf distribution. I must manually validate the ACME challenge, and then I must manually create the mysite.example.com CNAME CFDistro.cloudfront.net record. I should ignore the alias key (as that will not work cross account and I’m manually setting it for now until I can research multi-account dns resolution).

yes

it’s different for multi-account

btw, you can set the alias to the CNAME since the cert is in the same account. As long as CloudFront sees the cert for sub-domain, it will allow you to add CNAME aliases to the distribution

https://www.youtube.com/watch?v=7zHEjMyBGdU

https://www.youtube.com/watch?v=w3JCjsIOMdM

https://www.youtube.com/watch?v=-MVplkiUoFQ

you think terraform knows the count because in your head you know how many instances you want. But TF is not so smart as you

there is not a good way of dealing with that

in many cases, we ended up adding a new var count_of_xxx and explicitly providing the count to TF

also we have some docs on that https://docs.cloudposse.com/troubleshooting/terraform-value-of-count-cannot-be-computed/

(it’s relatively old, was created for TF 0.11. TF 0.12 is much smarter, but still can’t do it in all cases)

Makes sense, I was grasping at straws here, though I think I knew the answer all along

so yea, in those cases we were not able to “fix” it, we ended up adding explicit count OR splitting the code into two (or more) folders and using remote state

The annoying part with it, is sometimes it works when you add a new resource to the existing statefile, but then when you run from scratch, you hit this.

Makes sense why that is, but it’d be good if TF at least gave a warning in those cases

We also used one other way of fixing it. If for example, the count depends on some resource IDs, the IDs are not know before those are created

Try to use names for example

Or any other attributes that you provide

If I understand correctly, I think you are referring to a different issue; the depends_on one.

Very similar though in level of frustration;)

No, if in the count expression you use resources IDs, those are not known before the resources are created

But let’s say you provide resources names to terraform

Those are known before the resources are created

If you use names in the count expression, it might work

But not always

obviously it will work if the names are in an input variable

what I’m referring to, you can reference ResourceA.name in the count, and it could work in some cases even before the ResourceA are created since terraform could figure it out

Oh I see what you mean

Good tip

for example:

all those (known after apply) you can’t use in counts

name - you can, and TF would figure it out

e.g. count = lenght(aws_organizations_account.default.*.name) might work in some cases

count = lenght(aws_organizations_account.default.*.id) will never work

Brilliant, thanks

we use a repo + remote state for each microservice

so we can independently change that service config without having to commit to a big repo

repos are environment agnostic

thanks @jose.amengual, how do you handle changes that apply to every microservice?

pr to the repo, review and once approve terraform apply

you can use different methos to run terraform

does that become cumbersome when you have a lot of micro services? right now we have a monorepo with ~40 microservices… anytime we need to make a change that impacts all of them it is a huge PITA to plan and apply terraform everywhere

we use atlantis… but it ends up being 3 (environments) *40 plans and applies

trying to see if there is a better way

well we have 4 so is not much for us

I guess if you have one project that calls all the other microservices TFs as modules you will endup having VERY LONG plan runs

now I will argue that not for every software deployment you will have to change infrastructure every time

but I do not know your needs

I think with Terraform Cloud/Enterprise you can point workspaces to track individual folders, so if you have one workspace per microservice, a monorepo could work.

It will soon be possible with Spacelift, though using a policy-based approach.

BTW I’d probably rather avoid having a separate project for each microservice, and would try to group them by product area - ie. responsible org/team/tribe.

@Dhrumil Patel take a look at this module https://github.com/cloudposse/terraform-aws-elastic-beanstalk-environment

complete working example https://github.com/cloudposse/terraform-aws-elastic-beanstalk-environment/tree/master/examples/complete

Terratest for the example (it provisions it on real AWS account) https://github.com/cloudposse/terraform-aws-elastic-beanstalk-environment/blob/master/test/src/examples_complete_test.go

Ok thanks

Sidenote, I would recommend that you take out your aws accounts ids before posting outputs. It’s best to keep those secret.

Ya I forgot about it thanks for reminding me.

You should be able to edit them out

Is there any thing wrong with this code ?

I need to create code as my internship assignment and my mentor told me that I can’t using public registry modules thats why I am asking ?

Your EC2 instance is not launching correctly and/or not in time. Check for problems in /var/log/eb-activity.log

You also want to increase the Command timeout or disable health checks while you’re investigating.

Ok

Problem solved, I am providing AMI to autoscalling group and that AMI causing problem. Instance spawn using that AMI can’t communicate with elastic beanstalk. When I didn’t provide AMI in elastic beanstalk environment then it works perfectly fine. Don’t know why this is happennning any suggestion ?

Would need to see your error logs, however you must specify an AMI. I think your custom AMI may have a launch error.

Actually I am not using custom AMI I am using one of the ubuntu AMI from AMI store.

can you copy-paste the output of plan here ?

he’s explicitely using a local state file local_state.out , so that’s not it.

well, the question initially says that he is using remote backend

I could be wrong, I will wait for his confirmation if the backend is local-file

that’s not relevant, he posted his command line commands and he clearly pulls from remote to local file, and from that moment on uses the local state file terraform plan -state=local_state.out

not sure if he did partial init in that case

yes I have download the remote file with terraform state pull to mv everything and once the plan match my need I want to upload it back with terraform state push. then plan again to be sure and apply

@Pierre-Yves did you terraform state push already before running terraform plan ?

no I have specified -state=local_state.out

You in order to use local state, you might need to do terraform init afaik

with local state

that will consider your local state instead of remote state

https://www.terraform.io/docs/backends/config.html#partial-configuration

mhh exact and since I am moving a module it might requires it

terraform init -backend-config="path=local_state.out" => The backend configuration argument “path” given on the command line is not expected for the selected backend type.

seems better with explicitely having the file named terraform.tfstate

so it seems terraform don’t like I have a backend configured in the main.tf even when specifying -state=localfile or init with a local terraform.tfstate file

If I want to work locally I had to remove the backend block and terraform will ask to unconfigure and copy the current state to the local backend

`

terraform init
Initializing modules...

Initializing the backend...
Terraform has detected you're unconfiguring your previously set "azurerm" backend.
Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous "azurerm" backend to the
  newly configured "local" backend. No existing state was found in the newly
  configured "local" backend. Do you want to copy this state to the new "local"
  backend? Enter "yes" to copy and "no" to start with an empty state.

as a summary to move module resources on my laptop i have: • unconfigure the remote backend tfstate ( by commenting out the backend block • run terraform init • terraform propose then to copy the tfstate locally • I have move the resource, try and plan • re add the backend block for remote state • run terraform init and specify to copy back the state thanks for your help @aaratn an @maarten

this was only needed because moving resource module requires a terraform init

Ya that’s your best bet

Avoid testing things that terraform already covers in its own tests.

E.g. creating a bucket results in bucket. It’s safe to skip this kind of test

80/20 rule applied to testing terraform: You get 80% of the benefit and catch 80% of the problems by just running plan/apply/destroy. You have to spend 80% more effort to test the remaining 20%

i still think this presentation is really great for getting folks started… https://www.infoq.com/presentations/automated-testing-terraform-docker-packer

I have look as well to do terraform test and will experiment later the test from the terraform vscode extension which can do lint and “end to end” test . see the bottom of the page : https://docs.microsoft.com/en-us/azure/terraform/terraform-vscode-extension

Thanks, @Erik Osterman (Cloud Posse) @loren @Pierre-Yves

the infoq video above mention as well “conftest” for gke : https://github.com/instrumenta/conftest/tree/master/examples/terraform

https://github.com/cloudposse/terraform-aws-rds-cluster/pull/59 Please!

@Adam Crews https://github.com/cloudposse/terraform-aws-rds-cluster/pull/59#issuecomment-584830577

ugh, sorry about that, fixed and code pushed.

@Adam Crews https://github.com/cloudposse/terraform-aws-rds-cluster/releases/tag/0.21.0

This might come handy

https://github.com/awslabs/aws-limit-monitor

Consider writing instead:

workspace = coalesce(var.workspace, terraform.workspace)

https://www.terraform.io/docs/configuration/functions/coalesce.html

Tell me about it. They must’ve known it would cause a bunch of confusion

TFC workspace is basically a project and locally you can pull in multiple projects to 1 code-base mapped to workspaces.

I completely forgot about this distinction until @btai brought it up.

that’s why we use Admin permissions

yeah, I always go for the admin permission

I am already doing VCS+CI/CD, PRs, etc which works great and also means not many people can actually do any harm as we have a process BUT, as humans can make mistakes, someone, including myself, could mistakely give access to Jenkins (both ssh or URL) using the wrong permissions which may allow them to get admin access to all AWS accounts.

It is actually a complicated situation because you don’t want to have a bottleneck by having to always update the IAM policy to allow an extra action but at the same time you want to avoid risks of someone being able to do something they shouldn’t be doing.

@Erik Osterman (Cloud Posse) do yo guys use Open Policy Agent?

one thing we are struggling now is the user management part and SSO in AWS, how to better and easier to manage user/group policies trough SSO or other means, it is a hot topic in our world right now

a bit offtopic from this original thread

I have had a look at ita while ago and it can get really complex… to be honest, my company does not have that many users that need to access AWS so I am not using SSO but I did have a look to integrate with GSuite and it is not like a “next next finish” IMHO

we have about 300 users

some of them cross group boundaries or have multiple account access etc

it gets pretty complicated

I can imagine it can get really complex, not just the SSO part but the Security side of things

exactly

I was once at an AWS event and s security team from a company was there and they were talking about it

similar to what you have said

and it was so complex that even the AWS SA was lost

because you have all the security requirements too.. not just users/sign in

@jose.amengual (Just to be clear, these concepts are all relatively recent developments for IaC) we haven’t had a chance to adopt it yet, but this is what we’re planning on incorporating to our latest pipelines we’re developing for a customer

it is incredible to be that there isn’t a simple solution for this yet, it is still a bit rough

if a hard problem but Active Directory solved it many many years ago

their policies are incredible granular, although there is a lot of clicks involved

I wish there was a simpler way to integrate and manage users like AD, like you said

better having clicks involved than googling for a solution that we never found

hahaha lol

very true

AD is probably the best service MS has ever done

user management and gourp/users policieswork so so well

agree

@Gui Paiva SAML provider from GSuite to AWS works nicely. The only downsides are that you seemingly can’t attach policies to groups, only users individually. In practice, this is not so big of an issue for us since we are working on deploying GSuite users via ansible anyway. Also, I haven’t been able to determine yet how to add multiple GSuite apps for AWS yet

it is incredible to be that there isn’t a simple solution for this yet, it is still a bit rough

@jose.amengual can you elaborate?

Well if you look at the example of MS AD, they had this for years, fine grane policies , group, policies, identity, authentication and authorization

what I’m talking is basically an AWS solution that is easy to use , easy to understand and that solves the use cases that people have and that has a good programatic API access that is easy to program

IAM if far from being easy

SSO in AWS is ok-ish but then you have issues where you can attach policies to groups and such

there is alway quirks

and there is many SaaS product that tackle to solve this problem for you

the fact that there is that many, tells you that there is a need for something easier

that is what I mean

wont that destroy the existing bucket in order to recreate it with the cors policy?

no

if you create bucket previously with terraform

cool. let me test it out..

Also no if you import an existing bucket into the config

@Igor Bronovskyi verified it. Thanks.

@Igor Bronovskyi i spoke too soon. This toggles the application of the CORS policy to the bucket on every other terraform apply. Just discovered it after releasing the code

so removes it and appends it ….

that would be a bug, either in the tf aws provider, or in your config…

I dont think i explained myself well. The problem was the the bucket created as part of a collection of resources. Eventually i ended up issuing a PR to create a new resource to handle this issue. https://github.com/terraform-providers/terraform-provider-aws/pull/12141

you don’t have a list it looks like, you are constructing the list. If that’s the case, then the syntax is OK, you get substring from a string and put it into a list

correct. perfect, thanks!

@Andriy Knysh (Cloud Posse) I just realized that my list totally wouldn’t work because i’d be truncating the end of the dns name. So…

[lower(substr(“${var.service}-${var.branch}“, 0, 32))“.${var.stage}.${var.domain}“]

that hurts my head

would that work?

I think separating the quotes like that would make a list of two items, not one

or maybe not, since there’s no comma

make a local var in locals`

then use it to put into the list

more readable

yeah, you’re right

good call

there is a PR to fix that, and @Maxim Mironenko (Cloud Posse) is working on it

@Olivier fix is on the way, will let you know when ready

thank you

@Olivier here it is: <https://github.com/cloudposse/terraform-aws-elasticache-redis/releases/tag/0.16.0>

thanks

Isn’t this what you’re looking for - https://www.terraform.io/docs/providers/aws/r/autoscaling_group.html#wait_for_elb_capacity ?

I’m also thinking you could do this with lifecycle hooks - https://docs.aws.amazon.com/autoscaling/ec2/userguide/lifecycle-hooks.html - though you will need an external entity (eg. a Lambda) to control the action that’s taken.

One other thing I’d suggest looking at is https://aws.amazon.com/codedeploy/ - haven’t used it personally but it’s supposedly designed to handle deployments just like yours.

@Chris Fowles, I attempted the following with no luck -any thoughts?

resource "aws_api_gateway_domain_name" "domain" {
  certificate_arn          = data.aws_api_gateway_rest_api.api.endpoint_configuration == "EDGE" ? var.certificate_arn : null
  regional_certificate_arn = data.aws_api_gateway_rest_api.api.endpoint_configuration == "REGIONAL" ? var.certificate_arn : null
  domain_name              = var.domain_name

  tags = var.tags
}

error:

Error: Error creating API Gateway Domain Name: BadRequestException: A certificate was not provided for the endpoint type EDGE.

could be many diff reasons, did you check workspace, workspace_key_prefix for terraform_remote_state? Also, if the state bucket is in diff account, might be wrong permissions (assume_role)

Well, this is really out of left field. I’ll check those things, looking through the remote config and the data sources to see if I have introduced a bug in there but:

• I don’t use workspaces

• State bucket is in the same account

• This same set of modules provisioned without issue in a different account

• I’m provisioning as an admin

• Other modules using remote state work fine, only modules using remote state from this specific module (my RDS module) are acting like the state file is non-existant

It will be the stupidest thing… always is.

Thanks for responding

did you check that subfolder in the AWS S3 console?

Yeah, and pulled the tfstate file. It is there, looks good. I can’t dig into this right now, have to get back to it in the morning. Might just tear down my sandbox and see if I can reproduce there first. thx.

couple of tips:

From the source remote state: terraform output

From the state bucket that wants to consume the remote state: terraform console > data.terraform_remote_state.xxx.outputs

other things != terraform?

(another consideration is SSM, if you’re on AWS)

yep, things other than TF that need to programmatically access outputs. I think I’m good now, thanks for all the responses folks. Great Slack community here.

what’s the issue?

Is it possible to pass remote state values to vpc_id variable in this module? I alwasy get Unsupported Argument error while trying to pass value of VPC in the vpc_id

Thanks. I figured out issue was on my end only (knew it was something stupid)

yes. you can use any valid expression in an output value, https://www.terraform.io/docs/configuration/expressions.html

https://www.terraform.io/docs/configuration/outputs.html#declaring-an-output-value

For this purpose and many other we have created a docker image. I will just share some things of the whole README file that we are using in my team:

Introduction

To keep the infrastructure operational we use some tools to provide changes to the infrastructure and manage the settings.

The development environment will make sure that you don’t need to install all the tools we need, but just the ones you already use:

• Docker
• Docker-Compose

We are using ASDF in a Docker Container to get a fully operational development environment with all the tools needed.

Getting started

$ ./runtask.sh init # to initialize the development environment
Building infrastructure-console
...
Successfully tagged aws_infrastructure-console:latest
$ ./runtask.sh console # to get a console on the development environment
asdf@a8da505fb2ab$
asdf@a8da505fb2ab$ packer version
Packer v1.1.3
asdf@a8da505fb2ab$ terraform version
Terraform v0.11.2

All necessary folders are linked as docker volumes into the running docker container via docker-compose.

Note: New folders in the projects main folder need to be added in the docker-compose.yml and require a fresh console in order to be available in the docker container.

We basically use this docker image all the time for doing anything in the terraform and in general with the infra.

In this case everyone is using the same hierarchy, same version tools, etc…

Hi @Ognen Mitev, thanks. We do something similar but not specific to Terraform only. Is this image available online anywhere? so that I can take a look…

Hi Adrea, we also no not have it to terraform only, rather for everything that we use. https://hub.docker.com/r/zeppelinlab/ops_shell

No that’

’s something else

see the documentation link at the bottom? check it out -> lets turn this into a thread

@Francisco Montada

@MattyB which btn ?

https://docs.aws.amazon.com/AmazonS3/latest/dev/cors.html

I cannot have multiple origen ?

If you check out the documentation it suggests that you need multiple CORSRules to do what you want to do. I think this is a bug in the CloudPosse module. Let’s try something out. Delete lines 5 & 6 so there’s only 1 origin - *.gapcommerce.com

ok

now go to cloudfront and invalidate your cache (you can do this 1000 times per month before they charge you)

done

ok

yep

did not help

still showing Failed to contact the origin.

can you access your static assets using the cloudfront link?

let me chck

https://d1gqh37wdylbj0.cloudfront.net/

same

let’s go to DM if you don’t mind

what’s the last part of your ‘origin domain name and path’ in cloudfront?

s3.us-east-1.amazonaws.com ?

i had to set bucket_domain_format = “%s.s3.${var.region}.amazonaws.com”

yes it is

I added -website- and did not work

@MattyB I noticed my s3 endpoint has -website- on it

Have you seen the working example: https://github.com/cloudposse/terraform-aws-elastic-beanstalk-environment/blob/master/examples/complete/main.tf

this is what the terratests are based off of

Looks like this will do the trick: https://www.terraform.io/docs/providers/aws/r/api_gateway_account.html

terraform did this loooooong before cloudformation. create a provider block for each account using aliases, and pass the provider alias to the resource/module

https://www.terraform.io/docs/configuration/providers.html#alias-multiple-provider-instances

Yeah, saw this. So one terraform apply would generate a diff for n number of accounts?

well, it’s not as easy as defining a provider and it just working. You’d need to either declare a module per provider or declare resources that consume specific providers. This repo implements it about as cleanly as it gets.

https://github.com/nozaq/terraform-aws-secure-baseline/blob/2276b6db68b055bfb7c94023fe293e22c21cd19d/config_baselines.tf

This is cool. Thanks!

is it an OS-level limit?

Yes

echo "abc

...
"

is the same as

cat <<EOF
abc

...
EOF

except the latter is a heredoc which acts as a (very slow) way to do standard input

BTW, do you know what receives the “too long” argument list? The shell (local exec running sh -c) or echo ?

yeah, sh

isn’t it possible to render a local file as a state rather than writing it with a local-exec command?

Well I need that file for other purpose

https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/data.tf#L68 - rendered template as data source

https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/kubectl.tf#L1 - written file as a state

its swagger document that i use to import to api_gw with customized stuff, but in same time i need to manupilate with that file to use it in swagger online.

so its kinda tricky

even if you cannot directly use resource "local_file" , you can write it with local_file, and then copy it with local-exec

let me try local_file

thanks

it worked as a charm

thanks so much

thats all i needed. I tried with file, and it didnt work

Why is tf cloud not an option for you at this stage?

@Laurynas you can access the state of the different deployment

vscode

it depends on the plugin I am using neovim and it works perfect.

do you have the terraform plugin installed?

neovim for vscode?

vscode

which terraform extension do you guys use with vscode? I’m using mauve.terraform and it’s fine, but everything for .12 doesn’t work perfectly. The official extension is garbage. Hoping someone has a recommendation.

I’d say patch your editor to run terraform fmt upon save of a file that has a .tf extension. That should cover 80% of the issues: )

my main issues with this extension are around syntax highlighting. After certain blocks of code, it just stops working. Intellisense and sniffing out errors before running would also be helpful. I like Terraform, but the user experience makes it feels very unfinished / unpolished.

I am using neovim and it’s working just fine, what I miss is though the auto detection of missing variables like the pluging for inteliJ but IDK if it works with 0.12 as it’s supposed to.

i remember when such things didn’t work for terraform <=0.11. give it some time, these are community-managed plugins, not hashicorp managed. 0.12 was a big shift for hcl. it will get there

here’s a trick for supporting both 0.11 and 0.12 in vscode on a project-by-project basis. use the old plugin for 0.11 and the new language server for 0.12… https://github.com/mauve/vscode-terraform/issues/157#issuecomment-587125278

the language server is definitely the way to go for 0.12 though

let me enable that language server…haven’t done that

thx thx

I have switched to using IntelliJ TF module. I use it just for TF and nothing else. It’s great for 0.12

I installed Terraform forked for VSCode. And I`m satisfied

Super interested in this question as well. I’ve just started on my terraform journey and only have a short time to try to come up with a proof of concept. I’ve written and copied a lot of example code just to get a single working vpc up and running with both public / private subnets spanning multiple AZ’s. I keep coming back to this idea that I am spending un-necessary cycles re-inventing the wheel and could simply use the ones available in the terraform registry.

right now, we use some from the registry and it’s definitely a good idea. either official modules or cloudposse modules seem to be useful

the registry is great. im more curious about private modules that leverage the registry. so then devs would use the private modules with company centric defaults that then use the registry. does that make sense?

@RB - if we find a module that we want to modify, we clone it, (check the license), remove .git and create a file that references the original module. Cloudposse is a great place to start, but if your requirements differ a lot, you’ll need to fork or clone.

just keep in mind, forking is not a copy, so your infrastructure depends on another Github org or the registry

We just reference our own private modules from a git tag. I definitely would love to have a true private artifact repository for terraform modules. We use Artifactory - I’m hoping https://www.jfrog.com/jira/browse/RTFACT-16117 gets traction.

so does each developer service use the private modules? do devs contribute to private modules? also how do you update all terraform that depends on your modules?

@Joe Hosteny there are some good alternatives at the bottom of that ticket. i was looking into the terrafrom-aws-tf-registry

We aim to have private modules only near the root of the dependency graph. If we need something changed in a dependent module, we’ve been contributing those back to OSS modules. While PRs are open, we fork all the way down the graph through to the dependency, and run with the forks that point to our module.

Not sure if that addresses your question. We have moved to using CloudPosse as much as possible, for standardization. The init-from-module works fine with the private repo as well.

interesting. but when the parent module, in this case your fork or cloudposse, are updated, do you have to manually apply all the terraform that depends on those modules? is it automated?

manual, which is probably a good thing with most of our modules because it allows for a quick code review when you copy / merge. We have quite a bit of mods to some of these.

we plan on automating the detection of updates soon. something like: if tag on upstream project is greater than X, throw an alert or log or something.

During development, I generally reference the modules via filesystem path, so if I update any of them in the hierarchy at any point, I can just run make reset && make deps at the geodesic container for the desired AWS account, and all dependencies will get updated. Otherwise, yeah, you have to tag the intermediate repos and push the references up to the root.

I don’t think it’s any worse than any other dependency chain, TBH. If I have multiple logical changes in a dependency, I’ll do the PRs in multiple branches, then create a local branch that merges both of those (only locally) so I can get all of them for test at a single time in the local filesystem.

I did this pretty extensively for the CP ECS web app and its dependencies while we made a module for concourse

We use private modules without registry, just pointing source to git refs (tags), no need for more yet… Didn’t investigate any registry

I used Terraform Cloud for our private TF modules and it worked great

i’d been thinking about something similar lately. maybe a tag for the source location and for the tfstate location

We leverage many of the Cloudposse modules, and pin most of our stuff around their label module: https://github.com/cloudposse/terraform-null-label When looking at the console it is very easy to see things that were named with the output from that module. We then pair that with tags on all resources that support tags, and it becomes very clear how a resource was created and how to find the source in our tf codebase.

disaster recovery should be treated as an abnormal event - i.e. don’t try and bake it into (at least the recovery bit) your infrastructure provisioning. you should be following a careful process with checks along the way during a disaster situation so as to avoid making the situation worse.

you probably want to consider freezing all automation around a disaster event, as the system is in an unexpected state and you don’t want automation to propagate or exacerbate the issue

but how to recover the restore in the new terraform state?

terraform import to bring things back into statefile management

We have 4 worker groups, 1 per AZ plus a special one. Are all the subnets tagged? Think both with cluster and shared. All security groups allowing all others?

You could stand up a Debian pod in each node in each AZ and work out the combo

We have the following tags:

  private_subnet_tags = {
    "kubernetes.io/cluster/eks-cluster" = "shared"
    "kubernetes.io/role/internal-elb"   = true
  }

  public_subnet_tags = {
    "kubernetes.io/cluster/eks-cluster" = "shared"
  }

  vpc_tags = {
    "kubernetes.io/cluster/eks-cluster" = "shared"
  }

Good work! Did it so many times but didn’t automate it because I had few resources to move. I don’t know if there’s such tool so you should create a repo for it. If in future I will have such case I might PR with improvements

thanks. there is some funky logic in it but it made some module migrations a lot easier

I’ve seen that comment

You make repo, I’ll star it

I used this script today. Had to rewrite it a little bit to work with GCP but other than that, pretty good.\

it’s a bit difficult to guess what the import statements are and it’s currently using stdout to parse instead of converting to json first so it can definitely be improved. if you folks like, i can make a repo, and we can all contribute to it.

id love to see your fork too @Marcin Brański