#terraform (2021-06)

Me. For way too long. Found out about this a week or two back — Stoked it finally shipped

Ditto. We have long given up and moved to self managed S3 + CloudFront

I opened the initial issue, so yeah I’ve been waiting.

Sucks that there’s no integration for env vars with param store / secret mgr (in amplify) so moving to terraform would mean committing some tokens to source..

@Michael Warkentin Use sops + the sops provider. Better way of dealing with secrets then PStore or secrets manager IMO.

@Matt Gowie will you be joining us for #office-hours today?

Yeah @Erik Osterman (Cloud Posse) — I’ll be on.

https://github.com/cloudposse/terraform-aws-s3-website

oh perfect, thanks @Andriy Knysh (Cloud Posse)

https://github.com/cloudposse/terraform-aws-cloudfront-s3-cdn

see examples folders

hello Rob, I am using tfsec as a pre commit to validate my terraform code before pushing to Azure. I didn’t integrate it in CICD “yet” but will do it

Curious if it is just you authoring, one team, many teams? Does tfsec do about what you need? Are you writing a lot of custom policy?

one team but I have split code in several repos. default tfsec rules fits to me and yes I am authoring mainly for our infra team . for now tfsec is just taken as a warning and will not block the ci/cd. When it will then it should be in the cicd

The space here 1024 65000 is suspcicious

@emem I’m not familiar with cloudflare resources but I’m wondering, what is the resource/variable/object/etc named rule? Seems as though your are not referencing it correctly… :thinking_face:

Is rule a value that you are creating or is this from a third party module you are using?

thanks @managedkaos was able to find the issue

no problem! glad you worked it out

have u encountered this before

nil entry in ImportState results. This is always a bug with
the resource that is being imported. Please report this as
a bug to Terraform

Something isn’t right. That should return the respective account’s id. I use it all the time. I also use the aws-cli implementation of the same command all the time to check the current account.

aws sts get-caller-identity --profile dev
aws sts get-caller-identity --profile prod

do you use AWS SSO?

Here is a quick to test…

provider "aws" {
  region  = "us-east-1"
  profile = "sandbox"
}

data "aws_caller_identity" "current" {}

output "account_id" {
  value = data.aws_caller_identity.current.account_id
}

output "id" {
  value = data.aws_caller_identity.current.id
}

Yes. I have for 2-3 years.

AWS SSO (not old school SSO via IAM).

yeh ok - i’ll do some more digging then

you are correct - i was looking at this at 11pm last night and came to the wrong conclusion when i saw something change

it was another issue

thanks for diving so deep on this to help me out

Np

#pr-reviews

Hey there, you should be able to pass CLI args using the TF_CLI_ARGS as a variable: https://www.terraform.io/docs/cli/config/environment-variables.html#tf_cli_args-and-tf_cli_args_name

I’m consulting a customer to not use TF cloud. The business plan costs an arm and a leg

@msharma24 We’d love for you and your customers to check out our pricing models at env0 if the TFC quotes have your head spinning

https://www.env0.com/pricing

Disclaimer: I’m the DevOps Advocate at env0

Note: The Enterprise tier pricing isn’t listed because these are 100% customized agreements from top to bottom, so we don’t know what one looks like until we spec out what is needed.

They’ve been tracking that, I think… Patched some instances in 0.15.5, but sounds like there are still some occasions… https://github.com/hashicorp/terraform/issues/28776

my plans got worse after 0.15.5

Terraform Cloud, I’m assuming

do you mean rolling update in ASG or k8s deployment?

Any updates on this yet?

@RB and @Andriy Knysh (Cloud Posse) I think are taking a look at this right now

(we encountered this as well)

did you upgrade this module from a previous version, also is this happening on plan or apply

on plan. I figured it out – if you pass in a hostname with uppercase letters, you get this error

ahh good to know

pr in for anyone following along at home https://github.com/cloudposse/terraform-aws-acm-request-certificate/pull/46

quite the weird error

Yes, you can pass outputs from modules as inputs to other modules

You can use the terraform cloud if you are looking for a Paid service… https://www.hashicorp.com/blog/learn-ci-cd-automation-with-terraform-and-circleci

If you don’t want terraform cloud, you can try something like this..

https://victorops.com/blog/a-ci-cd-template-for-terraform

if you want to validate and find configuration issues of your terraform in the CI process.. you can use our free product https://get.soluble.cloud/

awesome! I’ll do some reading, thank you

I’d suggest against Terraform Cloud. They’re getting better, but are still fairly behind their competitors. Scalr or Spacelift are the way to go IMO:

https://scalr.com/

https://spacelift.io/

Spacelift has quite few disadvantages compared to scalr and Terraform cloud… I like the terraform cloud triggers which I use a lot that doesn’t exist in Scalr but if you are more into OPA, shared modules, Custom policies .. scalr might be a good fit..

You can check out our product ( disclaimer - i am CEO of env0) at www.env0.com which allows you to do much more than Terraform Cloud imho.

You can check out this video which presents all 4 solutions - Terraform Cloud, env0, Scalr, Spacelift https://youtu.be/4MLBpBqZmpM

We use the Fargate module for deploying atlantis: https://www.runatlantis.io

https://www.youtube.com/watch?v=uxL48EzjNFo

at last

Feels unexciting as there isn’t much new being released, but at least we’ll finally stop hearing jokes about terraform not being 1.0

exactly

it’s important to know that we can now stop having to consider a version upgrade as a major activity - which is nice

you called it @Erik Osterman (Cloud Posse) !

wonder if terraform test is still beta in v1.0 or staying with v0.15

@Chris Fowles
it’s important to know that we can now stop having to consider a version upgrade as a major activity - which is nice Yes/no.

Now we’re back to 0.11 and 0.12 style version upgrades - the kind that happen every year and are scary.

With regular breaking changes, we got much better at handling them.

but at least we’ll finally stop hearing jokes about terraform not being 1.0 But now I lose my excuse for why cloudposse modules are 0.x

Anyone got any thoughts on this? Would a GH Issue be more suitable for this?

Threading this to reduce noise.

good call

so your vars are in multiple files?

yeah

for different environments

How exactly are you structuring your terraform?

each app/env should have its own set of terraform.tfvars files

something like

app –terraform – dev – terraform.tfvars – main.tf – outputs.tf – variables.tf – stage – terraform.tfvars – main.tf – outputs.tf – variables.tf – prod – terraform.tfvars – main.tf – outputs.tf – variables.tf

(your experience may vary this is what we use basically)

Or use something like terragrunt where you can define them all in a single place and it keeps it a bit more DRY.

you should be able to import your module in the main.tf call, and expose the locals to the module there, where it can generate the local tags.

that seems to have worked, however my instances dont have any of the tags after apply

the plan didnt show a change either

I am not sure if there is a clean way to make it work with terraform. a hack/workaround could be to determine if there are changes some other way, if yes taint/replace the resource. or just handle helm separately :D

@Adnan So is this a known “side-effect” of using the helm provider? That is a big issue imo…

yes, i think its a known issue https://github.com/hashicorp/terraform-provider-helm/issues/372 not aware if it has been fixed somewhere

Here is good

Thanks, I’ve reshared.

Resources have the lifecylce meta-argument, with which you can use ignore_changes - I know this doesn’t answer your question, but the reason for me mentioning is; https://www.terraform.io/docs/language/modules/syntax.html - This mentions that the lifecycle argument is reserved for future releases.. so perhaps lifecycle is/will (be) available for modules

sadly just stumbled upon this.. hopefully it’ll get incorporated somehow

https://www.reddit.com/r/Terraform/comments/mrzsbg/how_to_use_lifecycle_feature_with_ec2instance/

any other shady hacks for this? or am i doomed to monkey patch this mess…?

Could you use terraform state mv to move resources into a new module which represents what is in state?

thanks, checking it out, im pretty newb when it comes to tf…

slight update: i ended up messing with the state file instead of monkey patching the tf code… im not endorsing my actions in anyway shape of form lol

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/route53_zone

https://github.com/terraform-aws-modules/terraform-aws-route53/tree/master/modules/zones

Thank you both..https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/route53_zone..I tried this and its working…

It’s using the null-label module. This is a module which doesn’t create infrastructure, but is designed to create a consistent name based on inputs

the module is instantiated as “this” and id is one of the null-label outputs. Specifically the one that outputs the “consistent name”

I’m having a hard time trying to narrow down the error I’m seeing when I use the eks-workers module.

It calls EC2-Autoscale-Group.

Which has a name prefix, which it gets from module.this.id

However the error I’m seeing suggests it’s getting a no value from module.this.id

https://github.com/cloudposse/terraform-aws-eks-workers/blob/master/main.tf#L169

there’s no default name. Did you pass in any of the variables used by null-label module?

Starting to see this now.. there’s no namespace, environment or stage.. which are inputs.

I haven’t done anything, I’m just using a CP module.

namespace, environment, stage, name, attributes

yes, this requirement is not well documented

simplest approach is to set name only to specify the name you want to use for the module’s resources

I see.. So I must pass these attributes into the eks-workers module?

yes, you have to pass at least one of them

The confusion came because the offending module is nested

passing multiple of them, and the null label module’s other variables are designed for advanced workflows where you compose or nest multiple labels

Brilliant, thank you. That was a simple fix

I passed name, and now I’m onto the next error! But at least I’m passed that point.

Thanks again,

Error: NoSuchBucket: The specified bucket does not exist

The bucket doesn’t exist - Create it first

when I created the bucket I get this errors:

Acquiring state lock. This may take a few moments...
[31m
[1m[31mError: [0m[0m[1mError locking state: Error acquiring the state lock: 2 errors occurred:
	* ResourceNotFoundException: Requested resource not found
	* ResourceNotFoundException: Requested resource not found



Terraform acquires a state lock to protect the state from being written
by multiple users at the same time. Please resolve the issue above and try
again. For most commands, you can disable locking with the "-lock=false"
flag, but this is not recommended.[0m

So you created the bucket and now you’re getting that error from TF?

I created it from AWS console with the exact name that was expected by the terraform form for storing the state.

That’s an issue with DynamoDB - I’ve not used that module before, but it looks as though it creates the bucket for you if you follow the guide.

https://github.com/cloudposse/terraform-aws-tfstate-backend#create

cloudposse/tfstate-backend/aws should have created the bucket for you if you followed this?

yes that what it should do but I don’t know why its not creating that

resource "aws_s3_bucket" "default" {
....

According to the module it does create the bucket.

resource "aws_dynamodb_table"
...

It also sorts the dynamo table.

Step one on the readme… where did you put it?

I added it in a folder called backend and the created a main.tf and added it there

this is my terraform structure

So the first step is in the main.tf in the backend folder and then the second step is in the backend.tf

That’s probably your problem.

add that module to management-site/main.tf

not sure thats the problem the jenkins deploy job calls it first before the management site.

Have you tried just following the steps in the module first? To see if it works that way? Then moving things around once you know it works?

I’m not sure why you’ve got logic in a script to check whether the bucket exists, and it it doesn’t exist; run the backend module. To me that doesn’t make sense… the whole point of Terraform is that it creates things which don’t exist and doesn’t re-create things which do exist.

What you’ve done won’t work though, you’ve created a backend with no state to put into it.

You need to bin that backend directory and bring the module into your [main.tf](http://main.tf)

Then run terraform init followed by terraform apply followed by terraform init -force-copy

The first init pulls the module down, the apply creates the S3 bucket and the second init copies the backend to the bucket.

This has been resolved thank you but I didn’t use cloudposse again as the issue persisted even after put all in same file as you advised. thanks for the help.

This was the one I used to achieve that. https://github.com/stavxyz/terraform-aws-backend

Check if the create_before_destroy is set and is true. If it is set and true, then it there is little to no down time downtime.

https://www.terraform.io/docs/language/meta-arguments/lifecycle.html

hmm dont think that would work in this case as it would try to create a NATGW with the elastic ip hooked up to the original NATGW…

the problem here is that i have whitelisted the original elastic ip somewhere else and messing with the original NATGW in any way shape of form would break this link

shoulda google the source… TLDR = SOL

https://github.com/terraform-aws-modules/terraform-aws-vpc/issues/506

looks like your kubernetes provider is not configured correctly or is expecting k8s on localhost

however before importing the state to Terraform Cloud it worked.

There is nothing on the docs, that the provider needs a config file, just pass the host, cluster_ca_certificate and token from the data resources

correct, the config file is not necessary if you set those properties. I was just going by the error message where it show that it is trying to connect to 127.0.0.1

it does not make sense, it tries to connect to localhost… the only change I made was addition the remote backend pointing to terraform cloud

Is your backend initialized? There’s a one-time step to push your s3 config.

Yes it is. I see the state file and the resources in the TFC GUI. When I run plan it executes it remotely

But then it throws the error

It’s a little hard to say, but it fees like a configuration problem. I’ve often found those out by looking at the TF_LOG=TRACE output. There’s a lot of info given about each call, including where the variables are referenced from.

Thx, would give it a try

I found the cause, the missing config path: Add this env var to fix

export KUBE_CONFIG_PATH=~/.kube/config

Nice catch!

Well I needed to go through 70k lines of trace logs

That does seem like a lot. I’ve been thinking of a tool that might help make that easier – something that parses for diagnostic info, and helps the user interpret the logs a bit better. It might be something that could be added to the utils provider, given time.

Here’s a TF_LOG=TRACE output. I’ve found this particular issue to be more difficult than most.

This line seems to be central to the issue:

path=.terraform/providers/registry.terraform.io/cloudposse/utils/0.8.0/linux_amd64/terraform-provider-utils_v0.8.0 pid=16307 error="exit status 2"

The provider seems to fail without a lot of additional info. In this case, I’ve linted the yaml, checked the variable dependencies, etc. I might try looking for version compatibilities next – or perhaps rebuild the module to add a bit more debugging, if available.

I figured this out. There were pieces of terraform-yaml-stack-config I didn’t understand. Once I found the TF_LOG_PROVIDER flag and the terraform-provider-utils source, I realized the merge tool could actually be a powerful scanning/diagnostic/debug tool to help with bad configurations like my own. Might be a win for other people down the road. I’ll try and post a few example diagnostics – like maybe something that warns about bad remote-state configs.

Only issue I had was with ecs on govcloud

Perpetual diffs when changing a resource not related to the tag change? Are you using terraform-null-label at the mo on all resources and passing them around into other modules etc?

e.g. https://github.com/hashicorp/terraform-provider-aws/issues/18311

ohai https://github.com/hashicorp/terraform-provider-aws/issues/19185

Yup that’s the one

Code snippet?

What are you supplying as the value for the availability_zone if anything? That looks like your problem.

resource "aws_subnet" "main_subnet" { vpc_id = aws_vpc.main.id cidr_block = var.subnet_cidr map_public_ip_on_launch = “true” availability_zone = var.availability_zone

tags = { name = “main Subnet” } }

variable “availability_zone” { type = string default = “eu-west-2a” }

are you passing any vars when calling terraform apply? do you have a terraform.tfvars file setting the availability_zone?

the error is coming from the aws api, it’s not really a terraform error, exactly. might inspect the console to see what’s up…

actually, the error is saying you are passing the literal string "var.availability_zone" as the value for var.availability_zone…

InvalidParameterValue: Value (var.availability_zone)

so i’d take another look at your aws_subnet block, and make sure it’s not actually this:

availability_zone = "var.availability_zone"

I haven’t got a .tfvars file

Any ideas why the variable I have set isn’t being picked up?

module "vpc" { source = “./modules/vpc” vpc_id = “var.vpc_id” vpc_cidr = “10.0.0.0/24” subnet_cidr = “10.0.1.0/24” availability_zone = “eu-west-2a” }

this is incorrect syntax:

vpc_id = "var.vpc_id"

(at least, it is incorrect if you mean to pass the value of the variable vpc_id, instead of the literal string "var.vpc_id")

remove the quotes:

vpc_id = var.vpc_id

quotes have been removed, but error still occurs

New/different error message received -

module.network.aws_internet_gateway.igw: Creation complete after 1s [id=igw-065812ef10aa22484] ╷ │ *Error: error creating subnet: InvalidVpcID.NotFound: The vpc ID 'aws_vpc.vpc.id' does not exist* *│* status code: 400, request id: b5a36b6d-fe2f-4f46-b2c5-eb6452e7b23e │  │  with module.vpc.aws_subnet.main_subnet, │  on modules/vpc/main.tf line 11, in resource “aws_subnet” “main_subnet”: │  11: resource “aws_subnet” “main_subnet” {

We use spacelift to setup dependencies using rego policies

we label all of our components and then make other components dependencies

plus spacelift provides graph visualization across root modules

I guess I need to go look at spacelift thanks!

checkout env0, scalr, and TFC too

https://www.youtube.com/watch?v=4MLBpBqZmpM

there’s probably a parameter to pass in an existing SNS topic

I guess the module expects you to either specify create_sns_topic = true or to specify a pre-existing SNS topic

Ahh okay, Saw the readme and it says to add this var findings_notification_arn if I pass false for create_sns_topic . Thanks for the insight!

cc: @matt

that’s this week, now

Ideas…. what do you not like about terraform?

What are the hardest parts getting terraform adopted by your organization?

What’s it like behind the scenes running such a popular open source project with thousands of contributors?

What are the benefits of using CDK for Terraform over vanilla Terraform? I understand the case for using a programming language a developer might be more familiar with compared to Terraform, but is there any other huge benefit to the CDK that might be overlooked by someone already using Terraform? Also, are there any anti-patterns that can be avoided by someone just starting out with the CDK? Thank you!

Good one

do you mean this variable - service_account_namespace?

https://github.com/cloudposse/terraform-aws-eks-iam-role/blob/debd970108254a59950c7e5f98dcfffc1719270e/variables.tf#L6

also see the comment here https://github.com/cloudposse/terraform-aws-eks-iam-role/blob/debd970108254a59950c7e5f98dcfffc1719270e/main.tf#L8

This is the kind of thing we want to add to a forthcoming terraform-aws-utils provider (nothing there yet, but we’re pretty close to pulling the trigger on it)

https://github.com/msharma24/multi-env-aws-terraform

Using Terraform workspaces sucks so I figured crating a wrapper script to do multi env deployment, and here is reference I posted on my github

I think workspaces are on the way out. The Terraform 1.0 compatibility guarantees explicitly call our workspaces as not being guaranteed.

The recommended approach from Hashicorp is one directory per environment.

Personally, I use a single directory and a different backend configuration per environment – so like Terragrunt, but we’re using Spacelift.

how do you handle promotion of changes with the different backends?

Generally, we continuously deploy our infra, so once you merge a PR, it will automatically deploy to dev, then stg, then prod. So there’s no real promotion in the usual case.

If we are making a major change which does require promotion, generally we rely on logic in the Terraform configuration which creates something only if environment is “dev” or whatever.

awesome, thanks!

in my team we are doing the same as @Alex Jurkiewicz

This is an interesting discussion! @Alex Jurkiewicz @Pierre-Yves I have followed a similar approach but only auto deploying feature/* branches to dev, and then when a PR is opened we run a TF plan against prod, so that we can review the plan + code review in the PR and when merged auto deploy to Prod

hm, you deploy feature branches? What if the PR is not later merged

@Alex Jurkiewicz can you comment on this?
The Terraform 1.0 compatibility guarantees explicitly call our workspaces as not being guaranteed. I just started a project that is using TF 1.0 and workspaces so reading this gives me pause

Typically, I have one environment per directory but I’m setting up a project in automation that uses workspaces as it simplifies the project structure. The project will have multiple (>5) environments at a time and they will come and go randomly as devs work on them so I figured using workspaces would be the best approach.

There is a section in the Terraform docs about their compatibility guarantees for Terraform 1.x. Search that for “workspace”

I think workspaces are the only viable native-terraform approach to dynamic environments like you describe, and I still use them for this purpose. But I’m trying to move away for static environments

thanks

I have static environment stage and production, bu no workspace . I think I’ll need workspace to be able to test my module and destroy everything after the test. but did not found other usage …

@Ian Bartholomew Which direction did you end up going? I’m about to hit the same question.

I’m going with terragrunt and the folder per environment, the nested folder per domain, and using external modules to handle promotion of changes

Which external modules?

our own. basically having a separate repo for a given domain, and version it using git tags, and referencing it the primary infrastructure repo, and using the ref option to point to a specific release of that. that way you can promote up changes in your infra with PRs, changing the version to point to a new one. This section in their docs does a good job at explaining it

Thanks. I’m a bit wary of creating a separate repo for each domain, but the flow makes sense otherwise.

yah, i feel you. The reason i am is that in the past, when we have put all the resources into a single state, it gets to be hard to manage pretty quick, both in terms of actual state size, as well as just understanding what all is changing. like on one project, doing a plan required redirecting the plan output to a separate file to inspect, since the output would exceed the console line limit

so, for me, breaking it up makes it easier to manage

I’ve always separated out state by env/module. Is that not the norm?

i think it is. we might be talking about the same thing. i just mean domain as in area of concern

Yeah, I think we are, gotcha.

I agree with @Ian Bartholomew split the tfstate amongst repos = light weight tfstate + better maintainability + improve access right

this is awesome reference https://github.com/maddevsio/aws-eks-base I highly recommend it

@Mohammed Yahya Thank you!

Skipping directories To skip a whole directory, use the environment variable CKV_IGNORED_DIRECTORIES. Default is CKV_IGNORED_DIRECTORIES=node_modules,.terraform,.serverless

you know where modules path are right? in .terraform folder > modules you can skip the whole folder or pass down the specific module.

also you can skip the check it self.

another solution would be to exit with non-zero

perfect! thank you

Did you update the variables file?

yes

Actually I fixed it. It’s a version issue. I will do a PR to the repo

https://github.com/cloudposse/terraform-aws-elastic-beanstalk-environment/pull/183

Can you show the complete module code?

Separate out the values: “public_subnet_a” = “10.10.0.0/28” “public_subnet_b” = “10.10.0.16/28”

into the variable.tf and then use the: for_each = var.{name of variable}

For what it’s worth not that it solves your immediate issue, but might help… I wrote up how to use for-each a bit more on my blog and it’s to date got a ton of traffic as many people get confused on this.

Your mileage may vary, but maybe you’ll find something useful there and if not add a comment and let me know. I had to write that down as it’s super intuitive.

I feel like the for each design reflects the Go developers that built it, but not normal users so behavior isn’t similar to many other tools I’ve used. Once I started learning Go it made a lot more sense.

https://www.sheldonhull.com/how-to-iterate-through-a-list-of-objects-with-terraforms-for-each-function/

Hey thanks for the responses - I will read your blog and see if I can figure it out @sheldonh and if not I will post the code after I sanitize it @Joe Niland

or is it just the step scaling strategy

If you have a look at the kube-state-metrics chart, you can see that the name is composed of the release name and chart name, if you don’t provide a fullnameOverride value. In your case that would be prometheus (release name) and kube-state-metrics (chart name) which results in prometheus-kube-state-metrics.

If you don’t want that you will have to set the fullnameOverride in your values file.

in this where can i pass?

I haven’t worked with the helm Terraform provider yet, but it would be something like

set {
  name = "fullnameOverride"
  value = "my-full-name"
}

I reckon

The stack resource does use a suffix (see here for example), so you might be out of luck there.

the value should be what i want to give the name of prometheus service right?

Yes, in case of kube-state-metrics it would be the full name, for the stack it would probably be prometheus (to get prometheus-operator).

but the single helm chart is installing all the things

all the services

i tried by passing like this

set {
  name = "fullnameOverride"
  value = "my-full-name"
}

but not worked

Is it possible or not?

Like without changing anything in repo

did you also see findings_notification_arn?

someone else reported this same confusion recently, sounds like the docs need an improvement here

In the README, the type for this variable is defined as:

list(object({
    rolearn  = string
    username = string
    groups   = list(string)
  }))

You need to provide values in that format

eg, you are missing username and groups values

I tried that way, but it failed too.

I’m checking the hashicorp documentation to see if I can get a light

how did it fail? Did you get the same error message, or a different one?

the error is different

Error: Variables not allowed                                                                                                       
                                                                                                                                   
  on vars.tf line 137, in variable "map_additional_iam_roles":                                                                     
 137:     rolearn = "arn:aws:iam::xxxxx:role/JenkinsRoleForTerraform"                                                       
                                                                                                                                   
Variables may not be used here.                                                                                                    
                                                                                                                                   
                                                                                                                                   
Error: Missing item separator                                                                                                      
                                                                                                                                   
  on vars.tf line 137, in variable "map_additional_iam_roles":                                                                     
 136:   default = [                                                                                                                
 137:     rolearn = "arn:aws:iam::xxxxxxx:role/JenkinsRoleForTerraform"                                                       
                                                                                                                                   
Expected a comma to mark the beginning of the next item.                                                                           
                                                                                                                                   
ERRO[0002] Hit multiple errors:                                                                                                    
Hit multiple errors:                                                                                                               
exit status 1 

sounds like you have some syntax errors then. Can you post your full code for this module block?

I’m guessing you are trying to wrap this module and pass in this value as a variable. And that you have some syntax errors in the wrapping part.

To start with, I suggest you hardcode the values in your module definition.

With your tip it worked perfectly. Thanks a lot for the help, @Alex Jurkiewicz

was it just moved to a dedicated “null provider” resources, so I guess you just have to load the provider to get it working. https://registry.terraform.io/providers/hashicorp/null/latest/docs https://registry.terraform.io/providers/hashicorp/null/latest/docs/data-sources/data_source

In certain versions of terraform a warning is displayed saying it’s deprecated when using it

https://github.com/hashicorp/terraform-provider-null/blob/949cab0972c1ee9b9d708adebdc4128bd0b2d322/internal/provider/data_source.go#L12

Huh, weird that it doesn’t show in the docs

Somewhere I believe I read that there would be no more development in ways of enhancements or feature requests too

oh well, anyways the Locals should work for him

Yes, TF is complaining that it’s deprecated. I’ve tried using locals to get the cluster name, but it does not have the same effect. Which is strange.

If I add a local cluster_name to pull from the eks_cluster module, and then set the relevant field in the worker nodes module.. I get the config map error

what is “the config map error”?

https://github.com/cloudposse/terraform-aws-eks-cluster/blob/c25940a8172fac9f37bc2a74c99acf4c21ef12b0/examples/complete/main.tf#L84

I wonder if there’s a difference with where a local would get it’s value from and where a data source would? Presumably a data source get’s it’s inputs from the API, which means the resource has to be up & local get it’s value from the module… so in theory a local variable could be populated before the cluster is fully up?

can you paste what you’re defining for the local

Then using local.cluster_name in eks_node_group…

module "eks_node_group" {
...
cluster_name = local.cluster_name
...
}

Ah, thats kind of what I was suspecting. The null data source was using both the cluster name AND the config map attribute, so that everything was synchronized. So you’d need a local defined that uses both those values

I did add a local for the config map ID

they need to be joined in single local though

Oh, that will be my problem.

Not this then?

yah join those into 1 string basically

that will cause the local to be undefined until both those values are available

Not completely sure what you mean?

You mean join using join() ?

I see.

Got it, excellent, thank you. I’ll test that now

That’ll only help you part-way though

you’ll have to then somehow use that variable in the other resource, so that it is dependent on it

might mean you do some ugly string splitting to get the cluster-id (or whatever it is you need)

Ah yeah, that’s true…

Seems logical to use null_data_source really, since it gives what we want. But having the deprecated warning is annoying.

Something like this I guess? element(split("-", "${wait_for_thing}"),0)

yah

try:

  for_each = var.enable_ingress_alb ? { for domain in aws_acm_certificate.cert[0].domain_validation_options : domain.domain_name => { name = domain.resource_record_name, record_value = domain.resource_record_value } } : {}

?

though note, if you start using subject alternative names, the validation record for root.zone and for *.root.zone are the same, which can lead to a race condition… here’s how we handled that: https://github.com/plus3it/terraform-aws-tardigrade-acm/blob/master/main.tf#L40-L46

Interesting thanks man I think the [0] doesn’t work so looking at options

did you change something?

resource "aws_acm_certificate" "cert" {
  count             = var.enable_ingress_alb ? 1 : 0

count is right there

I removed it by accident

Schoolboy error

I’d love to know this as well. I believe people have asked for the functionality, but I’m not sure if it’s been added lately. CloudPosse modules allow you to pass variables in like “enabled” or “<resource>_enabled”.

A common pattern in many Terraform modules is to add some sort of “create” or “enabled” parameter as mentioned above, but in recent Terraform versions it’s possible to use for_each and count on modules: https://www.terraform.io/docs/language/meta-arguments/for_each.html

I found a solution. We can use count on modules for Terraform versions 0.13 +

So I was able to do something like count = var.myflag ? 0 : 1 in the module definition.

https://www.terraform.io/docs/language/meta-arguments/count.html The very first section in the above link mentions this

also consider for_each to preserve index position when applying the same module multiple times

here is an example using count to trigger a module for a given environnment variable

module "servers" {
  count                              = var.env == "stage" ? 0 : 1
  source                             = "[email protected]:v3/your_module?ref=v1.67"
  location                           = var.location
  env                                = var.env

Are you using remote state?

use this module instead: https://github.com/cloudposse/terraform-aws-ssm-tls-ssh-key-pair

it writes the keys to SSM

put a count on the data resource?

or does it not support that?

oh it looks like it does

easy then

100% inserts embarrassed face

i guess this is what i get for working for a medical procedure this morning

@Nikola Milic the tfstate-backend module is typically (and recommended) to be invoke by itself as a single root module in isolation once for all of your other root modules. You can store all state files in a single bucket and utilize workspace_key_prefix in your backend configuration to properly separate root modules. Your workspaces will then create the various folders for each environment that you create.

My suggestion is to create a new root module to invoke tfstate-backend, transition your state for your root module to that new backend, and then remove + destroy the tfstate-backend module usage in your environment root module.

How do you isolate a root module?

https://www.terraform.io/docs/language/modules/index.html

Root modules are terraform projects. They store state.

Child modules are modules you consume in root modules.

So what I’m saying is you’ll create a separate terraform project alongside your existing one called bootstrap or whatever you want to call it that then creates your state bucket in isolation from the rest of your terraform resources.

I see. Let’s say that I create that brand new folder for creating new state bucket (which will be correctly called x-terraform). If I follow bootstraping guide on the repo, there will be a backend.tf file as a result of that process. Should that file replace current backend.tf in this original folder?

Yeah.

Then do a tf init and it’ll ask you to transition the state to the newly configured bucket.

At what point do i delete the module declaration from my original main.tf?

When doing this method, be sure to look into the workspace_key_prefix argument for the s3 backend configuration. You’ll need that to manage multiple root modules in the same bucket.

After you transition state to the new bucket. Once you do that, your legacy state bucket is no longer necessary.

You can keep it around for historical purposes if you want by just removing it from state, but everything should get dup’d over to the new bucket so that’s up to you.

workspace_key_prefix argument does not exist on the cloudposse module?https://github.com/cloudposse/terraform-aws-tfstate-backend

No, you’ll need to supply that yourself for each root module’s [backend.tf](http://backend.tf) file.

The other way is to create a tfstate-backend for each root module that you have, but we’re moving away from that as it’s unnecessary.

Hm I’m kind of confused by the terminology, let me see if we are on the same page. When you say “each root module that you have” what do you exactly mean?

If i understand you correctly i should have this:

/infra bootstrap/ main.tf <- backend config main/ dev/ dev.tfvars prod/ prod.tfvars backend.tf <- copied from bootstrap main.tf <- declaration of app resources

what are those root modules in this scenario

main/ and bootstrap/ are the root modules.

so there should be two backend.tf files in those two folders, each of them having additional workspace_key_prefix value same as the name of the folder

In smaller projects, having one root module can work. But in larger environments where you’re managing 1000s of resources it quickly becomes a huge headache so the community best practice is to separate root modules for areas of concern to decrease blast radius (think of it as having a root module for your various tiers of infra: Network, Data, Application Cluster, Monitoring, etc.)

Cloud Posse themselves goes with very fine grained root modules where they create one for each type of AWS service (check out terraform-aws-components for that), but that isn’t necessary for all projects.

so there should be two backend.tf files in those two folders, each of them having additional workspace_key_prefix value same as the name of the folder Yeah

Gotcha. Thanks for the really well explained solution.

I’ll try and make it work, if I get stuck, expect me here for more answers

Np. This is confusing stuff for anybody newer to Terraform. Hashi doesn’t do a good job in pushing best practices as well as they should.

Btw, one more quick question, do i need to worry about workspaces in my bootstrap project? I guess not?

@Matt Gowie All went smoothly, thanks again!

Ideally I’d use the cloudposse backend config module, except I’m not ok with having to run that first to init then generate the tf file. I’m half tempted to just flip back to using terraform cloud for remote runs and be done with it.

I could use Go, but more code to write and tear down, which feels like a stubborn refusal to then benefit from pulumi/terragrunt at that point

Terragrunt output is so messy it’s hard to debug at times, so I’m considering backing out some of the terragrunt for native terraform stacks. I have tf var files already and wrapper for handling this… but I don’t have backend s3 creation handled.

Even if I’m using PR workflow with Azure pipelines, it might just make sense to just leverage terraform cloud and be done with it i guess.

The pattern nowadays with the tfstate backend module is to just create it once and use the one bucket for all root modules using the workspace_key_prefix. I and others are digging it as you only need to create the backend once and then it’s a fairly untouched root module going forward.

Does that not work for you?

Ok, so one “stack” = backend creation and just use that going forward. I thought it would cause locks due to 1 dynamo provisioned, but I’m guessing the dynamo part is per backend state path instead, so I can still not be locked to a single non-parallel run for all stacks using bucket

“stack” in the SweetOps sense or stack in some other sense? Damn terminology is overused so I have to check

But in general, one tfstate backend creation period.

Dynamo locking still works the way you would want it to as long as you’re utilizing workspace_key_prefix.

Hmm. One directory containing this action and then that’s it for that aws account. Everything else is purely path changes. Back to using backendconfig file/vars to set to this at runtime.

thanks for confirming the key prefix is used with dynamo. Thought I’d be locking everything up with a single run at a time. Wasn’t aware the key prefix was the way it was locked. thanks!

just curious what others think, any 2c welcome

Confused on what pattern you’re asking exactly? Concourse overall or some tf pattern within the files you shared?

check out the inputs. it’s all yaml driven.

not a common pattern.

at least haven’t seen this in the wild before.

It’s not uncommon I’d think. Actually moved towards using yaml for a bit and merging but there’s some gotchas and it adds an additional abstraction that can be problematic at times to troubleshoot. I ended up trying to stick mostly with TFVars when possible.

Ah using YAML as a datasource is what you mean. Yeah this is becoming a more common approach IMO. I use it a lot for the same purpose: team.yaml, repos.yaml, accounts.yaml, etc.

(ya, tfvars cannot be loaded selectively at run time, which makes yaml better plus it’s portable)

We have a module for this pattern https://github.com/cloudposse/terraform-yaml-config

and we employ it all over the place:

• https://github.com/cloudposse/terraform-datadog-monitor

• https://github.com/cloudposse/terraform-aws-config

• https://github.com/cloudposse/terraform-opsgenie-incident-management

thanks for the pointers there folks.
I ended up trying to stick mostly with TFVars when possible. i feel like this should still be the default to not throw away the “typing” checks and the potential vars validation. if granular loading at runtime is not an issue i guess this is not a pattern to adopt then. very cool though for when needed.

what we do is use native tf variables in our open source modules, but leverage YAML for the configuration in our components (aka root modules)

that way we get both.

our modules validate the types, while our configuration is separate from the code.

the wafv2 resources in Terraform are quite poor. We actually stopped using them because they were so slow

interesting

do you still not stream it to firehose?

we had firehose streaming for a while, but dropped it. We use access logs from cloudfront level now instead

we don’t use cloudwatch unfortunately

We actually stopped using them because they were so slow Slow to plan / apply?

yes

cloudfront access logs write to s3 fwiw

We are implementing the terraform-aws-firewall-manager with WAFv2. We stopped short of kinesis only due to time, but will probably add it.

https://github.com/cloudposse/terraform-aws-firewall-manager

https://github.com/cloudposse/terraform-aws-firewall-manager/pull/4

adds firehose

cc: @Ben Smith (Cloud Posse)

Cc: @ohad congrats!

Thanks @Erik Osterman (Cloud Posse)

Thanks a lot @Erik Osterman (Cloud Posse)!!

Released as 0.19.0 — Thanks @Ryan Ryke!

https://github.com/cloudposse/terraform-aws-cloudtrail-s3-bucket/releases/tag/0.19.0

i lied… my problem is a flow logs issue in gov cloud…

i usually set assume role in the provider section

i have an issue in gov cloud with the kms key being cranky at me

looks like the bucket policy was updated to add the arn:aws-gov-cloud option but the kms key does not

trying to roll with just cp modules on this customer

cc @Erik Osterman (Cloud Posse) again (sorry)

@Andriy Knysh (Cloud Posse)

@Ryan Ryke thanks for the PR, it looks good, @Matt Gowie and myself made some comments

updated, i added iam in (missed that thanks) not sure what you meant about changing the format

@Ryan Ryke shipped as 0.12.1

thanks a bunch

thanks @Ryan Ryke

by changing the format I meant you can provide your own value in the variable

since the code now uses the var, it will work

it’s a cloudposse convention to use https://github.com/cloudposse/terraform-null-label as module "this"

essentially, it’s a module with no resources that exports a consistent name you can use for other resources

I guess that’s what I’m confused about. I expected to find something like module "this" in the .tf files, but I searched through all the code and can’t find that. I see module "label" and all kinds of other references, but not the reference to this. Sorry if I’m a bit slow on the uptake.

there is, in context.tf

we include this file in all our modules https://github.com/cloudposse/terraform-aws-ec2-autoscale-group/blob/master/context.tf

and then you have module.this

which is also, by convention, the same file & filename. You can see the original in the null-label repo

haha… wow… it’s right there. I guess GitHub search failed me. Sorry about that.

Thanks for taking the time!

this way, we don’t have to specify all the common vars in each module, the file just provides all the vars we use in ALL modules, so we have consistent naming convention for the common vars

Yeah makes perfect sense actually.

https://www.youtube.com/watch?v=V2b5F6jt6tQ

We use Terraform to deploy our ECS services as part of our Ci/CD pipeline. To be honest Terraform is exactly deigned for the deployment of apps (Waypoint maybe better for this). That said, we update the ECS task definition (in HCL) with the new service tag number which is passed in as a variable during runtime. This then updates the service and a new one is rolled out.

A downside of this approach is that it will delete the previous task definition, which means a roll back would require a redeployment of the previous version.

@Bruce Thanks for sharing your approach! Indeed, that’s definitely a downside. It would be nice to keep the task definition revisions.

Seems like some good discussion about just this has been happening https://github.com/hashicorp/terraform-provider-aws/issues/258#issuecomment-655764975

I’ve just started last week, but still experimenting with it. I love what it does so far, but I’m struggling a bit to get the multi-region issues worked out.

add force_destroy = true to the config, run the apply, then destroy, https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket#force_destroy?

We have force_destory set to true but it still take a long long long long long time to complete.

oh yeah, it is sloooooow

i guess you could terraform state rm <bucket> and then destroy? and handle the bucket removal out-of-band?

not ideal, but it is a work around

aws knowledge center says to use a lifecycle policy to expire/delete all objects and versions after one day

https://aws.amazon.com/premiumsupport/knowledge-center/s3-empty-bucket-lifecycle-rule/

Yea for really large versioned buckets I end up running a boto script to empty the bucket before running the destroy, this is a lot quicker but certainly not ideal and have yet to find a better way

Not sure if it’lll help, but Terraform has this feature: https://registry.terraform.io/providers/hashicorp/time/latest/docs/resources/sleep

But if the S3 command itself times out, not sure that’ll help.

that seems like it provides human readable definitions of rules, which Sentinel doesn’t provide, right?

we’ve decided to go with OPA for writing rules directly against Terraform. It’s not the easiest language to work with. But I don’t have a lot of confidence there’s a clear winner in this space yet

Me either, not investing time in anything just yet

Probably worth looking at Checkov too. Yeah OPA + Conftest is a decent shout.

Well… What region are you using? How many AZ names are returned by the data resource? Is that the same number as the number off cache clusters?

ap-south-1

should i enable it ?

this is related to a a mismatch with AZ == cache clusters?

I am facing the same issue and this says adding element works but it is not

Okay, so found that it should be adjusted in the task definition. But there I seem to run in to other issues..

Error: Invalid value for module argument
  on main.tf line 90, in module "thehive_service_task":
  90:   volumes = [
  91:     {
  92:     host_path = "/",
  93:     name      = "efs-ecs"
  94:     efs_volume_configuration = [
  95:       {
  96:       file_system_id = "fs-XXXXX"
  97:       root_directory = "/"
  98:       transit_encryption = "ENABLED"
  99:       transit_encryption_port = null
 100:       authorization_config = []      
 101:       }
 102:     ]
 103:   #  docker_volume_configuration = null
 104:   },
 105:   ]
The given value is not suitable for child module variable "volumes" defined at
.terraform/modules/thehive_service_task/variables.tf:205,1-19: element 0:

attribute “docker_volume_configuration” is required.

So I need to enable that of the docker_volume_configuration - And that gets me in to this issue:

Sooo.. kind of deadlocked there right now as I cannot null the for_each neither.

Okay.. So this apparently works.. Since it’s still implicitly applying source-dest mapping even though you apply EFS config.. It’s weird, and not the nicest way or am I mistaken? empty vars needing to be set even though the code already implied them to be empty? Feels nasty and cluttery at least.

module "test_service_task" {
  source    = "cloudposse/ecs-alb-service-task/aws"
  version   = "0.57.0"
...
  volumes = [
    {
    host_path = null
    name      = "efs-ecs"
     efs_volume_configuration = [
       {
       file_system_id = "fs-XXXX"
       root_directory = "/"
      transit_encryption = "ENABLED"
      transit_encryption_port = null
      authorization_config = []
      }
    ]
    docker_volume_configuration = []
...

Will post to #office-hours if no real good answers.

will generate a lock.hcl for all platforms

I had the issue too while using TFE, it sounds like it was because I was using the plugin_cache_dir configuration which was forcing in some way the platform of my providers to my local terraform and not the remote one. I resolved my issue by using a .terraformignore file specifying to exclude .terraform.lock.hcl* for remote execution.

I’m not using it for now, I’m adding it to gitignore lot of issues when using CI

until it mature I guess.

Huh interesting. I will try out the providers lock CMD. Maybe that’ll help…

My issue also could be plugin_cache_dir as I of course use that as well.

You also need to lock Darwin arm64 now

@Matt Gowie this is the closest I know https://github.com/hashicorp/terraform/issues/27769