we have many terraform root modules and they each have their own terraform state

putting everything in a single root module is an antipattern for a number of reasons (large blast radius, very long plans and applies, difficult to modify)

@Ben Smith (Cloud Posse) is the SME on this (because he largely implemented the second iteration of our OpsGenie Terraform modules / components with @Andriy Knysh (Cloud Posse))

So opsgenie-team is our current component, opsgenie is basically deprecated

Long story short opsgenie-team is also better by design since in OpsGenie the toplevel construct is a team, and this component captures that

Figured that out, but thanks for explaining.

Sadly… configuring OG’s data model from TF seems overly complex for internal my needs so I think I am going to punt on it. But I will likely use it in client projects in the future as I can see the utility when it’s a larger organization with many teams.

Yeah we took a deep dive into opsgenie and learned that everything is centered around a team, and theres a log of standards that are needed. since everything resolved around a team we started from there and added each additional resource to it. coming up with a stack yaml that the base component is your defaults, and you override for each team.

Essentially the opsgenie component is deprecated, but still fully functional if it provides use to someone. the opsgenie-team should provide alot more of a whole solution for each team, which could just be one, in an org

hmmm perhaps fork all the modules ? but then youd also have to repoint all the dependent modules to the new org

what if you ran a terraform init and then committed the .terraform/modules directory

I actually thought about doing that…..

I been in similar situations before as a consultant. Decisions like these are nearly always based on the fear of a supply chain attack. Sometimes it’s possible to bend the rules and get an exception. Showing the compliance badges from crew bridge on the modules should help. As well as explain you can build it all from the bottom up, but this will both delay the project significantly as well as increase the price. All while they are introducing a risk of human error in to the code. Getting all the certifications on your own code will take even more time and cost a lot more. Hope you will succeed in arguing modules are safe also from an external repo. You can ask difficult questions as how they protect them self against rouge software in the terraform providers. Or how they handle software dependencies.

agreed but this is not going to change and to be honest I do not care much if it does not change

why? once they try to update one module and realize they need to update 20 they are going to fix it themselfs

right now they just do not get it and they are understaffed

so it is hard for them to focus

Everytime we go down this path, we decide to walk it back, but there’s a need for it.

Hashicorp has a tool for vendoring providers, which is now built in to the core with terraform providers mirror

old command was https://github.com/hashicorp/terraform/tree/main/tools/terraform-bundle

There is a pretty easy workaround though

use artifactory as a git proxy or fork every cloud posse module repo you need. Then use the [url] section in the .git/config to use the local endpoint insteadOf <https://github.com/cloudposse/>

For this “policy” to be enforceable, they should block access to github directly.

https://www.jvt.me/posts/2019/03/20/git-rewrite-url-https-ssh/

I like this strategy because they can still pull in changes when they need to, or fork when they need to

interesting

yeah, you can manage if all your modules are one-level deep, but if any module refers to another module, then it gets a bit hairy. hopefully the reference is a git reference, then you can use the gitconfig insteadOf trick…

i can’t find it now, but i remember seeing a tool somewhere that would help declare module sources and versions in a file, and clone them to a local directory. so your own references would be to the local location, and you could use the tool to create that “cache” and manage its lifecycle separately

https://github.com/devopsmakers/xterrafile

(archived)

Haha, they recommend vendir

https://github.com/coretech/terrafile

haha, looks like i commented on this 4 years ago https://github.com/coretech/terrafile/issues/1

vendir is what i had in mind, https://github.com/vmware-tanzu/carvel-vendir

@Erik Osterman (Cloud Posse) the insteadof is not going to work, because cloudposse is an org and if you can’t create an org in your company where you can clone the repos then is a no go

ya, if you cannot clone/fork the repos, that would be a problem. but i feel less sympathetic on that.

I was talking to someone recently who mentioned that Artifactory was supposedly coming out with some sort of a passthru terraform registry. Not sure though to what extent.

Starting to feel like @RB had the right idea, just init and commit the modules dir…

haha, @loren I think you’re right!!

You can play dns and proxy tricks also, but at some point it’s just not worth it

and vendir is not quite there

you will have to somehow dig on the code, find the versions of all the sources and create a yaml for vendir

Some of that isn’t all that bad… I’ve used a little trick loading hcl into python, then inspected the content, constructed an object and written an override file to json…

But it’s still more about the remote source and whether they have more remote sources … Now you’re deep into reproducing terraform init already …

yep

and the docs in vendir are terrible

you find more info in the issues

there you go

now if you could do something fancy with source = "git....." and use cloned repos in a repo…..

https://github.com/jasonwbarnett/terraform-registry-proxy

Finally!! someone has done it.

nice! this implements what we spoke about before pepe (and they laughed at it lol). it’s basically a MITM to overwrite the registry with a custom url

well then…..when is this going to end…..

https://github.com/outsideris/citizen

but this have the same problem with submodules

I’ve worked around this using insteadOf for the terraform-null-label module

git config --global url.git@your-git-server:path/to/registry/modules.insteadOf <https://github.com/cloudposse> 

@David does this work even when using the registry notation for modules?

Hi @Erik Osterman (Cloud Posse) https://github.com/cloudposse/terraform-aws-ssm-parameter-store/blob/master/context.tf#L24 Is this what you are referring to?

yes

Then yes,

source  = "cloudposse/label/null"

translates to

<https://github.com/cloudposse/terraform-null-label?ref=0.25.0>

running

git config --global url.git@your-git-server:path/to/registry/modules.insteadOf <https://github.com/cloudposse> 

forces the clone through your own source control server rather than going to Github, the colon is important rather than the /

tldr, yes

what is path/to/registry/modules ? is it under it’s own org?

We use Gitlab, and this is the path to the project so these are just groups for structure

does that make sense?

so under path/to/registry/modules you have a bunch of cloudposse modules?

some yes, they are mirrored from github

i think this works if you do have access to the terraform registry endpoint, or if you override using .terraformrc. something has to intercept/change the default value of [registry.terraform.io](http://registry.terraform.io), unless you do have access to that site

the problem is when you can’t clone to an org/group in your VCS, or if you have the module releases for example in a s3 bucket

we don’t have that issue as we have a forward proxy to fetch modules for us

yeah if you need to wholesale change the source url and proto, that’s trouble. can’t think of anything that will do that except some kind of custom, specialized proxy

our egress rules for dev are more lenient than they are for prod our shared components environment allow egress to github which allows us to mirror repos from github but we don’t permit that anywhere else other than dev, so we are not so strict as your configuration

we also permit .[hashicorp.com](http://hashicorp.com) and [registry.terraform.io](http://registry.terraform.io) in all our envs

aye, yeah, that’s what i figured. that’s why your gitconfig trick is working. it is hitting registry.terraform.io to get the manifest for the module, and that manifest is what supplies the github git url. then your gitconfig is able to swap the git url

@jose.amengual i wonder if you could use the tf registry protocol to implement your own registry, and a .terraformrc config to force its use instead of the upstream registry, and have your registry return the s3 url (or wherever)… https://www.terraform.io/internals/module-registry-protocol

i think that’s kinda what apparentlymart was saying over in hangops

yes, that is what I want to try

but can you change the registry url in terraformrc?

good question, there is a “provider_installation” block… not sure if that impacts modules also… https://www.terraform.io/cli/config/config-file#explicit-installation-method-configuration

so my idea was to add [registry.terraform.io](http://registry.terraform.io) in my host table and point it to my api gateway that will then return a 301/302 to point to my registry url and then see if it worked

Makes sense

the proxy thing is cool but I do not like it much since is yet another thing I need to deploy BUT maybe is even possible to do it in NGIX pure or Varnish(I know it can do it)

which instead of relaying in some golang code that is too new, it relays on well known tooling

that is in the case the redirect does not work

I still think this should be implemented in the TF with an option to change the default registry url for any call to the registry( including modules)

Totally agree, been a long time pain point. We ended up recompiling the terraform binary to change it

ya, @jose.amengual i think it would be interesting if the proxy-technique could just be reimplemented in native a native AWS API gateway with rewrites

I might have got it:

curl <https://registry.terraform.io/.well-known/terraform.json> -X GET -k
{
      "modules.v1": "<https://iklqoxlmui.execute-api.us-west-2.amazonaws.com/live/modules.v1/>"
}⏎                                                       

That’s cool

Are you going to try implement it this?

I was trying to avoid custom configs in /etc/hosts

Very cool!

but I still need to figure out how to avoid this :

Failed to retrieve available versions for module "consul" (main.tf:1) from registry.terraform.io: Failed to request discovery document: Get
│ "<https://registry.terraform.io/.well-known/terraform.json>": x509: certificate is valid for registry.amengual.cl, not registry.terraform.io.

which is the blocker now

I’m trying to implement it but I want minimal custom config but since TF Cli is too smart now it looks like I will have to install CA certs and other suff which I do not really want to

I remember the good old days of source = git://.......

heh. yeah. too smart, for its own good!

easier to change the registry host in source code, and/or disable the cert check, and recompile. then give your team your binary

i don’t have code for recent terraform versions handy, but here’s what we did back for terraform 0.13… eyeballing the current code base, looks like only minor adjustments to the file paths maybe, and should otherwise apply cleanly

it is looking like that

looking at the code it pisses me off because it will be very easy for them to offer the option to pass that as an argument

yep. in our actual implementation (what i posted is from an older patchset), i think we read it from an env, with a default value if the env is unset

Loren did you ever created and issue to get this added?

negative, we did it in tf 0.11 days and just kept it up. meanwhile, tf was doing a bunch of stuff around the registry and made it sound like they were going to incorporate the use case. but if they did, i never could figure it out

ok, I will create and issue and I will start making a lot of noise……

https://github.com/hashicorp/terraform/issues/31134

Can you post the error you’re receiving from Terraform @Chin Sam?

sure, absolutely, i use Terragrunt to inject the inputs, so here is the inputs

error

So it looks like your issue is not a Terraform or module problem, but more a problem with AWS’s validation / what you’re passing to AWS.

InvalidEventPatternException: Event pattern is not valid. Reason: Filter is not an object

If you go and try to create the same CloudWatch event / rule pattern via the console — you would get the same error.

I would suggest you try doing what you want via the console first, get it working there, and then try to reverse engineer what you did in the console back to your Terraform code.

To be more clear —

InvalidEventPatternException: Event pattern is not valid. Reason: Filter is not an object

That error is an AWS API error — their validation code is rejecting the Terraform made API request because it doesn’t like that event_rule_pattern.

yeah i understand that, so i might passing it in wrong way cloudwatch_event_rule_pattern = "cron(0 0 * * ? *)" , so this input accepts cron and i am passing incorrectly ?

@Chin Sam check out these docs — I believe they match up to that variable (I didn’t confirm):

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_event_rule#event_pattern

They mention the event rule pattern should be JSON. They also point you to this documentation: https://docs.aws.amazon.com/eventbridge/latest/userguide/eventbridge-and-event-patterns.html

@Chin Sam it looks like your value for that attribute needs to be a map — See this line inside the module: https://github.com/cloudposse/terraform-aws-cloudwatch-events/blob/0.5.0/main.tf#L15

i see, thank you @Matt Gowie, much appreciated!

CC @Alex Jurkiewicz @loren @ekristen @Erik Osterman (Cloud Posse) I would particularly like your feedback.

Hooray!

Ultimately, your roadmap sounds like a lot of work for the CloudPosse team. If you have the resources, it’s a nice roadmap. But personally, I get little value from the bits beyond “re-release the current version as 1.0.0 and follow semver going forward”.

Stuff like coordinated update of all modules sounds hard. The null-module updates took months to roll out – is another coordinated update really what you want to take on?

Thanks, @Alex Jurkiewicz, but I’m not quite sure you what you mean by “coordinated updates of all modules”. You may have read something into what I wrote that I did not mean to imply.

The releases of v1, v2, and v3 of s3-log-storage, for example, are just renumberings of existing versions. Not hard at all. In general, on our roadmap, the release of v1 will be the current module, pinned to AWS v3 if it doesn’t work with v4. The v4 compatible release will incorporate all the other changes in our backlog, just so we can give ourselves a break on writing migration documents.

oh! right, sounds then

You must be Nuru on github

( don’t tell anyone)

Just commenting on one thing I saw you say on Github. I don’t think you should fear major version bumps, or try to minimise them. The value I get from semver is informative. I wouldn’t want the migration to semver to slow development of modules, or require more changes to be backwards compatible

sounds like a plan… will the v1 releases on all modules be cut more or less at the same time, or will the v1 release for any given module wait until the v2/v3 releases of that module are ready?

I think @Jeremy G (Cloud Posse) is for now focusing on just modules affected by s3 changes. Long term all modules, but the scope for now is smaller. The s3 changes mean anyone not watching the output of terraform plan closes might delete their bucket!

@Jeremy G (Cloud Posse) Thanks for the ping! Also thank you for taking the time to type this up, I know it’s been a tough thing to sort out. This sounds like a solid plan. I believe the rapid fire is likely to be a rare occurrence and I believe the community can and will understand the reasons for it this time and should this happen again for AWS V4->V5 I suspect that it will still be considered rare.

Regarding the possible rapid fire of majors due to AWS version and refactoring I would say this … since progress is being made towards 1.x and using major releases, I’m very flexible. If you all feel like it’s more beneficial to refactor on 0.x and pin to 3.x AWS and once refactor is done rev to 1.x to reduce the rapid fire of major releases and then rev to 2.x for the AWS 4.x upgrade, I can see that working out too, it would be status quo to the current model to make breaking changes under 0.x and I would qualify this situation as unique and not likely to occur again any time soon.

Thanks for following back up on this. Thank you to the team and company for the work being done on these modules and the people that help support. Appreciate all the effort.

define spam?, is that basically me?

haha promoting bitcoin wallets!

Interesting comparison break down —

https://github.com/nike-inc/pterradactyl#comparison-between-pterradactyl-vs-terragrunt-vs-terraspace

That’s cool that they included that comparison table.

Have you looked at or tried out Opta yet?

I like where they’re headed, but we haven’t used it to build/deploy anything to prod yet.

I hadn’t heard of Opta. After a quick browse — It looks to high level IMHO. It seems to abstract too much away and then you run into having to peel back a bunch of Opta’s code and various layers that you don’t have the best visibility into to actually understand how you can X. That works for some teams and more power to em, but if you’re trying to do something more complicated it feels like it will break down.

module "dynamodb_label" {
  source = "./modules/labels"

  enabled = var.ENABLE_TABLE
  name    = var.dynamodb_name

  context = module.this.context
}

locals {
  json_data               = file("./items.json")
  items                 = jsondecode(local.json_data)
}

module "dynamodb_table" {
  source = "./aws-dynamodb"

  count = var.ENABLE_TABLE ? 1 : 0

  hash_key                      = "schema"
  hash_key_type                 = "S"
  autoscale_write_target        = 50
  autoscale_read_target         = 50
  autoscale_min_read_capacity   = 5
  autoscale_max_read_capacity   = 1000
  autoscale_min_write_capacity  = 5
  autoscale_max_write_capacity  = 1000
  enable_autoscaler             = true
  enable_encryption             = true
  enable_point_in_time_recovery = true
  ttl_enabled                   = false

  dynamodb_attributes = [
    {
      name = "schema"
      type = "S"
    }
  ]

  context = module.dynamodb_label.context
}

resource "aws_dynamodb_table_item" "dynamodb_table_item" {
  for_each   = var.ENABLE_TABLE ? local.items : {}
  table_name = module.dynamodb_table.table_name
  hash_key   = "schema"
  item       = jsonencode(each.value)

  depends_on = [module.dynamodb_table]

}

The JSON file

{
  "Item1": {
    "schema": {
      "S": "<https://schema.org/government-documents#id-card>"
    },
    "properties": {
      "S": "{\"documentName\":{\"type\":\"string\"},\"dateOfBirth\":{\"type\":\"string\"}}"
    }
  },
  "Item2": {
    "schema": {
      "S": "<https://schema.org/government-documents#drivers-license>"
    },
    "properties": {
      "S": "{\"documentName\":{\"type\":\"string\"},\"dateOfBirth\":{\"type\":\"string\"}}"
    }
  }
}

The error

Error: Inconsistent conditional result types

  on dynamodb-table.tf line 173, in resource "aws_dynamodb_table_item" "dynamodb_table_item":
 173:   for_each   = var.ENABLE_TABLE ? local.items : {}
    ├────────────────
    │ local.items is object with 13 attributes
    │ var.ENABLE_TABLE is a bool, known only after apply

The true and false result expressions must have consistent types. The given expressions are object and object, respectively.

----------
Error: Unsupported attribute

  on dynamodb-table.tf line 174, in resource "aws_dynamodb_table_item" "dynamodb_table_item":
 174:   table_name = module.dynamodb_table.table_name
    ├────────────────
    │ module.dynamodb_table is a list of object, known only after apply

This value does not have any attributes.

I’ve tried many options to pass this error even change the variable type from bool to object. If I remove the condition in for_each and just pass local.items the aws_dynamodb_table_item tries to create regardless of the depends_on and it fails of course to create because table_name is returned empty because of count = module.dynamodb_label.enabled ? 1 : 0 in dynamodb_table module

I want the aws_dynamodb_table_item to be skipped if var.ENABLE_TABLE is set to false

What am I missing here?

Your main issue is this:

The given expressions are object and number, respectively

Try something like this:

resource "aws_dynamodb_table_item" "dynamodb_table_item" {
  for_each   = var.ENABLE_TABLE ? local.items : {}
  table_name = module.dynamodb_table.table_name
  hash_key   = "schema"
  item       = each.value

  depends_on = [module.dynamodb_table]

}

@Matt Gowie Thanks for your reply. I’m sorry that I failed to include this. I have actually tried that and I just updated it. I’m actually using this and the value of ENABLE_TABLE is false of type bool

resource "aws_dynamodb_table_item" "dynamodb_table_item" {
  for_each   = var.ENABLE_TABLE ? local.items : {}
  table_name = module.dynamodb_table.table_name
  hash_key   = "schema"
  item       = each.value

  depends_on = [module.dynamodb_table]

}

I still get this error.

Error: Inconsistent conditional result types

  on dynamodb-table.tf line 173, in resource "aws_dynamodb_table_item" "dynamodb_table_item":
 173:   for_each   = var.ENABLE_TABLE ? local.items : {}
    ├────────────────
    │ local.items is object with 13 attributes
    │ var.ENABLE_TABLE is a bool, known only after apply

The true and false result expressions must have consistent types. The given expressions are object and object, respectively.

@Grubhold check out https://github.com/hashicorp/terraform/issues/27877 and https://github.com/hashicorp/terraform/issues/23364

They describe your issue and how you can work around it.

@Matt Gowie Thanks for your reply. Unfortunately none of that worked I had checked them. Its very poor error that Terraform is failing to tell exactly what type it needs even when its consistent and it says that its object and object. Because local.items and {} are both objects but they don’t have the same attributes, and therefore they are not the same object type.

I finally found a way to get around this obscure error by adding this condition for_each = { for k,v in local.items : k => v if var.ENABLE_TABLE } for anyone with the same requirement this is a gem that I’ve missed you might find it useful as well.

I’m guessing it’s the same issues reported here? https://github.com/terraform-aws-modules/terraform-aws-iam/issues/193

thank you!

this:

  unique_animals = [
    for store in local.pet_store :
    store.animal_type
  ]

can just be:

  unique_animals = distinct(local.pet_store[*].animal_type)

oh wow, even better

oh right and i forgot the distinct. my original code just grabbed all the animal types and didnt deduplicate. i admit that i didn’t run the code above

i feel like it should be possible to use the grouping operator for this kind of use case, but it’s always hard for me to wrap my head around how it works… https://www.terraform.io/language/expressions/for#grouping-results

I was trying to use the grouping operator … to deduplicate previously.

question - is that syntax on the if statement after the for expression correct… terraform is complaining

what is the complaint? it looks right at first glance to me

https://www.terraform.io/language/expressions/for#filtering-elements

==?

solid ^ that was it

Thank you @RB and @loren

i could see how grouping could get there if the input looked like this:

pet_store = {
  fluffy = { animal_type = "cat" }
  blah   = { animal_type = "cat" }
  bingo  = { animal_type = "dog" }
}

then you could get the list with:

pets_by_type = { for name, pet in local.pet_store : pet.animal_type => name... }

and that should result in:

{
  cat = ["fluffy", "blah"]
  dog = ["bingo"]
}

and i guess you could construct that data structure and transform from there, somehow

It doesn’t look like our module supports the log_delivery_configuration setting yet — https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/elasticache_cluster#log_delivery_configuration

You can implement it yourself by forking the module and then PRing back. Then you’ll have implemented it for your project and for others that may want that in the future. It is always appreciated!

Ping me if you implement it and I’ll check out your PR.

Thank you @Matt Gowie for your reply… I talk with my team….

I guess there isn’t much to it - https://github.com/cloudposse/terraform-aws-ecs-alb-service-task/blob/master/examples/complete/main.tf#L28

i don’t believe we have a module for it. just like you said, there’s not much to it so we just take the cluster as an input instead of creating

we use atmos with Spacelift (to generate TF workspaces and varfiles)

in any case, atmos don’t care about the backend, being it S3 or Tf Cloud

the backend is configured in TF as usual

atmos has the functionality to generate the TF backend file from YAML config, but you don’t have to use it

Okay, thanks, that makes sense, so I can only use tf cloud as remote since I have to plan/apply from within Atmos because terraform needs a .tf file in the working directory to read from.

We have an unmaintained module for TFE that probably doesnt work any more https://github.com/cloudposse/terraform-tfe-cloud-infrastructure-automation

there’s no other way

Having read a bit more, I now believe it works like this….

… I create ALB and IP target group separately. This is then referenced via ecs_load_balancers and linked to the ecs service.

Nvm, my silly mistake with

terraform_backend_config_file_path
terraform_backend_config_file_name

it should be a .tf file

glad you figured it out

funny, people are asking what I think

and the second time I see it

they have atlantis integration docs

@matt

That’s pretty cool…thanks!

try this

kube_exec_auth_enabled = !var.kubeconfig_file_enabled

also try this

  kube_data_auth_enabled = false
  # exec_auth is more reliable than data_auth when the aws CLI is available
  # Details at https://github.com/cloudposse/terraform-aws-eks-cluster/releases/tag/0.42.0
  kube_exec_auth_enabled = !var.kubeconfig_file_enabled
  # If using `exec` method (recommended) for authentication, provide an explicit
  # IAM role ARN to exec as for authentication to EKS cluster.
  kube_exec_auth_role_arn         = coalesce(var.import_role_arn, module.iam_roles.terraform_role_arn)
  kube_exec_auth_role_arn_enabled = true
  # Path to KUBECONFIG file to use to access the EKS cluster
  kubeconfig_path         = var.kubeconfig_file
  kubeconfig_path_enabled = var.kubeconfig_file_enabled

Thanks so much for the response. Last night we identified that our dev cluster had successfully updated to 1.22 despite the terraform error described above.

Kind of frustrating, but after looking through all the cluster configs and tests it seems to be running in a stable fashion and of course because it couldn’t access the aws-auth cm, the SSO roles in the cluster are all in tact.

Of course when we perform the changes to our prod env I’ll be sure to keep the above information on hand. I’ll also take a read through the link in the comments.

Out of curiosity, is this an expected error that we can safely ignore or should I suspect the worst and continue to investigate?

Thanks again for the feedback. Really appreciated.

Does the label module lowercase your input name… I cant’ remember.

Can you pass in the name as all caps so it matches your imported object?

Unfortunately, your other option would be to fork and add an ignore_changes for the name attribute, but that would be a shame and you wouldn’t get as much use of using the open source module.

Somehow I wasn’t tracking that issue yet, subscribed and gave it a

Best response:

Haha, when writing variable objects, it often occurs to me that I just want this, https://github.com/hashicorp/terraform/issues/19898#issuecomment-1084628191

Jeremy’s point about that experiment dragging on from 0.14 onward is a good one. It’s sad to me how slow to react they’ve been on this considering it’s one of the top requested features. We’re really not asking for much it seems to me.

hmm this seems like a recent error

cc: @Ikana

cc: @jose.amengual you added this recently, no ?

It works if I created the policy with -target

and then I created everything else

also tried this:

mmmm

I can try to reproduce in like 1 hour

I will report back

when an arn in var.custom_iam_policy_arns is created in the same state, that for_each expression will generate that error because the arn is not known until apply (as it says), and for_each key expressions must all be fully resolved in the plan phase

I discussed this recently on one of Anton’s modules also… https://github.com/terraform-aws-modules/terraform-aws-iam/issues/193#issuecomment-1063421930

ok, my original pr was better @RB LOL

I was just using a string…..

and no for_each

we might have to switch back to one policy and pass the arn and if there is a need for the user to have more than one policy they can merge them together in one before passing it

@Ikana I can make a branch with the changes and maybe you can try it out?

you might want to also play with the moved block for that change, or otherwise it is backwards incompatible…

it was merged last week and is broken

so my guess no one have been able to use it

unless the policy already existed……

it’s only partially broken. you could have supplied an aws-managed-policy-arn as those always exist. or you could use -target. or you could create the policy in another tfstate.

correct

sure, I’m down to try

@jose.amengual

@Ikana https://github.com/cloudposse/terraform-aws-lambda-function/tree/fix_custom_lambda_policy

you will need to use the github source

yes because it can’t be calculated….

can you add a depends_on to the policy you are creating?

ahhh but the count will not work either..

ahhh I have a bad logic there, I will PM you

I do not like this much but I guess this is what we will have to do https://github.com/cloudposse/terraform-aws-lambda-function/blob/fix_custom_lambda_policy/variables.tf#L244

please see this example https://github.com/cloudposse/terraform-aws-lambda-function/blob/main/examples/complete/main.tf

to allow attaching the policy using the module, the arn would have to be contrived instead of passed in

to allow attaching the policy outside the module, the arn can be passed in using the resource

We added 2 new outputs to attach a policy from outside, the role_name, and role_arn which gets around the issue you saw above

thanks @RB

That looks like it will be subject to a race condition on the creation of the policy, and its attachment to the lambda role?

I’d recommend the outside method personally but the example serves to be both a test and example.

wait how could there be a race condition? there’s a depends on on the module for the inside policy to be complete

Ahh, I missed the depends_on. That is handling the race condition for most cases. I’ve come to avoid depends_on anywhere I can, especially at the module-level, so just overlooked it.

the inside method seems better for managed aws policies and the outside method seems best for custom policies. no depends on required for either method

my personal preference, as a pattern for this kind of thing (not necessarily suggesting to change anything for this module), is still just to have the variable be a list of objects with name and arn, or a map of name => arn, instead of a list of arns if i’m gonna for_each over it, the key can’t be a value that is likely to be an output of a resource

so instead of

  custom_policy_arns = [
    "arn1",
    "arn2",
  ]

it would be

  custom_policy_arns = {
     role1 = "arn1"
     role2 = "arn2"
   }

and this would avoid the for each / count error ?

yep that’s how it works. only the keys in the for_each expression need to be known. the values do not. so role1 and role2 in your example are the keys and arn1 and arn2 are the values. that’s what i was attempting to demonstrate in this comment… https://github.com/terraform-aws-modules/terraform-aws-iam/issues/193#issuecomment-1063421930

that makes sense. i wouldn’t be against a solution like that instead of a straight list of arns.

terraform code… so much of it are workarounds.. sheesh

there are way fewer rough edges than there used to be, but this is certainly one of them

isn’t this supposed to be solved in one of the newer versions? I remember someone talking about a coming fix or solution

there’s work/thought ongoing about something like a “progressive” apply. it’s a bit messy though, since there are things that are not known explicitly up front, so the initial plan is not terribly accurate. what is it going to look like? an interactive, guided plan/apply/plan/apply until everything that can be known is known, or a real error is encountered? i’m not sure…

well you could apply data resources first and then pull right away to get the values and then continue

they already do that. data sources are known values, unless the inputs to the data sources are themselves attributes of un-applied resources…

https://github.com/hashicorp/terraform/issues/4149

ahhhhhh right sorry in this case is the arn of the resource aws_iam_policy which is not a data resource

ufta that could get messy

maybe is a feature that depends_on triggers but no matter what it could get messy

I use: https://tfswitch.warrensbox.com/Quick-Start/

just run tfswitch in the tf dir and it’ll switch to the latest supported tf version in that dir

that looks good! will try

ok, good for start! ideally would love to skip the tfswitch part, but will stick with this for now. Thank!

it’s the best thing I’ve used so far - all the others are clunky and don’t work so well. I got used to running tfswitch quite quickly. Ideally though, one would use Terraform Cloud or ’Atlantis right?

@ikar AFAIK, there is nothing that does what your describe. Tfenv/tfswitch/etc are what is available.

Are you using .terraform-version files with tfenv? If not, I would check that out.

My suggestion would be to solidify on a recent version and then upgrade all your root modules to that version. Otherwise you create too much mental burn on your team who is trying to remember which version supports what.

Hey @Matt Gowie, we had nothing, now we have tfswitch and we’ll see how that works. Keeping .terraform-version in my notes, thanks!

i use asdf for this problem and it works great.

oh, that must be why asdf stinks in one of my TODO browsers tabs

For future travelers, I was able to use the aws_route data type and I found the corresponding route table id through the aws cli via

aws ec2 describe-route-tables

Sounds like you aren’t using a VPC?

I personally would prefer the second option. It follows the same pattern as what database does my application connect to.

I think it also depends on a sort of chicken-egg situation… are you building both modules in same state or seperately?

In my totally humble opinion, I prefer to keep databases in their own state away from the state where the app/web/presentation layer is handled. that way, i can wack the app and not working about stepping around the data layer.

So with that approach, I would know the database SG ID and would pass that into the ASG module as a variable… or if you have good tagging and/or naming, you can look up the database SG as a data source and use that to wire up your ASG SG to the DB SG.

Yet another approach I have used, create an SG that is common to all apps and use that as a “backplane” network… kinda like the “default” SG in an AWS default VPC. So any resource attached to the backplane SG can talk to any other resource attached to the backplane SG. Of course, that changes security requirements, but if your traffic is localized, this type of “openness” might not be a problem.

@Bhavik Patel Why does it sound like I’m not using a VPC?

@managedkaos Ya we have everything in separate states.

create an SG that is common to all apps and use that as a “backplane” network It seems the “best” practice is to create a security group for each resource. It’s what we’ve adopted at our company and I kind of wish we didn’t. Having generic security groups is a lot easier to maintain in some cases

@Bhavik Patel you can do both. and yes, i do create an app/resource specific SG if i am using the default SG approach. the default SG is there for cases that can benefit from it. however, in most cases, the DBs and other data sources have their own, dedicated SG with specific rules.

@Tyler Jarjoura I’ve just seen some companies set their security groups to accept all traffic within their VPC instead of the single resource they are using

I don’t think you’ll see too many modules supporting multiple different private subnet tiers – I don’t think it’s a very common pattern in cloud to have multiple networks like this. It’s a bit of a DC-anachronism

simplest approach is probably to use the cloudposse named subnets module with a for_each containing explicit configuration for each subnet

what I’m dealing with here is most VPCs will have ‘private’ and ‘data’ subnets as there is no use of routable IP space as there are no NGW or IGW except in the security VPC that has the firewall. All VPCs are connected to a TGW with a route table allowing them to reach the security gateway in a hub/spoke model. Existing setup was done manually so we’re not trying to redo it with Terraform

If the infra already exists, might be simplest to use terraformer to convert your existing infra into a 1:1 mapping of hardcoded resources

Existing has been setup manually and by many hands at this point. We’re getting to deploy into a fresh region giving us the chance to work it all out in a consistent manner. After it’s deployed and fully tested in the new region we’ll be able to fail over to the new region and re-dress the old region.

@matt

Ah, I believe we didn’t finish it

https://github.com/cloudposse/terraform-aws-sqs-queue/pull/2

@ let’s finish this one off

the pr is ready to be merged i think

last time we left off on it was whether or not to use the single item list inputs like in the sg module

Hi @RB, This might not be correct - https://github.com/cloudposse/terraform-aws-sqs-queue/blob/init/main.tf#L15

The resource is after a json blob for this variable - https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sqs_queue

Sorry, ignore my last 2 comments!

What’s the correct format for redrive_policy ?

it’s a single item list(string) input

so you’ll have to input the value like this

module sqs {
  redrive_policy = [jsonencode({
    deadLetterTargetArn = aws_sqs_queue.terraform_queue_deadletter.arn
    maxReceiveCount     = 4
  })]
}

Awesome, thanks.

if you tag all your terraform resources and your non-terraformed esources differently, you can use the awscli to retrieve the non-terraformed resources.

https://docs.aws.amazon.com/cli/latest/reference/resourcegroupstaggingapi/get-resources.html

I think what you are looking for is https://driftctl.com/

thanks

couldn’t you use an output instead of a file?

Correct me if I’m wrong but output is only visible after an apply I think. And I’d like to view the diff at plan time.

it should show the diff during plan time in terraform 1.x

Try wrapping it in an array. Something like the following —

instance_security_group_ids = [ module.security_group.securitygroup_id["private"] ] 

Ah that’d be a much appreciated fix! There is a lot of issues / workarounds in the CP modules for those issues.

this resource doesnt have an argument for name prefix

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/db_instance

but the others do

Right. Is there some kind of policy to use name_prefix? It seems like name would be the more useful / human-friendly choice

maybe to prevent conflicts?

yes they backported the new s3 resources with deprecation warnings

ahh nice, so i can just blindly proceed to 4.x and worry about it at some future point? :D

Ahh, for futurelings - jump to >=4.9.0 https://registry.terraform.io/providers/hashicorp/aws/latest/docs/guides/version-4-upgrade

probably best to use the latest if youre upgrading

Oh I see, it’s another bot that updates documentation.

i don’t think the problem is your for_each, it’s in how the module you are calling is determining the value of its internal local: local.create_exec_role

Hi Loren, Thanks. Do you know why it’s happening and if there is a work-around? Trying to keep it dry

without looking, i am guessing it is related to:

  task_exec_role_arn = module.role_ecs_task_exec.arn

and it is testing var.task_exec_role_arn != null or something to determine whether or not to create the data source “aws_iam_policy_document” “ecs_task_exec”

you’ll have to look at the module source to see exactly how it is determining the value of that local

but once you know which variable it is, presuming it is an arn, you can construct the arn so it is completely known at plan time, instead of passing the attribute/output from the module that creates it

e.g. "arn:aws:iam:...:${role_name}"

Thanks again, I’ll take a look.

see also this issue for more context: https://github.com/hashicorp/terraform/issues/30937

awesome, cheers.

This is very interesting. I understand the setup now for a few individual secrets. Do you have any sense on how well it’d handle setting 100 or so secrets at once? e.g. to build an application’s env file. I use an ansible-vault at the moment.

I think there’s support for building out an env file, let me try to look for that article

I haven’t used it for more than a couple variables thus far

maybe something like this https://developer.1password.com/docs/cli/secrets-config-files

ok interesting, so it’s reducing the calls by storing them all within one item, that implies to me that it’d be slow still to pull in many secrets at once. Still cool

I wonder if you can store the whole config in one secret, pull it once and then place it. Of course that’s less flexible than having their individual items in a vault

No easy answer, see this long thread… https://sweetops.slack.com/archives/CB6GHNLG0/p1649184530092499?thread_ts=1649184530.092499&cid=CB6GHNLG0

See the last few messages in that thread

there’s now a proxy for the registry

need two tricks… a proxy for registry sources, and also gitconfig insteadOf for git sources

Thanks, was hoping for an easier solution

no internet access === more complexity and $$$

https://sweetops.slack.com/archives/CB6GHNLG0/p1651225161784949?thread_ts=1649184530.092499&cid=CB6GHNLG0

When running the example:

sounds like you are passing in an invalid policy document

Hey Alex, When running something really simple example too:

In the simple example, no policies are being passed to the role and it errors. Should the simple example work?

I read the code, and it looks like you probably need to specify at least one trusted principle

Yep, I think you might be right. https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role#assume_role_policy

I’ll check and confirm. Thanks.

read the source luke

There isn’t a cloud posse module AFAIK. But there are others out there that I’ve seen before. I’d Google it.

Cheers, Matt. I’ll take a look.

Invented @ CloudPosse AFAIK

Seen lots of other orgs use their own version since.

yes, we were the first ones to create any sort of label module

the pattern has now caught on

we’re also discussing internally how to generalize it so we can support any labels

also, we invented the context pattern for terraform

that’s awesome!

you can’t

the output is in HCL format. You can view outputs in JSON format if you want, then you could easily parse and re-display in your preferred format

not sure if this helps, but using the terraform output with the raw option might help:

output.tf

output "text" {
    value = "awesome"
}

bash/zsh

$ export TEXT=$(terraform output -raw text)
$ echo "this is \"${TEXT}\""
this is "awesome"