#terraform (2021-07)

I think it’s all pretty standard stuff. Except null label. That definitely deserves more documentation as it is complex and opinionated

You might want to read the readme for that, I don’t know any better resources to learn

https://docs.cloudposse.com/tutorials/first-aws-environment/ still WIP by the way.

but definitely like the video addition

@Erik Osterman (Cloud Posse) ^^

The modules are pretty standard. The null label like @Alex Jurkiewicz says is opinionated. And how we actually use our modules is pretty radical compared to what I see most companies are doing. The tutorial @Mohammed Yahya points how is how we actually manage our environments.

output.tf

## DB Output Variables
output "db_admin_user" {
  value       = tomap({
  for k, v in module.rds : k => v.db_admin_user
  })
}

output "db_password" {
  value       = tomap({
  for k, v in module.rds : k => v.db_password
  })
  sensitive = true
}

output "db_port" {
  value       = tomap({
  for k, v in module.rds : k => v.db_port
  })
}

output "db_url" {
  value       = tomap({
  for k, v in module.rds : k => v.db_url
  })
}

output "db_name" {
  value       = tomap({
  for k, v in module.rds : k => v.db_name
  })
}

output "db_config" {
value = {
  user = aws_db_instance.database.username
  password = aws_db_instance.database.password
  database = aws_db_instance.database.name
  hostname = aws_db_instance.database.address
  port = aws_db_instance.database.port
}
}

module.rds.db_config[“xyz”].user module.rds.db_config[“xyz”].password …

in your case use each.value.db_config.port

can you state how my output.tf will look like, it would be great?

because there’s also a for each loop

see this https://www.terraform.io/docs/language/meta-arguments/for_each.html

https://www.terraform.io/docs/language/expressions/dynamic-blocks.html

getting this error with my above code

can you please tell what do i need to try

use something like

  dynamic "subnet_mapping" {
    for_each = var.subnet_mapping

    content {
      subnet_id            = subnet_mapping.value.subnet_id
      allocation_id        = lookup(subnet_mapping.value, "allocation_id", null)
      private_ipv4_address = lookup(subnet_mapping.value, "private_ipv4_address", null)
      ipv6_address         = lookup(subnet_mapping.value, "ipv6_address", null)
    }
  }

var.db_config in you case

Used dynamic block still gave error

use count as if statement like here https://github.com/terraform-aws-modules/terraform-aws-alb/blob/e643496d985dc07a406cac025bc63c4bb10371e7/main.tf#L57

Thanks @Mohammed Yahya

Hey Zach, I’m the DevOps Advocate at env0. Yes, the pro tier is currently set per environment (state file) per month, but we’re currently talking internally about adding more options. We would love to get your feedback as to what you would like to see. This tier is also unlimited users, and unlimited concurrent runs. Also access to SAML, and our Custom Flows which enables a vast area of extensibility, including Policy as Code.

hello! Just looking at pricing of the TACO vendors and features, was trying to make sense of the env0 one. The issue on my side from cost perspective is that ‘per state’ if we are following a lot of terraform standard practices means I have lots of small ‘low blast area’ states multiplied by ‘n’ environments

I guess I might be looking at around $3k/month in that case

You’re absolutely right. We’re entertaining a possible per user type thing as well.

right on.

I’m still just poking into the idea of hosted IAC runs anyways, we’re a small org so we’ve gotten by w/o it for awhile. Just starting to hit the problems of “hey … did someone not run this updated state?”

If you don’t mind sending me a DM with how many users and state files you think you would be working with, I can take it to our CEO and CTO to see what they think. We know pricing is a major concern for the TACoS, so we want to make sure we are doing right by everybody.

personally, i like pricing per runtime minute (like codebuild), or per simultaneous job (like travis-ci). per user sucks, just means we’ll optimize to minimize individuals actually touching the service (not good for you either!)

That makes sense @loren. We’re also exploring per deployment models as well.

you can use default subnets,right? you can take them from the console

to kinda go after the obvious first, does the lambda execution role have permissions to perform the action? and does the s3 bucket policy allow the lambda to perform the action?

if the lambda does not access to the bucket AND the code file is not on the bucket you will get that error

s3_key is something like /dev/app.zip

so if the file is not in <s3://module.lambda>_artifact_bucket.outputs.bucket_name//dev/app.zip you get that error

and obviously the bucket permissions need to be correct

the bucket and key exist

does the bucket policy need to allow the role used by the lambda to access the file from s3 to use?

or does the IAM role we use to execute terraform need access to get the file from s3?

you said you are “executing” a lambda. as in, invoking an existing lambda. if that’s correct, the lambda execution role needs permissions, and the bucket policy needs to allow the lambda execution role (or the account) as the principal

if you are attempting instead to “create” or “update” a lambda, then it is the principal running terraform that needs permission to the bucket, and the bucket policy needs to allow that principal

@loren is spot on, one is creation, another es executing ( where the lambda does something to that s3 bucket)

We are getting access denied when creating the lambda via terraform

then you need to check permissions on the bucket

the lambda role needs access to the bucket

Is the issue the lambda role not being there in the policy or the IAM role we assume terraform with not being there?

Is the bucket in the same region?

Another approach I suggest is to create a temporary bucket and use aws_s3_object_cooy to stage into a bucket you control

Also, check cloudtrail. The full error will be there

its in the same region

@Steve Wade (swade1987) sometimes TF_LOG=debug terraform apply is useful to see the underlying AWS API calls

I figured it out it was regarding object ownership that the lambda wanted to use so had to fix the ACLs

thanks for all the help guys was much appreciated

jsonencode would work, but you need = as delimiter, so ¯_(ツ)_/¯

Yeah, not what I need

My keys have colons in them too which doesn’t help { "foo:bar" = "badgers", "environment" = "dev"}

Probably a string template

We are using spacelift to CD

We use spacelift to run terraform, and terraform to configure spacelift

We use the stack YAML configuration to configure everything.

checking the code, it doesn’t seem like code signing is specified at all if the user doesn’t specify it: https://github.com/hashicorp/terraform-provider-aws/blob/main/aws/resource_aws_lambda_function.go#L453

you might want to use TF_LOG=trace or read your cloudtrail logs to check details of why your function is failing. Perhaps it’s not code signing related

oh, this is an error while loading the resource: https://github.com/hashicorp/terraform-provider-aws/blob/main/aws/resource_aws_lambda_function.go#L849

It seems like the case of regions without code signing support is not handled by the provider. Fixing this would probably require an update to the provider code. Someone might have submitted a PR for this already.

yeah exactly! it’s been submitted already. The issue is that AWS doesn’t support it in osaka and other regions. So I was wondering if there is a way around it rather than wait for AWS to actually enable this in the region.

there’s an attached pic where someone suggests the use of dynamic blocks to disable code signing.. not sure how this can be achieved! was wondering if anyone had an idea!

you can’t work around the issue while still using that resource. You would have to use some other structure in terraform. For example, create a cloudformation stack and manage that with terraform, or use awscli and shell out with data external

ok. good to know. Thanks! (y)

i just call them “security_group_rules” and define it as a list of objects, with attributes mapped to the security group rule resource

i am needing to allow a security group in account X to communicate with a RDS db in account Y

if you don’t want to define the object, you can do it as list(any) and then use try(object.value.attr, null) to default the value to null if not provided

i am trying to work out what the source_security_group_id will be if its cross account

can you pass <account id>/<sg id> ?

is that an attribute of a resource, or of the module you are writing that i don’t have access to?

so the module that i am creating creates the SG attached to the DB (as well as the DB itself)

i want to provide a way to add additional SG rules to that SG

if you use vpc peering, you can use references in a rule to a security group in another account, yes.

yes we are peering

if you use transit gateway instead, you are SoL

yes, the format for the cross-account sg reference is <account id>/<sg id>

perfect thanks man much appreciated

i am trying to work out the foreach as i really want to pass a list of objects containing the account_id and the security group id

honestly, for security group rules, i would just force the caller to define the label as one of the object attributes

then your for_each is just:

for_each = { for rule in var.rules : rule.label => rule }

i was thinking of doing …

# Allow additional security groups to access the database.
resource "aws_security_group_rule" "rds_ingress_cross_account" {
  for_each                 = toset(var.cross_account_security_group_ids)
  description              = "Ingress: Allow access from ${aws_security_group.pods.name}."
  from_port                = 3306
  protocol                 = "tcp"
  security_group_id        = aws_security_group.rds.id
  source_security_group_id = each.key
  to_port                  = 3306
  type                     = "ingress"
}

aws_security_group["pods"].name

what does the code look like in the submodule? and what attribute from the resource do you want?

i’ll grab it, 1 sec

here is the call to the submodule

module "s3-static-site" {
	for_each = lookup(var.root_true, var.env) ? {} : local.static-site-template
	source = "./modules/static-sites"

        env             = var.env
        region          = var.region
        is_root_account = var.is_root_account
	server_access_logs_bucket = aws_s3_bucket.bucket-server-access-logs.id
 
        kms_key_s3 = var.kms_key_s3
        site_name     = "${lookup(var.static_bucket_prefix, var.env)}each.key"
	acl            = each.value.acl
        index_document     = each.value.index_document
        error_document     = each.value.error_document
        cors_allowed_headers     = each.value.cors_allowed_headers
        cors_allowed_methods     = each.value.cors_allowed_methods
        cors_allowed_origins     = each.value.cors_allowed_origins
        cors_max_age_seconds     = each.value.cors_max_age_seconds
}

Here is the submodule main.tf

resource "aws_s3_bucket" "static-site" {
  bucket = "${lookup(var.static_bucket_prefix, var.env)}${var.site_name}"
  acl    = var.acl
  website {
    index_document = var.index_document
    error_document = var.error_document
  }
  cors_rule {
    allowed_headers = [var.cors_allowed_headers]
    allowed_methods = [var.cors_allowed_methods]
    allowed_origins = [var.cors_allowed_origins]
    max_age_seconds = var.cors_max_age_seconds
  }
  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        sse_algorithm     = "AES256"
      }
    }
  }
  logging {
    target_bucket = var.server_access_logs_bucket
    target_prefix = "logs/${var.env}/${var.site_name}/"
  }
  tags = merge(
    local.common_tags,
    {
      "Name" = "${lookup(var.static_bucket_prefix, var.env)}${var.site_name}"
    },
  )
}

I’m trying to do something like this in the outputs: output from submodule:

output "static-site-url" {
	value = aws_s3_bucket.static-site.website_endpoint
}

output from calling module (which will be later referenced in the top level main)

output "static-site-url-capture" {
  value = module.s3-static-site.static-site-url
}

i’m basically trying to pull the website_urls (4 of them) and then declare them in my cloudflare module for DNS CNames

I could just move this submodule up to be a main module, I just wanted to try this and see if it worked

I’m also finding this for_each output construct a bit challenging to understand based on the docs

when you use for_each, the result becomes a map of objects instead of a single object. the key in the map is the same as the each.key value from the for_each expression

sure, so like [“theitemcreated”]

exactly, and your for_each is basically local.static-site-template , which appears to be a map. so each key in that map can be used as a key on module["s3-static-site"]["your key"]

for your outputs to be usable and make sense, you’ll almost certainly want them to also be a map, otherwise you won’t know which static-site bucket the output is coming from…

it’s dynamically referencing it that I’m struggling with I think

So if I had an output like this

output “website_dns” value = aws_s3_bucket.static-site[“websiteA”].website_endpoint

and in the main.tf where I use that variable I could do this

my_website = module.bucket.aws_s3_bucket.website_dns

ah, I’m seeing your post (I was typing at the time)

this will give you a map of { name : website_endpoint }

output "static-site-url" {
	value = { for name, site in module.s3-static-site : name => site.static-site-url }
}

i often recommend using outputs just to see the shape of the object, even before you try to use it as a reference somewhere else

once you see the shape, it makes a lot more sense as far as how you get at it

I thought you couldn’t print outputs anymore from modules

you could for example just output the whole module resource:

output "s3-static-site" {
	value = module.s3-static-site
}

(I’m clearly missing something)

i’m having to guess at what you’re meaning, but i think you’re getting at the change from terraform 0.11 to terraform 0.12, when the outputs became top-level attributes of the module instead of being nested under the outputs attribute… you can still access them, you just won’t find them under .outputs anymore cuz it doesn’t exist

@loren This was extremely helpful. Thank you

This was the secret sauce I needed

output "static-site-url" {
	value = { for name, site in module.s3-static-site : trimsuffix(name, ".domain.com") => site.bucket-websites }
}

then declaring the variable

static_site_url =           module.buckets.static-site-url

then to create the records

resource "cloudflare_record" "static-sites" {
  for_each = var.static_site_url
  zone_id  = lookup(var.cloudflare_zone_id, var.env)
  name     = "${lookup(var.cloudflare_dns_previx, var.env)}${each.key}" # record name, site
  value    = each.value # s3 dns name
  type     = "CNAME"
  proxied  = true
  ttl      = 1
}

You’re issue log points out that terraform is pulling an old version of the null label module. Update your refs, you aren’t actually using 0.24.1

Here’s your problem: https://github.com/phillhocking/hubsub.io/blob/dev/main.tf#L8

Try version 0.7.0

It’s not possible as far as I know. AWS has lots of documentation about the fact that when you associate an ES cluster with a VPC then it’s intrinsically private.

Here is a little bit of background info: https://www.jeremydaly.com/access-aws-vpc-based-elasticsearch-cluster-locally/

@Matt Gowie thank you very much! so as I understood I have to use this example https://github.com/cloudposse/terraform-aws-elasticsearch/tree/master/examples/non_vpc

Yeah if you want a public cluster I don’t believe you can associate it with a VPC.

Though I would question if you really want a public cluster vs being able to access that cluster yourself. But I’ll leave that to you to figure out.

@Matt Gowie thank you so much. I want to create index and mapping with same terraform soI have to have access. But thank you for your advice, I will think how to done in privately.

@Vikram Yerneni replied in the issue. See if that helps you.

Thanks a lot @Matt Gowie. Appreciate the help here

I am testing the example u mentioned

@Matt Gowie Instead of all four variables namespace, stage, environment, name (based on my colleagues advice), I went with name variable and it worked.

I updated the ticket with same

I do one per account

I have used one s3 bucket for the org wide in the past but have had issues where in some one accidently ovwewrote the state file keys

I use versioned bucket

Yeah - versioned buckets helps for rollback and DR

one per account as well

Thank you for your response. I figured using a CF stackset would be the best option to deploy TF state bucket and dynamodb in each account org wide

3 options here( lot of tradeoffs and debates) :

• 1 bucket with all state files when using a shared service account where IaC pipeline lives there. you can have very tight access on this bucket.

• 1 bucket per account with 1 state file for all resources in that account

• 1 bucket per account with x states files based on tf workspaces in that account.

always reduce the blast radius in your state file, you don’t want to put all the eggs in one basket.

We used to create one bucket per account, but when you get into programatic creation of accounts, creating that S3 bucket for maintaining state is such a major blocker for automation. Thus we back tracked on this and predominantly use a single S3 bucket.

programatic creation of backend configurations also helps mitigate misconfiguration.

One versioned bucket in an “admin” account which has no other infra basically. Them make use of a wrapper to terraform which handles the -backend-config and the state file key part for me. It mirrors the path in my repo almost 1:1. /foo/infra/prd == s3://bucket/…/foo/infra/prd/terraform.tfstate

Thank you for your response. I m using a CFN stackset to create to the S3 and Dynamodb , deployed to targeted OU from the management account which has Control Tower enabled - so the cfn automatically made use of the ControlTower Execution IAM roles and , it has auto deployment feature so that when new accounts are added to OU - the stack will auto create S3 bucket and DynamoDB in new accounts

I also published this blog post ~https://msharma24.github.io/blog/terraform-backend-for-multi-aws-accounts/~https://mukeshsharma.dev/2021/07/12/aws-terraform-s3-backend.html>

one per acct..

yes, do terraform state list figure out which resource you don’t want and remove it with terraform state rm x.y.z

then remove the resource from datadog, manually

I would consider using a different module if you want directly addressable instances in something like an autoscale group.

https://registry.terraform.io/modules/cloudposse/ec2-instance-group/aws/latest

IMO, this is more of a use case for what kubernetes calls “Stateful Sets”, which is why we have this “instance group” module.

Note, it does not auto-scale.

with this module, you can even assign each instance in the group an elastic ip

but in my case, i am using private local IP and route53 record will route traffic to that IP.. lets say, we hosted web app on ec2 instance and that app instance local IP get change when every its get terminated and created new one. so i want to create route53 records in a way that do have to go again and again to update the IP in records. is this possible ?

why not use a private load balancer? why do the instances in the autoscale group need to be directly addressed by their private IP address?

let’s understand the business use-case first, then see what architectural choices will support that.

I have trying to deploy sensu-go server with RDS and database will be hosted into RDS using Postgres and Yes. you are right. we dont need to put ec2 instance in auto autoscale group. then in this case. problem will get solve automatically. am i right ?

well, instances will preserve their private IP for the lifetime of the instance. the risk of that instance going away, however, in an autoscale group is heightened because the objective there is to provide fault tolerance and availability, not preservation of state (including IPs). I don’t know enough about sensu - so what part about it needs/wants static IP addresses? my assumption is that the sensu server itself is stateless, and uses RDS to preserve state. And then any agents just need to address the sensu server endpoint, either by DNS or IP. If that’s all correct, then placing the load balancer infront of sensu, will be what you want.

Cool. Thanks a lot. i got it. let me try and see the results.

which version? they do have docs for v0.11 and older (basically HCL1 instead of HCL2…) https://www.terraform.io/docs/configuration-0-11/index.html

0.14

sigh. my bad. poor reading

that’s cool

the hack i’ve used in the past is to browse the repo by branch or tag… it’s not always pretty, and sometimes you have to view the raw files when there is embedded html to really see what’s up, but it works… https://github.com/hashicorp/terraform/tree/v0.14/docs

basically the code for multiple providers here is not working in 0.14 https://www.terraform.io/docs/language/providers/configuration.html

ya that’s basically what I’m doing

was wondering if there was something better

thanks

for that, do not specify configuration_aliases in the required_providers block. that argument is not backwards-compatible

instead, define the provider block with an alias for the second name. everything else should be the same

I’m writing a module that takes in two aws providers and it’s not working. I tried

  required_providers {
    aws = {
      source = "hashicorp/aws"
    }
    aws-dns = {
      source = "hashicorp/aws"
    }
  }

with

resource ... {
  provider = aws-dns
...
}

and

module ... {
  providers = {
    aws     = aws
    aws-dns = aws.dns
  }
  ...
}

but that didn’t work, although there’s no syntax errors

it just ignores the second provider

any ideas on passing in multiple providers to modules?

with version 0.14

  required_providers {
    aws = {
      source = "hashicorp/aws"
    }
  }

provider "aws" {
  alias = "aws-dns"
}

with

resource ... {
  provider = aws-dns
...
}

and

module ... {
  providers = {
    aws     = aws
    aws-dns = aws.dns
  }
  ...
}

just checking if that worked for you?

oh sorry, didn’t see you post, I’ll check now..

I can’t get that to run.

well, that’s the gist of doing it in 0.14. the reusable module declares the provider block with the alias. and the caller (e.g. the root module) passes the providers inside the module block

Just to settle this conversation, this is what worked: mod1/main.tf:

provider "aws" {
  alias = "other"
}

resource "aws_s3_bucket" "bucket1" {
  bucket_prefix = "bucket1"
}

resource "aws_s3_bucket" "bucket2" {
  provider      = aws.other
  bucket_prefix = "bucket2"
}

main.tf:

provider "aws" { ... }

provider "aws" {
  alias   = "account1"
  ...
}

module "mod1" {
  source = "./mod1"
  providers = {
    aws       = aws
    aws.other = aws.account1
  }
}

thanks again

How are you running that command? If it’s a shell pipeline, you probably want to make sure you have set -o pipefail or similar

that command is in makefile and i am calling it from a config file in yml

Yeah you can delete and manage versions via the registry UI if you’re the owner —

To support null-label chaining and passing around terraform project “context”, the terraform-null-label module exports a special file called [context.tf](http://context.tf) that you can add to any child or root module to be able to utilize a top label. It’s used in all of the cloudposse modules and provides a consistent interface for naming across all the modules.

The [context.tf](http://context.tf) file is specifically generated so it can be easily copy / pasted into any other module. So it’s referred to as an “export”.

Thanks for responding. I got these in my .terraform/module/module.json

    {
      "Key": "terraform_state_backend.dynamodb_table_label",
      "Source": "cloudposse/label/null",
      "Version": "0.22.0",
      "Dir": ".terraform/modules/terraform_state_backend.dynamodb_table_label"
    },
    {
      "Key": "ecs_alb_service_task_myweb_server.task_label",
      "Source": "cloudposse/label/null",
      "Version": "0.24.1",
      "Dir": ".terraform/modules/ecs_alb_service_task_myweb_server.task_label"
    },

Could you elaborate a little bit on how the the exports.tf leads to an entry in this file with different versions? What I’m trying to do is to get an inventory of modules and their versions of the repo by inspecting this file. Thanks @Matt Gowie

I know the reason I have different versions is because the modules that use the null_label module require different versions. But not sure how they get here

So what your module.json is telling you is that your terraform_state_backend module is utilizing null-label as a child module at version 0.22.0 AND your ecs_alb_service_task module is utilizing null-label as a child module at version 0.24.1

Does that make sense?

They both consume that null-label module and for both consumptions, it requires an instance of that child module on your local.

Yes, I get that. But I also have other entries like this: { "Key": "default_label", "Source": "git::<https://github.com/cloudposse/terraform-null-label.git?ref=tags/0.21.0>", "Dir": ".terraform/modules/default_label" }

They are the same module, right? I wonder why they have different form in the ‘Source’ field. Why it’s named ‘couldposse/label/null’? Where is that from?

every single module invocation specifies a source, and every source could vary. It’s just a matter of the author/maintainer to update it and we don’t always do that.

for example, if we cut a new release of terraform-null-label that has a trickle down effect of literally hundreds of PRs and updates, since modules depend on modules.

'couldposse/label/null' is the registry notation.

git::<https://github.com/>... is the git source notation

historically, we used git. but now we don’t use that anywhere. So anything using the git:: notation is definitely out of date (probably 1+ years)

Thanks @Erik Osterman (Cloud Posse) I just realized it’s notation difference. So you are officially switched to registry notation? Good to know.

Can you md5 the combo? https://www.terraform.io/docs/language/functions/md5.html

It solves my issue if I don’t need numbers or convert to string so look a good way for me. In the end, it seems that we cannot define ourself the index of these target in the module. I’m trying to do with it but thank you for the solution I may reuse it

cc @Erik Osterman (Cloud Posse) — I’m assuming Cloud Posse has had to do this before but you likely deployed it via K8s?

Do you have a default value assigned?

Importing resources means that Terraform will be used to manage those. If a lookup is the objective then data sources can be used. https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/ec2_transit_gateway

I know this, but after that if I do a destroy, then it will destroy it too right? No other solution? Can it get from another terraform statefile? So create one for base network and one for new objects that can use the other tf’s resources

You also can read outputs of external / other statefiles.

Indeed destroying imported resources is destructive. HOWEVER you can find and consume resources using data sources.

will try that. thanks

You could trace the API Calls in Cloudtrail and even use it for an IAM policy. https://securingthe.cloud/aws/automatically-generate-aws-iam-policies/

@Alex Jurkiewicz

aha, concat and spread, nice

Hey, but that is input from terragrunt file, and i want to iterate over it in terraform file.

I want to use that here in postgres db

resource "postgresql_role" "user" {
  ---->for_each = keys(var.sql_db)[0]
  name     = lookup(keys(var.sql_db)[0],"db_names")
  login     = true
  password  = random_string.suffix.result
}

resource "postgresql_database" "postgres" {
  ---->for_each = keys(var.sql_db)[0]
  name     = lookup(keys(var.sql_db)[0],"db_names")
  owner    = postgresql_role.user[each.value].name
}

small note: i would suggest you put for_each at the top of any module/resource block, since it modifies the whole block

Okay, but how can i iterate over list of db to make dbs and assign roles for them

so your second code block is code in the “../rds” module?

yes

when you use for_each with a module block, it’s going to create two copies of the module

actually no they are in same script

I’m sorry, but I’m really confused. Can you provide a simpler example?

yea, So on a wider picture if there are 2 db instances then that loop inside module will create 2 db instance on the basis of sql_db map, right

But i want to create multiple dbs within db instance and for that i am using resource of postgres db and role in this script with provider of postgres

for that i need to iterate over that list of dbs that present inside that sql_db map

ok, there is a limitation of Terraform you will run into here. You can’t create a postgres instance and then create DBs inside it in a single Terraform configuration

in resource block of postgres db and role

Terraform needs to initialise all providers at startup. In this case, the Postgres provider will be unable to initialise because the Postgres instance doesn’t exist. So Terraform will fail.

The only solution is to split these tasks into two separate Terraform configurations

the creation of db instance is in different script and the creation of db and roles in root file itself

Just want a way to iterate over that list, it is creating the roles and db but on the key name but i need it for the db_names list

based on this block you posted before:

resource "postgresql_role" "user" {
  ---->for_each = keys(var.sql_db)[0]
  name     = lookup(keys(var.sql_db)[0],"db_names")
  login     = true
  password  = random_string.suffix.result
}

I think you want something like this:

resource "postgresql_role" "user" {
  for_each = var.sql_db.db_names
  name     = each.key
  login     = true
  password  = random_string.suffix[each.key].result
}
resource "random_string" "suffix" {
  for_each = var.sql_db.db_names
}

got this error

After using this

oh, you need var.sql_db.postgres.db_names

Getting this error

After using this

resource "postgresql_role" "user" {
  for_each  = var.sql_db["postgres"].db_names
  name      = each.value
  login     = true
  password  = random_string.suffix.result
}

resource "postgresql_database" "postgres" {
  for_each = var.sql_db["postgres"].db_names
  name     = each.value
  owner    = postgresql_role.user[each.key].name
}

instead of

for_each  = var.sql_db["postgres"].db_names

try

for_each  = toset(var.sql_db["postgres"].db_names)

Note that this will make each.key and each.value the same. Specifically, each.key/value will hold db1 and then db2 on each iteration.

the problem you are running into is that without casting ["db1","db2"] into a set, or indexing into it as an array, you are operating on it as the complete list.

For example: (note this quick example does not use for_each but the concept of using toset() is still the same:

variable "sql_db" {
  default = {
    postgres = {
      node_type  = "db.t3.micro"
      type       = "postgresql"
      disk_size  = 6
      db_names   = ["db1", "db2"]
      admin_user = "postgresadmin"
    }
  }
}

output "dbnames" {
  value = [for i in toset(var.sql_db["postgres"].db_names) : i]
}

dbnames = [
  "db1",
  "db2",
]

Thanks, it worked!

awesome

@Andrew Miskell thanks for reporting this

we need to test the module with TF 1.0.2

currently we are working on this PR https://github.com/cloudposse/terraform-aws-eks-cluster/pull/119 to improve the module

Excellent, thanks!

I believe you have to init your backend before importing anything,

is the vpc attached to the transit gateway?

oh… you are right. that is missing. thanks

I ran into the same issue going from an existing cluster with v0.39.0 to the same one with v0.42.1 of the terraform-aws-eks-cluster module

terraform plan fails with the same error message. cc @Andriy Knysh (Cloud Posse)

Prior to Nuru’s new PR - Enhance Kubernetes provider configuration (#119)…. I worked around it by manually passing a KUBE_CONFIG_PATH (which was something new I had to do)

KUBE_CONFIG_PATH=~/.kube/config terraform plan

Also tried it now (defining the KUBE_CONFIG_PATH) and plan ran successfully.

(Wondering if I should always be defining that env var everywhere now like in pipelines, etc but I’m not sure yet if it will work in all cases, first-time run vs updates)

Interesting. The new module inputs:

+  kubeconfig_path_enabled = true
+  kubeconfig_path = "~/.kube/config"

seems to do the equivalent: KUBE_CONFIG_PATH=~/.kube/config terraform plan without having to define KUBE_CONFIG_PATH

I wound up using kube_exec_auth_enabled = true property in the cluster configuration to get it working.

I tried that one and got “Unauthorized” Was there another input to set?

No, although check with aws eks get-token works in the shell

All it’s supposed to do is call out to aws eks get-token for the token and use it.

You might need kube_exec_auth_aws_profile and kube_exec_auth_aws_profile_enabled if you have multiple profiles in your aws cli config.

Yes I do (for multiple profiles). From the cmd line, aws eks get-token works. I’ll give it a try thanks!

At the moment, I don’t… I just have a single default profile.

Is it also normal for terraform to want to update/replace the OIDC thumbprints and the cluster tls data on every run?

Thanks! This also works!

  kube_exec_auth_enabled = true
  kube_exec_auth_aws_profile_enabled = true
  kube_exec_auth_aws_profile = "project-1-admin"

Do people also standardize on profile names on teams? Wondering that the way I have the profile name defined… may differ from the way others may have that profile name defined …may differ with a CI environment (default)

Not sure, likely not… may need to have a default value and each user overrides the default if they have a different name.

I’m sure there’s probably a better way to do it… But for some reason the default wouldn’t work for me.

Yes. So a matter of passing something extra terraform ... -var=profile-name-goes-here each time. For the kubeconfig_path that config location is pretty standard

yeah, and I have mine in the standard location, but it doesn’t pick it up for some reason.

Normally, no I don’t notice TF replacing the OIDC thumbprints (when there are no code changes). But during the plan of updating the module version, yes I notice it is trying to replace it.

thank you @Andrew Miskell! I think settled on

  kube_exec_auth_enabled          = true
  kube_exec_auth_role_arn_enabled = true
  kube_exec_auth_role_arn         = var.role_arn

Turns out I did have a role defined in .tfvars which works when passed in.

From memory there is a page in the docs going through per instance size limitations

Just use aurora though

If only … I would if I could believe me!

I’m trying to find the docs as then I’ll do a lookup in my terraform module

In your first app_admins section it looks like there’s an extra ] - is that an intentional extra closing square bracket?

When you say “it is mapping all the values from the list of string to both the maps” what do you mean? Can you paste the result?

It may be to do with you using the same variable name (user) for iterating through each array. Try renaming to admin_user, viewer_user and editor_user to rule that out.

It also looks like you’re mixing users and roles in your arrays in the first post you made which probably helps answer one of the questions above.

Checkout the “filtering elements” section here: https://www.terraform.io/docs/language/expressions/for.html

Also see regexall for testing the elements of the lists to identify roles vs users.
regexall can also be used to test whether a particular string matches a given pattern, by testing whether the length of the resulting list of matches is greater than zero.
https://www.terraform.io/docs/language/functions/regexall.html

now using this

 map_users = concat([
    for user in local.user_app_admins:
    {
      userarn  = user
      username = split("/",user)[1]
      groups   = ["system:masters"]
    }
  ],
  [
    for user in local.user_app_viewers:
    {
      userarn  = user
      username = split("/",user)[1]
      groups   = ["cluster-viewer"]
    }
  ],
  [
    for user in local.user_app_editors:
    {
      userarn  = user
      username = split("/",user)[1]
      groups   = ["cluster-editor"]
    }
  ]
  )
//substr(split("/",user)[0], -4, -1) == "user"

  map_roles = concat([
    for role in local.role_app_admins:
    {
      rolearn  = role
      username = split("/",role)[1]
      groups   = ["system:masters"]
    }
  ],
  [
    for role in local.role_app_viewers:
    {
      rolearn  = role
      username = split("/",role)[1]
      groups   = ["cluster-viewer"]
    }
  ],
  [
    for role in local.role_app_editors:
    {
      rolearn  = role
      username = split("/",role)[1]
      groups   = ["cluster-editor"]
    }
  ]
  )

Not worked

│ Error: Error putting IAM role policy zs-test-k8sAdmins-policy: ValidationError: The specified value for roleName is invalid. It must contain only alphanumeric characters and/or the following: +=,.@_-
│       status code: 400, request id: 1dc44f02-4c71-4a64-9814-c106b4add52c
│ 
│   with aws_iam_role_policy.admin_policy["0"],
│   on rbac.tf line 3, in resource "aws_iam_role_policy" "admin_policy":
│    3: resource "aws_iam_role_policy" "admin_policy" {
│ 
╵
╷
│ Error: Error putting IAM role policy zs-test-k8sAdmins-policy: ValidationError: The specified value for roleName is invalid. It must contain only alphanumeric characters and/or the following: +=,.@_-
│       status code: 400, request id: 238e3a81-1306-44e4-96fd-93bb9fae34d2
│ 
│   with aws_iam_role_policy.admin_policy["1"],
│   on rbac.tf line 3, in resource "aws_iam_role_policy" "admin_policy":
│    3: resource "aws_iam_role_policy" "admin_policy" {
│ 
╵
╷
│ Error: Error putting IAM role policy zs-test-k8sDevs-policy: ValidationError: The specified value for roleName is invalid. It must contain only alphanumeric characters and/or the following: +=,.@_-
│       status code: 400, request id: 088ce065-d0b8-46d2-bbdd-8432eb597681
│ 
│   with aws_iam_role_policy.dev_policy["1"],
│   on rbac.tf line 33, in resource "aws_iam_role_policy" "dev_policy":
│   33: resource "aws_iam_role_policy" "dev_policy" {
│ 
╵

getting this error

that’s interesting!

i recommended running a second time with plan to catch any “read” calls that only happen when the resource is already in the tfstate, and also destroy if your role needs that level of access…

I find iamdump really useful for this.

https://github.com/claranet/iamdump

iamlive is another one. i just like not having to use any tool or start any server… https://github.com/iann0036/iamlive

i usually point people here, https://learn.hashicorp.com/collections/terraform/modules

alright i’ll read it

thanks

https://sweetops.slack.com/archives/CB3579ZM3/p1626892223047500

TLS handshake timeout When a node is unable to establish a connection to the public API server endpoint, you may an error similar to the following error.

server.go:233] failed to run Kubelet: could not init cloud provider "aws": error finding instance i-1111f2222f333e44c: "error listing AWS instances: \"RequestError: send request failed\\ncaused by: Post  net/http: TLS handshake timeout\""

The kubelet process will continually respawn and test the API server endpoint. The error can also occur temporarily during any procedure that performs a rolling update of the cluster in the control plane, such as a configuration change or version update. To resolve the issue, check the route table and security groups to ensure that traffic from the nodes can reach the public endpoint.

so check nodes sg, if not working try to update k8s version to 1.20

I finally sort of figure it out. I think my problem was some sort of race condition. I was following the examples from different CP modules but they are very confusing. Some mention you have to create a null resource, some say you have to set depends_on on the node groupe module. Turns out since I’m using module_depends_on I don’t get the error anymore. I’m not sure if this is what fixed the issue though.

Thanks for your help @Mohammed Yahya

let me know if posting like this one is not appropriate

all good, @Mohammed Yahya

We use it via Spacelift. Works great

Do you maintain a single rego rule set? Multiple? How do you test/validate rego changes to prevent unintended regression?

We define a set of rules centrally, and stacks can opt in to using them. Almost all stacks do.

At the moment our rules are all warnings only, so they don’t block merge. We don’t change the rules often so dealing with regressions is ignored

Thanks. The team authoring the rego is deciding what rules to add in the spirit of unit/integration testing to maximize quality? Or mandate? Presume the former. Really appreciate the feedback.

We aren’t really at the scale you are talking about. There are 10 SREs who write and own the Terraform files. A larger number of developers can contribute pull requests.

But I don’t think we have any policy’s around what rules to add or not. It’s a best effort thing to reduce security issues. There’s none outside our team driving this

check this helpful blog post about opa and conftest https://blog.styra.com/blog/policy-based-infrastructure-guardrails-with-terraform-and-opa

sample opa policies https://github.com/Scalr/sample-tf-opa-policies

you can rebuild your state by hand using terraform import

not fun, but doable

You can also try Terraformer - https://github.com/GoogleCloudPlatform/terraformer

they are multiple resources

eks,vpc,iam,subnet,fargate

how do i go about it in that case

https://github.com/GoogleCloudPlatform/terraformer/blob/master/docs/aws.md#use-with-aws

nice thanks alot. let me try this out

@omry quick question have u tried this with eks before. seems like its not supported. But its part of the suppported resourced on terraformer doc

No, sorry but I have not tried it with EKS. If it’s not working I suggest you should open an issue.

It’s a stretch to say that lambda uses docker containers. Lambda uses container images as an alternate packaging format and distribution mechanism to the zip file in S3. AFAIK only ECR is supported.

that is what i meant my bad, also that is what I thought as well

This might work for you, once implemented, https://github.com/aws/containers-roadmap/issues/939

The AWS provider can only work on a single region at a time. Look up multi pp let provider configuration and provider aliases.

You need two AWS providers in your code, one for each region

Hi alex, i tried to work with aliases,

provider "aws" {
  alias      = "fr"
  access_key = "${var.AWS_ACCESS_KEY_ID}"
  secret_key = "${var.AWS_SECRET_ACCESS_KEY}"
  region     = "eu-central-1"

Then i tried to call it within the module - but i cannot call the provider from within the module

 module "db1" {
  source  = "terraform-aws-modules/rds/aws"
  version = "3.3.0"

  providers {
    aws = "aws.fr"
  }

  identifier = "db1"

This is the detailed error

│ Error: Unsupported block type
│
│   on main.tf line 221, in module "db1":
│  221:   providers {
│
│ Blocks of type "providers" are not expected here.

providers = {

that was it! thank you very much!!

Launching on Monday! Will let you know as soon as it’s live

woot woot

Any details? PR comments on plan or anything else? Looking forward to checking things out

Integration with Azure DevOps repos and webhooks are there, PR comments will be coming a bit later; hope we will be able to get some feedback from you and a few other customers and iterate (=add what’s needed) very quickly

https://docs.spacelift.io/integrations/source-control/azure-devops here you go!

your best option maybe to create a new branch and test with terraform plan, without applying, and see changes, if anything will be replaced then you need to plan out the migration to new changes.

alright, sounds like a solid plan, thanks mate!

i am getting the error

│ Could not retrieve the list of available versions for provider hashicorp/aws: no available releases match the given constraints >= 3.40.0, 3.42.0, >= 3.43.0

but in our providers.tf file we got

required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "3.42.0"
    }

any ideas what might be wrong please?

I saw this today, funny, I tried lower the version like 3.41.0 then run init, and update it back to 3.42.0

i had to stop because seems that some breaking changes take place

 Error: Invalid function argument
│
│   on .terraform/modules/eks/local.tf line 188, in locals:
│  187:     for index in range(var.create_eks ? local.worker_group_count : 0) : templatefile(
│  188:       lookup(
│  189:         var.worker_groups[index],
│  190:         "userdata_template_file",
│  191:         lookup(var.worker_groups[index], "platform", local.workers_group_defaults["platform"]) == "windows"
│  192:         ? "${path.module}/templates/userdata_windows.tpl"
│  193:         : "${path.module}/templates/userdata.sh.tpl"
│  194:       ),
│  195:       merge({
│  196:         platform            = lookup(var.worker_groups[index], "platform", local.workers_group_defaults["platform"])
│  197:         cluster_name        = coalescelist(aws_eks_cluster.this[*].name, [""])[0]
│  198:         endpoint            = coalescelist(aws_eks_cluster.this[*].endpoint, [""])[0]
│  199:         cluster_auth_base64 = coalescelist(aws_eks_cluster.this[*].certificate_authority[0].data, [""])[0]
│  200:         pre_userdata = lookup(
│  201:           var.worker_groups[index],
│  202:           "pre_userdata",
│  203:           local.workers_group_defaults["pre_userdata"],
│  204:         )
│  205:         additional_userdata = lookup(
│  206:           var.worker_groups[index],
│  207:           "additional_userdata",
│  208:           local.workers_group_defaults["additional_userdata"],
│  209:         )
│  210:         bootstrap_extra_args = lookup(
│  211:           var.worker_groups[index],
│  212:           "bootstrap_extra_args",
│  213:           local.workers_group_defaults["bootstrap_extra_args"],
│  214:         )
│  215:         kubelet_extra_args = lookup(
│  216:           var.worker_groups[index],
│  217:           "kubelet_extra_args",
│  218:           local.workers_group_defaults["kubelet_extra_args"],
│  219:         )
│  220:         },
│  221:         lookup(
│  222:           var.worker_groups[index],
│  223:           "userdata_template_extra_args",
│  224:           local.workers_group_defaults["userdata_template_extra_args"]
│  225:         )
│  226:       )
│  227:     )
│     ├────────────────
│     │ local.workers_group_defaults["platform"] is "linux"
│     │ path.module is ".terraform/modules/eks"
│     │ var.worker_groups is tuple with 1 element
│
│ Invalid value for "path" parameter: no file exists at #!/bin/bash -xe
│
│ # Allow user supplied pre userdata code
│ ${pre_userdata}
│
│ # Bootstrap and join the cluster
│ /etc/eks/bootstrap.sh --b64-cluster-ca '${cluster_auth_base64}' --apiserver-endpoint '${endpoint}' ${bootstrap_extra_args} --kubelet-extra-args '${kubelet_extra_args}' '${cluster_name}'
│
│ # Allow user supplied userdata code
│ ${additional_userdata}
│ ; this function works only with files that are distributed as part of the configuration source code, so if this file will be created by a resource in this configuration you must instead obtain this result
│ from an attribute of that resource.
╵

So having one module use another module?

https://github.com/mumoshu/variant2/tree/master/examples

this is the best place to start

I wouldn’t use our variant code to start - it’s a master class and more complicated.

Thank you!

That explains my confusion The last time i used the yaml stack config it was brand new and I got seriously confused after 2-3 days of work on it. Will revisit now.

it’s less confusing now

It is definitely a different way to think about terraform, but it’s proven to be immensely popular with our customers.

Thank you! I tried it when it still wasn’t mature, but really wanted to avoid terragrunt for this project if I could stick with native, as it’s so hard to convert back later on.

Will take a look! Appreciate all the insight as always

@Erik Osterman (Cloud Posse) is this part of the tutorial still relevant with atmos? https://docs.cloudposse.com/tutorials/atmos-getting-started/#5-invoke-an-atmos-workflow

Was trying to avoid using the entire atmos config/docker setup and focus just on what’s being run with variant/cli without dependency on the geodesic container for now.

NOTE: I built my entire terragrunt stack to start with the terraform-null-label + random pets so I’m feeding those values into every single part of deployment from thereafter. That’s another reason I’m trying to make sure I know where the logic is being set for the workflow, because ideally I don’t gut all of that work.

Geodesic is not required

It just makes it easier to start up since we document all the requirements in the Docker file

Also check out geodesic today. We invested a lot into usability. It’s hard to make a mistake now

That was about 4 months ago

There is also a geodesic tutorial

I can. Trying to keep it easy to plug into azure pipelines + spacelift and don’t need any role assumption and all the rest. If I can execute the planning commands with zero need to customize then it might be worth using. Otherwise, I still have to call it externally.

Are you saying basically the entire stack stuff is just native terraform, other than the workflows, which is variant still?

Oh, I think I’m close. My problem is most likely that I don’t see any variant files for the examples on plan-all. I’m trying to figure out where this stuff sits. I see the atmos repo has terraform-plan files, which are useful and show me that the config/vars are being set. Going to go look in github org and search for more on where the plan-all type logic is coming in.

variable "default-args" {
    type  = list(string)
    value = [
      "-out",
      "${opt.stack}-${param.component}.planfile",
      "-var-file",
      "${opt.stack}-${param.component}.terraform.tfvars.json"
    ]
  }

Maybe this an import or abstracted somewhere i’m missing

There we go… logging my progress for others in case this is useful.

… so the plan-all = https://github.com/cloudposse/reference-architectures/blob/63b99a8f7f8e8dc06de6de4cda1465154ab00632/stacks/workflows-ue2.yaml#L9

This is actually calling the -job: terraform plan vpc. In looking at the atmos repo you can see:

job "terraform plan" {
  concurrency = 1
  description = "Run 'terraform plan'"

  parameter "component" {

....

So I’m seeing that these job names (which suprised me with a space in it, making me think some subcommands) are actually variant run but configured from the yaml file to organize the list of jobs to iterate on.

And lastly the way this is being built is from the atmos/modules/workflow/workflow.variant file. https://github.com/cloudposse/atmos/blob/a5fd6ea6515e40b0ac278e454db86009bdcb3286/atmos/modules/workflow/workflow.variant#L55

This shows it does a hcl items ` items = conf.env_config.workflows[param.workflow].steps` and uses this to build the collection of jobs to run 1 at a time from what I see.

Geez. No wonder my first go around was confusing. Super flexible, but was expecting this in code/Go/scripts. HCL made this a bit confusing since I wasn’t aware it was building the entire flow inside the HCL variant files.

cc @Erik Osterman (Cloud Posse) there’s my logged results I know things are changing still. Unless you say something is way off I plan on trying out the workflow file + terraform variant files for now with some terraform stack yaml from your tutorials repo and see how it goes. I think this is all compatible with spacelift and the work you’ve been doing.

Thank you!

Getting there! variant run terraform plan tfstate-backend --stack euc1-root ran. The inheritance with yaml stuff adds complexity to do natively. Pretty awesome how the generate command sets up the dynamic key info in the backend each time. This is the stuff I was using terragrunt to solve.

If it works out I’ll post up a blog post on my experience. I’m thankful for the terraform module variant files as that’s pretty darn cool! Can’t wait to try out the spacelift stuff too. Thank you!

@sheldonh could you please share your blog url here? I’m quite interested on knowing how this process ends. Thanks!

Sure! Don’t have a post yet. We’ll see how it goes . I post a weekly update here https://app.mailbrew.com/sheldonhull?aff=sheldonhull

@Erik Osterman (Cloud Posse) real quick, just to confirm I’m not on the wrong track….

With the new work y’all are doing am I going down a path that y’all are moving away from by using variant to run the defined workflow from the yaml (like ue1.dev.yml) file? I finally figured out that stuff yesterday. (Avoiding Atmos for now to understand the workflow itself better)

I’m guessing that I define the workflow for deploy (probably already there) and it’s basically running the back end generate initialization plan and apply.

My option using the yaml stack config seems to be use that variant based workflow (that Atmos is a wrapper for I think) and it manages my backend state and commands and hopefully this sets everything up with spacelift next week.

Otherwise I write tasks to do the same steps that variant is doing in my own Task library which I’m trying to avoid so I can benefit from work y’all have done. I see that I can’t just run the yaml stack config without variant/wrapper because it requires the dynamic building of the backend.tf.json.

It’s making a lot more sense after digging through it yesterday.

it’s basically running the back end generate initialization plan and apply. Yes, we write a backend.tf.json that is derived from the deep merged configuration of backend. Then we write a terraform.tfvars.json which is based on the deep merged configuration of vars

This deepmerging is what’s so powerful. It’s what allowed us to build a service catalog. We can deploy services this easily:

import:
- catalog/eks
- catalog/rds
- catalog/vpc

And we’ll have a new environment with a VPC, EKS cluster and RDS cluster in 4 lines of code.

Plus, we can override settings for anything very succinctly as well.

Everything is ultimately pure terraform using our custom terraform provider.

the stack config allows us to define configuration in a tool agnostic way. so we have written atmos as one tool. but we also have written many other things that operate on the stack configuration. stack configs provide something impossible to do in OEM terraform: deep merging, imports, YAML anchors, etc. And yet, since it’s portable, we’re able to use pure terraform with it.

• https://github.com/cloudposse/terraform-provider-utils

• https://github.com/cloudposse/terraform-yaml-stack-config

• https://github.com/cloudposse/terraform-spacelift-cloud-infrastructure-automation

• https://github.com/cloudposse/terraform-tfe-cloud-infrastructure-automation (not currently maintained, but shows how we defined a portable config format, and then we were able to use it interchangably with spacelift, atmos, and TFC).

• Also, I think the word workflow is used interchangeably, so I’m not sure of the usage in the context.

What we were talking about yesterday on the call is to think more in terms of policies than workflows.

Atmos implements something we called workflows, which are not policy based. They just run a set of steps in order. But that is what I say we should move away from. Also, just a reminder: we’re rewriting atmos in native go and moving away from variant. Variant served it’s purpose: it helped us prototype what we wanted, but now that we know what the interface is, we’re making a cli for it. Also, atmos is not to be confused with a general purpose task runner like make or gotask.

With spacelift, we implement a policy driven approach. But it won’t make much sense until you see it first hand. This stuff is radically father ahead than the way 99% of companies use terraform or even think of using terraform.

even the way most companies use policies with terraform is different. e.g. atlantis added support for policies, but like most other solutions, those policies are scoped to what one run of terraform is doing. Imagine if you had a policy that governed the every execution of terraform (plan/apply) across all root modules, and was aware of all other root modules. so you could have an overarching collection of policies that govern when/how everything runs.

…that’s possible with spacelift

@Erik Osterman (Cloud Posse) This really helped. Variant is neat, but the “workflow” was very confusing initially because variant was newer to me. It duplicates what I coded in my Go based cli task and so abstracting from Go was a bit clunkier than just writing the code for me.

My problem is that:

• root module/core infra is very hard to refactor once you start relying on it across the team. Thus trying to get this ready for something like Spacelift.
• I like terragrunt for DRY, but trying to stick with native so the work I do will benefit more of the org without relying on more Terragrunt quirks.

Right now as part of the an application team rather than cloud operations I own one complete aws app. This means I’m doing the entire stack in dev/prod for VPC -> ECS tasks. So my takeaway:

1. Variant, Make, Mage, whatever to call terraform native init, plan, and apply at this time. That’s basically what variant was trying to do.
2. Variant workflows are just another form of the same problem with cli driven vs policy driven. Policy driven is more the future, BUT right now it’s not a plug and play to do this. It’s likely something I’ll need to do after I get things working with spacelift and my stack configs.

Therefore my action item is use whatever to do the run commands right now, leverage the yaml stacks to be compatible with this new approach being taken, and come back soon to look into the policy driven approach before I scale to team.

Sound about right? On the same page? Thanks again, I’ve really enjoyed the creative thinking you have on this topic and appreciate all the pointers!

Sound about right? Is your new Go based cli open sourced yet so I can look sometime as that’s what I’m now working in full time. Would like to see what’s there to learn and maybe eventually contribute if I see something I can help with.

Love how much of what I’ve been doing crosses over into similar area. Just noticed the randompet script that’s using yq. I love that I used yq to convert a powershell object into datadog configuration files dynamically. It’s a great tool! I also get a laugh everytime folks look strangely at me for itchy-elephant or the like in the vpc name

thanks @Ryan Ryke

please run

make init
make github/init
make readme

and terraform fmt

apparently my developer tools are broken… reinstall

nvm, got it.

{ for k,v in module.label.tags … }

Yup

brain fart. Thanks.

thankfully (for me) for structures also sort lexicographically so zipping lists works out.

A yaml object has no intrinsic sorting, so “why” is probably just “that’s what the deserialiser happens to do”

also, in GO, order is not preserved, so they probably just sort it by default

the docker hub image has existed for ages, but it went through a long period of neglect. Nice to see it’s being updated again.

Personally I use tfenv everywhere, even in CI

I’ve been using tfswitch. I like tfenv too. I guess I liked that tfswitch read the version information from the terragrunt or terraform plans and would allow dynamic switching. Practically I realize I haven’t had need for that as much as I thought

Was thinking docker would simplify versioning among many environments though with less issues, so I’ll have to give it a shot and see.

I would suggest to look towards using Atlantis as it also supports multiple TF versions and is quite nicelly integrated into CI/CD pipelines.