#terraform (2023-02)

Might be more of a terragrunt question?

The module itself was designed to use a YAML catalog.

Examples are here: https://github.com/cloudposse/terraform-aws-service-control-policies/tree/master/catalog

discovered this as well https://github.com/hashicorp/terraform-provider-aws/issues/15563 looks like i might be out of luck with this

Let’s move this to a new channel I created called refarch

It would be better to use our terraform provider instead

This module should be updated to use it, but it hasn’t been updated yet.

https://github.com/cloudposse/terraform-provider-utils

The assumption is you are attempting to deep merge YAML in terraform

I opted for a simple yamldecode because I don’t even want any of the deep merge features at this point, it’ll just be overkill given we’re not really doing yaml tf anywhere outside this one library. thanks for the insight!

Makes sense

I m getting below error as its unable to identify depends on

for_each = { for entry in local.zone_appvpc: "${entry.zone_id}.${entry.app_vpc}" => entry }
│     ├────────────────
│     │ local.zone_appvpc is a list of object, known only after apply
│ 
│ The "for_each" value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be created. To work
│ around this, use the -target argument to first apply only the resources that the for_each depends on.

it’s not related to depends_on

this is a common “problem” with terraform

https://www.google.com/search?q=The+%22for_each%22+value+depends+on+resource+attribu[…]rome..69i57j0i512j0i30j69i64.189j0j4&sourceid=chrome&ie=UTF-8

That might be tricky to maintain but yeah I’d be interested if i could just get the output i want without having to make code changes

Would it support gitops or would it be console ops but for terraform?

Of course, you see both GitHub and GitLab icons in the corner? It would sync your Terraform code directly to your repo. So you can work both from the UI and VS Code.

I actually started building this tool recently, so if you want to play with it, its available here. https://codesmash.studio/schema

Its still in early development so you can’t save your code just yet! :)

As a DevOps/Infra engineer I find it hard to see a real use case where I would want to use this tool over my current setup using Vs code + terraform cli + terraform extensions for Vs code + copilot/codex. I do wonder if this is something that would appeal to devs more than infra engineers.

Maybe ask in #office-hours ?

Okay, one question, what does your current setup give you that the no-code approach doesn’t?

If I knew how you are using your system, I might be able to implement that experience into my tool out of the box! :)

Sure! Happy to share. My typically work flow looks like this…

• learn about the underlying resource I am trying to provision (ex. Mysql rds from AWS)

• Look at the terraform docs for that resource/provider

• Implement happy path of that resource to get something deployed quickly

• Iterate and fine tune the resource with the settings I want

• Convert it into a module for reuse

• Pin all my versions to help ensure things remain consistent/reproducible.

• Deploy using the module into one or more environments (dev/prod).

• Ensure it’s all living in source control. Currently I use terraform cloud for deployments. And in some cases I run it locally and have a backend in s3 or another storage provider.

give me some time to read this through

Oh yeah I’ve also added chatGPT to a lot of my workflows as well. So an AI assistant would be nice too but I imagine I could still use chatGPT with your tool as I do today.

ok, first things that comes to my mind is that you mentioned the documentation. I should add a link to each resource which would point to the official Terraform documentation page

All the other things you mentioned would be supported, since the whole thing saves everything in a git repo by default. So your CICD process can run as usual

but one thing which would offer you which you currently might not have is a package manager like NPM. You would actually be able to install several repos together into a larger repo, which you would then simply provision into a new infrastructure. Do you currently have something like that?

This is different than using modules, correct? I could say deploy this repo and then this other repo right after?

Yes, so here’s the idea. You would have many modules in many different repos. But you could then combine them into another repo based on your needs. Then you would just lauch this larger repo and have your infrastructure up and ready.

for example, you have lets say API repo and DB repo. Then you create APP1 repo which consists of both API and DB repo, and then create APP2 repo which consists of also API repo and DB repo but with different parameters.

Sorry. I can’t see much use for this compared to using terraform-lsp & Copilot (for autocompletion) and modules (for composition)

Visual display could be useful for visualising dependencies, which are hard to track when reading linear code spread over multiple files. But this is a niche use-case for me.

Visual editors can make sense for people less familiar with the underlying tooling. But then you end up needing to support people/code ejecting from your system once they get more experienced/complex.

And more broadly, I am all for lowering the barrier to entry for Terraform, but I don’t want it more widely used at work. The hard part about Terraform is understanding the underlying AWS (et al) APIs. Not the language itself.

the idea is more that you would have a packaging system for your modules. The UI really does lower the barrier of entry but isn’t the main feature. The main feature is that you would be able to compose your whole infrastructure from smaller modules in a more maintainable way.

I think I’d need to see a demo of this to understand. But I wouldn’t say composition is a painful part of the Terraform experience

okay, that’s good. I’ll be sure to let you know once I have an MVP ready.

This reminds me of working with CAD tools from my days designing circuits with VHDL. Same scenario: we had code that represented circuits (like TF modules) and we connected them using inputs and outputs (like TF variables and outputs).

We used code editors to do the design work but it was nice to have tools that visualized the code as a system, showing what modules were connected and how.

I see the same benefit from this tool. Not sure I would do the design work in a GUI but I would appreciate being able to see the complete system as the design came together.

In the GUI view, links to the source of a module (or TF docs for base modules) would be a nice touch.

One thing I would suggest is putting the inputs along the left edge of the block and the outputs along the right edge of the block. considering flow diagrams, that may make it easier to read (for left to right reading) than having the inputs and output in a top to bottom list.

I will be interested to see what are the benefits but as other said I will need to see a demo to compare with my current setup and see the benefits

Another UI/UX point I would suggest: indicate if a module is a resource module or if it contains other modules.

I imagine a tool like this (again, thinking about CAD tools I’ve used) would start at the top level of the system. The top level might be just one block with inputs and outputs. inside that block (say, double click to get into it) would be other modules that are either base resources or perhaps contain other modules, and so on going down until there are only base resources.

not sure you have that in mind but I think that would be a useful feature for super modules that collect multiple submodules into a single, callable module.

@managedkaos yup, that’s the idea. But in my case, I would totally leave it up to the user how his module behaves. The community can create any type of module they want and use the git url as a source to it. Once its loaded into the UI, only the top level input and output variables will be picked up. Then you can connect them with other repos to form a cohesive infrastructure.

You would then be able to save this templete to your own repo and reuse it in other projects.

@jose.amengual thanks, I’ll contact you when my MVP is ready

Just to clarify some things. This is not a Terraform specific project. Terraform is just the starting point. The community would be open to making their own providers which would be able to handle other tools as well. So you would then be able to chain multiple repos and process them using this system. You would then package them together to create full-stack apps where you would have all your code available in your repos. So, no more-lock in like with Bubble. This is more of an App Package Manager, but with a visual UI editor. Also, there would ba an API/CLI access as well.

ive always thought a single bucket is the easiest. ive had clients that required multiple buckets for compliance reasons

Yeah it’s definitely easiest.

Thanks @RB I can always count on you for good insights

We use a monorepo which both contains the application code and most of the supporting infra. We got a state bucket in each of our accounts. So far we are happy with that set up.

+1 for the state bucket per account. Within that bucket, each workload has its own folder. Teams I’ve worked with used this approach for dozens of accounts and it helped with knowing where to look for things.

I went with per account to folks – wow: I APPRECIATE the input.

No problem! I wanted to share an example at the time but was away from a computer. Here’s a sample of how we set the backend up with S3:

terraform {
  required_version = "~> 1"

  required_providers {
    aws = {
      source = "hashicorp/aws"
    }
  }

  backend "s3" {
    bucket = "acme-terraform-state-files"
    key    = "project-name-here/infrastructure.tfstate"
    region = "us-west-2"
  }
}

each project uses the same format and updates the key section.

I’d also add a version constraint on the aws provider and add dynamodb locking

If you decide to use multiple s3 buckets, id still use a single locking dynamodb

If you decide to use multiple s3 buckets, you may also want to decide where you want to keep those buckets (all in one account or one in each account), and the bucket naming convention

If youre not using a terraform wrapping tool like atmos (made by cloudposse), or terragrunt (by gruntworks), or terramate, then you may also want to define the key naming convention and/or the workspace key structure (if using workspaces)

https://developer.hashicorp.com/terraform/language/functions/try

@Andriy Knysh (Cloud Posse) you’re a mensch, buddy. .

nice to see you again @Jonas Steinberg. Looks like you are working with GCP now

always a pleasure to see you my friend – yes GCP for the time being, but AWS is always in the background.

data "aws_iam_policy_document" "trust" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRoleWithWebIdentity", "sts:TagSession"]
    principals {
      type        = "Federated"
      identifiers = [aws_iam_openid_connect_provider.this.arn]
    }
  }
}

This sort of trust policy is what you need

i am using IAM identity center though e

If I can help it I don’t want to have to add AWSReservedSSO_AdministratorAccess_REDACTED/[[email protected]](mailto:[email protected]) for each of the people I want to have access

oh, whoops. https://aws.amazon.com/premiumsupport/knowledge-center/iam-identity-center-federation/ might be related

btw, please don’t delete questions after they get answered

No, policies are not principals

it doesn’t seem that a permission set in IAM identity center allows inline policies that include assume roles

at present we have …

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::REDACTED:role/atlantis-ecs_task_execution",
                    "arn:aws:iam::REDACTED:root",
                ]
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

I want to remove the root and add a specific group from SSO

it is not possible to add a group from sso. the principal will be either a role arn or an sts session arn

the combo of the group and account can be assigned to a permission set. that assignment will create a role named for the permission set in that account. when the user accesses the permission set, they assume that role in that account. that is the role you can use as a principal for assume-role actions

gotcha that makes sense

the issue i have is that SSO role arn is random in its naming per account.

do you have many of them you want to trust? or just one?

its just one

but i can’t find a way of “finding the arn” which some regex

if you have many, you can use a condition instead of a principal…

			"Condition": {
				"ArnLike": {
					"aws:PrincipalARN": [
						"arn:*:iam::*:role/aws-reserved/sso.amazonaws.com/*/AWSReservedSSO_ROLENAME_*",
					]
				}
			}

if you have just one, create the assignment, and then hardcode the value

in the Principal field, you cannot use wildcards, that’s not allowed

but it works fine in conditions

what do i set the identifier block to?

sorry a little bit of an IAM n00v

probably leave it as:

               "AWS": [
                    "arn:aws:iam::REDACTED:root",
                ]

where REDACTED is the current account?

REDACTED is the account to which the permission set is assigned. that’s where the role will be created

the permission set lives in the users account

so it that case it needs to be the user account id?

oh the role is created inside the specific account itself

permission sets “live” in the sso account, nowhere else. the assignment of a permission set to a group:account creates a role in that account. roles live in accounts

yeh so redacted should be the current account id

let me take that for a spin, thanks @loren

cheers

define compliant

The readme of each module

in which sense is not compliant?

yep, that is the standard

Thankx

Thanks

Though it is not aws 4.x provider compatible

or 3.x

Using https://github.com/nozaq/terraform-aws-remote-state-s3-backend

why does it have to be compatible with 4?

To use a supported and developed version of provider

3.x is still supported

3.76.1 PUBLISHED a month ago

by the way did you checked the [versions.tf](http://versions.tf), it is not pinned to version 3

fresh install :

Initializing provider plugins...
- Finding hashicorp/local versions matching ">= 1.3.0"...
- Finding hashicorp/aws versions matching ">= 2.0.0, >= 3.0.0"...
- Finding hashicorp/time versions matching ">= 0.7.0"...
- Installing hashicorp/local v2.3.0...
- Installed hashicorp/local v2.3.0 (signed by HashiCorp)
- Installing hashicorp/aws v4.53.0...
- Installed hashicorp/aws v4.53.0 (signed by HashiCorp)
- Installing hashicorp/time v0.9.1...
- Installed hashicorp/time v0.9.1 (signed by HashiCorp)

apply is failing not thte init

because of s3 migrate

anyway i used nozaq

I use this every day and no issues, if you can’t post more details, is hard to tell

I have the raw YAML approach working:

# works

resource "datadog_service_definition_yaml" "service_definition" {
  
  service_definition = <<EOF
    schema-version: v2
    ...

can you just use yamlencode?

service_definition = yamlencode(each.value)

I just cracked it.

Thanks @loren

It turns out that the file built-in function returns a string literal which removes the need for a HEREDOC so all I have to do is simply pass my yaml file to file function and then return my value to the input variable, not HEREDOC needed.

yeah i assumed you were doing some manipulation on the object, since you read in the file and decode it above…

module "my-mod" {
  source = "../foo"
  files = yamldecode(file(files.yml))
}

if not, then yeah, just file() will give you your serialized yaml string

Try this:

# files.yml
file1:
  key1: "value1"
  key2:
    - subkey1: "subvalue1"
      subkey2: "subvalue2"

# module instantiation
module "my-mod" {
  source = "../foo"
  files = yamldecode(file("files.yml"))
}

# module source
locals {
  service_definitions = { magic = "magic" }
}

resource "datadog_service_definition_yaml" "service_definition" {
  for_each = local.service_definitions

  service_definition = <<EOF
${each.value}
EOF
}

ah – files is the for_each source. anyway thanks && fwiw I solved it.

turns out file returns a string lol. makes sense!

https://github.com/hashicorp/terraform-provider-aws/blob/v4.54.0/CHANGELOG.md

see atmos

The path variable would need to be passed within the s3 module. If it’s not then PR time.

Alternative would be passing the user into the module.

Right, thanks. I took out the S3 user part and do that separately. Had to create a PR for passing permission boundary on the S3 user-module, but I’ll let that follow it’s intended workflow

like :

locals {
  permissions_policies = [for policyname in var.permissions_policies : format("data.aws_iam_policy_document.%s.json", policyname)]
}

if I pass the local to the module

  policy_documents = local.permissions_policies
  policy_document_count = local.policy_document_count

I get :

Error: "override_policy_documents.0" contains an invalid JSON: invalid character 'd' looking for beginning of value
│ 
│   with module.github_dev_role.data.aws_iam_policy_document.default[0],
│   on .terraform/modules/github_dev_role/main.tf line 42, in data "aws_iam_policy_document" "default":
│   42:   override_policy_documents = var.policy_documents

try?

locals {
  permissions_policies = [for policyname in var.permissions_policies : data.aws_iam_policy_document[policyname].json]
}

and I think is because is taking the names as string instead of resouce names

but I have 2 data resources

2 different policies

is not one policy

I don’t use that module, so am not sure what it expects. Just commenting on what you’re sharing here…

what I mean is :

permissions_policies = [ data.aws_iam_policy_document.AllowLambda.json, data.aws_iam_policy_document.AllowECS.json]

if I pass that it works

Yes, I believe my edit will get you to that

the module input var is a list(string) so I thought it should work

But the way you’re using format() does not work

effectively, your approach using format() is passing this:

permissions_policies = [
  "data.aws_iam_policy_document.AllowLambda.json",
  "data.aws_iam_policy_document.AllowECS.json"
]

not this:

permissions_policies = [
  data.aws_iam_policy_document.AllowLambda.json,
  data.aws_iam_policy_document.AllowECS.json
]

my approach gets you to the latter

let me try

I added the . but did not worked either

Oh I see what you’re doing. You have two separate, named data sources. You’ll need to use one, with for_each to handle both. Or just hardcode the named references

I was trying to build a list of recources ids basically

There’s no way to dynamically generate/resolve the data sources name like you’re trying

no issues, I thought there was a way

Would you include your full or partial data source definition?

sure

Oh i see what you’re doing now. I kept invisioning a for each for some reason.

Why not just combine those policies into an aggregate then consume the aggregate?

Then you can use this

data.aws_iam_policy_document.aggregated.json

because this component can create roles that can have many different policies attached

Try this

provider "aws" {
  region = "us-east-1"
}

variable "policies" {
  type    = set(string)
  default = []
}

locals {
  policies = {
    "1" = data.aws_iam_policy_document.example_1.json
    "2" = data.aws_iam_policy_document.example_2.json
  }

  override_policy_documents = compact([
    for p in var.policies:
    lookup(local.policies, p, "")
  ])
}

data "aws_iam_policy_document" "example_1" {
  statement {
    sid = "1"

    actions = [
      "s3:ListAllMyBuckets",
      "s3:GetBucketLocation",
    ]

    resources = [
      "arn:aws:s3:::*",
    ]
  }
}

data "aws_iam_policy_document" "example_2" {
  statement {
    sid = "2"

    actions = [
      "s3:ListAllMyBuckets",
      "s3:GetBucketLocation",
    ]

    resources = [
      "arn:aws:s3:::*",
    ]
  }
}

data "aws_iam_policy_document" "aggregated" {
  count = length(local.override_policy_documents) > 0 ? 1 : 0

  override_policy_documents = local.override_policy_documents
}

output "aggregated" {
  value = one(data.aws_iam_policy_document.aggregated[*].json)
}

I tested this with the following inputs

terraform plan -var='policies=[]'
terraform plan -var='policies=[""]'
terraform plan -var='policies=["1"]'
terraform plan -var='policies=["1","2"]'

I will give it a try

Updated thread https://sweetops.slack.com/archives/CB6GHNLG0/p1676470678175129

can anyone help what am i missing?

I want to pick my lambda code from s3

as I see in docs

  s3_existing_package = {
    bucket = aws_s3_bucket.builds.id
    key    = aws_s3_object.my_function.id
  }

keys are bucket and key, not s3_bucket and s3_key

oh i completely missed that . Thanks @Andrey Taranik

@Sebastian Stadil can maybe make some intros or connect you with someone

nice, thanks erik.

@Jonas Steinberg Sorry for the delay, I’m happy to help

Yes, we call those “integration secrets” and they are typically (manually) stored in SSM or ASM.

Alternatively you can use something like sops secrets with KMS+IAM.

There are a number of new tools on the market (open source) that slap a UI ontop of the secrets to make them easier for develoeprs/teams to manage.

see announcements in #office-hours

I think you can do this by using the respective IAM access credentials per workspace?

seams that resource aws_vpc_peering_connection_accepter can’t work without aws_vpc_peering_connection, as it trying to spin up a new vpc peering, or am I wrong

did you test this kind of workflow?

I have not tested this. Have you tried using a data resource instead? https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/vpc_peering_connection

good point, will test it and give you feedback

was good idea but there are 2 different remote states, and the accepter part don’t know about the requester

so I think the idea of splitting the requester and accepter in 2 different terraform cloud workspaces that one can create the request and the second one can accept it any time it’s not doable

So looking at the TF resources the vpc_peering_connection is a requirement for the accepter however you should be able to fetch that data using data resources. So you will definitely have to create the connection request first but I imagine you will be able to accept anytime after in the separate workspace using that data resource I shared. If you want to accept before the connection request was made, I do not believe this would be doable.

https://registry.terraform.io/providers/Mastercard/restapi/latest/docs

Other options:

• improve the provider

• Don’t use Terraform for this, deal with it your current way and do other higher value work for a year

I’d actually encourage you to engage with the creators of the provider. If you can, fix/improve it so the community can benefit.

Thanks for the responses, they confirm what I suspected were the only options. Too bad the mastercard/restapi is somewhat clunky, it could be good to use while a better solution (like improving the providers is in progress)

@jonjitsu just saw this one also, terracurl… https://www.hashicorp.com/blog/writing-terraform-for-unsupported-resources

@Erik Osterman (Cloud Posse) that one is new, might be good for #office-hours and a linkedin share

Neat! Will add

IMO, the context part of null-label is extremely complex and mostly useful when you are doing something multi-tenanted or with extremely large number of practitioners.

Are you sure your needs are complex enough that simply composing values isn’t sufficient?

variable "ansible_defaults" {
  type = object({
    ssh_user = string
    ssh_extra_args = string
  })
}
module "ansible" {
  ssh_user = "centos"
  ssh_extra_args = var.ansible_defaults.ssh_extra_args
}
module "ansible2" {
  ssh_user = "ec2-user"
  ssh_extra_args = var.ansible_defaults.ssh_extra_args # duplicated from the above module, but is that so bad?
}

relevant: https://sandimetz.com/blog/2016/1/20/the-wrong-abstraction

Yeah, i guess that’s what i’m tossing up with right now. In our org we have essentially multi-tenant(s) w.r.t to the different teams leveraging some of these shared modules.

i’m trying to avoid other teams sooking that they need a specific argument in ansible to get it to work etc and trying to shield the module author from having to take that into consideration, it could probably be avoided with a good set of input vars and some locals logic, but as we approach potentially a hundred or so of these modules that the use of an ansible a shared module responsible for generating a valid ansible-playbook command is appealing.

Good feedback tho - seems like more thought is required.

Probably incompatible with your OS and/or architecture

If you download the same binary from terraform releases, does it work?

All we do is repackage that.

Its an older version of the shell, seems like its a problem with the apk from that. we were on a very old version (0.136.1) I upgraded to 0.152.5-alpine and everything worked.

I’ll close the ticket with the solution. thanks!

Ok, great!

don’t believe there is a resource for that. you can set it easily in the console as part of the bootstrap process

Yep, looks like there isn’t an AWS api endpoint for it.

Just another manual step to add to the bootstrap process

what does this have to do with Terraform?

The error seems straightforward even if the use case isn’t. The field is passing a string instead of an int.

did you set the correct Dynamo permissions? https://developer.hashicorp.com/terraform/language/settings/backends/s3#dynamodb-table-permissions

also, for cross-account access, the permissions must be on both sides

The role that Atlantis assumes must have permissions to the Dynamo table

and the table must allow the other role to access it

The permissions are correct in account B. I’ll have to double check in account A tomorrow. thanks for the tip

this is usually done using roles: RoleA (in account A) has the permissions to assume RoleB (in accountB). RoleB has the permissions to access the bucket and Dynamo, and also has a trust policy to allow RoleA to assume it

then for S3 backend, you specify RoleB

Yep… exactly how I’m doing it

so when TF runs under RoleA, it assumes RoleB to access the backend

Hey @Andriy Knysh (Cloud Posse) I checked all the permissions in my roles and they are all fine. I used the policy simulator in Account A and cheked the role there and it shows that I have the correct permissions. Same thing in Account B. I already know the role in Account B works because that is what our current Jenkins uses for terraform plans/applies. Is there some other mechanism that might keep someone from acccessing a table from outside on an account? I looked and from what im reading Ddb doesnt use ACLs so its not that. The table is actually there and works…again its what we already currently use for state locking. I just keep running into this resource not found error. I even tried just describing the table from my admin user that assumed the role in Account A and it wasnt able to access it either. Kind of unsure what else to check or try. Any ideas?

This is what I have in a policy in both accounts and those policies are attached to the roles in each account.

{
            "Sid": "ReadWriteLocks",
            "Effect": "Allow",
            "Action": [
                "dynamodb:PutItem",
                "dynamodb:GetItem",
                "dynamodb:DeleteItem",
                "dynamodb:DescribeTable"
            ],
            "Resource": [
                "arn:aws:dynamodb:us-east-1:111111111:table/table-name"
            ]
}

Im wondering now if you can even use dynamodb state locking if that table is in another account. There isnt a way to specify a different account in the backend provider.

Dynamo table in another account works ok. You need to make sure you add a role to the backend config

in the example above, the bucket and the table are in the root account in us-east-2

role_arn is added to the backend config of the component in accountA

so TF itself assumes RoleA to provision the resources in accountA, but for the backend it assumes role_arn in the other account to access the s3 bucket and Dynamo table. (role_arn must have all the required permissions as described here https://developer.hashicorp.com/terraform/language/settings/backends/s3

so having the S3 and Dynamo in another account works all the time

1) add a role to the backend, 2) check the role’s permissions, 3) make sure the RoleA can assume the role_arn for the backend and 4) role_arn has a trust policy to allow RoleA assuming it

Use attributes and label_order if required.

using the name like that is perfect ok

or as mentioned, you can use name and attributes

You can use a join() and for separator use the null label separator.

Thanks for the detailed explanation @managedkaos

the example https://github.com/cloudposse/terraform-aws-eks-cluster/tree/master/examples/complete is a working example, it gets deployed to AWS on every PR, and we don’t merge a PR unless the example gets deployed successfully. Please try it

I tried it and it is working now. Earlier I was referring to README.yaml file. I made a custom main.tf file using Readme file and was using latest cloudposse modules of vpc(2.0.0),subnet etc..

I noticed in vpc 2.0.0 ipv4_primary_cidr_block is used instead of cidr_block used in vpc 1.1.0

Hi Radha, this article is using terraform-aws-modules/eks/aws module and I was trying to do with cloudposse/eks-cluster/aws module.

But I need to pass some user_data to bastion server
I also tried using user_data_template variable. I think you’ll want to use an absolute path that points to your template file.

https://github.com/cloudposse/terraform-aws-ec2-bastion-server/blob/master/user_data/amazon-linux.sh

how can I do kubeconfig with the output of eks-cluster module aws eks update-kubeconfig --region <region> --name <cluster-name>

I tried like this: user_data_template =”my_user_data/install_kubectl.sh”

But I’m getting this error:

are you using the module variable user_data ?

https://github.com/cloudposse/terraform-aws-ec2-bastion-server/blob/master/examples/complete/fixtures.us-east-2.tfvars#L17 is an example of how to add more lines to the boot-time script

Actually I tried both user_data and user_data_template

In user_data how can I add multiple lines? My user data is : (file attached)

@tamsky Bro you got any idea ?

Seeing this now. Can you share your attempts to use user_data – share your full terraform config – either inline or as a gist?

and have you tried using filebase64("install_kubectl.sh") and providing that output to the module using the module parameter user_data_base64 ?

It’s something of an intricate dance, creating the vpc attachment (tgw member), accepting the attachment (tgw owner), and creating routes in the tgw route tables (tgw owner). but should be possible.

We’ve published our own module for tgw, which has invidual sub-modules for each of those steps. You’re welcome to see if it suits your use case. Here’s the cross-account module, which is just composed of calls to the other submodules, which you could certainly split out in your own configs… https://github.com/plus3it/terraform-aws-tardigrade-transit-gateway/blob/master/modules/cross-account-vpc-attachment/main.tf

the tests/ directory has complete working examples

I’m wondering how much resource_id leakage there is across accounts… and if I’ll need to read state from the other account to accomplish what I’ve described.

(thank you for the link to your module, btw)

in the vpc attachment, it’s pretty much just the transit_gateway_id, and any vpc routes you want added to the member vpc.

and then in the vpc_accepter, you need the attachment id from the member vpc_attachment

I think I’m also wishing someone had a timeline-type diagram that describes the order of operations. It’s all still a bit confusing to me.

you could mock up an environment that uses that cross-account module, and generate a terraform graph, and use graphviz to plot it?

less of a resource graph, and more of a sequence diagram

mostly the same thing, since terraform is a DAG

but I guess a resource graph would show that

did you have an opinion on whether I’d need to use remote_state to read resource values from the other account?

it is confusing though, took me quite a while to work out the kinks in that module

no opinion really, no. i use terragrunt. you could use terraform_remote_state, or parameter store. or just hardcode the value (unless you’re expecting the tgw id to change frequently)

i ended up just hardcoding the value in terragrunt, rather then messing with the dependency block

yeah - that’s the notion I have coming into this – that some number of values/strings need to be shared across terraform runs

shared (or hardcoded)

thanks for validating my ideas and sharing yours

sure thing!

Our root module implementation of TGW using our modules is here: https://github.com/cloudposse/terraform-aws-components/tree/master/modules/tgw

Back in the day, we did it the way you described, but the DX was poor and more complicated. Setting it up this way allows us to keep the state of each TGW spoke separate, but also automate the end-to-end connection in one fell swoop.

Using IAM assumed roles, it’s able to apply that changes in both accounts.

Absolutely, this:
Setting it up this way allows us to keep the state of each TGW spoke separate, but also automate the end-to-end connection in one fell swoop. That is also why we keep this cross-account piece together, in one tfstate

Thanks for sharing your experience @Erik Osterman (Cloud Posse)! – it’s super useful to hear (ahead of time) that my design is actually a footgun…