#terraform (2024-04)

one example is to use the github provider

https://github.com/cloudposse/terraform-aws-components/blob/main/modules/argocd-repo/versions.tf#L9

https://github.com/cloudposse/terraform-aws-components/blob/main/modules/argocd-repo/provider-github.tf

then use the provider resources to provision

https://github.com/cloudposse/terraform-aws-components/blob/main/modules/argocd-repo/git-files.tf

Yea I have a req of redeploying Github inside our boundaries and so I have a chance to do it clean. I was about to start clicking but I was finding some of what you’re saying above.

someone emoted nope, hmmmm

ill have to read more on the code, getting to a control state would be cool

the auth token for the github provider is read from SSM or ASM in case of AWS, the rest should be straightforward (variables configured with Terraform and/or Atmos), let us know if you need any help

We recently rolled this out for cloud posse: https://github.com/repository-settings/app

https://github.com/cloudposse/.github/blob/main/.github/settings.yml

That is very slick

This combined with GitHub organizational repository rulesets works well for for us and eliminates the need to manage branch protections at the repo level

From a compliance perspective, I have to enforce any baseline I have

So like this might work well to enforce

If you decide to go the route with terraform, there are some gotchas. I know there are bugs/limitations with the HashiCorp managed provider for GitHub that others have worked around, but those forks are seemingly abandoned. We would recommend a single terraform component to manage a single repo, that using atmos to define the configuration for each repo with inheritance. This would mitigate one of the most common problems companies experience when using terraform to manage GitHub repos: GitHub API rate limits. Defining a factory in terraform for repos will guaranteed be hamstrung by these rate limits. So defining the factory in atmos instead and combining it with our GHA will ensure only affected repos are planned/applied when changes are made.

But I rather like the approach we took instead.

I’m using Terraform to manage all our internal Github Enterprise repos and teams etc. Happy to answer any questions.

@joshmyers what issues have you run into along the way and are you using the official hashicorp provider?

Mostly slowness in the provider, against Github Enterprise on several gigs over the last few years. Sometimes this comes from API limits but mostly slow provider when you start managing 50-100+ repos and plenty of settings. We managed to hunt down a few of the slow resources and switch them out for others e.g. graphql for github_branch_protection_v3 resource. Things got a bit better in more recent provider versions. Can also split the states at some business logical point so each doesn’t get too big. I like the drift detection and alignment here. Plenty of manual tweaks/bodges forgotten to be removed got cleaned up.

The whole ownership move to integrations happened quite a while ago, not too many limitations I’ve hit other than the obvious resources and lack of certain things (most that you’d want is available). It’s the main provider. No competing fork silliness.

I think https://github.com/repository-settings/app looks cool in terms of getting some consistency across multiple repos, but I wouldn’t call the model enforcing. Certainly lighter touch than TF. Wanted to manage teams too so we’re already here…

Ironically you could use the provider to manage the <https://github.com/cloudposse/.github/blob/main/.github/settings.yml> files in repos, but have the probot do the heavy lifting.

I think this statement is easily misunderstood.

https://sweetops.slack.com/archives/CB6GHNLG0/p1712345595034359?thread_ts=1712075779.976419&cid=CB6GHNLG0

The same is :100: true of any implementation that manages repository settings via GitOps.

It’s entirely mitigated with CODEOWNERS and branch protections (e.g. via organizational repository rulesets).

Yea josh I was specifically thinking of watching drift

Thanks for clarifying. Yes, that’s true.

I have to play with the provider a bit

https://sweetops.slack.com/archives/CB6GHNLG0/p1712348663742749?thread_ts=1712075779.976419&cid=CB6GHNLG0 ish, the repo that manages the other repos can be managed/run by a separate team/different policies etc vs directly applying in your own repo. Aye CODEOWNERS/protections help.

fwiw, that’s the approach taken by https://github.com/github/safe-settings

It’s the centralized/decentralized argument.

Note, the same exact teams required to approve changes to the centralized repo, can/should be the same teams required to approve the PRs in decentralized repos. About the only difference I can see is visibility. In the centralized approach, the configuration can be private, which is beneficial. But the controls guarding the changes to the configuration itself, are the same in both cases: CODEOWNERS & branch protections with approvals.

´ Ironically you could use the provider to manage the <https://github.com/cloudposse/.github/blob/main/.github/settings.yml> files in repos, but have the probot do the heavy lifting. This would be pretty awesome.

Ah, TIL https://github.com/github/safe-settings - nice

I wasn’t across safe-settings or repository-settings.

As of this week we have about 1k resources being managed in github via terraform and our plans take close to 10mins.

Eeeekkkk that’s a long time. Btw, did you publish your modules for this?

It’s private and it’s pretty much code that is unmaintable. I just spent the past hour looking at repository-settings and for a moment I thought I was going to switch but it looks like there is a known issue with branch protection rules. I wasn’t able to get it to work in my repo: https://github.com/repository-settings/app/issues/857

Other then branch protection (a big feature that we need to have) it seems like a pretty good tool. Given this recent experience I’m wondering even if it worked how would i know when it “stops” working in the future. Ex. they fix it and in 3 months it breaks again and my new repos don’t have branch protection.

@venkata.mutyala you don’t even need those, since your GHE. We don’t use those. We use Organizational Repository Rulesets, which IMO are better.

Why manage branch protections on hundreds of repos when you can do it in one place.

To be clear, Organizational Repository Rulesets implement branch protections

But they are more powerful. You can match on repository properties, wildcards, etc. You can easily exclude bots and apps to bypass protections. You can enable them in a dry-run mode without enforcement, and turn on enforcement once they look good.

Dangggggg. How did i forget about this.

Thanks Erik! This is going to clear out a lot of TF that i didn’t need to write. I recall seeing this before but totally forgot about it.

Note, we also have an open PR for repository-settings. It will probably be a while before it merges. The repo is maintained, but sparingly.

https://github.com/repository-settings/app/pull/910

I recall seeing this before but totally forgot about it. And I just talked about it on #office-hours!! I must have really sucked presenting how we use it. I even did a screenshare

That said, safe-settings (a hard fork of repository-settings i believe and also in probot) is very well maintained and by GitHub

I just didn’t like the centralized approach.

How are you managing membership into each team?

That’s still manual for us.

Did you automate that too?

Yeah we have

Yeah we have some automation but it’s not very good. I was hoping to copy a cloudposse module hence my tears’ :sob:.

@joshmyers did you go super granular with your permissions management and just use github_repository_collaborators or do you use teams/memberships?

Teams and membership, this is GHE internal if that makes a difference

(We would accept contributions for cloudposse style modules for managing github repos and teams)

We being me too

We do this and use https://github.com/mineiros-io/terraform-github-repository.

We have a very light root module around it here: https://github.com/masterpointio/terraform-components/tree/main/components/github-repositories

Nice, looks very similar to ours. We also manage a few things like the CODEOWNERS/PR template files there too

@Matt Gowie are you using the forked (abandoned?) provider or the official hashicorp provider

(because mineiros also forked the provider)

https://github.com/mineiros-io/terraform-github-repository/blob/main/versions.tf#L11-L12 < the official provider.

Cool! Then I’m more bullish on it

I think cw logs for event bridge need to have the aws/events/ prefix. Fairly sure I had the same problem and noted that when I created it through the console it created a log group with that prefix

Oh, interesting.. Let me try that

Unfortunately that wasn’t the issue..

log_configuration = {
    cloudwatch_logs_log_destination = {
      log_group_arn = aws_cloudwatch_log_group.pipes_log_group_raw_data_stream_transformer.arn
    }
    level                  = "ERROR"
    include_execution_data = ["ALL"]
  }

It was mostly a syntax error. This ended up being the solution

Wow!

Cc @Jeremy G (Cloud Posse) @Andriy Knysh (Cloud Posse)

Great news this is something being worked on

According to the latest Hashicorp earnings call - this feature will be TFC/E specific and will not come to BSL TF cli.

Uff, thanks.

@Jeremy G (Cloud Posse) where’s your recent post on count-of issues?

Also, you’ll dig this. https://sweetops.slack.com/archives/CB6GHNLG0/p1712272028318039

It might be less of a problem in the future.

@Piotr Pawlowski I wrote this article to help people understand the issue you are facing. I hope it helps you, too.

You can use multiple providers, or Terraform workspaces. There may be another approach too via some 3rd party tooling.

honestly, type that exact same question into ChatGPT.. it will give you some options

Haha.. did that and it wasn’t helpful . Thought of checking internally as some of you might have wanted the same.

lol right on, yeah the feedback it gave me seamed reasonable, but there has to be a better way to manage a scenario like that for sure, tons of people have to be doing that exact same thing

I’ve also seen this popping up in my feed on LinkedIN and on Cloud Posse Repos.. this may be worth looking into (I plan on looking further into it as well): https://github.com/cloudposse/atmos

yeah, seems like a good solution… see “Use Cases” on that URL

I solved it by adding custom aws_iam_policy with proper policy defined with aws_iam_policy_document datasource and pass it to the module via node_role_policy_arns variable, not sure if this is the proper approach, though

I have it declared like so:

node_role_arn                 = [module.eks_node_group_main.eks_node_group_role_arn]

that is in my node group module code, and my eks cluster module is also in the same .tf file as well.

oh snaps, you’re talking about the autoscaler, not only the node group.. never mind

I ended up here - so is the plan DDB table the same as the internal state/locks table TF uses, in your case?

tl;dr Need to stop storing the plan as an artifact. This action looks nice. Need to provision the DDB table (if indeed is different to the locks table)

Aha, @Igor Rodionov @Dan Miller (Cloud Posse) can you get unblocked on the actions and their dependencies.

Need to stop storing the plan as an artifact. Why do you need to stop storing the planfile as an artifact? That ensures what you approve in the Pull Request is identical to what you apply upon merge to the default branch

As a workflow artifact attached to the PR, because putting in S3/DDB would allow finer grained access control.

Would rather store/fetch from S3.

As a workflow artifact attached to the PR Aha! Yes, that should be doable, but not something we’ve yet implemented. We would like that though… What we tried to do was something even more extreme, which was using artifact storage for everything, which proved challenging because how limited the artifact API is. But only storing the planfile as an artifact, and not trying to use the artifact storage to also replace dynamodb, that should be much easier.

Would rather store/fetch from S3. Oh, wait, I think we’re saying the same thing. We use S3 in our customer implementations, so that is already supported.

I thought you wanted to use artifact storage

So there must be a feature flag somewhere not set.

Already use artifact storage but would rather S3 - the above action looks like it’ll do that, right? Just not sure what schema the metadata DDB table is expecting…?

Nono, we use both dynamodb and S3.

We use dynamo to store the metadata, so we can find planfiles and invalidate planfiles across PRs.

Use use S3 because I think dynamodb has limits on blob storage. Also, it serves as a permanent record, if so desired.

We didn’t want to use S3 as a database and have to scan the bucket to find planfiles.

Note that 2 PRs could merge and affect the same components therefore we need some way to invalidate one or the other. Technically, a merge queue could alleviate some of the problems, but we don’t yet support that.

Let me find the schema for dynamo

Yeah makes sense. Where do you actually create that DDB table/whats the schema? e.g. the Terraform DDB state lock table has a partitionKey of LockID of type String

Yes, so here’s how we do it. Sicne we do everything with re-useable root modules (components) with atmos, here’s what our configuration looks like.

import:
  - catalog/s3-bucket/defaults
  - catalog/dynamodb/defaults

components:
  terraform:
    # S3 Bucket for storing Terraform Plans
    gitops/s3-bucket:
      metadata:
        component: s3-bucket
        inherits:
          - s3-bucket/defaults
      vars:
        name: gitops-plan-storage
        allow_encrypted_uploads_only: false

    # DynamoDB table used to store metadata for Terraform Plans
    gitops/dynamodb:
      metadata:
        component: dynamodb
        inherits:
          - dynamodb/defaults
      vars:
        name: gitops-plan-storage
        # These keys (case-sensitive) are required for the cloudposse/github-action-terraform-plan-storage action
        hash_key: id
        range_key: createdAt

    gitops:
      vars:
        enabled: true
        github_actions_iam_role_enabled: true
        github_actions_iam_role_attributes: ["gitops"]
        github_actions_allowed_repos:
          - "acmeOrg/infra"
        s3_bucket_component_name: gitops/s3-bucket
        dynamodb_component_name: gitops/dynamodb

The gitops component is what grants GitHub OIDC permissions to access the bucket and dyanmodb table

Oh, let me get those defaults

@Matt Calhoun where do we describe the entire shape of the dynamodb table?

Doh, I’d seen the above components but missed vars passing hash_key and range_key - thank you!!

aha, ok, sorry - also, this question has sparked a ton of issues on the backend here. Some docs weren’t updated, others missing.

No worries, thank you for checking

Think got it

OK, some progress…

Plan runs, can see object in S3

but nothing in DDB…no writes, can see a single read. What am I not grokking about what this thing does? I kinda expected to see some metadata about the plan/pr in DDB? Action completed successfully.

Yes, you should see something like that. Our west-coast team should be coming on line shortly and can advise.

^^ debug output

Hi Josh…just to be clear, do you have a PR open? It should write metadata to the DDB table whenever you push changes. And another sanity check, does your user/role in GHA have access to write to that table in DDB?

Hey Matt - thanks so much. Yes PR is open (internal GHE), chaining roles and yes it has access to DDB and S3. I was getting explicit permission denied writing to DDB before adding it - so something was trying…

Hmm, I wonder if same PR meant it didn’t try to re write on subsequent push? Let me push another change.

Committed a new change, can see S3 plan file under the new sha…but still nothing in DDB (Item count/size etc 0) …

Actually maybe the previous DDB IAM issues was on scan…which would make sense as I can see reads on the table but no writes.

Yup, confirmed previous fail was trying to scan. (since succeeded)

Bah - so sorry, problem between keyboard and computer. I’m looking in the wrong account - this is working as intended

When you’re ready, we have example workflows we can share for drift detection/remediation as well.

Working nicely - thank you!

No but that sounds very cool.

Perhaps code pilot can be trained to do it?

Would be super useful, right?

I think I tried to use copilot to do it, but it only wanted to do it in the same file.

I’m sure this would be a no-brainer for somebody who has built a proper VSCode app.

Or even just something on the command line

For this specific, ask I am 99% with the right prompt, and a shell script, you could iterate over all the files with mods because the prompt would be easy, and the chances to get it wrong are small.

https://github.com/charmbracelet/mods

In your prompt, just make sure convey to respond in HCL and to only focus on parameterizing constants. You can give it the convention for naming variables, etc.

I’m working on this extension for the monaco editor, and will port it to a vscode plugin among other things soon

Will check out mods for this… Charmbracelet

@george.m.sedky awesome to hear! Happy to be a beta tester when you’ve got a VSCode plugin.

awesome matt! will text you as soon as it’s ready

@Matt Gowie - report back if it works out, or was a flop

@Matt Gowie this is how it works now with monaco before the VS code plugin https://youtu.be/ktyXJpf36W0?si=xJaaQ5Pn1i7L0m_j

What is the issue Its trying to create…… Is there any error ur facing??

Hi @tamish60, I’ve managed to make it work, this region doesn’t seem to have the support for the kind of API Gateway that I’m trying to do, I moved a different region and it was able to work just fine

(@Stephan Helas best to use atmos)

So we’ve seen more and more similar requests to these, and I can really identify with this request as something we could/should support via the atmos vendor command.

One of our design goal in atmos is to avoid code generation / manipulation as much as possible. This ensures future compatibility with terraform.

So while we don’t support it the way terragrunt does, we support it the way terraform already supports it. :smiley:

That’s using the [_override.tf](http://_override.tf) pattern.

We like this approach because it keeps code as vanilla as possible while sticking with features native to Terraform.

https://developer.hashicorp.com/terraform/language/files/override

So, let’s say you have a [main.tf](http://main.tf) with something like this (from the terragrunt docs you linked)

module "example" {
  source = "github.com/org/modules.git//example"
  // other parameters
}

To do what you want to do in native terraform, create a file called [main_override.tf](http://main_override.tf)

module "example" {
  source = "/local/path/to/modules//example"
}

You don’t need to duplicate the rest of the definition, only the parts you want to “override”, like the source

So this will work together with the atmos vendor command.

1. create components/terraform/mycomponent
2. create the [main_override.tf](http://main_override.tf) file with the new source (for example)
3. configure vendoring via vendor.yml or component.yml
4. run atmos vendor This works because atmos vendor will not overwrite the [main_override.tf](http://main_override.tf) file since that does not exist upstream. You can use this strategy to “monkey patch” anything in terraform.

https://en.wikipedia.org/wiki/Monkey_patch#<i class="em em-~"</i>text=In%20computer%20programming%2C%20monkey%20patching,altering%20the%20original%20source%20code>.

@Stephan Helas also consider the following: if you are developing a new version of a component and don’t want to “touch” the existing version. Let’s say you already have a TF component in components/terraform/my-component and have an Atmos manifest like this

components:
  terraform:
    my-component:
      metadata:
        component: my-component  # Point to the Terraform component (code)
      vars: 
        ...

then you want to create a new (complete diff) version of the TF component (possibly with breaking changes). You place it into another folder in components/terraform, for example in components/terraform/my-component/v2 or in components/terraform/my-component-v2 (any folder names can be used, it’s up to you to organize it)

then suppose you want to test it only in the dev account. You add this manifest to the top-level dev stack, e.g. in the plat-ue2-dev stack

Wow, thx very much. i’ll try the vendor aproach.

in fact, my second question was, how to develop a newer component versions

you can have many different versions of the “same” Terraform component, and point to them in Atmos stack manifests at diff scopes (orgs, tenant, account, region)

i think i got the idea, but i need to test it though. its a lot to take in. i love the deep yaml merging approach a lot. i’ll try it later and will probably come back with new questions .)

i didn’t knew about the override feature of terraform. thats awesome - and yes - i’ll totally use that. So, as soon as i have mastered the handling of the remote-state (something terragrunt does for me) i think i can leave terragrunt behind

as soon as i have mastered the handling of the remote-state (something terragrunt does for me) Are you clear on the path forward here? As you might guess, we take the “terraform can handle that natively for you” approach by making components more intelligent using datas sources and relying less on the tooling.

That said, if that wouldn’t work for you, would like to better understand your challenges.

in fact, my second question was, how to develop a newer component versions We have a lot of “opinions” on this that diverge from established terragrunt patterns. @Andriy Knysh (Cloud Posse) alluded to it in his example.

This probably warrants a separate thread in atmos, if / when you need any guidance.

as short summary, i use terragrunt (like many others i suppose) mainly for generating backend and provider config. on top of that i use the dependency feature for example to place a ec2 instance in a generated subnet. as i don’t have to deal with remote states manually - i don’t know how it works behind the terragrunt curtain.

right now i try to understand the concept of the “context” module. after that i’ll try to create the backend config and use remote state. but i thing i will at leaat need another day for that.

Yes, all good questions. We should probably write up a guide that maps concepts and techniques common to #terragrunt and how we accomplish them in atmos

Backends, is a good one - like we didn’t like that TG generates it, when the whole point is IAC So we wrote a component for that.

@Erik Osterman (Cloud Posse)

thx again. i’ll open the next thread in atmos

The best write up on context might be by @Matt Gowie https://masterpoint.io/updates/terraform-null-label/

(you keep nerd sniping me in this thread!)

ok. thx soo much. its a real schame, that i didn’t find out about atmos sooner. could have saved me from a lot of weird things i did in terragrunt.

Atmos already generates many things like the backend config for diff clouds, and the override pattern (e.g. for providers)

https://atmos.tools/core-concepts/components/terraform-backends

the remote-state question comes up often. We did not want to make YAML a programming language, but use it just for config, so the remote state is done in the TF module and then configured in Atmos, see https://atmos.tools/core-concepts/components/remote-state

having said that, Atmos now supports Go templating with Sprig and Gomplate functions and datasources

https://atmos.tools/core-concepts/stacks/templating

which brings YAML with Go templates close to being a “programming” language . (w/o hardcoding anything in Atmos to work with functions in YAML, just embedding the existing template engines)

having said that, we might consider adding an Atmos-specific remote-state datasource to get the remote state for a component in Atmos directly (YAML + Go templates) w/o going through Terraform.

we might support both approaches (no ETA on the “remote-state” template datasource yet)

@Andriy Knysh (Cloud Posse)

how would i get access to the provisioning documentation? I’ve created a account to login.

@Stephan Helas that part of our documentation (docs.cloudposse.com/reference-architecture) is included with our commercial reference architecture.

That’s how we make money at Cloud Posse to support our open source. If you’d like more information, feel free to book a meeting with me at cloudposse.com/quiz

Aaaah i understand

@Erik Osterman (Cloud Posse)

i’ve got a design question:

so i tried to use s3 backend and versioned components. for that i simply created folders in components like this:

.
└── terraform
    ├── infra
    │   ├── account-map
    │   └── vpc-flow-logs-bucket
    ├── tfstate-backend
    │   └── v1.4.0
    └── wms-base
        ├── develop
        ├── v1.0.0
        └── v1.1.0

I then created the stack like this:

▶ cat stacks/KN/wms/it03/_defaults.yaml
import:
  - mixins/_defaults
  - mixins/tennant/it03

components:
  terraform:
    it03/defaults:
      metadata:
        type: abstract
      vars:
        location: Wandsbek
        lang: de
    base:
      metadata:
        type: real
        component: wms-base/v1.0.0
        inherits:
          - base/defaults
          - it03/defaults

if i include the s3 backend without a key prefix definition, the prefix would be named like the component (as described in the documentation). this would lead to a loss of state, if i changed the component version (as a new s3 prefix was created).

so i integrated a tenant mixin for the prefix like this:

import:
  - catalog/component/wms/defaults
  - mixins/region/eu-central-1

vars:
  tenant: wms
  environment: it03
  tags:
    instance: it03

terraform:
  backend:
    s3:
      workspace_key_prefix: wms-it03

So that the bucket prefix name stays stable.

questions:

• is the a sound design or am i doing it wrong?

You identified and understand the gist of the problem. The workspace_key_prefix is how to keep the terraform state stable, when versioning multiple components.

So in your example

    └── wms-base
        ├── develop
        ├── v1.0.0
        └── v1.1.0

This is the correct way to version components. The idea is ultimately that you want all stacks that use a given component to converge on the same one.

Switching versions is as easy as your example, with the caveat that the location in TF state would change.

    base:
      metadata:
        type: real
        component: wms-base/v1.0.0

So we need to ensure the backend.s3.workspace_key_prefix is stable for that component, but not all components.

So we allow component to vary and point to the version. And we require backend.s3.workspace_key_prefix to point to the a constant of the component name without the version like wms-base

You’re doing part of that, but this seems wrong to me

terraform:
  backend:
    s3:
      workspace_key_prefix: wms-it03

@Andriy Knysh (Cloud Posse) would be best to chime in here.

@Gabriela Campana (Cloud Posse) we should have a “design pattern” that describes how to use multiple versions of a component while keeping state stable

thx. this was helpful. a design pattern would be awesome.

we’ll describe such a pattern in the docs

@Stephan Helas workspace_key_prefix is by default the terraform component name (in components/terraform/<component>) , pointed to metadata.component attribute

so yes, if you want to kep it the same regardless of the versions, you can specify it in backend.s3.workspace_key_prefix and use for all versions. For example

don’t use globals like the following since it will be applied to ALL components in your infra

terraform:
  backend:
    s3:
      workspace_key_prefix: wms-it03

in the example above, the TF state S3 bucket will look like this:

bucket:
  wms-base:  # this is the key prefix, which becomes a bucket folder for the TF component
    <stack-name>-it03-v1.0:  # subfolder for the `it03/v1.0` component in the stack
      `tf_state` file
    <stack-name>-it03-v1.1:  # subfolder for the `it03/v1.1` component in the stack
      `tf_state` file    
    <stack-2-name>-it03-v1.1:  # subfolder for the `it03/v1.1` component in another stack `stack-2-name`
      `tf_state` file

if you want to keep not only workspace_key_prefix the same for all versions of the component, but also the Terraform workspace, you can do the following:

components:
  terraform:
    it03/defaults:
      metadata:
        type: abstract
      vars:
        location: Wandsbek
        lang: de
      backend:
        s3: 
          workspace_key_prefix: wms-base
    it03/v1.0:
      metadata:
        type: real
        component: wms-base/v1.0.0
        inherits:
          - base/defaults
          - it03/defaults
        # Override Terraform workspace
        terraform_workspace_pattern: "{tenant}-{environment}-{stage}"
    it03/v1.1:
      metadata:
        type: real # type 'real' is the default and optional, you can omit it
        component: wms-base/v1.1.0
        inherits:
          - base/defaults
          - it03/defaults
        # Override Terraform workspace
        terraform_workspace_pattern: "{tenant}-{environment}-{stage}"

refer to https://atmos.tools/core-concepts/components/terraform-workspaces

note that in this case, both workspace_key_prefix and TF workspace will be the same for it03/v1.0 and it03/v1.1 components, so you will not be able to provision both at the same time (or if you do so, one will override the other in the state since they will be using the same state file in the same folder)

https://linear.app/cloudposse/issue/DEV-1944/create-a-design-pattern-that-describes-how-to-use-multiple-versions-of

Thx again for all the information.

@Andriy Knysh (Cloud Posse)

i can’t inherit the terraform_workspace_pattern, right? so i need to overwrite the terraform_workspace_pattern on every component in the stack

no, you can’t inherit it b/c it’s in metadata section which is per component only and is not inheritable

so i need to overwrite the terraform_workspace_pattern on every component in the stack

let me know why you would want to do it in the first place

in my ./modules/ses folder I have copied the following example files exactly:

context.tf
outputs.tf
variables.tf
versions.tf

My main.tf looks like this:

module "ses" {
  source  = "cloudposse/ses/aws"
  version = "0.25.0"

  domain        = var.domain
  zone_id       = ""
  verify_dkim   = false
  verify_domain = false

  context = module.this.context
}

I don’t want the route53 verification stuff set up (as we will need to set up the MX record ourselves) so I didn’t add that resource, or the vpc module. Is that where I’ve messed up?

and I also have a terraform.tf file with this content (matching how others have set up other modules in our project):

provider "aws" {
  region = var.region
}

provider "awsutils" {
  region = var.region
}

I’m really stuck on why this error is being thrown inside this ses.ses_user module. Any help at all would be greatly appreciated!

I removed the terraform.tf file and moved the providers into the main.tf file, and added in the vpc module and route53 resource to see if that was the issue, but it wasn’t.

provider "aws" {
  region = var.region
}

provider "awsutils" {
  region = var.region
}

module "vpc" {
  source  = "cloudposse/vpc/aws"
  version = "2.1.1"

  ipv4_primary_cidr_block = "<VPC_BLOCK - not sure if confidential>"

  context = module.this.context
}

resource "aws_route53_zone" "private_dns_zone" {
  name = var.domain

  vpc {
    vpc_id = module.vpc.vpc_id
  }

  tags = module.this.tags
}

module "ses" {
  source  = "cloudposse/ses/aws"
  version = "0.25.0"

  domain        = var.domain
  zone_id       = ""
  verify_dkim   = false
  verify_domain = false

  context = module.this.context
}

Really not sure what’s going wrong here. I’ve matched the examples pretty much exactly. Is it because it’s running via terragrunt or something?

Am I supposed to set up every parameter in the context.tf file by passing in variables? It looks like there’s a “name” var in there which defaults to null - that could explain why I’m getting the error maybe?

I think that might be what I was missing, passing in the name variable. I just set it to our application name and it worked! I’ll refactor things to pass these variables down from the environment so it should be good, finally got the basic ses setup up.

Now the question is how to configure all the other SES stuff.

Is it possible to configure the ‘Monitoring your email sending’ (publish to SNS topic) via the terraform script? This required a configuration set to be applied too, is that possible via the module? I can’t see any options for the inputs for this.

You should give it a simple name ses-user and it should work

The name goes from the context as an input, to being used in the module.this null label, and then gets passed to the ses module using the context input. The module.this.id is the fully qualified name composed of the namespace, environment, stage, name, and other inputs. Technically none of those are required but at least one needs a value such as name for the id to have a value so you don’t get that error message

Awesome thanks for confirming. I had it as “ses” to start with, then I’ve changed it to use the same iam user name that the backend uses for s3 as well so it has the perms to talk to both services. One thing I haven’t checked yet is whether it deletes the other perms on that user or not.

There’s also a deprecated warning when running the terraform job:

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
╷
│ Warning: Argument is deprecated
│ 
│   with module.ses.module.ses_user.module.store_write[0].aws_ssm_parameter.default["/system_user/ccp-dev/access_key_id"],
│   on .terraform/modules/ses.ses_user.store_write/main.tf line 22, in resource "aws_ssm_parameter" "default":
│   22:   overwrite       = each.value.overwrite
│ 
│ this attribute has been deprecated
│ 
│ (and one more similar warning elsewhere)
╵

looks like a deprecation in the ssm parameter write module

https://github.com/cloudposse/terraform-aws-ssm-parameter-store/issues/58

I know one of our customers tested it (accidentally) in their github actions by not pinning their terraform version. When 1.8 was released, their terraform started throwing exceptions and crashing. Pinning to the previous 1.7.x released fixed it.

The new release - 1.8.2 - has solved the performance issue. In my case it’s even faster than 1.7.5

extremely powerful.

You can write your own function in golang within a provider and then run the function in terraform.

In OpenTofu 1.7, where provider functions are coming as well, we have some cool functional expansions on it that allow use cases like e.g. writing custom functions in Lua files side-by-side with your .tf files.

We’ll have a livestream next week on Wednesday diving into some of the details: https://www.youtube.com/watch?v=6OXBv0MYalY

opentofu 1.7 is still in beta tho, no ?

Yep, the stable release is coming out in ~2 weeks

https://sweetops.slack.com/archives/C063TG2DYTC/p1713450876539809

Beta 1 released this morning!

writing custom functions in Lua files side-by-side with your .tf files.

https://github.com/opentofu/opentofu/pull/1491

We’re working to get the experimental Lua provider on the registry as we speak, so you’ll be able to play around with it when test driving the beta!

are there other languages being supported besides Lua?

It’s not support for any specific languages per se, we’re just enabling the creation of providers that can dynamically expose custom functions based on e.g. a file passed to the provider as a config parameter.

So like here

provider "lua" {
    lua = file("./main.lua")
}

So the community will be able to create providers for whatever languages they want.

However, the providers still have to be written in Go, and after investigating options for embedding both Python and JavaScript in a provider written in Go, it’s not looking very promising.

I’ll be doing a PoC for dynamically writing those in Go though, via https://github.com/traefik/yaegi

how about security side with functions?

do you mean how does opentofu prevent supply chain attacks with providers that integrate new languages ?

yeah

I don’t think that’s any different to normal providers and their resources though, is it?

Btw. Lua provider is in and ready to play with https://github.com/opentofu/terraform-provider-lua

awesome

thinking if that is an issue if the codes in lua will replace Terraform binaries etc

It is awesome! To be clear: This will likely only be available for OpenTofu and not for Terraform because the OpenTofu devs are taking the provider functions feature further with this update that they came up with. I haven’t see anything that says Terraform will do the same, which will be an interesting diversion!

Oh, I didn’t catch that.

I was more looking to see provider functions adding uniformity and improving interoperability, than introducing incompatbilities

This requires some enhancements to the HCL library and is currently pointing at my personal fork, with a PR into the main HCL repository: hashicorp/hcl#676. As this may take some time for the HCL Team to review, I will likely move the forked code under the OpenTofu umbrella before this is merged [done].

Got it

OpenTofu will still likely add provider functions that the providers define and open up and AFAIU that should be available across OTF + Terraform, BUT these dynamically defined functions that you can create yourselves and are project specific will be OTF only.

This is my understanding so far, so not 100% on that.

Shouldn’t this be in the opentofu channel?

they are still quite related, aren’t they?

What a shame.

Break out the forks

Break out the forks Now’s a good time to mention that the opentofu channel exists

what a bad timing for an acquisition!

https://sweetops.slack.com/archives/CB6GHNLG0/p1713990571160029

more fun

AFAIK there is not such a data source. The Access Policies are, IMHO, only useful for bootstrapping access to a new cluster or one where you lost access to the admin IAM role. You really should be using Kubernetes RBAC for access control in most cases.

What does “using Kubernetes RBAC” look like from the Access Entry perspective? I only enabled API mode.

See AWS docs on access entries and Kubernetes docs on RBAC for details.
If an access entry’s type is STANDARD, and you want to use Kubernetes RBAC authorization, you can add one or more group names to the access entry. After you create an access entry you can add and remove group names. For the IAM principal to have access to Kubernetes objects on your cluster, you must create and manage Kubernetes role-based authorization (RBAC) objects. Create Kubernetes RoleBinding or ClusterRoleBinding objects on your cluster that specify the group name as a subject for kind: Group. Kubernetes authorizes the IAM principal access to any cluster objects that you’ve specified in a Kubernetes Role or ClusterRole object that you’ve also specified in your binding’s roleRef.

That sounds hard. I’m still a kubernetes noob. We have no k8s dedicated platform team for this work. Just using built-in cloud stuff as much as possible. I think I’d rather just spin up more accounts and/or clusters than try to bother with rbac at that level

I understand. In that case, I’d just leave the list of policies hard coded for now. There are only 4.

That’s what I did, yeah. A map of policy name to arn. Just would be nice to have a data source. I think there’s like 6 or 7 policies now. On my phone so don’t have my code in front of me

Oh and we do work in govcloud and iso partitions. So a hardcoded map gets cumbersome…

I see 6 policies now. But if you’re not going to lock things down, then you can just give everyone AmazonEKSClusterAdminPolicy which is full control.

it’s kinda “cluster as a service” in this case. I’m just writing the terraform, not managing the cluster or any apps. I don’t care what they do with it afterwards. maybe they’ll use eks rbac. I’m just giving them the option to manage the eks access entries, and trying to simplify the input where I can

You can look at what we did for guidance and inspiration

Is this in the wrong place. Terraform isn’t mentioned.

Yea more for #random , although this was discussed in #office-hours

if this is viewed as a protest, it is a correct place Lol

Odd, I thought this was #terraform, not #protests.

Haha, yea let’s keep #terraform Focused on specific questions, news and events directly tied to Terraform.

Valkey fork discussed on April 4th.

https://sweetops.slack.com/archives/CHDR1EWNA/p1712214652259389

So I am trying to look at it optimistically. IBM is behind OpenBao (fork of Vault), which is a vote for open source. While healthy competition is good, I don’t like the current hostilities between HashiCorp and OpenTofu. Best case is if the projects merge and move back to MPL or into CNCF as a unified project. Kind of like what happened with NodeJS and IO.js. Vault is where the money is at for them.

I like your perspective Erik and I hope for the best out of this. Regardless, there is a path forward here and that is what matters to me.

An interesting theory I saw on LinkedIn is that it was a contingency for the acquisition to take place; that IBM required the move to BSL. To preserve their image, it is better that HashiCorp do it than RedHat. It would also explain the “business logic” for HashiCorp, making such a drastic/sweeping change, with seemingly little regard for the downstream effects and virtually no advance notice.

It’s just confusing to me that IBM would then also be a backer of OpenBao and going to the extent it has…

Yeah those two pieces seem counter-intuitive.

https://en.wikipedia.org/wiki/False_flag

It’s all rather new, I can’t really speak for the company or others.

My personal stance is “wait and see”. If we continue on the trajectory we’ve set for ourselves from a strategy perspective, I think it’ll be great.

I know a lot of folks seem to think we’ve stagnated, but it’s far from the truth.

I also hope is for the best

Sorry to see you go. We have plenty of places to debate open source and other project forks. The point is just not in this channel.

this error message says that EnableNonSecurity isnt supported with Windows. You can disable non security with var.patch_baseline_approval_rules, but the default option is set like this: https://github.com/cloudposse/terraform-aws-ssm-patch-manager/blob/main/variables.tf#L191-L211

Thanks Dan. How does this work with a Windows asset? Just trying to patch one windows box. No Linux is involved.

it should work the same way, but you should change the default option to what you need for windows. for example

  patch_baseline_approval_rules = [
    {
      approve_after_days  = 7
      compliance_level    = "HIGH"
      enable_non_security = false
      patch_baseline_filters = [
        {
          name   = "PRODUCT"
          values = ["WINDOWS"]
        },
        {
          name   = "CLASSIFICATION"
          values = ["Security", "Bugfix", "Recommended"]
        },
        {
          name   = "SEVERITY"
          values = ["Critical", "Important", "Medium"]
        }
      ]
    }
  ]

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ssm_patch_baseline

https://docs.aws.amazon.com/systems-manager/latest/APIReference/API_DescribePatchProperties.html

This is what I get now.

I’d recommend trying one of the options listed under “valid values”

Quick and dirty is a symlink, but you still need to figure out the backend stanza for each since they are different environments

That could be done via cli flag or custom wrapper

Hi @Joe Perez, Thank you for the quick response. Do you have an example or link for a document.

I don’t have exactly that, but I did write a terraform wrapper article on how to do this with templates https://www.taccoform.com/posts/tf_wrapper_p1/

Atmos would be the cloudposse way. There’s also terragrunt and terramate

thank you @loren But, we are trying to stick with the vanilla terraform.

Good luck with that. You’ll end up with your own wrapper or script, to meet that requirement

I was trying terraform -chdir, not sure how it can help us in our scenario.

It doesn’t, really

Terraform Cloud workspaces help here as well. Stacks will go even further.

@DevOpsGuy working with vanilla terraform and no other tooling is the best and only way to cut your teeth learning terraform. In order to truly appreciate the challenges of working with terraform, there’s no substitute for “learning the hardway”. However, I really do encourage you to read our write up on this, if for no other reason to be aware, so that as you start to encounter all the rough edges you’ll know - you’re not alone. This is a tried and true path, and multiple solutions out there exist, paid or open source. As Jake mentions, this is one of the problems the commercial Terraform Cloud/Enterprise offering solves, but there are alternatives as well which are open source.

https://atmos.tools/reference/terraform-limitations/

Teams often flounder with Terraform because they’re usually starting from scratch—a costly and outdated approach. Imagine if you tried to build a modern website without frameworks like React or Angular? That’s the uphill battle you’re fighting, without standardized conventions for Terraform, which other tooling brings to the table.

Plus 1 to this. It’d definitely helpful to have better organization in variable files

cc @Erik Osterman (Cloud Posse) (for v2 components)