#terraform (2024-04)

one example is to use the github provider

https://github.com/cloudposse/terraform-aws-components/blob/main/modules/argocd-repo/versions.tf#L9

https://github.com/cloudposse/terraform-aws-components/blob/main/modules/argocd-repo/provider-github.tf

then use the provider resources to provision

https://github.com/cloudposse/terraform-aws-components/blob/main/modules/argocd-repo/git-files.tf

Yea I have a req of redeploying Github inside our boundaries and so I have a chance to do it clean. I was about to start clicking but I was finding some of what you’re saying above.

someone emoted nope, hmmmm

ill have to read more on the code, getting to a control state would be cool

the auth token for the github provider is read from SSM or ASM in case of AWS, the rest should be straightforward (variables configured with Terraform and/or Atmos), let us know if you need any help

We recently rolled this out for cloud posse: https://github.com/repository-settings/app

https://github.com/cloudposse/.github/blob/main/.github/settings.yml

That is very slick

This combined with GitHub organizational repository rulesets works well for for us and eliminates the need to manage branch protections at the repo level

From a compliance perspective, I have to enforce any baseline I have

So like this might work well to enforce

If you decide to go the route with terraform, there are some gotchas. I know there are bugs/limitations with the HashiCorp managed provider for GitHub that others have worked around, but those forks are seemingly abandoned. We would recommend a single terraform component to manage a single repo, that using atmos to define the configuration for each repo with inheritance. This would mitigate one of the most common problems companies experience when using terraform to manage GitHub repos: GitHub API rate limits. Defining a factory in terraform for repos will guaranteed be hamstrung by these rate limits. So defining the factory in atmos instead and combining it with our GHA will ensure only affected repos are planned/applied when changes are made.

But I rather like the approach we took instead.

I’m using Terraform to manage all our internal Github Enterprise repos and teams etc. Happy to answer any questions.

@joshmyers what issues have you run into along the way and are you using the official hashicorp provider?

Mostly slowness in the provider, against Github Enterprise on several gigs over the last few years. Sometimes this comes from API limits but mostly slow provider when you start managing 50-100+ repos and plenty of settings. We managed to hunt down a few of the slow resources and switch them out for others e.g. graphql for github_branch_protection_v3 resource. Things got a bit better in more recent provider versions. Can also split the states at some business logical point so each doesn’t get too big. I like the drift detection and alignment here. Plenty of manual tweaks/bodges forgotten to be removed got cleaned up.

The whole ownership move to integrations happened quite a while ago, not too many limitations I’ve hit other than the obvious resources and lack of certain things (most that you’d want is available). It’s the main provider. No competing fork silliness.

I think https://github.com/repository-settings/app looks cool in terms of getting some consistency across multiple repos, but I wouldn’t call the model enforcing. Certainly lighter touch than TF. Wanted to manage teams too so we’re already here…

Ironically you could use the provider to manage the <https://github.com/cloudposse/.github/blob/main/.github/settings.yml> files in repos, but have the probot do the heavy lifting.

I think this statement is easily misunderstood.

https://sweetops.slack.com/archives/CB6GHNLG0/p1712345595034359?thread_ts=1712075779.976419&cid=CB6GHNLG0

The same is :100: true of any implementation that manages repository settings via GitOps.

It’s entirely mitigated with CODEOWNERS and branch protections (e.g. via organizational repository rulesets).

Yea josh I was specifically thinking of watching drift

Thanks for clarifying. Yes, that’s true.

I have to play with the provider a bit

https://sweetops.slack.com/archives/CB6GHNLG0/p1712348663742749?thread_ts=1712075779.976419&cid=CB6GHNLG0 ish, the repo that manages the other repos can be managed/run by a separate team/different policies etc vs directly applying in your own repo. Aye CODEOWNERS/protections help.

fwiw, that’s the approach taken by https://github.com/github/safe-settings

It’s the centralized/decentralized argument.

Note, the same exact teams required to approve changes to the centralized repo, can/should be the same teams required to approve the PRs in decentralized repos. About the only difference I can see is visibility. In the centralized approach, the configuration can be private, which is beneficial. But the controls guarding the changes to the configuration itself, are the same in both cases: CODEOWNERS & branch protections with approvals.

´ Ironically you could use the provider to manage the <https://github.com/cloudposse/.github/blob/main/.github/settings.yml> files in repos, but have the probot do the heavy lifting. This would be pretty awesome.

Ah, TIL https://github.com/github/safe-settings - nice

I wasn’t across safe-settings or repository-settings.

As of this week we have about 1k resources being managed in github via terraform and our plans take close to 10mins.

Eeeekkkk that’s a long time. Btw, did you publish your modules for this?

It’s private and it’s pretty much code that is unmaintable. I just spent the past hour looking at repository-settings and for a moment I thought I was going to switch but it looks like there is a known issue with branch protection rules. I wasn’t able to get it to work in my repo: https://github.com/repository-settings/app/issues/857

Other then branch protection (a big feature that we need to have) it seems like a pretty good tool. Given this recent experience I’m wondering even if it worked how would i know when it “stops” working in the future. Ex. they fix it and in 3 months it breaks again and my new repos don’t have branch protection.

@venkata.mutyala you don’t even need those, since your GHE. We don’t use those. We use Organizational Repository Rulesets, which IMO are better.

Why manage branch protections on hundreds of repos when you can do it in one place.

To be clear, Organizational Repository Rulesets implement branch protections

But they are more powerful. You can match on repository properties, wildcards, etc. You can easily exclude bots and apps to bypass protections. You can enable them in a dry-run mode without enforcement, and turn on enforcement once they look good.

Dangggggg. How did i forget about this.

Thanks Erik! This is going to clear out a lot of TF that i didn’t need to write. I recall seeing this before but totally forgot about it.

Note, we also have an open PR for repository-settings. It will probably be a while before it merges. The repo is maintained, but sparingly.

https://github.com/repository-settings/app/pull/910

I recall seeing this before but totally forgot about it. And I just talked about it on #office-hours!! I must have really sucked presenting how we use it. I even did a screenshare

That said, safe-settings (a hard fork of repository-settings i believe and also in probot) is very well maintained and by GitHub

I just didn’t like the centralized approach.

How are you managing membership into each team?

That’s still manual for us.

Did you automate that too?

Yeah we have

Yeah we have some automation but it’s not very good. I was hoping to copy a cloudposse module hence my tears’ :sob:.

@joshmyers did you go super granular with your permissions management and just use github_repository_collaborators or do you use teams/memberships?

Teams and membership, this is GHE internal if that makes a difference

(We would accept contributions for cloudposse style modules for managing github repos and teams)

We being me too

We do this and use https://github.com/mineiros-io/terraform-github-repository.

We have a very light root module around it here: https://github.com/masterpointio/terraform-components/tree/main/components/github-repositories

Nice, looks very similar to ours. We also manage a few things like the CODEOWNERS/PR template files there too

@Matt Gowie are you using the forked (abandoned?) provider or the official hashicorp provider

(because mineiros also forked the provider)

https://github.com/mineiros-io/terraform-github-repository/blob/main/versions.tf#L11-L12 < the official provider.

Cool! Then I’m more bullish on it

I think cw logs for event bridge need to have the aws/events/ prefix. Fairly sure I had the same problem and noted that when I created it through the console it created a log group with that prefix

Oh, interesting.. Let me try that

Unfortunately that wasn’t the issue..

log_configuration = {
    cloudwatch_logs_log_destination = {
      log_group_arn = aws_cloudwatch_log_group.pipes_log_group_raw_data_stream_transformer.arn
    }
    level                  = "ERROR"
    include_execution_data = ["ALL"]
  }

It was mostly a syntax error. This ended up being the solution

Wow!

Cc @Jeremy G (Cloud Posse) @Andriy Knysh (Cloud Posse)

Great news this is something being worked on

@Jeremy G (Cloud Posse) where’s your recent post on count-of issues?

Also, you’ll dig this. https://sweetops.slack.com/archives/CB6GHNLG0/p1712272028318039

It might be less of a problem in the future.

@PiotrP I wrote this article to help people understand the issue you are facing. I hope it helps you, too.

You can use multiple providers, or Terraform workspaces. There may be another approach too via some 3rd party tooling.

honestly, type that exact same question into ChatGPT.. it will give you some options

Haha.. did that and it wasn’t helpful . Thought of checking internally as some of you might have wanted the same.

lol right on, yeah the feedback it gave me seamed reasonable, but there has to be a better way to manage a scenario like that for sure, tons of people have to be doing that exact same thing

I’ve also seen this popping up in my feed on LinkedIN and on Cloud Posse Repos.. this may be worth looking into (I plan on looking further into it as well): https://github.com/cloudposse/atmos

yeah, seems like a good solution… see “Use Cases” on that URL

I solved it by adding custom aws_iam_policy with proper policy defined with aws_iam_policy_document datasource and pass it to the module via node_role_policy_arns variable, not sure if this is the proper approach, though

I have it declared like so:

node_role_arn                 = [module.eks_node_group_main.eks_node_group_role_arn]

that is in my node group module code, and my eks cluster module is also in the same .tf file as well.

oh snaps, you’re talking about the autoscaler, not only the node group.. never mind

I ended up here - so is the plan DDB table the same as the internal state/locks table TF uses, in your case?

tl;dr Need to stop storing the plan as an artifact. This action looks nice. Need to provision the DDB table (if indeed is different to the locks table)

Aha, @Igor Rodionov @Dan Miller (Cloud Posse) can you get unblocked on the actions and their dependencies.

Need to stop storing the plan as an artifact. Why do you need to stop storing the planfile as an artifact? That ensures what you approve in the Pull Request is identical to what you apply upon merge to the default branch

As a workflow artifact attached to the PR, because putting in S3/DDB would allow finer grained access control.

Would rather store/fetch from S3.

As a workflow artifact attached to the PR Aha! Yes, that should be doable, but not something we’ve yet implemented. We would like that though… What we tried to do was something even more extreme, which was using artifact storage for everything, which proved challenging because how limited the artifact API is. But only storing the planfile as an artifact, and not trying to use the artifact storage to also replace dynamodb, that should be much easier.

Would rather store/fetch from S3. Oh, wait, I think we’re saying the same thing. We use S3 in our customer implementations, so that is already supported.

I thought you wanted to use artifact storage

So there must be a feature flag somewhere not set.

Already use artifact storage but would rather S3 - the above action looks like it’ll do that, right? Just not sure what schema the metadata DDB table is expecting…?

Nono, we use both dynamodb and S3.

We use dynamo to store the metadata, so we can find planfiles and invalidate planfiles across PRs.

Use use S3 because I think dynamodb has limits on blob storage. Also, it serves as a permanent record, if so desired.

We didn’t want to use S3 as a database and have to scan the bucket to find planfiles.

Note that 2 PRs could merge and affect the same components therefore we need some way to invalidate one or the other. Technically, a merge queue could alleviate some of the problems, but we don’t yet support that.

Let me find the schema for dynamo

Yeah makes sense. Where do you actually create that DDB table/whats the schema? e.g. the Terraform DDB state lock table has a partitionKey of LockID of type String

Yes, so here’s how we do it. Sicne we do everything with re-useable root modules (components) with atmos, here’s what our configuration looks like.

import:
  - catalog/s3-bucket/defaults
  - catalog/dynamodb/defaults

components:
  terraform:
    # S3 Bucket for storing Terraform Plans
    gitops/s3-bucket:
      metadata:
        component: s3-bucket
        inherits:
          - s3-bucket/defaults
      vars:
        name: gitops-plan-storage
        allow_encrypted_uploads_only: false

    # DynamoDB table used to store metadata for Terraform Plans
    gitops/dynamodb:
      metadata:
        component: dynamodb
        inherits:
          - dynamodb/defaults
      vars:
        name: gitops-plan-storage
        # These keys (case-sensitive) are required for the cloudposse/github-action-terraform-plan-storage action
        hash_key: id
        range_key: createdAt

    gitops:
      vars:
        enabled: true
        github_actions_iam_role_enabled: true
        github_actions_iam_role_attributes: ["gitops"]
        github_actions_allowed_repos:
          - "acmeOrg/infra"
        s3_bucket_component_name: gitops/s3-bucket
        dynamodb_component_name: gitops/dynamodb

The gitops component is what grants GitHub OIDC permissions to access the bucket and dyanmodb table

Oh, let me get those defaults

@Matt Calhoun where do we describe the entire shape of the dynamodb table?

Doh, I’d seen the above components but missed vars passing hash_key and range_key - thank you!!

aha, ok, sorry - also, this question has sparked a ton of issues on the backend here. Some docs weren’t updated, others missing.

No worries, thank you for checking

Think got it

OK, some progress…

Plan runs, can see object in S3

but nothing in DDB…no writes, can see a single read. What am I not grokking about what this thing does? I kinda expected to see some metadata about the plan/pr in DDB? Action completed successfully.

Yes, you should see something like that. Our west-coast team should be coming on line shortly and can advise.

^^ debug output

Hi Josh…just to be clear, do you have a PR open? It should write metadata to the DDB table whenever you push changes. And another sanity check, does your user/role in GHA have access to write to that table in DDB?

Hey Matt - thanks so much. Yes PR is open (internal GHE), chaining roles and yes it has access to DDB and S3. I was getting explicit permission denied writing to DDB before adding it - so something was trying…

Hmm, I wonder if same PR meant it didn’t try to re write on subsequent push? Let me push another change.

Committed a new change, can see S3 plan file under the new sha…but still nothing in DDB (Item count/size etc 0) …

Actually maybe the previous DDB IAM issues was on scan…which would make sense as I can see reads on the table but no writes.

Yup, confirmed previous fail was trying to scan. (since succeeded)

Bah - so sorry, problem between keyboard and computer. I’m looking in the wrong account - this is working as intended

When you’re ready, we have example workflows we can share for drift detection/remediation as well.

Working nicely - thank you!

No but that sounds very cool.

Perhaps code pilot can be trained to do it?

Would be super useful, right?

I think I tried to use copilot to do it, but it only wanted to do it in the same file.

I’m sure this would be a no-brainer for somebody who has built a proper VSCode app.

Or even just something on the command line

For this specific, ask I am 99% with the right prompt, and a shell script, you could iterate over all the files with mods because the prompt would be easy, and the chances to get it wrong are small.

https://github.com/charmbracelet/mods

In your prompt, just make sure convey to respond in HCL and to only focus on parameterizing constants. You can give it the convention for naming variables, etc.

I’m working on this extension for the monaco editor, and will port it to a vscode plugin among other things soon

Will check out mods for this… Charmbracelet

@george.m.sedky awesome to hear! Happy to be a beta tester when you’ve got a VSCode plugin.

awesome matt! will text you as soon as it’s ready

@Matt Gowie - report back if it works out, or was a flop

@Matt Gowie this is how it works now with monaco before the VS code plugin https://youtu.be/ktyXJpf36W0?si=xJaaQ5Pn1i7L0m_j

What is the issue Its trying to create…… Is there any error ur facing??

Hi @tamish60, I’ve managed to make it work, this region doesn’t seem to have the support for the kind of API Gateway that I’m trying to do, I moved a different region and it was able to work just fine

(@Stephan Helas best to use atmos)

So we’ve seen more and more similar requests to these, and I can really identify with this request as something we could/should support via the atmos vendor command.

One of our design goal in atmos is to avoid code generation / manipulation as much as possible. This ensures future compatibility with terraform.

So while we don’t support it the way terragrunt does, we support it the way terraform already supports it. :smiley:

That’s using the [_override.tf](http://_override.tf) pattern.

We like this approach because it keeps code as vanilla as possible while sticking with features native to Terraform.

https://developer.hashicorp.com/terraform/language/files/override

So, let’s say you have a [main.tf](http://main.tf) with something like this (from the terragrunt docs you linked)

module "example" {
  source = "github.com/org/modules.git//example"
  // other parameters
}

To do what you want to do in native terraform, create a file called [main_override.tf](http://main_override.tf)

module "example" {
  source = "/local/path/to/modules//example"
}

You don’t need to duplicate the rest of the definition, only the parts you want to “override”, like the source

So this will work together with the atmos vendor command.

1. create components/terraform/mycomponent
2. create the [main_override.tf](http://main_override.tf) file with the new source (for example)
3. configure vendoring via vendor.yml or component.yml
4. run atmos vendor This works because atmos vendor will not overwrite the [main_override.tf](http://main_override.tf) file since that does not exist upstream. You can use this strategy to “monkey patch” anything in terraform.

https://en.wikipedia.org/wiki/Monkey_patch#<i class="em em-~"</i>text=In%20computer%20programming%2C%20monkey%20patching,altering%20the%20original%20source%20code>.

@Stephan Helas also consider the following: if you are developing a new version of a component and don’t want to “touch” the existing version. Let’s say you already have a TF component in components/terraform/my-component and have an Atmos manifest like this

components:
  terraform:
    my-component:
      metadata:
        component: my-component  # Point to the Terraform component (code)
      vars: 
        ...

then you want to create a new (complete diff) version of the TF component (possibly with breaking changes). You place it into another folder in components/terraform, for example in components/terraform/my-component/v2 or in components/terraform/my-component-v2 (any folder names can be used, it’s up to you to organize it)

then suppose you want to test it only in the dev account. You add this manifest to the top-level dev stack, e.g. in the plat-ue2-dev stack

Wow, thx very much. i’ll try the vendor aproach.

in fact, my second question was, how to develop a newer component versions

you can have many different versions of the “same” Terraform component, and point to them in Atmos stack manifests at diff scopes (orgs, tenant, account, region)

i think i got the idea, but i need to test it though. its a lot to take in. i love the deep yaml merging approach a lot. i’ll try it later and will probably come back with new questions .)

i didn’t knew about the override feature of terraform. thats awesome - and yes - i’ll totally use that. So, as soon as i have mastered the handling of the remote-state (something terragrunt does for me) i think i can leave terragrunt behind

as soon as i have mastered the handling of the remote-state (something terragrunt does for me) Are you clear on the path forward here? As you might guess, we take the “terraform can handle that natively for you” approach by making components more intelligent using datas sources and relying less on the tooling.

That said, if that wouldn’t work for you, would like to better understand your challenges.

in fact, my second question was, how to develop a newer component versions We have a lot of “opinions” on this that diverge from established terragrunt patterns. @Andriy Knysh (Cloud Posse) alluded to it in his example.

This probably warrants a separate thread in atmos, if / when you need any guidance.

as short summary, i use terragrunt (like many others i suppose) mainly for generating backend and provider config. on top of that i use the dependency feature for example to place a ec2 instance in a generated subnet. as i don’t have to deal with remote states manually - i don’t know how it works behind the terragrunt curtain.

right now i try to understand the concept of the “context” module. after that i’ll try to create the backend config and use remote state. but i thing i will at leaat need another day for that.

Yes, all good questions. We should probably write up a guide that maps concepts and techniques common to #terragrunt and how we accomplish them in atmos

Backends, is a good one - like we didn’t like that TG generates it, when the whole point is IAC So we wrote a component for that.

@Erik Osterman (Cloud Posse)

thx again. i’ll open the next thread in atmos

The best write up on context might be by @Matt Gowie https://masterpoint.io/updates/terraform-null-label/

(you keep nerd sniping me in this thread!)

ok. thx soo much. its a real schame, that i didn’t find out about atmos sooner. could have saved me from a lot of weird things i did in terragrunt.

Atmos already generates many things like the backend config for diff clouds, and the override pattern (e.g. for providers)

https://atmos.tools/core-concepts/components/terraform-backends

the remote-state question comes up often. We did not want to make YAML a programming language, but use it just for config, so the remote state is done in the TF module and then configured in Atmos, see https://atmos.tools/core-concepts/components/remote-state

having said that, Atmos now supports Go templating with Sprig and Gomplate functions and datasources

https://atmos.tools/core-concepts/stacks/templating

which brings YAML with Go templates close to being a “programming” language . (w/o hardcoding anything in Atmos to work with functions in YAML, just embedding the existing template engines)

having said that, we might consider adding an Atmos-specific remote-state datasource to get the remote state for a component in Atmos directly (YAML + Go templates) w/o going through Terraform.

we might support both approaches (no ETA on the “remote-state” template datasource yet)

@Andriy Knysh (Cloud Posse)

how would i get access to the provisioning documentation? I’ve created a account to login.

@Stephan Helas that part of our documentation (docs.cloudposse.com/reference-architecture) is included with our commercial reference architecture.

That’s how we make money at Cloud Posse to support our open source. If you’d like more information, feel free to book a meeting with me at cloudposse.com/quiz

Aaaah i understand

in my ./modules/ses folder I have copied the following example files exactly:

context.tf
outputs.tf
variables.tf
versions.tf

My main.tf looks like this:

module "ses" {
  source  = "cloudposse/ses/aws"
  version = "0.25.0"

  domain        = var.domain
  zone_id       = ""
  verify_dkim   = false
  verify_domain = false

  context = module.this.context
}

I don’t want the route53 verification stuff set up (as we will need to set up the MX record ourselves) so I didn’t add that resource, or the vpc module. Is that where I’ve messed up?

and I also have a terraform.tf file with this content (matching how others have set up other modules in our project):

provider "aws" {
  region = var.region
}

provider "awsutils" {
  region = var.region
}

I’m really stuck on why this error is being thrown inside this ses.ses_user module. Any help at all would be greatly appreciated!

I removed the terraform.tf file and moved the providers into the main.tf file, and added in the vpc module and route53 resource to see if that was the issue, but it wasn’t.

provider "aws" {
  region = var.region
}

provider "awsutils" {
  region = var.region
}

module "vpc" {
  source  = "cloudposse/vpc/aws"
  version = "2.1.1"

  ipv4_primary_cidr_block = "<VPC_BLOCK - not sure if confidential>"

  context = module.this.context
}

resource "aws_route53_zone" "private_dns_zone" {
  name = var.domain

  vpc {
    vpc_id = module.vpc.vpc_id
  }

  tags = module.this.tags
}

module "ses" {
  source  = "cloudposse/ses/aws"
  version = "0.25.0"

  domain        = var.domain
  zone_id       = ""
  verify_dkim   = false
  verify_domain = false

  context = module.this.context
}

Really not sure what’s going wrong here. I’ve matched the examples pretty much exactly. Is it because it’s running via terragrunt or something?

Am I supposed to set up every parameter in the context.tf file by passing in variables? It looks like there’s a “name” var in there which defaults to null - that could explain why I’m getting the error maybe?

I think that might be what I was missing, passing in the name variable. I just set it to our application name and it worked! I’ll refactor things to pass these variables down from the environment so it should be good, finally got the basic ses setup up.

Now the question is how to configure all the other SES stuff.

Is it possible to configure the ‘Monitoring your email sending’ (publish to SNS topic) via the terraform script? This required a configuration set to be applied too, is that possible via the module? I can’t see any options for the inputs for this.

You should give it a simple name ses-user and it should work

The name goes from the context as an input, to being used in the module.this null label, and then gets passed to the ses module using the context input. The module.this.id is the fully qualified name composed of the namespace, environment, stage, name, and other inputs. Technically none of those are required but at least one needs a value such as name for the id to have a value so you don’t get that error message

Awesome thanks for confirming. I had it as “ses” to start with, then I’ve changed it to use the same iam user name that the backend uses for s3 as well so it has the perms to talk to both services. One thing I haven’t checked yet is whether it deletes the other perms on that user or not.

There’s also a deprecated warning when running the terraform job:

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
╷
│ Warning: Argument is deprecated
│ 
│   with module.ses.module.ses_user.module.store_write[0].aws_ssm_parameter.default["/system_user/ccp-dev/access_key_id"],
│   on .terraform/modules/ses.ses_user.store_write/main.tf line 22, in resource "aws_ssm_parameter" "default":
│   22:   overwrite       = each.value.overwrite
│ 
│ this attribute has been deprecated
│ 
│ (and one more similar warning elsewhere)
╵

looks like a deprecation in the ssm parameter write module

https://github.com/cloudposse/terraform-aws-ssm-parameter-store/issues/58

I know one of our customers tested it (accidentally) in their github actions by not pinning their terraform version. When 1.8 was released, their terraform started throwing exceptions and crashing. Pinning to the previous 1.7.x released fixed it.

extremely powerful.

You can write your own function in golang within a provider and then run the function in terraform.

In OpenTofu 1.7, where provider functions are coming as well, we have some cool functional expansions on it that allow use cases like e.g. writing custom functions in Lua files side-by-side with your .tf files.

We’ll have a livestream next week on Wednesday diving into some of the details: https://www.youtube.com/watch?v=6OXBv0MYalY

opentofu 1.7 is still in beta tho, no ?

Yep, the stable release is coming out in ~2 weeks

https://sweetops.slack.com/archives/C063TG2DYTC/p1713450876539809

Beta 1 released this morning!

writing custom functions in Lua files side-by-side with your .tf files.

https://github.com/opentofu/opentofu/pull/1491

We’re working to get the experimental Lua provider on the registry as we speak, so you’ll be able to play around with it when test driving the beta!

are there other languages being supported besides Lua?

It’s not support for any specific languages per se, we’re just enabling the creation of providers that can dynamically expose custom functions based on e.g. a file passed to the provider as a config parameter.

So like here

provider "lua" {
    lua = file("./main.lua")
}

So the community will be able to create providers for whatever languages they want.

However, the providers still have to be written in Go, and after investigating options for embedding both Python and JavaScript in a provider written in Go, it’s not looking very promising.

I’ll be doing a PoC for dynamically writing those in Go though, via https://github.com/traefik/yaegi

how about security side with functions?

do you mean how does opentofu prevent supply chain attacks with providers that integrate new languages ?

yeah

I don’t think that’s any different to normal providers and their resources though, is it?

Btw. Lua provider is in and ready to play with https://github.com/opentofu/terraform-provider-lua

awesome

thinking if that is an issue if the codes in lua will replace Terraform binaries etc

It is awesome! To be clear: This will likely only be available for OpenTofu and not for Terraform because the OpenTofu devs are taking the provider functions feature further with this update that they came up with. I haven’t see anything that says Terraform will do the same, which will be an interesting diversion!

Oh, I didn’t catch that.

I was more looking to see provider functions adding uniformity and improving interoperability, than introducing incompatbilities

This requires some enhancements to the HCL library and is currently pointing at my personal fork, with a PR into the main HCL repository: hashicorp/hcl#676. As this may take some time for the HCL Team to review, I will likely move the forked code under the OpenTofu umbrella before this is merged [done].

Got it

OpenTofu will still likely add provider functions that the providers define and open up and AFAIU that should be available across OTF + Terraform, BUT these dynamically defined functions that you can create yourselves and are project specific will be OTF only.

This is my understanding so far, so not 100% on that.