#terraform (2024-11)

terraform Discussions related to Terraform or Terraform Modules

Archive: https://archive.sweetops.com/terraform/

2024-11-06

Release notes from terraform avatar
Release notes from terraform
02:03:31 PM

v1.11.0-alpha20241106 1.11.0-alpha20241106 (November 6, 2024) EXPERIMENTS: Experiments are only enabled in alpha releases of Terraform CLI. The following features are not yet available in stable releases.

terraform test accepts a new option -junit-xml=FILENAME. If specified, and if the test configuration is valid enough to begin executing, then Terraform writes a JUnit XML test result report to the given filename, describing similar information as included in the normal test output. (<a…

Release v1.11.0-alpha20241106 · hashicorp/terraformattachment image

1.11.0-alpha20241106 (November 6, 2024) EXPERIMENTS: Experiments are only enabled in alpha releases of Terraform CLI. The following features are not yet available in stable releases.

terraform tes…

2024-11-07

setheryops avatar
setheryops

One of my devs created a security group in AWS using terraform. He has create_before_destroy set to false Now he needs to replace that SG but can not cause downtime by having the first SG deleted and the recreated. If he sets create_before_destroy to true after the SG already exists will terraform honor that and create the new SG along with rules before destroying the older SG? Im thinking it will but ive never tried it and I dont want to test in prod

loren avatar

safest option would be to make only the change to create_before_destroy, set it to true. apply that. then make the other “replacing” changes and apply.

setheryops avatar
setheryops

Hmm….Ill see what I can do with that. I think he already has some changes that went through for rules he changed. Thx though.

loren avatar

you could perhaps also set up a similar situation yourself, on a non-prod environment, and see how it goes

2024-11-11

2024-11-12

2024-11-13

Naima A avatar
Naima A

Hello everyone, I’m using terraform to deploy all my infrastructure code , from k8s cluster creation to pod deployment, we have multiple environments : develop, staging and prod but we don’t use diffrent branches for our repository, we want to automate the terraform applies, specially for develop and staging and we are wondering what are the existing open source tools that worked for you

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

What you want to look at are mono repo strategies and apply those concepts to terraform

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We also don’t use multiple branches

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We use atmos at Cloud Posse and provide our GitHub actions. There is also #atlantis. And a whole slew of other TACOS.

shayyanmalik13 avatar
shayyanmalik13

Atlantis would work well here for you with GitHub actions as Erik mentioned. We use the same setup for one of our monorepo deployments

phate avatar

Hi everyone,

I’m wanting to use the CloudPosse AWS Config module - https://github.com/cloudposse/terraform-aws-config/tree/main

I’ve read through all the documentation and looked through the complete example provided, but I’m not clear on something, hoping someone has used this before:

I want to set this up at the Org level. We have AWS Organizations setup with several accounts, 2 main ones and some smaller ones, and I want to use this module to setup a proper centralised setup like it recommends on the AWS docs using trusted access via Organisations etc, and what I’m not clear on looking at this module code is if this does this for you automatically or if you need to apply this code to all accounts, the examples provided don’t really mention anything about multi-account setups

Michal Tomaszek avatar
Michal Tomaszek

I briefly checked the component for AWS Config and it looks it allows for the configuration you’d like to set: https://github.com/cloudposse/terraform-aws-components/tree/main/modules/aws-config

Release notes from terraform avatar
Release notes from terraform
05:43:30 PM

v1.10.0-rc1 1.10.0-rc1 (November 13, 2024) NEW FEATURES:

Ephemeral resources: Ephemeral resources are read anew during each phase of Terraform evaluation, and cannot be persisted to state storage. Ephemeral resources always produce ephemeral values. Ephemeral values: Input variables and outputs can now be defined as ephemeral. Ephemeral values may only be used in certain contexts in Terraform configuration, and are not persisted to the plan or state files.

ephemeralasnull function: a function takes a…

Release v1.10.0-rc1 · hashicorp/terraformattachment image

1.10.0-rc1 (November 13, 2024) NEW FEATURES:

Ephemeral resources: Ephemeral resources are read anew during each phase of Terraform evaluation, and cannot be persisted to state storage. Ephemeral r…

2024-11-14

josh.reichardt avatar
josh.reichardt

Does anyone know of a decent tool that creates decent diagrams from code? The graph command just doesn’t cut it

shayyanmalik13 avatar
shayyanmalik13

following

sergeyro avatar
sergeyro

Is it an open source tool ?

Enrique Lopez avatar
Enrique Lopez

For backend calls diagraming I use this one:

https://sequencediagram.org/

SequenceDiagram.org - UML Sequence Diagram Online Tool

Free sequence diagram online tool. Create sequence diagrams using textual notation or draw quickly via Drag and Drop with an easy to use interface.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Michael was working on a tool: https://github.com/RoseSecurity/Terramaid

RoseSecurity/Terramaid

A utility for generating Mermaid diagrams from Terraform configurations

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

But it’s a hard problem

Michael avatar
Michael

I’m also a big fan of Rover: https://github.com/im2nguyen/rover

Like Erik said, it’s a difficult problem to solve. Transitive reduction in graph theory can only help so much when there are so many resources

im2nguyen/rover

Interactive Terraform visualization. State and configuration explorer.

2024-11-15

2024-11-17

2024-11-18

2024-11-19

Josh Simmonds avatar
Josh Simmonds

:wave: Atmos newbie here, loving the product so far and enjoying being able to consume your open source modules! They’re helping make quick work of our new environmental standup and the hiera-style hierarchy is a beautiful thing to see working with TF.

I’m currently working my way through a little bit of a side questing exercise (importing our existing resources into the foundational account, account-map, and tfstate-backendmodules) and am trying to get the account-map module to work correctly (I’m very close, having already gotten account and tfstate-backend functional against existing accounts and OU mappings for our AWS Org).

Currently, I’m running into an issue with account-map where it seems to be trying to load different atmos.yaml settings from some default than what I have specified at my top level and where I run the atmos commands from. Specifically, I’m seeing the following error when trying to run atmos terraform plan account-map --stack sre-bootstrap :

│ Error: failed to find a match for the import '/atmos/components/terraform/account-map/stacks/orgs/**/*.yaml' ('/atmos/components/terraform/account-map/stacks/orgs' + '**/*.yaml')

The plan technically succeeds, producing the desired plan output, but then atmos itself seems unhappy about the result. Looking at the pathing above, I’m running my commands from within /atmos and my atmos.yaml is located at /atmos/atmos.yaml (and I don’t define or expect any stacks to be defined within the vendored account-map module, nor do we organize based on org at our toplevel).

stacks:
  base_path: "stacks"
  included_paths:
    - "**/*"
  excluded_paths:
    - "**/_defaults.yaml"
    - "mixins/**/*"
    - "catalog/**/*"
  name_pattern: "{environment}-{stage}"

Any guidance ya’ll can provide would be tremendously helpful as to where or how to address this!

Samuel Than avatar
Samuel Than

Hey @Josh Simmonds, i’m sort of in the similar journey with you. I manage to get around this problem by following the guideline at https://atmos.tools/core-concepts/components/terraform/remote-state/#recommended-options , this is if you’re using remote state as a way of “importing” existing resources. After placing the atmos.yaml and setting up the required ENV - ATMOS_BASE_PATH, should be able to resolve the failed to find a match error

Using Remote State | atmos

Terraform natively supports the concept of remote state and there’s a very easy way to access the outputs of one Terraform component in another component. We simplify this using the remote-state module, which is stack-aware and can be used to access the remote state of a component in the same or a different Atmos stack.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Aha, so I can chime in with a little bit of contextual information here

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

You might have realized that some of our components also rely on atmos.yaml (via the cloudposse/terraform-provider-utils).

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

When Terraform calls a provider, it’s executed from a subfolder of the .terraform directory, so it cannot “auto discover” the atmos.yaml very easily.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

That’s why this environment variable must be set. However, @Andriy Knysh (Cloud Posse) I though we set it automatically, if not set. If not, I’ll ensure we add that to our shortlist.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think the envar I’m thinking about is ATMOS_CLI_CONFIG_PATH

Josh Simmonds avatar
Josh Simmonds

wave Thanks for the follow up, Erik! That’s exactly what got me around the issue, thanks to Samuel’s input. Adding

env:
  ATMOS_CLI_CONFIG_PATH: "../../.."
  ATMOS_BASE_PATH: "../../.."

to my setup here did skirt me around the nested atmos.yaml issue I pointed out earlier! I’ve been able to make much more progress today as a result, which is great, and I’m up against some different problems, which I think is largely related to the Brownfield implementation I’m working from here. More work to do on that front to get things to align to the cloudposse opinionated modules, but unblocked for now!

Josh Simmonds avatar
Josh Simmonds

Again, thanks to you and Samuel for your time and input

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Great to hear!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Related to brownfield, a big reason we’ve bit the bullet and moved all of our components to a new GitHub org is as a precursor for improving brownfield support

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Brownfield support will require making breaking changes to our components, and we need a way to support both earlier generations of components and newer ones

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Also, I just realized we’re in the #terraform channel - let’s use atmos moving forward

1
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@Erik Osterman (Cloud Posse) we set the variables automatically in geodesic

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
Configure Project Repository | atmos

Atmos supports both the monorepo and polyrepo architectures when managing the configurations for components

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Andriy Knysh (Cloud Posse) we should not assume geodesic in atmos though

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

we are not assuming it, it’s just geodesic has more info and sets the env variables. For all other cases (e.g. running locally or on CI/CD) the above doc describes a few ways to do it (either place atmos.yaml in a location known by Atmos, or use the env vars)

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

if you are using a Dockerfile, then this is what we usually do:

we place atmos.yaml in rootfs/usr/local/etc/atmos in the repo, and then in the Dockerfile we execute

COPY rootfs/ /
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

inside the container, atmos.yaml will be in /usr/local/etc/atmos (which is one of the locations known and searched by Atmos and the utils provider)

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

2024-11-20

Release notes from terraform avatar
Release notes from terraform
07:53:32 PM

v1.10.0-rc2 1.10.0-rc2 (November 20, 2024) NEW FEATURES:

Ephemeral resources: Ephemeral resources are read anew during each phase of Terraform evaluation, and cannot be persisted to state storage. Ephemeral resources always produce ephemeral values. Ephemeral values: Input variables and outputs can now be defined as ephemeral. Ephemeral values may only be used in certain contexts in Terraform configuration, and are not persisted to the plan or state files.

ephemeralasnull function: a function takes a…

Release v1.10.0-rc2 · hashicorp/terraformattachment image

1.10.0-rc2 (November 20, 2024) NEW FEATURES:

Ephemeral resources: Ephemeral resources are read anew during each phase of Terraform evaluation, and cannot be persisted to state storage. Ephemeral r…

2024-11-22

kylan11 avatar
kylan11

Is there any way I can use templates to get either a “plan” or “apply” value depending on the atmos command that’s been ran? Something like {{ .command }} (which right now outputs “tofu” for me since I’m using OpenTF) ?

Gabriela Campana (Cloud Posse) avatar
Gabriela Campana (Cloud Posse)

@Andriy Knysh (Cloud Posse)

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@kylan11 it’s not supported natively in Atmos.

can you provide more details why and how you want to use it, and we might point you at how to achieve it

kylan11 avatar
kylan11

I have a multi-account AWS org setup with IAM groups/roles already implemented (so no way for me to switch to aws-teams for the foreseeable future), and my idea is to dynamically assume either a “tf-plan-role” or “tf-apply-role” on the target account based on the command, doing something like this:

# …. terraform: providers: aws: region: “eu-west-1” assume_role: role_arn: ‘arnawsiam:role/tf-{{ .command }}-role’

That’s to allow developers to plan locally without having to publish a PR etc, but leaving enough flexibility to allows us admins to apply locally without having to manually edit the stack files. I am sure there is a way better solution out there, but given my constraints on using the current IAM groups / roles setup I haven’t thought of it yet.

Miguel Zablah avatar
Miguel Zablah

this might be a silly question but is why dose CloudPosse modules have some tags with prefix v and other do not but all release have the prefix, is this a bug?

Like this module cloudposse/terraform-null-label releases are with v prefix (v0.25.0) but tag is without (0.25.0) but then we have this module cloudposse/terraform-aws-alb where it matches and both tag and release have the prefix v (v2.1.0)

what is the prefer way?

RB avatar

That seems like a bug or an artifact of how they used to do it vs how they do it now. The good news is that if you use the registry source, it always works without the v even if the upstream tags contain the v

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Igor Rodionov

Gabriela Campana (Cloud Posse) avatar
Gabriela Campana (Cloud Posse)

@Igor Rodionov bumping this up

2024-11-25

Moritz avatar

Terraform 1.9.0 (released June 2024) added support for variable validation rules referencing other objects/variables. Is there any guidance/policy/plan to bump the required_version to enable those in cloudposse modules?

Michal Tomaszek avatar
Michal Tomaszek

Can’t speak for Cloud Posse, but due to Terraform BSL licensing I don’t think that anything higher than Terraform 1.5.7 (the last MPL-licensed version) will be supported. All versions of OpenTofu are supported.

this1
Gabriela Campana (Cloud Posse) avatar
Gabriela Campana (Cloud Posse)

@Dan Miller (Cloud Posse) pls confirm

Dan Miller (Cloud Posse) avatar
Dan Miller (Cloud Posse)

Yes Michal is right. We can’t advise anyone to use a later version of Terraform higher than 1.5.7. However OpenTofu is an excellent option. Please see this document related to choosing the best option for your own use case

https://docs.cloudposse.com/layers/project/design-decisions/decide-on-terraform-version/

Decide on Terraform Version | The Cloud Posse Reference Architecture

Choose Terraform version for compatibility and consistency

Moritz avatar

so even if opentofu implements support for it in 1.9 (it’s on the roadmap), the required_version will never be increased past 1.5.7?

opentofu currently does not have a separate tofu block and uses the same version constraint: https://opentofu.org/docs/language/settings/#specifying-a-required-opentofu-version

Does this mean cloudposse modules will never implement the improved validation?

Gabriela Campana (Cloud Posse) avatar
Gabriela Campana (Cloud Posse)

@Dan Miller (Cloud Posse)

Dan Miller (Cloud Posse) avatar
Dan Miller (Cloud Posse)

This is a better question for @Erik Osterman (Cloud Posse)

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

At this time we will not adopt breaking features in BSL terraform not supported by OpenTofu.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We will accept breaking changes from 1.5.7 provided they work in opentofu

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Breaking changes require a new major release

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Every module is tested against MPL versions of Terraform and OpenTofu.

Release notes from terraform avatar
Release notes from terraform
05:23:40 PM

v1.10.0-rc3 1.10.0-rc3 (November 25, 2024) NEW FEATURES:

Ephemeral resources: Ephemeral resources are read anew during each phase of Terraform evaluation, and cannot be persisted to state storage. Ephemeral resources always produce ephemeral values. Ephemeral values: Input variables and outputs can now be defined as ephemeral. Ephemeral values may only be used in certain contexts in Terraform configuration, and are not persisted to the plan or state files.

ephemeralasnull function: a function takes a…

Release v1.10.0-rc3 · hashicorp/terraformattachment image

1.10.0-rc3 (November 25, 2024) NEW FEATURES:

Ephemeral resources: Ephemeral resources are read anew during each phase of Terraform evaluation, and cannot be persisted to state storage. Ephemeral r…

2024-11-26

2024-11-27

Release notes from terraform avatar
Release notes from terraform
10:03:38 AM

v1.10.0 1.10.0 (November 27, 2024) NEW FEATURES:

Ephemeral resources: Ephemeral resources are read anew during each phase of Terraform evaluation, and cannot be persisted to state storage. Ephemeral resources always produce ephemeral values. Ephemeral values: Input variables and outputs can now be defined as ephemeral. Ephemeral values may only be used in certain contexts in Terraform configuration, and are not persisted to the plan or state files.

ephemeralasnull function: a function takes a value…

2
Matt Gowie avatar
Matt Gowie

Hey folks – we’re doing one of our Terraform Audits for one of our clients. As part of that audit, we run tflint + a trivy scan against their Terraform codebase to get some artifacts about how they’re doing regarding conventions and security. We’re looking for other tools that we might want to add to that list. Does anyone here have any suggestions on similar static analysis tools in the TF space that you think provide value?

omry avatar

You can check a list of them here - https://github.com/shuaibiyy/awesome-tf?tab=readme-ov-file#tools

Some come to mind from the top of my head:

  1. Checkov
  2. Kics
  3. terrascan
  4. tfsec - I think it’s part of Trivy anyway now.
Michal Tomaszek avatar
Michal Tomaszek

I use these for years now and they complement each other quite well. There’s some overlapping in functionalities, but each of them shines as well in specific scenarios.

  • terraform fmt (canonical format)
  • terraform validate (config files validation)
  • TFLint (linter, detecting errors specific to the resources, custom checks)
  • tfsec/Trivy (security vulnerabilities)
  • Checkov (scanning against industry benchmarks)

Remarks:

  • tfsec is no longer continued and it became a part of Trivy (see: https://www.prnewswire.com/news-releases/aqua-security-acquires-tfsec-301328713.html).
  • Checkov was acquired by Prisma Cloud. It’s fine to use it for local development to see where you’re at. For CI, if you want to set severity level, API key is needed. You basically need to be corporate to buy API key AFAIK.
  • I highly recommend using terraform fmt and TFLint as a part of pre-commit hooks and terraform validate and Trivy as a part of pre-push hooks. That way, commiting is a fast operation still. Pushing to remote repo takes a little longer, but the code is already scanned and ready for PR creation, especially, when you run the same toolset as a part of CI workflow.
2

2024-11-28

2024-11-29

    keyboard_arrow_up