The situation is that we have a terraform component, used by lots of stacks, but each stack has its own json configuration file for that component. These files are several hundred lines long, so it would be unwieldy to store in a variable. Currently, we store all these files inside the component, but this means if I change the file for one stack, atmos describe affected returns all stacks with that component, leading to a bunch of no-op plan and apply steps in our git automation.

I’d like to store the files elsewhere in the repo, and have each stack’s yaml configuration point to the file. When the file is changed, it should trigger a plan or apply of just that stack. Does anyone know of a good way to do this?

@Andy Wortman give me a few< I’ll show you how to do it

https://atmos.tools/cli/commands/describe/affected

oh, that is awesome!

this is how to specify deps on external files and folders in the YAML stack manifests using the settings.depends_on attribute, which is used in atmos describe affected

note that you can also have dependencies on external (but local) terraform modules in your TF components - but in this case Atmos detects that automatically, no need to specify anything in YAML stack manifests

Exactly what I was looking for. Thanks @Andriy Knysh (Cloud Posse)!

and try again

that code was taken from a larger component whicj allows provisioning one VPC flow logs bucket for many VPCs, and the example did not change it to the case where the bucket is deployed in the same stack as the VPC

(we’ll update the code and the docs in the next release)

Thanks! That lead to the following error..

│ Error: Attempt to get attribute from null value
│
│   on vpc-flow-logs.tf line 5, in resource "aws_flow_log" "default":
│    5:   log_destination      = module.vpc_flow_logs_bucket[0].outputs.vpc_flow_logs_bucket_arn
│     ├────────────────
│     │ module.vpc_flow_logs_bucket[0].outputs is null
│
│ This value is null, so it does not have any attributes.
╵
exit status 1

after running: atmos terraform deploy vpc-flow-logs-bucket -s org1-plat-ue2-prod then atmos terraform deploy vpc -s org1-plat-ue2-prod

@Dave this is another thing that we are going to add to the “Quick Start” in the next release. In fact, it’s documented here https://atmos.tools/core-concepts/components/remote-state

see https://atmos.tools/core-concepts/components/remote-state#caveats

in short, the remote-state module uses the utils provider to read Atmos components, and Terraform executes all providers from the component’s folder

atmos.yaml does not exist in the component’s folder, hence the utils provider can’t find the remote state

if you are on the host, put atmos.yaml to /usr/local/etc/atmos/atmos.yaml

Oh, I totally read that the other day, apologies for forgetting… best practices would be then to copy atmos config there any time it is updated?

or just set the ENV var ATMOS_CLI_CONFIG_PATH (this is what geodesic does automatically)

Oh, I totally did that…

best practices would be to use a Docker container and use the rootfs pattern

https://github.com/cloudposse/atmos/tree/default-atmos-yaml/examples/quick-start

https://github.com/cloudposse/atmos/blob/default-atmos-yaml/examples/quick-start/Dockerfile#L45

https://github.com/cloudposse/atmos/blob/default-atmos-yaml/examples/quick-start/rootfs/usr/local/etc/atmos/atmos.yaml

or (especailly if you are on the host), use ATMOS_CLI_CONFIG_PATH to set the path to atmos.yaml to whatever location you like

Yeah I have that set, I am using the container.

(this is an anooying feature, but that’s how Terraform works with the providers)

Still having the issue, I’m clearly missing something….

I have done the following (automatically set on my docker run command)
just set the ENV var ATMOS_CLI_CONFIG_PATH (this is what geodesic does automatically)

# Tried both of the following
 √ . [NLS] (HOST) _repotest ⨠ echo $ATMOS_CLI_CONFIG_PATH
/_repotest/

 √ . [NLS] (HOST) _repotest ⨠ echo $ATMOS_CLI_CONFIG_PATH
/_repotest/atmos.yaml

I have tried this

if you are on the host, put atmos.yaml to /usr/local/etc/atmos/atmos.yaml

ls -l /usr/local/etc/atmos/
total 4
-rwxr-xr-x 1 root root 1931 Feb  2 09:08 atmos.yaml

Does using my current atmos.yaml suffice, or is there something special about your atmos.yaml from here?

https://github.com/cloudposse/atmos/blob/default-atmos-yaml/examples/quick-start/rootfs/usr/local/etc/atmos/atmos.yaml

with

/usr/local/etc/atmos/atmos.yaml

what error do you see?

√ . [NLS] (HOST) _repotest ⨠ echo $ATMOS_CLI_CONFIG_PATH
/_repotest/atmos.yaml

╷
> │ Error: Attempt to get attribute from null value
> │
> │   on vpc-flow-logs.tf line 5, in resource "aws_flow_log" "default":
> │    5:   log_destination      = module.vpc_flow_logs_bucket[0].outputs.vpc_flow_logs_bucket_arn
> │     ├────────────────
> │     │ module.vpc_flow_logs_bucket[0].outputs is null
> │
> │ This value is null, so it does not have any attributes.
> ╵
> exit status 1

√ . [NLS] (HOST) _repotest ⨠ echo $ATMOS_CLI_CONFIG_PATH /_repotest/

╷
│ Error: Attempt to get attribute from null value
│
│   on vpc-flow-logs.tf line 5, in resource "aws_flow_log" "default":
│    5:   log_destination      = module.vpc_flow_logs_bucket[0].outputs.vpc_flow_logs_bucket_arn
│     ├────────────────
│     │ module.vpc_flow_logs_bucket[0].outputs is null
│
│ This value is null, so it does not have any attributes.
╵
exit status 1

are you using the ENV variables, or your atmos.yaml is in /usr/local/etc/atmos/atmos.yaml?

I’ve tried both, are they mutually exclusive?

if you are using the ENV vars, you need to set 2 vars:

Initial Atmos configuration can be controlled by these ENV vars:

ATMOS_CLI_CONFIG_PATH - where to find atmos.yaml. Path to a folder where the atmos.yaml CLI config file is located
ATMOS_BASE_PATH - base path to components and stacks folders

Yes read that

Those are both set.

if Atmos sees ATMOS_CLI_CONFIG_PATH, it will not try to use ``/usr/local/etc/atmos/atmos.yaml`

what’s ATMOS_BASE_PATH?

Also tried

echo $ATMOS_CLI_CONFIG_PATH
/repotest

echo $ATMOS_BASE_PATH
/repotest

Also tried

unset ATMOS_CLI_CONFIG_PATH
unset ATMOS_BASE_PATH

cp /repotest/atmos.yaml /usr/local/etc/atmos/
base_path: "/repotest" # atmos.yaml

let’s review this:

pick one of the methods

note that ATMOS_BASE_PATH must be an absolute path

this is what geodesic is doing:

• In the Dockerfile, copies the rootfs to the container so we have /usr/local/etc/atmos/atmos.yaml in there • Sets ATMOS_BASE_PATH to /localhost/...../infra - NOTE: this is absolute path

(again, sorry that this is complicated, but Terraform executes providers from the component folder, and we don’t want to place atmos.yaml in every component’s folder)

Are we sure PATHING had anything to do with it?

As soon as I disabled vpc_flow_logs_enabled: false it worked just fine.

I went through each of the OPTIONS here

For this to work for both the `atmos` CLI and the Terraform `utils` provider, we recommend doing one of the following:

- Put `atmos.yaml` at `/usr/local/etc/atmos/atmos.yaml` on local host and set the ENV var `ATMOS_BASE_PATH` to point to the absolute path of the root
  of the repo

- Put `atmos.yaml` into the home directory (`~/.atmos/atmos.yaml`) and set the ENV var `ATMOS_BASE_PATH` pointing to the absolute path of the root of
  the repo

- Put `atmos.yaml` at a location in the file system and then set the ENV var `ATMOS_CLI_CONFIG_PATH` to point to that location. The ENV var must
  point to a folder without the `atmos.yaml` file name. For example, if `atmos.yaml` is at `/atmos/config/atmos.yaml`,
  set `ATMOS_CLI_CONFIG_PATH=/atmos/config`. Then set the ENV var `ATMOS_BASE_PATH` pointing to the absolute path of the root of the repo

- When working in a Docker container, place `atmos.yaml` in the `rootfs` directory
  at [/rootfs/usr/local/etc/atmos/atmos.yaml](<https://github.com/cloudposse/atmos/blob/master/examples/quick-start/rootfs/usr/local/etc/atmos/atmos.yaml>)
  and then copy it into the container's file system in the [Dockerfile](<https://github.com/cloudposse/atmos/blob/master/examples/quick-start/Dockerfile>)
  by executing the `COPY rootfs/ /` Docker command. Then in the Dockerfile, set the ENV var `ATMOS_BASE_PATH` pointing to the absolute path of the
  root of the repo. Note that the [Atmos example](<https://github.com/cloudposse/atmos/blob/master/examples/quick-start>)
  uses [Geodesic](<https://github.com/cloudposse/geodesic>) as the base Docker image. [Geodesic](<https://github.com/cloudposse/geodesic>) sets the ENV
  var 

and same results everytime.

so it might be something else. You can DM me your setup and i’ll review

@Dr.Gao if you look at this doc https://atmos.tools/integrations/github-actions/atmos-terraform-drift-detection, you can see that the action uses this GH action https://github.com/cloudposse/github-action-atmos-terraform-select-components/blob/main/action.yml

which executes atmos describe stacks --format json, which returns all components in all stacks

https://atmos.tools/cli/commands/describe/stacks

How can I specify all components?

the action returns all components, which is what you need for drift detection

on the other hand, to detect only the affected component (affected byu the changes in a PR), you can use https://atmos.tools/integrations/github-actions/affected-stacks

and then to plan the affected components, https://atmos.tools/integrations/github-actions/atmos-terraform-plan

Thanks very much! That is very helpful!

Is Github action with Atmos production ready? @Andriy Knysh (Cloud Posse)

they are used in production

https://github.com/cloudposse?q=github-action-atmos&type=all&language=&sort=

let us know if you need help or find any issues

Thanks! Andriy!

I will reach out again if we have any issues

this is not related to Atmos since you want to use the same Terraform backend and have already configured backend.s3 section so all state will be stored in the same backend (even for the external account). You are probably using different IAM roles to access the backend and the AWS resources

for AWS resources, see an example here https://github.com/cloudposse/terraform-aws-components/blob/main/modules/aurora-mysql/providers.tf

you need to provide an IAM role in assume_role that Terraform will assume to access the external account

if you are using just provider "aws" you can always provide that role in assume_role

if you are using

module "iam_roles" {
  source  = "../account-map/modules/iam-roles"
  context = module.this.context
}

then the account-map component needs to be configured to return that IAM role for a specific context, e.g. for a diff Org, or diff tenant, or diff account - depending on how you model the external account (is it a separate Org, or a separate tenant, or just a separate account)

then in Atmos manifests, you create those configurations for the new Org/tenant/account

so you can alway model an external account as a separate Org, or tenant, or just a separate AWS accou nt in the existing Org/tenant

see https://atmos.tools/design-patterns/organizational-structure-configuration for the explanation of that design pattern

Thank you for the detailed reply, I think I understand what you mean, I’m just not sure how I would make the account-map component return the external role to assume? How would I add the external AWS account (which is part of the customers AWS Org) within my own AWS Org setup configured using atmos + cloud components?

So I would just see it as a separate external account, which I want to provision resources on using my existing atmos+cloudposse project. So adding it using the manifests and allowing it to be using the assume_role (terraform_role_arn) with an external role would perfectly fit my needs. I just don’t know how to set that up and can’t immediately find any examples either.

if you are using the account-map component, there is no examples like that. To use an existing external account as a separate stage and to use an existing terraform IAM role, you will need to modify the account-map component

the output https://github.com/cloudposse/terraform-aws-components/blob/main/modules/account-map/modules/iam-roles/outputs.tf#L1

some logic needs to be aded here https://github.com/cloudposse/terraform-aws-components/blob/main/modules/account-map/modules/iam-roles/main.tf#L43

e.g. if stage=<new_stage> return the existing Terraform role ARN

the account-map needs to be modified before you configure the new functionality with Atmos, eg. by providing it with a new input - a map of existing accounts to the terraform roles to assume

then use that input in the component to add it to the outputs

and then you can configure it with Atmos (which is just to configure that new input variable)

for eaxmple:

what you are asking is brownfield environment

see this PR https://github.com/cloudposse/terraform-aws-components/pull/945 which discuss that

also this https://github.com/cloudposse/terraform-aws-components/pull/943

Interesting, thank you!

as you can see, the PRs are under discussion (which means there is no consensus on how to do this, so you need to make changes to the components by yourself for now, using any approarch, and configure it with Atmos)

terraform init -reconfigure is used to be able to use a component in many stacks

Re-running init with an already-initialized backend will update the working directory to use the new backend settings. Either -reconfigure or -migrate-state must be supplied to update the backend configuration.

you can always set it to false in atmos.yaml

ok thanks, I’m still learning, not quite I understand why it needs to if it’s re-running in the same stack, unless it’s not detecting that it’s in the same stack and just running it always because the config var says true

-reconfigure was introduced (it’s configurable) for the cases when we use multiple Terraform backends per Org, per tenant, or per account. Then we can provision the same TF component into multiple stacks using diff TF backends for each stack

and to make it configurable (and disable for the cases like yours), it was added to atmos.yaml config

ok thx, my case is very simple right now as I’m bootstrapping. Hopefully warnings go away as I get more sophisticated infra in place.

@Dan Miller (Cloud Posse)

the waf component may not have everything that the terraform-aws-waf module has. We use components as root modules for customer implementations, so it likely only has what we’ve required for a given use case. You can likely add anything that the module supports to the component

https://github.com/cloudposse/terraform-aws-waf

rule groups are supported by the managed_rule_group_statement_rules input: https://github.com/cloudposse/terraform-aws-waf/blob/main/variables.tf#L361

@Andriy Knysh (Cloud Posse) do you have an example of using an and_statement or or_statement or not_statement?

i think those or and and statements are not supported by the module. Prs are welcome

@Dan Miller (Cloud Posse), I think the rule groups are not supported in the terraform-aws-components WAF module. The managed rule group are the ones that are managed by AWS, but not the ones we create.

So it sounds like we’ll need to customize it to do both rule groups and multi statements then. @johncblandii

https://github.com/cloudposse/terraform-aws-waf/blob/main/rules.tf#L1031

@Gabriel Tam are those not what you are describing?

https://github.com/cloudposse/terraform-aws-waf/blob/main/variables.tf#L691

https://github.com/cloudposse/terraform-aws-components/blob/main/modules/waf/variables.tf#L713

Those are the rule group referencing rules to point to an existing rule groups (arn). I couldn’t find where I can create the rule groups. And the and, or, and not statement are needed for our use cases. Those are the ones we will need to create / customize.

if you modify/update/improve it, your contribution to the WAF module would be greatly appreciated (taking into account that WAF is a complex thing, it would benefit many people)

so it seems support for this resource is what @Gabriel Tam is referring to.

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/wafv2_rule_group

@Erik Osterman (Cloud Posse) created it, I enjoyed reading it, after reading it you will want to use Atmos :)

You surely do! And you’re better equipped to do so with all the XP distilled in this page!

I know there is a native GitHub way to do it by clicking on actions, just wondering if there is another frontend

We do not have any immediate plans for a front end

Focused on adopting GitHub actions and GitHub enterprise functionality as much as possible right now

What specifically are you missing that a UI would solve? Visualizing atmos stacks is a bit broad.

I was thinking that it would be nice to have the ability to see which stacks have drifted visually

We believe we’ve solved that. We open GitHub Issues. This way drift is actionable (_it can be assigned to someone to remediate, and supports remediation from issues, when possible) and _visual. We also create issues from failures.

The problem with existing UIs, is they are not actionable. They just show you, you have a bunch of drifted stacks.

Pro tip, you can use projects with github issues.

Github Issues can be synced to jira, if not using github issues.

(note our actions support this out of the box)

If so, how do i vendor from upstream and copy in my local mixin?

@RB are you asking about how to copy a local file to the component’s folder?

@RB are you asking about how to copy a local file to the component’s folder?
Yes

Oh i see, i can use the ../../ expression?

Hmm i tried this and i got an error last time

is it working for you?

Btw have you also experimented with the new vendor manifest?

What’s nice about that is you can use yaml anchors within the file to DRY up provider copying

https://atmos.tools/core-concepts/vendoring/

Ah yes, @Andriy Knysh (Cloud Posse) shows an example there:

Option 1

- component: "context"
      # Copy a local file into a local file with a different file name
      # This `source` is relative to the current folder
      source: "components/terraform/mixins/context.tf"
      targets:
        - "components/terraform/vpc/context.tf"
        - "components/terraform/alb/context.tf"
        - "components/terraform/ecs/context.tf"
        # etc...
      # Tags can be used to vendor component that have the specific tags
      # `atmos vendor pull --tags test`
      # Refer to <https://atmos.tools/cli/commands/vendor/pull>
      tags:
        - context

Since there can only be one source, there’s no way to do the copying from a different location in each - component definition

I could see this improving, but it’s not a pattern we right now use and dissuade against using due to the impossibility of testing it with open ended mixins.

We’re adding a terratest helper to do atmos testing for components and stacks

Ah that looks better than my current way which is to use a make target

i.e.

make vendor COMP=ecr
make vendor/edit COMP=ecr

.PHONY: vendor
vendor: ## Vendor the component
	@mkdir -p components/terraform/$(COMP)
	@sed 's,ecr,$(COMP),g' ./mixins/component.yaml > components/terraform/$(COMP)/component.yaml
	@atmos vendor pull -c $(COMP)

.PHONY: vendor/edit
vendor/edit: ## Vendor the component and edit the files
	@hcledit block rm 'variable.region' -f components/terraform/$(COMP)/variables.tf -u
	@cp ./mixins/providers.tf components/terraform/$(COMP)/

On the one hand, make is nice because i understands modification times. Albeit, not in your implementation. But it’s an over optimizing for the most part to do that. Give the vendoring a shot.

In make though, I would do it like this: (functional looking pseudo code)

ALL_PROVIDERS = $(shell find . -type f -name 'providers.tf)

$(ALL_PROVIDERS): mixins/providers.tf
  cp $< $@

.PHONY : providers
providers: $(ALL_PROVIDERS)

That would have only copied it if the mixins/providers.tf is newer than the [proviers.tf](http://proviers.tf) inside of a component.

Thanks Erik. You got some make skills!

The other one is using hcledit to remove the variable.region from [variables.tf](http://variables.tf) , after vendoring, as I have moved that into the client’s [providers.tf](http://providers.tf) file

hcledit block rm 'variable.region' -f components/terraform/$(COMP)/variables.tf -u

Any chance vendoring can also include running a cli command via post_vendor key or similar ?

Since component-level generation (like terragrunt, terramate) is not something we ascribe to, it’s not yet something we can prioritize. I think we will inevitably support it, but for now recommend doing that in make, or like @Hans D does with Gotask

I can see supporting it, primarily for the reason making it easier for companies to migrate from other tools into atmos

no worries, for now I will look into the new vendoring yaml, and tie that back into a make target and i should be good to go

Oh, and also for very advanced vendoring requirements, there’s https://carvel.dev/vendir/docs/v0.39.x/vendir-spec/

(which atmos vendoring is based on)_

we can add pre and post hooks to vendoring (no ETA, but in the near future)

Hi, I’ve encountered the same issue here, not sure if this thread resolved the issue from within the ComponentVendorConfig?

I’m trying to replace the supplied providers.tf file with one that sources the iam_roles module from a private registry on vendor (to kill relative paths to the iam_roles module):

  mixins:
    - uri: ../../helpers/providers.registry.tf
      filename: providers.tf

I get the following error

Pulling the mixin '../../helpers/providers.registry.tf' for the component 'my-component' into '/localhost/path/to/components/terraform/my-component/providers.tf'

relative paths require a module with a pwd

This is on Atmos v1.60.0

make sure the relative path is correct

see https://github.com/cloudposse/atmos/blob/master/examples/tests/components/terraform/infra/vpc/component.yaml#L30

atmos vendor pull -c infra/vpc

@Matthew Reggler if that still is not working for you, please DM me with your config and I’ll take a look

also, update Atmos to the latest

This is my yaml

jobs:
  terraform:
    runs-on: ubuntu-latest
    steps:
    - name: Plan Atmos Component
      uses: cloudposse/github-action-atmos-terraform-plan@v1
      with:
        component: "github-oidc-role/cicd"
        stack: "gbl-prod"
        component-path: "component/terraform/github-oidc-role"
        terraform-plan-role: "arn:aws:iam::123456789012:role/org-gbl-prod-gha-cicd"
        terraform-state-bucket: "org-state-bucket"
        terraform-state-role: "arn:aws:iam::123456789012:role/org-gbl-prod-gha-cicd"
        aws-region: "us-east-1"

It looks like it’s failing in cloudposse/github-action-atmos-get-setting

But if I run this, it works

    runs-on: ubuntu-latest
    steps:
    - uses: hashicorp/setup-terraform@v2
  
    - name: Setup atmos
      uses: cloudposse/github-action-setup-atmos@v1
      with:
        install-wrapper: true
  
    - name: Run atmos
      id: atmos
      run: atmos terraform plan github-oidc-role/cicd --stack=gbl-prod

@Igor Rodionov please take a look

I think these moved into the config.

(the aim being to keep config out of workflows so they are more easily disributed)

cc @Matt Calhoun who has also been working on github-action-atmos-get-setting

Yes that makes sense. I was following the readme. I also tried only the component and the stack and received the same weird issue in github-action-atmos-get-setting

@Dan Miller (Cloud Posse) @Igor Rodionov I think the readme may be out of date?

(it does have the part about the config up above)

@RB have you created the config file >

?

https://github.com/cloudposse/github-action-atmos-terraform-plan?tab=readme-ov-file#config

Oh boy no i did not. I thought i could give it that info as inputs to the workflow

If i create that config, wouldn’t i just be duplicating the atmos stack yaml in the gitops config?

@Igor Rodionov we should have a friendlier error message

If i create that config, wouldn’t i just be duplicating the atmos stack yaml in the gitops config? We are moving most of the gitops config into the atmos stack config. And some of it will be moved into atmos.yaml

(this was based on feedback we received, and it makes sense in hindsight)

That way the GHA should work more out-of-the-box

The key thing we’re aiming for is that GHA workflows should not need to be edited.

So we moved over to the .github/config/atmos-gitops.yaml pattern but still having the same error

Run cloudposse/github-action-atmos-get-setting@v1
  with:
    component: s3/some-new-bucket
    stack: ue1-devops-prod-01
    settings-path: settings.github.actions_enabled
  env:
    ATMOS_CLI_CONFIG_PATH: 
  
Error: SyntaxError: Unexpected token 'F', "
Found stac"... is not valid JSON
Error: SyntaxError: Unexpected token 'F', "
Found stac"... is not valid JSON
    at JSON.parse (<anonymous>)
    at getSetting (/home/runner/work/_actions/cloudposse/github-action-atmos-get-setting/v1/src/lib/settings.ts:28:1)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at processSingleSetting (/home/runner/work/_actions/cloudposse/github-action-atmos-get-setting/v1/src/useCase/process-single-setting.ts:40:1)
    at /home/runner/work/_actions/cloudposse/github-action-atmos-get-setting/v1/src/main.ts:30:1

Our atmos-gitops.yaml file

  atmos-version: 1.45.3
  atmos-config-path: ./rootfs/usr/local/etc/atmos/
  terraform-state-bucket: bucket
  terraform-state-role: arn:aws:iam::OMMITTED:role/org-gbl-prod-cicd
  terraform-plan-role: arn:aws:iam::OMMITTED:role/org-gbl-prod-cicd
  terraform-apply-role: arn:aws:iam::OMMITTED:role/org-gbl-prod-cicd
  terraform-version: 1.6.0
  aws-region: us-east-1
  enable-infracost: false
  sort-by: .stack_slug
  group-by: .stack_slug | split("-") | [.[0], .[2]] | join("-") 

I do notice that ATMOS_CLI_CONFIG_PATH is empty in the call to get-setting, not sure if thats part of the issue

Looking at this documentation too it seems we may need to setup our atmos.yaml to output json so the settings github action can correct parse it https://atmos.tools/cli/configuration/

@Igor Rodionov can you please look at this

(also, heads up, @Brett Au - sorry to do this to you, very soon we’re moving the configuration into atmos.yml for consistency)

Hello @Brett Au What version of cloudposse/github-action-atmos-terraform-plan are you using?

Sorry getting back to this, @RB and I moved into the aws-team-roles and aws-teams modules and we didn’t want to pollute this issue with something custom we did.

I have re-tried this today with cloudposse/github-action-atmos-terraform-plan@v2

I added the configuration into atmos.yaml

integrations:
  github:
    gitops:
      terraform-version: 1.8.1
      infracost-enabled: false
      artifact-storage:
        region: us-east-1
        bucket: bucket
        #table: cptest-core-ue2-auto-gitops-plan-storage
        role: arn:aws:iam::OMITTED:role/l360-gbl-identity-cicd
      role:
        plan: arn:aws:iam::OMITTED:role/l360-gbl-identity-cicd
        apply: arn:aws:iam::OMITTED:role/l360-gbl-identity-cicd
      matrix:
        sort-by: .stack_slug
        group-by: .stack_slug | split("-") | [.[0], .[2]] | join("-")

I still have atmos-gitops.yaml as it appears uses: cloudposse/github-action-atmos-affected-stacks@v3 still requires it.

I get the following error when running the plan github action

SyntaxError: Unexpected token 'p', "path: /hom"... is not valid JSON
 at JSON.parse (<anonymous>)
    at getSetting (/home/runner/work/_actions/cloudposse/github-action-atmos-get-setting/v1/src/lib/settings.ts:28:1)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at processSingleSetting (/home/runner/work/_actions/cloudposse/github-action-atmos-get-setting/v1/src/useCase/process-single-setting.ts:40:1)
    at /home/runner/work/_actions/cloudposse/github-action-atmos-get-setting/v1/src/main.ts:30:1

I should state that my manual github action (bash) is working just fine

      - name: Run atmos
        if: ${{ steps.shouldrun.outputs.status == 'true' }}
        id: atmos
        run: |
          export ATMOS_CLI_CONFIG_PATH=${GITHUB_WORKSPACE}
          export ATMOS_BASE_PATH=${GITHUB_WORKSPACE}
          atmos terraform plan ${{ matrix.component }} --stack=${{ matrix.stack }} -no-color

So I know the environment can properly run atmos, but it appears the output from the github-action-atmos-get-setting github action is not valid json and failing

We do have a custom name pattern in atmos

  name_pattern: "{environment}-{stage}"

Sorry for the tags, but just wanted to bubble up this old issue

@Andriy Knysh (Cloud Posse) @Erik Osterman (Cloud Posse) @Igor Rodionov

I still have atmos-gitops.yaml as it appears uses: cloudposse/github-action-atmos-affected-stacks@v3 still requires it. The atmos-gitops.yaml file has been entirely eliminated from all actions and replaced with the integrations.github section

Including for affected stacks.

https://github.com/cloudposse/github-action-atmos-affected-stacks?tab=readme-ov-file#config

ACK I think I may have been on a v2 branch

hopefully an easy fix

@Brett Au, have you succeeded in solving the issue?

This error is coming up now. I added some logs to it and using the same integrations key in the yaml file.

https://github.com/cloudposse/github-action-matrix-extended/issues/9

Only notable difference is that a dynamodb lock table is not supplied.

I see the issue.

⨠ atmos describe config -f json | jq .
parse error: Invalid numeric literal at line 1, column 6

which fails because of these 2 outputs shown

⨠ atmos describe config -f json | head -2
Found ENV var ATMOS_BASE_PATH=/localhost/git/work/atmos
Found ENV var ATMOS_BASE_PATH=/localhost/git/work/atmos

This is because

# atmos.yaml
logs:
  level: Trace

I also tried setting ATMOS_LOGS_LEVEL and get the same error

⨠ export ATMOS_LOGS_LEVEL=Off
⨠ atmos describe config -f json | jq .
parse error: Invalid numeric literal at line 1, column 6

Options

1. Set log level Off in atmos.yaml and set ATMOS_LOGS_LEVEL=Trace in geodesic
2. or fix atmos to allow using an environment variable to turn off the Found ENV in the output
3. or update to strip out Found ENV from the output I think I’ll just do (1) for now but i’d prefer (2) and (3) seems like a temp fix

Im following the Organization Design Pattern: https://atmos.tools/design-patterns/organizational-structure-configuration

atmos change provider configurations to gain access to my organization’s child accounts That’s up to the terraform.

Here’s how we do it https://github.com/cloudposse/terraform-aws-components/blob/main/modules/ecr/providers.tf

TL;DR providers support variables. So use other modules or variable inputs to supply the values.

Our guiding principle is for atmos to have no cloud-provider specific requirements

But if you put the provider in the configuration and have to backout the component, wont that inhibit that delete process since terraform will be looking for the provider to do that delete?

inhibit that delete process since terraform will be looking for the provider to do that delete? It’s using state from something outside of the current component (root module), so it does not inhibit the delete.

Okay awesome. Thanks for the info Erik.

I know what you’re referring to, however, and we have encountered that when we made mistakes. But it’s not something we encounter anymore.

some of our engineers will be more familiar than me on the specific implementations.

https://github.com/cloudposse/terraform-aws-components/blob/51ec634e6c7b9b1495cbac[…]2188401b56389aa0b/modules/account-map/modules/iam-roles/main.tf

Here’s one of those examples.

E.g. cannot have labels get disabled, if we need to successfully destroy

Okay this makes sense. Thanks Erik. Was banging my head against a wall there for a while.

as Erik mentioned, it’s up to the provider config

provider "aws" {
  region = var.region

  assume_role = var.assume_role
}

assume_role can be provided from other TF components (as in our examples), or from TF vars, or from Atmos stack manifests for diff Orgs/tenants/accounts

CloudPosse does not have GCP modules

we used CP modules and created this : https://github.com/slalombuild/terraform-atmos-accelerator

Thanks!

This is very helpful!

component.yaml is used to pull all files for a component. If you pulling two components, you can place them into separate filders and use 2 diff component.yaml files

or use one component.yaml and mixins

with mixins, you can pull anything from multiple sources (but just one by one)

Thanks! I think compoent.yml with minxins is the way I am going to try

Since it is not pull multiple component, it is myltiple module in one component

This is really helpful, thanks @Andriy Knysh (Cloud Posse)

How the mixins is structured when I have multiple modules from multiple components?

Do I create multiple mixin config files for each component?

a list of mixins is part of spec at the same level as source

https://atmos.tools/core-concepts/components/vendoring

Awesome!

short answer: yes, it will work

long answer:

the label order in the context (from null-label which is used in all components) is to uniquely and consistently name the cloud (AWS) resources, so your resource names/IDs will look like {namespace}-{tenant}-{environment}-{stage}-{name} (or in whatever order you want)

on the other hand, the Atmos manifests folder structure is for humans to organize the Atmos config and make it DRY. Atmos does not care about the stacks folder structure, it’s for people to config, organize and manage

what Atmos cares about is the context variables defined in the stack manifests - that’s how Atmos finds the stack and component in the stack when you execute commands like atmos terraform plan <component> -s <stack>

https://atmos.tools/quick-start/final-notes/

as described in https://atmos.tools/design-patterns/, you can organize the stacks (Atmos manifests) in many different ways depending on your Organization/tenants/regions/accounts structure

https://atmos.tools/quick-start/provision/#stack-search-algorithm

Thanks!

I love your design !

all of that was done to be able to separately and independently configure 3 diff things: 1) cloud resource names; 2) Atmos manifests folder structure (for people); 3) Atmos stack names (e.g. plat-ue2-prod)

You make a fair point but from what I’ve seen, building your own IaC automation with GitHub Actions can get messy. First of all lack of standardization (which creates silos and bottlenecks across projects), too much reliance on individual knowledge (so when someone leaves, there’re always big gaps) and as things get more complex, it demands more and more resources, limiting what you can build on top of it. Also to mention that maintaining homegrown solutions in-house is a ton of work

we have Atmos handling different Terraform environments for many companies, starting with a simple case with one Org and just a few accounts, to multi-Org, multi-tenant, multi-OU, multi-account with hundreds of components deployed to thousands of stacks

we also have Atmos working with Spacelift (handling tens of thousands of resources and thousands of stacks in some cases across multiple Orgs/OUs/regions/accounts), and with GitHub Actions (we can give you a demo on how to use Atmos with GHA) (discussion on GHA vs Spacelift vs other CI/CD tools is a completely diff topic, all of them have their own pros and cons, including cost, usability, user experience, access control, audit, messiness :) , etc.)

here are some docs describing the core concepts of Atmos:

https://atmos.tools/

https://atmos.tools/quick-start/

https://atmos.tools/reference/terraform-limitations

https://atmos.tools/design-patterns/

https://atmos.tools/design-patterns/organizational-structure-configuration

https://atmos.tools/integrations/spacelift

https://atmos.tools/integrations/github-actions/

https://atmos.tools/cli

https://atmos.tools/cli/commands/workflow

https://atmos.tools/core-concepts/vendoring/

https://atmos.tools/reference/schemas

https://atmos.tools/core-concepts/components/validation

let us know if you need more explanation or help (it’s difficult to answer/describe everything in one go)

this is a real example of Spacelift stacks using Atmos:

there are 1274 Spacelift stacks configured across many Orgs, each having many OUs with multiple accounts, deployed into many diff regions

(Definition: a Spacelift stack is an Atmos component provisioned into an Atmos stack, for example a vpc component can be provisioned multiple times into different Org/OU/account/region stacks, keeping the entire config reusable and DRY using the concepts like imports and inheritance:

https://atmos.tools/core-concepts/stacks/imports

https://atmos.tools/core-concepts/components/inheritance

https://atmos.tools/core-concepts/components/component-oriented-programming

(also take a look at why the tool is called Atmos https://atmos.tools/faq)

Thanks for sharing! I took some time to read through the resources and decided to stick to Terramate. In the end, I don’t see any significant benefits why I should be using Atmos over Terramate.

As said, we don’t want to use Spacelift because we don’t need any of the features Spacelift offers.

Here’s a bit of feedback:

• Atmos feels really cumbersome to get started with

• The orchestration and order of execution in Terramate helps us better to manage environments and split those up into stacks

• Using yaml for configuration sounds like a step back for us. We like the ability to manage all IaC related config with HCL, otherwise we wouldn’t use Terraform in the first place

• We need code generation and like the approach quite a bit. We also don’t see any issues with tests anyways, thanks for helping us out here!

ok, no problem

regarding “Using yaml for configuration sounds like a step back for us”

Note that in Atmos, YAML is used for configuration of stacks for diff environments. You still use plain Terraform for the components (which can be used with Atmos or without). There is a clear separation of concerns: code (terraform root modules) and configuration for diff environments (Atmos manifests).

Also, YAML is everywhere now (kubernetes, kustomize, helm, helmfile, etc.), it looks like it’s the modern way to define configurations (regardless of whether you like or hate YAML )

in the end, there is no one perfect tool for everything. Terramate has its advantages (the code generation, change detection and testing are cool, also using plain HCL), as well as Atmos (these are different approaches to solve similar problems in diff ways)

We need code generation and like the approach quite a bit. @Peter Dinsmore Can you help me understand how you leverage code generation?

Using yaml for configuration sounds like a step back for us. We like the ability to manage all IaC related config with HCL, otherwise we wouldn't use Terraform in the first place

Kubernetes uses yaml and is moving forward….

@jose.amengual for Kubernetes yaml makes sense too. For Terraform, I don’t see why I should use a different configuration language if HCL is exactly built for that? E.g. in Terramate, I can describe the purpose (metadata) and orchestration behavior of a stack as HCL in a stack configuration stack.tm.hcl.

Also, the code generation is configured with HCL, which allows the use of Terraform functions inside the code generation. I don’t think YAML would be suitable for that.

@Erik Osterman (Cloud Posse) all sorts of things, to mention a few:

• We generate native backend and provider configuration in stacks interpolating stack metadata. It’s super powerful.

• We use Terramate modules for generating templates based on e.g. provider version used (we render different attributes in resource configuration based on if we use the stable or beta google provider)

• We generate Kubernetes manifests using Terraform outputs

Anyways, there’s a ton of different tools in the market. Just because we favor a different approach doesn’t mean that Atmos isn’t a great tool! Any contribution to the ecosystem is appreciated.

@Peter Dinsmore I you shared all this. It’s easy to be surrounded by a lot of people who sing praises. The opportunity for improvement lies with constructive criticism.

@Erik Osterman (Cloud Posse) we should catch up at some point

Are you looking for workflows?

https://atmos.tools/core-concepts/workflows/

I am unaware of a atmos flag that just applies or plans all components defined for stack. However, atmos workflow allows you to synchronous run applies and plans for one or more components.

@pv please help us to understand what you mean by “apply all stacks in a pipeline”. You probably should not plan/apply all stacks at once (there could be thousands of them), but check what components/stacks have changed and plan/apply only those

see this for example https://atmos.tools/integrations/github-actions/affected-stacks

Yes the workflow is the second best option of just adding each plan/apply command there but yes apply all stacks in a mono repo is better phrased

Also what is the downside to running them all if there is only a change to one stack? Wouldn’t all the terraform show as no changes other than the new stack that you added?

the downside is it will take time and consume resources

Does it consume more resources than running regular terraform? Or is it running more in the background that consumes more resources?

i mean if you have hundreds or thousands of stacks, why trigger all of them if only one or a few are changed and should be planned/applied. Triggering all ofnthem will take a lot of time and probably cost money

(Atmos calls regular terraform for each plan/apply, there is no difference here, it does not do anything in the background)

I do not have hundreds of thousands of stacks. But none of this really answers my initial question so I’m assuming that means there is no way to run all stacks unless I create a workflow that includes an apply of each individual stack?

there is no option in Atmos to trigger everything (e.g. atmos terraform apply -all) (was not implemented, at least yet, for the reason that people usually have hundreds or thousands of stacks, and triggering all of them would be a waste of resources). Help us understand what exactly you want to do in the pipeline, and we would be able to offer a solution

Atmos workflow is one of the solutions

but if you have a small static set of stacks and you know all of them, you can just execute atmos terraform apply ... sequentially in a script (or in a workflow)

the point is, it’s usually not feasible/practical to list all stacks in a workflow or in a script since there could be too many of them

another solution:

the atmos list stacks and atmos list components commands you can find here and add them to your atmos.yaml:

https://atmos.tools/core-concepts/custom-commands/#list-atmos-stacks-and-components

this solution is dynamic (you don’t have to know and hardcode all stacks and components in advance)

@pv I think maybe what you want is to apply only the affected stacks in the Pull Request?

(that solution is mentioned above)

@pv let us know if any of the above would help you to implement what you want (and let us know if you need help)

But none of this really answers my initial question so I’m assuming that means there is no way to run all stacks unless I create a workflow that includes an apply of each individual stack? Yes, the gist of it is we have not implemented plan-all and apply-all workflows because they have multiple problems. • Plan all never works if your root modules have any interdependencies. At best it gives you a false-sense of what will happen. At worst, it just errors. • Apply-all is practical for cold-starts, but should never be used after that. And since it’s for a cold-start, there are often other considerations. Therefore atmos workflows have been how we address it. • From a CI/CD perspective, neither plan-all nor apply-all should ever be used.

I can go into more details any anyone of these. For example, there are alternative considerations for how to address apply-all in a safe, automated way in a CI/CD context, but that’s something solved in CI/CD and not atmos.

@Andriy Knysh (Cloud Posse) I think what you sent me is probably the best for what I am thinking of. I think what I prefer about straight up terraform in the past is just having my pipeline plan and apply and using separate repos for Landing Zones, and Products infra. This requires you to add extra steps and extra potential break points for the workflow but I think this more dynamic approach may help with that.

@Erik Osterman (Cloud Posse) I get those concern with how Atmos is setup but with traditional TF, when you run plan and apply, it looks over just any changes in your terraform path on the repo and only applies what is reflected as new, changed, or deleted compared to the state. Have you ever done a time or cost analysis showing how much money you save doing the atmos approach over traditional terraform? Also with the first point, isn’t that reason “depends_on” exists? Sure you don’t see all the values until apply but its faster than running one dependency and then the next one until all dependencies are accounted for

ut with traditional TF, when you run plan and apply, it looks over just any changes in your terraform path on the repo and only applies what is reflected as new, changed, or deleted compared to the state. We get that with our GitHub Actions.

1. Describe Affected
2. Run terraform plan on each affected stack
3. Apply each change with GitHub Actions. To be clear, this has the outcome you want. It’s just not implemented as a “plan-all” or “apply-all”. It’s implemented using GitHub Action matrixes.

Have you ever done a time or cost analysis showing how much money you save doing the atmos approach over traditional terraform @mike186 has some interesting numbers he likes to share about the minutes of terraform they run per month. In their case, it’s a combination of Atmos+Spacelift, but the same would be true of Atmos+GitHub Actions.

https://sweetops.slack.com/archives/C031919U8A0/p1705551267070989?thread_ts=1705353125.508609

Also with the first point, isn’t that reason “depends_on” exists? So atmos is designed to work with any number of systems, including spacelift. Not every underlying system can implement all the capabilities of the configuration. In this case depends_on is currently utilized by our Spacelift implementation. We plan on adding GHA support for this soon, but cannot commit to when.

Sure you don’t see all the values until apply but its faster than running one dependency and then the next one until all dependencies are accounted for Agree, so what we really want it to trigger downstream dependencies when upstream components are affected. This is supported today with Spacelift and Atmos.

regarding “Have you ever done a time or cost analysis showing how much money you save doing the atmos approach over traditional terraform?”. To be clear, Atmos is configuration on top on terraform root modules, in the end Atmos generates the varfiles/backend and executes plain Terraform (so you’d have the same number of runs using just plain terraform in Spacelift or GHA). But this is not about Atmos vs plain terraform, it’s about architecting your Terraform modules and splitting them into smaller parts to reduce complexity and plan/apply time and blast radius. Once you do it correctly in terraform, you can configure the root-modules with Atmos to be deployed into various environments (OUs, regions, accounts). And once you correctly design your Terraform root modules, you will save time and money when planning/applying them

@pv depending on what you decide to do, but if you still want atmos terraform apply-all (and atmos terraform plan-all) , I can impelement those as Atmos custom commands and put in the docs (so you could just copy them into your atmos.yaml) (but as mentioned, the best way forward is to use the GHA to execute atmos describe affected to trigger just the affected stacks, then atmos describe dependents to trigger all the dependencies )

@Andriy Knysh (Cloud Posse) No need to create a custom command if it is against practice. I will start with the documentation you sent me and should be good to work with that

I think what I prefer about straight up terraform in the past is just having my pipeline plan and apply Aha, I think I misunderstood what you meant initially. Do you mean having larger root modules, that have, for example, the entire landing zone defined?

“Agree, so what we really want it to trigger downstream dependencies when upstream components are affected. This is supported today with Spacelift and Atmos”

@Erik Osterman (Cloud Posse) but how is that more cost effective if you have to pay to use spacelift?

And yes larger root modules that are dynamic in nature. I have a Landing Zone that I like to maintain and modify based on the needs of the company. But I know atmos is intended to be more modular so I am pivoting from what I would normally do

That’s a larger calculus that will depend on what your needs are. Spacelift is clearly an enterprise solution for enterprise challenges. That’s why we created the GitHub Actions which are entirely self-hosted and have no SaaS option or tie-in.

The the larger root modules are definitely convenient from a developer perspective. Terraform handles the DAG. The problem is they do not scale as the complexity grows. This may not be a concern, then great - you can use them. It’s our experience, working in enterprise contexts, that these root modules grow and grow. The time to plan takes longer and longer, and they are more susceptible to transient errors. Additionally, the more that goes into a root module, the less reusable it is across organizations. Since Cloud Posse is primarily concerned with how to make infrastructure re-usable, atmos is optimized for this use-case.

The larger the root module, the harder it is to separate concerns, the harder it is to restrict what can change and when.

Every change risks changing everything everytime it’s plan/applied, which means a huge blast radius.

So the solution is to break it into smaller root modules, by lifecycle. But then the problem is as you say, the complexity is offloaded to the tooling that calls terraform.

The first recommendation is to reduce the coupling between the layers, when possible, reducing how often those dependencies are triggered. Then implement CD to roll out the changes. So terraform is responsible for provisioning foundations, and CD is responsible for how to orchestrate those changes.

@pv please take a look at https://atmos.tools/reference/terraform-limitations

@pv also want to invite you to our weekly office hours. https://cloudposse.com/office-hours (we’ve run them for ~4-5 years and never missed a week)

Thanks @Erik Osterman (Cloud Posse) and @Andriy Knysh (Cloud Posse). I’ll review the docs you sent and look into the office hours. Appreciate all the attentive support

Anytime! We spend a lot of time thinking about these things, and really need to write up more documentation on “Well-Architected Terraform (according to Cloud Posse)” to make it easier to understand why we do the things we do. Especially when they go against a lot of norms, that we no longer ascribe to.

hey sorry to revive an old thread a little bit but what about atmos terraform validate --all Just thinkin out loud… i’ll probably go the custom command path or something

you know what… nevermind…bad idea… custom commands will work in the pinch i’ve put myself in…

We do have plans to add support for commands that can be applied to a graph. But no eta yet.

But for now, a custom command will get you something close to that fastest

@Hans D is this what you were also attempting? (per our other DM)

atmos terraform validate --all

custom go binary … validate vpc connectivity when attaching vpc to transit gateway It sounds like the implementation should be flipped around. Terraform should be using your go as a provider. Since it’s already in Go, that lift shouldn’t be too bad.

However, something like what you want to do should be possible using atmos custom commands.

You can create a custom command that first calls your command, then calls terraform.

Custom commands have access to the config and accept standard command line parameters.

https://atmos.tools/core-concepts/custom-commands/

then you could simply run:

Many tools instead implement things like before/after hooks. We may implement it at some point, but it’s never been something we’ve required at Cloud Posse, and we do a lot of Terraform. Granted, for these types of situations, our route is to go with a Terraform provider.

So that’s why we’ve written our utils and awsutils providers, anytime we need an escape hatch. This is inline with how things should work in Terraform. You could say, our opinionated Best Practice.

As an aside, this sounds like a cool data source if you created one!

validate vpc connectivity when attaching vpc to transit gateway

This is great , thanks for validating this . I created it as a standalone go binary that leverages AWS api’s to test the vpc connectivity . I will need to figure out how to make it a tf provider . I am fairly new to the world of terraform . I would have to look into it . Not sure if it’s too much of an ask , have you come across any examples of making a go binary as a provider? I am guessing I need to leverage tf sdk?

Are there any non atmos custom command practical usecases that people take advantage of the atmos command feature? Or is it specially for atmos commands?

Well, I can!

We faced this same issue.

https://github.com/cloudposse/terraform-provider-awsutils

We took the HashiCorp AWS provider, stripped everything out, and added in what we needed that was custom.

And, since you mention you’re new to Terraform, there’s also the local-exec provider, which is the escape hatch when you don’t have time to write a provider.

We took the HashiCorp AWS provider, stripped everything out, and added in what we needed that was custom. This is exactly what I was thinking. Great .

I will try it out, thanks much for your help

we deploy website on release

maybe we can deploy it on merging to main

yea, I think we should change that.

Let’s deploy website always on merge to main

Without a release.

I updated it in my current PR to do that.

cc @Andriy Knysh (Cloud Posse)

thanks (I did the same in your PR :)

oh lol

https://github.com/cloudposse/atmos/pull/544

btw, @RB can you share the command I should add to the install page for nixos?

https://pr-544.atmos-docs.ue2.dev.plat.cloudposse.org/quick-start/install-atmos

or maybe @Jeremy White (Cloud Posse) if you’re around

https://pr-544.atmos-docs.ue2.dev.plat.cloudposse.org/quick-start/install-atmos/

found it.

nix-env -iA nixpkgs.atmos

@Andriy Knysh (Cloud Posse) good for re

approved

Thanks for the quick turnaround ! This should make the atmos cli releases easier to follow.

Yep, sorry about that!

@Matt Gowie @kevcube

can we do the same for awsutils

yes, @Erik Osterman (Cloud Posse) please put the PGP keys in 1pass

It should be the same

When running locally, I imagine it should be using the human -<role> suffixed iam roles in delegated accounts

But for some reason, it’s thinking that my laptop is a terraform/spacelift/cicd user and trying to assume terraform instead. Is this expected? Should human users assume the -terraform role or should it assume the same -<role> that it originally assumes.

i.e. if primary role is identity-admin then the role for ue1-dev should be gbl-identity-dev-admin role, instead of gbl-identity-dev-terraform role, right?

@Jeremy G (Cloud Posse) you implemented terraform dynamic roles for components, can you explain how to use it?

@RB without using the dynamic roles that Jeremy implemented, yes it’s expected that all components assume the terraform role

regardless of the initial role

Note that I deployed aws-teams and aws-team-roles using the readme yaml. I changed terraform to cicd in case I used tofu instead of terraform.

• https://github.com/cloudposse/terraform-aws-components/tree/main/modules/aws-teams

• https://github.com/cloudposse/terraform-aws-components/tree/main/modules/aws-team-roles

without using the dynamic roles that Jeremy implemented, yes it’s expected that all components assume the terraform role
regardless of the initial role Oh interesting!

But then wouldn’t all AWS SSO roles have the same IAM permissions if they were all able to assume to -terraform roles in all child accounts?

if you look at all components, the provider will use the terraform role by default https://github.com/cloudposse/terraform-aws-components/blob/main/modules/alb/providers.tf

all of this is configured in account-map/modules/iam-roles

Jeremy will explain how to use/switch to the dynamic roles (meaning that instead of the TF role, account-map/modules/iam-roles should return the role you have already assumed when executing TF commands)

I know we have https://github.com/cloudposse/terraform-aws-components/blob/main/modules/account-map/variables.tf#L100, but Jeremy has more context and knowledge

Thanks Andriy. That’s very helpful.

A separate issue I noticed is that I wanted to use cicd for the aws-team-roles and it looks like I have to use terraform because its currently hard coded in account-map

https://github.com/cloudposse/terraform-aws-components/blob/f32372b6a55797bdead5676bccf75ffc448e2687/modules/account-map/main.tf#L65-L90

The current version of account-map optionally supports having a “terraform” role and a “planner” role in each account. The former allows making all changes in the target account, while the latter is meant to provider read-only access. Roles which are allowed to assume the “terraform” role do so. Roles which cannot assume the “terraform” role but can assume the “plan” role assume the “plan” role. All other roles attempt to run Terraform using their existing role, without assuming a new role. You must have this feature enabled; it is disabled by support. Details are documented here .

The role names use to allow plan or apply access are configured in terraform_role_name_map and default to “planner” and “terraform”, but can be set to whatever you want.

A key principle remains that there is a uniform set of roles in all the accounts other than root and identity, and access is controlled by allowing (or not) access to those roles in those accounts. So if you want your cicd role to be able to run terraform apply, it should be allowed to assume the terraform (or “apply”) role in that account.

Note that this feature also requires support from tfstate-backend, which should be documented in the same document linked above regarding dynamic terraform roles.

Thanks Jeremy for taking the time to write your response.

I cannot access the cloudposse docs link above. Is it behind a paywall? It says my account needs to be approved

i do see the reference to the role map. I didn’t realize there was an apply and plan role.

It does seem like the terraform role, in spite of the role map, is hard coded in account-map, but that’s an easy fix. For now, we’ll create the cicd role as an aws-team in identity and use the terraform role as aws-team-roles per account.

Andriy also mentioned the terraform_dynamic_role_enabled toggle that we can also optionally flip if we want to go that route. I think i like the idea that only admin can use the apply role and everyone else has to use the planner role provided there are few admins and everyone else goes through the cicd.

@RB wrote:
It does seem like the terraform role, in spite of the role map, is hard coded in account-map, but that’s an easy fix. For now, we’ll create the cicd role as an aws-team in identity and use the terraform role as aws-team-roles per account. I’m not sure what you are talking about. The concept of a role used by default for people running Terraform is hard coded into our components, and referred to as the Terraform role, but the name of that role is configurable, as I pointed out previously.

Hi Jeremy. Sorry, I was referring to this code. I cannot change this string easily. Perhaps it needs to be exposed as an input? or maybe I’m using it incorrectly?

https://github.com/cloudposse/terraform-aws-components/blob/f32372b6a55797bdead5676bccf75ffc448e2687/modules/account-map/main.tf#L65-L90

That code supports the legacy terraform_roles output for people not using dynamic Terraform roles. The newer, preferred, Dynamic Terraform Roles ignore that output, and use terraform_role_name_map and terraform_access_map instead.

The existing providers.tf is not only an example of how to retrieve the role to assume, it is sufficient for nearly all our components to use without modification.

Thank you very much!

Could the corp (shared-services) account be a suitable alternative? What do you folks think?

(This is for a brownfield project so some of the accounts in this case already exist prior to starting)

We’re looking to change our strategy on this in v2 refarch

Plan is to deploy one in each account, in a hierarchical fashion off of root.

Now, in your case, you don’t have access to root, you’d need to designate some other account.

I don’t personally know the scope of this change. Others on the team might.

Could be corp or auto or artifacts maybe using some of our terms.

will impact some of the bootstrapping, as you need the new account to be available before you can move over to s3/dynamodb. But doing some split of priviliged, less priviliged sounds sane

personally would not mix it with one of the “core” “core” accounts, but a dedicated one if you want to have a central state.

Go templates are used just in imports https://atmos.tools/core-concepts/stacks/imports

it’s not supported everywhere in Atmos manifests… fro the reason that Go templates need context variables, and in general it’s not possible to provide the context variables in advance b/c it could be a circular dependency - to get all the variables, we need to import everything and process it, so we don’t know all the vars yet

Ok I so I can use templating or variable substitution in the vars: section and the import section in the stack file but not in other palces

vars: environment: {{ .environment }} stage: “{{ .stage }}”

and in the stack file

import: #- path: "catalog/dvsa" - path: "mixins/project" context: tenant: "dvsa" stage: "{{ .stage }}" app_region: "dev" environment: "dev01" namespace: "mts"

you can do it only in imports

meaning if you import a file that has workspace_key_prefix: "infra-{{ .tenant }}-{{ .namespace }}-{{ .environment }}-{{ .stage }}-init" and provide a context for it, it will work

Yep I get that. Thanks

It just how I structure the code

for the very first import with templates, you have to provide a context with all the vars defined

to make it work the way I want to

yep thats what I am doing

then you can use https://atmos.tools/core-concepts/stacks/imports/#hierarchical-imports-with-context to propagate the context to all the child imports

if you have specific questions, you can always DM me your code and I’ll try to help you

yep I am doing the same I create a set of files to import and pass in the context at the top level and this then get propagated down as it the has an import to import other files

also, let’s review your question. Why do you need workspace_key_prefix with templates?

usually, we have Terrafrom workspace that includes namespace, tenant, environment and stage

There is a naming convention to follow and was just trying to bake that in

the each component has workspace_key_prefix which is usually just the component name (added by Atmos automatically)

There is a naming convention to follow and was just trying to bake that in

ok

Ok I think I have the gist of it and a way forward thanks for the quick response

I should clarify: This is both a problem with workspace names AND with spacelift Stacks I believe. That ends up being our blocker here.

The spacelift bits are quite easy te replace (my experience), the more concerning bit is the terraform state including the workspace bit… (which is referenced in the spacelift config)

it’s possible to rename all Atmos components (change names to reflect regions, or prefix/suffix with something) and still keep the same Terraform workspace_key_prefix (so Tf will not attempt to destroy it). This means that the existing resources will be unde the old ``workspace_key_prefix , the new ones will be at diff workspace_key_prefix)`

for example, let’s say you had

    components:
      terraform:
        vpc:

deployed in us-east-2

now you want to change the Atmos component name for it and add another Atmos component for the other region

@Matt Gowie not sure if this would help you, let me know

(spacelift stacks will be recreated, but that should be ok)

and to keep the Spacelift stack names the same, there is another trick

    components:
      terraform:
        vpc/ue2:
          vars: {}
          settings:
            spacelift:
              workspace_enabled: true
              # `stack_name_pattern` overrides Spacelift stack names
              # Supported tokens: {namespace}, {tenant}, {environment}, {stage}, {component}
              stack_name_pattern: "{stage}-vpc"

Ah both of those help to know about @Andriy Knysh (Cloud Posse) – Thanks! I’ll look into what we would want to do here considering…

yes, some issues with the auto releaser, it did it for some reason (w/o having a patch label). Prob it was confused by the three commits

given the other why-is-this-a-patch that had afaik only a single commit, it might need some further investigation if this keeps popping up

yes, we had similar issues in other repos, your help is appreciated

and given that patch releases in our env only get picked up by renovate bot once a week it made it more stand out for me (given the release announcement where I missed the renovate pr becoming available)… not a biggy, just triggered me.

It’s a misunderstanding on how release drafter works. The first PR that merges drafts an unpublished release. That might have been a patch. The next PR that merges updates the draft release. Before publishing, the person who clicks publish should review and make sure it makes sense. For example the version. Unlike our terraform modules, we do not automatically publish the releases for atmos.

This is nice because we can bundle 3-4 PRs into one release and get release notes for all of them.

So I think it was our mistake on not reviewing the release before publishing.

it was my mistake. Before we always released it manually, now the autoreleaser tries to help creating Drafts releases, which in this case should have been discarded (the fact that it created a patch release is still a bug)

Discarded or edited? I am on my phone, so maybe not seeing the obvious.

since it was a patch release, discarded. We need to review this

the autoreleaser was confused by the patch label on the first PR in the release https://github.com/cloudposse/atmos/pull/544

It’s not confused though. It’s operating as designed.

The key is it’s a draft. It’s not finalized.

That means subsequent PRs “merge” into that release.

But we can change the release numbering before publishing

ok, should we release 1.65.0 manually (for Atmos aliases)?

This is a minor hiccup in the grander scheme of things.

It will get resolved in the next release. Let’s just pay attention to the release number before manually clicking publish on the draft release.

It doesn’t look like our GHAs are being leveraged for Atmos

Keep in mind, CI/CD with Terraform is non trivial. We’ve spent considerable time developing these actions, so it’s hard to answer what’s wrong with the simple case of just calling Atmos via a run statement.

Also, I see explicit calling of stacks. We developed a GitHub Action to describe affected stacks in the PR. Using this action, you can use a matrix to plan all affected stacks.

it wants to destroy the previous resource I created because ot os (not in the configuration): This sounds like a problem with the backend configuration.

By convention, we usually rely on Terrraform Workspaces. If the backend isn’t configured to use workspaces, then you would get that effect of overriding the state.

Ahh nvm my TF state got screwed for above, was way I was doing it the atmos command I messed it up pointing to path rather than stack name as appeared in atmos. I think we are all good

Ok! great to hear

Followup: The issue was we had changed from terraform apply to terraform deploy. When switching the pipeline between those commands, you run into that issue. We destroyed the resource and ran them both as a deploy to resolve the issue

thanks @Andrew Ochsner, will review it (thanks for testing it on Azure, any help on Azure and GCP is greatly appreciated)

if there’s a more idiomatic way of doing that in golang, lemme know… not my native programming language

https://github.com/cloudposse/atmos/releases/tag/v1.64.3

thank you @Andrew Ochsner for your contribution

@jose.amengual you are familiar with Atlantis (and Atmos), can you please take a look at ^

are you pushing the files after you generate them or after the pr is created ?

I am pushing file after i generate them

and you see that push on your PR and the variable file too?

Yes, I see

even on Atlantis server i checked the repo atlantis cloned has varfile exits

can you post the full command that Atlantis is running?

Sure

and maybe your github action code too

I am using Gitlab

ok, no problem

the command and workflow config

I have used below command to generate varfile and then i pushed that file into my gitlab repo

atmos terraform generate varfiles --file-template={component-path}/varfiles/{namespace}-{environment}-{component}.tfvars.json

I have generated atlantis file that i getting parsed too

version: 3
automerge: true
delete_source_branch_on_merge: true
parallel_plan: true
parallel_apply: true
allowed_regexp_prefixes:
  - dev/
  - staging/
  - prod/
projects:
  - name: test-uw1-root-tfstate-backend
    workspace: uw1-root
    workflow: workflow-1
    dir: /components/terraform/tfstate-backend
    terraform_version: v1.2
    delete_source_branch_on_merge: true
    autoplan:
      enabled: true
      when_modified:
        - "**/*.tf"
        - varfiles/$PROJECT_NAME.tfvars.json
    plan_requirements: []
    apply_requirements:
      - approved
workflows:
  workflow-1:
    apply:
      steps:
        - run: terraform apply $PLANFILE
    plan:
      steps:
        - run: terraform init -input=false
        - run: terraform workspace select $WORKSPACE || terraform workspace new $WORKSPACE
        - run: terraform plan -input=false -refresh -out $PLANFILE -var-file varfiles/$PROJECT_NAME.tfvars.json

Atlantis command running from gitlab

running "terraform plan -input=false -refresh -out $PLANFILE -var-file varfiles/$PROJECT_NAME.tfvars.json" in "/home/atlantis/.atlantis/repos/amit/devops/atmos-example/10/uw1-root/components/terraform/tfstate-backend": exit status 1: running "terraform plan -input=false -refresh -out $PLANFILE -var-file varfiles/$PROJECT_NAME.tfvars.json" in "/home/atlantis/.atlantis/repos/amit/devops/atmos-example/10/uw1-root/components/terraform/tfstate-backend": 

Error: Failed to read variables file

the problem is I think the $PROJECT_NAME is not matching the variable name

if you search in the atlantis.yaml , one project name, do you have a corresponding varfiles/$PROJECTNAME.tfvar.json?

But if you look into the screenshot i shared it is showing the correct tfvars file after the error

Yes, i have varfiles/test-uw1-root-tfstate-backend.tfvar.json file

you looked inside this dir? /home/atlantis/.atlantis/repos/amit/devops/atmos-example/10/uw1-root/components/terraform/tfstate-backend

Yes

and the atlantis command you run?

ok, that will plan all projects

can you try with one specific one?

with -p project-name

I have only one project

can you show the generated atlantis.yaml?

I just pinned it

I’m checking and comparing against my setup

seems to be the same

I don’t know why it is not taking varfile from varifile folder something seems to off

mmmm what is this : uw1-root?

ahhh that is your workspace

It’s my atmos stack name and workspace

so on the atlantis server you have /home/atlantis/.atlantis/repos/amit/devops/atmos-example/10/uw1-root/components/terraform/tfstate-backend/varfiles/test-uw1-root-tfstate-backend.tfvars.json

correct?

Let me confirm

Got it

you mean the file is there and it has content?

It was an issue with varfile i was generating with atmos that doesn’t include stage and generating a wrong file

File is there name of that file is not correct

ahhhhhhhh

ok, cool

I was shaking my head from last 4 hours Lol

Thanks @jose.amengual

no problem

TL;DR: is something missing from our atmos integration doc?

@jose.amengual thanks for the help

no, docs are good , this was a a mis label situation

For example when I do,

cloud_nat:
          subnetworks:
            - name: subnetworkname
              secondary_ip_range_names: ["name1", "name2"]

then the tf plan shows secondary_ip_range_names = []

i don’t see secondary_ip_range_names is used in the code https://github.com/slalombuild/terraform-atmos-accelerator/blob/main/components/terraform/gcp/network/nat.tf

it’s defined in the variable, but is not used

@jose.amengual

That variable is from the public module this component uses https://github.com/terraform-google-modules/terraform-google-cloud-nat

@Andriy Knysh (Cloud Posse) it actually is in variables. It is a part of the “subnetworks” variable

but if in our root module (SlalomBuild) repo is not used in the instantiation of the Google module, then it will not be respected

I can look at this tomorrow.

@pv ping me tomorrow

@pv your YAML config looks correct. Look at the varfile that Atmos generates to see if the variable is there. If not, something must be wrong with the stacks config

When I describe it, it shows the values but it is still not a part of the plan. Not sure what the issue is with that one

in our instantiation https://github.com/slalombuild/terraform-atmos-accelerator/blob/main/components/terraform/gcp/network/nat.tf#L11

we are not using secondary_ip_range_names

wait

is part of this variable

https://github.com/slalombuild/terraform-atmos-accelerator/blob/main/components/terraform/gcp/network/nat.tf#L22

you will need to figure out how to pass the value from yaml so it gets render correctly in json

this are some examples on how it should looks like https://github.com/slalombuild/terraform-atmos-accelerator/blob/main/components/terraform/gcp/network/example-vars.auto.tfvars

here is the example

https://github.com/slalombuild/terraform-atmos-accelerator/blob/main/stacks/gcp/dev/us-west-2.yaml#L105