#atmos (2023-12)

hey @Leo

How do I pass the outputs of one component as inputs to another? (within the same stack)

https://atmos.tools/core-concepts/components/remote-state - Atmos does not use YAML to define component remote state, it uses the TF remote-state module and YAML to define the input parameters

Does atmos automatically resolve dependencies?

https://atmos.tools/cli/commands/describe/dependents

use settings.depends_on for the components

does atmoshave a command to deploy all components in a stack?

use Atmos workflows https://atmos.tools/core-concepts/workflows/, and/or Atmos custom commands https://atmos.tools/core-concepts/subcommands/ - a workflow can call any script, any Atmos native or custom command, etc. Similarly, a custom command can call any script, any workflow, or Atmos native or other custom commands

Where can I find the full schema for stacks, workflows

we are working on that (and supporting the schemas in IDEs and in https://atmos.tools/cli/commands/validate/stacks command , we’ll have a new release this week with all the support and all the docs

Do you know where I could find a production-ready repo that uses atmos

an example of such a repo (not a full-blown infra, something to start with) will be in the new release as well. Take a look at https://atmos.tools/category/quick-start which describes the steps to take to start

Hi @Andriy Knysh (Cloud Posse), thanks for your help: I’ve got a atmos POC up and running now! :rocket:

I see that settings.depends_on is quite handy when describing components - does it play any role during terraform apply too?

Looking forwards to the new release!

settings.depends_on is not used directly in terraform apply, but it’s used in https://atmos.tools/cli/commands/describe/dependents, so you can create a custom command (and/or workflow) to use describe dependents and then plan/apply all the dependencies

depends_on is also used in CI/CD, e.g. GitHub Actions https://github.com/cloudposse?q=action-atmos&type=all&language=&sort= , or for Spacelift to automatically create stack dependencies https://docs.spacelift.io/concepts/stack/stack-dependencies from Atmos manifests

because my stage or tenant names have dashes in them - that will work with the dash in the stack name pattern

How do I deploy all components in a stack with atmos command? workflow is the only option?

workflows or custom commands. We could add a native atmos command terraform deploy all or similar, but since it could be very different for diff use-cases, you can do the same with custom commands according to your needs

deploy all components in a stack that would be very different for diff use cases even if you used plain terraform

with custom commands, you can do something like this:

add these commands to atmos.yaml:

then implement another custom command to use these custom commands:

you can modify and improve this command anytime (e.g. execute each step sequentially or concurrently, outputting debug/trace messages) w/o waiting on native implementation in Atmos (which is not easy to do to cover all possible use cases)

then execute atmos terraform deploy-all -s <stack>

or use https://atmos.tools/core-concepts/workflows/

you can combine custom commands with workflows, or workflows with custom commands in any combination

Thanks for the details…will get back after trying it

Could templates be used in stack configuration files

see https://atmos.tools/core-concepts/stacks/imports#go-templates-in-imports

Atmos stacks could it be used with Terraform enterprise or any documentation with that regard about integration?

what is special about Terraform enterprise? TF backend? Atmos can auto-generate TF backend from the stack manifests. But you can disable the auto backend generation and add a separate backend config file to each Terraform component (so you can use any backend besides those directly supported by Atmos)

@Andriy Knysh (Cloud Posse) thanks for this feedback. My followup question is, let’s say we develop catalogs & stacks based on atmos configuration..in future would we be able to use the same with Terraform enterprise? hopefully this questions make some sense, I am totally ignorant of Terraform Enterprise capabilities. I am trying to figure if we go down the path with atmos and in future if Terraform enterpise becomes an option would we have to start over or use any of the existing work

@Alcp the short answer to “if we develop catalogs & stacks based on atmos configuration..in future would we be able to use the same with Terraform enterprise” is yes

the long answer:

Terraform Enterprise is a self-hosted version of Terraform Cloud.

With regular OSS Terraform, you specify the backend, for example s3 in a file, similar to this (in the file backend.tf.json)

or, you add a setting to atmos.yaml to auto-generate the backend files automatically for each component (so no need to deal with the backend files0

now, with Terraform Cloud (and I guess with Terraform Enterprise), they have introduced the cloud block to be used instead of the backend block, similar to:

terraform {
  cloud {
    organization = "example_corp"
    ## Required for Terraform Enterprise; Defaults to app.terraform.io for Terraform Cloud
    hostname = "app.terraform.io"

    workspaces {
      tags = ["app"]
    }
  }
}

you can define it in the file [backend.tf](http://backend.tf) (or backend.tf.json) in each terraform component folder

currently, Atmos does not support autogeneration of TF cloud blocks (so it has to be added manually), but we are planning to add it to Atmos soon

to summarize:

1. When using the s3 backend, you can 1) provide the backend file for each component; or 2) ask Atmos to auto-generate it for each component automatically. The s3 backend can be used with the regular OSS Terraform, on the command line, or in CI/CD systems like GitHub Actions, Spacelift etc.

1. When using Terraform Cloud or Terraform Enterprise, you will probably use the cloud block. Right now, you can provide the cloud block for each component in a separate file, and we’ll add the auto-generation of the cloud block to Atmos soon

Ok cool thanks again for answering, so with backend change. The way we deploy stacks with atmos be the same

yes, I think the same (the only diff would be the backend config, but it’s not related to components and stacks)

Okay nice, I like atmos way of organizing components/catalog/stacks so it makes perfect sense to go the monorepo setup with atmos and in future the main impact would be the backend if enterprise comes in to picture

Once again thanks for the help and support.

also, when running in the SaaS systems like Spacelift or Terraform Cloud, they don’t know anything about Atmos (nor terragrunt or other wrappers), they just execute terraform commands. But we use the hooks that they provide to execute atmos generate varfile (https://atmos.tools/cli/commands/terraform/generate-varfile) and atmos generate backend (https://atmos.tools/cli/commands/terraform/generate-backend) commands before they execute terraform plan/apply commands

https://docs.spacelift.io/concepts/configuration/runtime-configuration/

https://developer.hashicorp.com/terraform/cloud-docs/agents/hooks

Thinking specifically about the account component so the account-map and aws-team* components can be used without a new org

Well, those are precisely the components that won’t work well in brownfield, at least the brownfield we plan to address.

However our plan for next quarter (2024) is to refactor our components for ala carte deployment in brown field settings. E.g. enterprise account management team issues your product team 3 accounts from a centrally managed Control Tower. You want to deploy Cloud Posse components and solutions in those accounts, without any sort of shared s3 bucket for state, no account map, no account management.

Oh very cool! Looking forward to it

Would it be a difficult transition to migrate from cloudposse atmos the way it’s setup now to the new landing zone setup?

It can be done in varying degrees of adoption. We will be updating all components to write to SSM and have setup a mechanism to replicate designated shared SSM parameters across all accounts and regions

Keep in mind that some enterprises do allow people to create roles ( not users) so whatever solution you guys create should allow you to create the roles and support Permission boundaries

Hmm if everything is saved to ssm does that mean the components will rely less on remote state so the remote-state.tf files will be replaced by data sources and ssm params?

One gap I’ve noticed in the readmes is that the output (and in the future ssm params), which is used by the remote state modules in descending components, are not documented in the readme.

It’s true that the outputs.tf is documented and can be looked at but only if the values are single scalar values. If they are maps, then it’s far harder to see what the output looks like unless you have access to an environment where it’s already deployed.

It would be nice to have a separate section in the readme to show the full sample output structure somehow.

I’m trying to rework the components for the brownfield env and one of the hardest things I’m trying to figure out is how to recreate the outputs for account component so the descending components will be and to read its remote state

It will all be wrapped in reusuable modules

all the outputs of the account-map component are not really secrets so it could be possible to just output to ssm to all accounts and then use that info to feed back into the existing modules without many changes

for me, i’m trying to do it for the account component since i do not know the format of the account-map output

unless there is an easy way to construct the account-map without the account component ?

@Andriy Knysh (Cloud Posse) do you recall how we could add existing accounts to the account component? I think we did that for one customer that was using Control Tower.

This was without importing.

I think we had to specify the account_id in the YAML or something

Atmos supports a backend of type static where we just define the values as if they were provisioned by other components

so it’s possible to “model” the entire account-map by using the static backend (as if the account-map was a real provisioned component)

example (from the tests) https://github.com/cloudposse/atmos/blob/master/examples/tests/stacks/catalog/terraform/test-component-override.yaml#L27

(it’s not documented in https://atmos.tools/, need to add a section about it)

Thats nice! I didn’t know about that type

The root issue is that we don’t know what the values need to look like for the component output. If we had that, then constructing a local account or account-map replacement would be straight forward

The other alternative, probably faster, is to add a way to NOT create an org or an account, for the account component

          organizational_units:
            - name: core
              enabled: false              # this will prevent creation of the OU
              accounts:
                - name: core-artifacts
                  enabled: false          # this will prevent creation of the account
                  tenant: core
                  stage: artifacts
                  tags:
                    eks: false

the account component has such a flag https://github.com/cloudposse/terraform-aws-components/blob/main/modules/account/variables.tf#L37

thats only for organization tho, not all accounts? or maybe im mistaken? ill give it a try! ty andriy

yes, that’s for Org only (to not create it if you want to use an existing one)

ah ok so ill need to do it for accounts too. ill play with it, if i can get it to work, ill upstream my changes

yesterday I imported 2 existing accounts into the accountcomponent

I’m about to apply soon, so I can keep you posted

Did the emails have to align to import them without recreating?

@jose.amengual did you fix the issues you were having with the import?

yes, I posted it here

pretty sure is a TF bug

yes they have to

example :

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # aws_organizations_account.organizational_units_accounts["pepe-nonprod"] must be replaced
-/+ resource "aws_organizations_account" "organizational_units_accounts" {
      ~ arn                        = "arn:aws:organizations::111111111:account/o-za5r5sz4rw/22222222222" -> (known after apply)
      + close_on_deletion          = false
      + create_govcloud            = false
      ~ email                      = "[email protected]" -> "[email protected]" # forces replacement
      + govcloud_id                = (known after apply)
      + iam_user_access_to_billing = "ALLOW"
      ~ id                         = "22222222222" -> (known after apply)
      ~ joined_method              = "INVITED" -> (known after apply)
      ~ joined_timestamp           = "2023-11-09T20:34:30Z" -> (known after apply)
      ~ name                       = "pepe-Non Prod" -> "pepe-nonprod" # forces replacement
      ~ status                     = "ACTIVE" -> (known after apply)
      ~ tags                       = {
          + "Environment"   = "global"
          + "Name"          = "pepe-nonprod"
          + "Namespace"     = "pepe"
          + "Stage"         = "root"
          + "component"     = ""
          + "contact"       = "[email protected]"
          + "expense-class" = ""
        }
      ~ tags_all                   = {} -> (known after apply)
        # (1 unchanged attribute hidden)
    }

I’m changing the account name ( because it has a space) and the email

Ah so the emails did change

And that’s causing the accounts to recreate…

this accounts have emails that do not match the new standard

so that needs to be changed

I wonder if adding a new key to the yaml to override the email convention to no-op the tf would do the trick

I think a hcl change would be needed to but i think that would do it

yes

by the way @RB the import worked after matching email and account name and I was able to add it to account map and sso and such

right but you had to match the email which means you had to update the email in the account itself to no-op it ?

I don’t want to change the email to the cp convention. I just want to reuse the existing accounts

I think you can add to the account component an email attribute to override

beautiful! exactly what i wanted

you will just need to change this :

# Provision Accounts for Organization (not connected to OUs)
resource "aws_organizations_account" "organization_accounts" {
  for_each                   = local.organization_accounts_map
  name                       = each.value.name
  email                      = format(each.value.account_email_format, each.value.name)
  iam_user_access_to_billing = var.account_iam_user_access_to_billing
  tags                       = merge(module.this.tags, try(each.value.tags, {}), { Name : each.value.name })

  lifecycle {
    ignore_changes = [iam_user_access_to_billing]
  }
}

hmm I don’t think that will work unless the email contains the full name of the account

it will take some tweaking but i can make it work

ill also need to change the other resources too like the OUs

why can’t just read it from the organizational_units if defined?

I did something similar for the Child OUs support I added

look at parent_id

if you add to local.organizational_units_accounts_map the email if defined in the account , you could just pass that or default to the each.value.account_email_format

right here:

 # Organizational Units' Accounts list and map configuration
  organizational_units_accounts = flatten([
    for ou in local.organizational_units : [
      for account in lookup(ou, "accounts", []) : merge({ "ou" = ou.name, "account_email_format" = lookup(ou, "account_email_format", var.account_email_format), parent_ou = contains(keys(ou), "parent_ou") ? ou.parent_ou : "none"}, account)
    ]
  ])

line 16 of the component

sorry I could send you the link in the code but I’m doing something else

Looks promising. I’ll dig more into that soon. Thanks Pepe! I’m hoping to put in a pr for it all this week and that should unlock all the other components

definitely and if you feel like I think that local could use some better formatting and move some of those lookups can be moved to other locals to make it easier to read

I have account and account-map working in a brownfield env. Could someone give me a review on the upstreamed changes?

https://github.com/cloudposse/terraform-aws-components/pull/943

@RB thanks for the PR, looks good, a few comments

@RB can you update the README with an example of how to configure brownfield accounts in the YAML?

(otherwise this will be lost)

Thanks @Andriy Knysh (Cloud Posse) and @Erik Osterman (Cloud Posse) for reviewing. I made the necessary changes, tested in a few cases, and updated the readme to reflect the additions and how to use them.

Please re-review.

@RB cc @Andriy Knysh (Cloud Posse) I’m very wary of modifying account like this, because it is extremely difficult to test with greenfield deployments, and we have seen plenty of issues with mixing data sources and resources, even when we try to be careful with enabling/disabling them via count.

Can we resolve this instead by better documenting the required output of account and showing people how to supply that to remote-state as input to account-map?

We can modify account-map to remove the need for anything but account_info_map from accounts.

Thanks for reviewing Jeremy.

Could the pr’ed version be tested with current greenfield environments by cloudposse? I’m almost positive it would result in no-changes.

However, if you folks don’t want to risk it. If the account component output was documented, we’d still need a component to output that format which would end in a different flavor of the component.

Would you folks then be willing to allow an upstream of a separate component (a brownfield flavor of account) instead of modifying the existing one to work with both existing and new accounts?

The hope is that we could upstream this to unlock a new suite of users to atmos and the cloudposse refarch. Up until now, brownfield users, the majority of cloud users, have been excluded and difficult to adopt the components without creating their own due to the dependence of the core components

So the changes in this PR are directly inline with our stated objectives to support brownfield environments. Our medium-term objective (big project) is to elminate reliance on account and account-map components. However, in order to accomodate brownfield environments in the shorter term, I am amenable to this type of change by @RB. @Jeremy G (Cloud Posse) brings up good points about the challenges of working with count, but I wonder if that matters a whole lot less, if taking into account that destruction of accounts barely happens.

I think @Jeremy G (Cloud Posse) brings up a good point about:

Can we resolve this instead by better documenting the required output of account and showing people how to supply that to remote-state as input to account-map? This is how we previously solved it. Although keeping it in account makes it a bit more consistent from a DX.

Maybe @Ben Smith (Cloud Posse) @Dan Miller (Cloud Posse) @Jeremy White (Cloud Posse) can test this PR in one of our upcoming implementations.

My concern with testing account is that we have to test that it works to create accounts where none but the root exist, and if we find a problem, we don’t have an easy way to create a new test environment. This has made it the most difficult component to get right. Adding data sources that reference the same resources that the component creates is asking for hard-to-find and hard-to fix bugs, because data sources fail when they reference resources that do not exist, which is only going to be the case when we are first provisioning accounts; it will not suffice to ensure the component works after accounts have already been created. Because of these things, I am very reluctant to make substantial changes to account.

If you wanted to create a separate component, say account-data, and build it to support brownfield only, I would be more open to that. You deploy account to create and manage accounts and organizations, and account-data to use existing ones. While it would mean maintaining 2 components in sync, account is very stable, so it should not be a big job to keep them in sync, and we can and will resolve this issue differently in the v2 architecture.

I also note that the PR regarding changes to account-map is unacceptable as-is because it removes support for the existing root account having a different name in AWS than it does in Atmos stacks. It’s kind of subtle, but it points to the kind of trouble we can be getting ourselves into without noticing.

in https://atmos.tools/category/quick-start, we used just a few TF components to show how to configure Atmos. The quick start does not show a full infrastructure b/c it would be too big for a quick start

the account-map component is used to store information about all the accounts https://atmos.tools/category/quick-start

before account-map, the account component needs to be provisioned https://github.com/cloudposse/terraform-aws-components/tree/main/modules/account

then, the account-map component is used in all other components to get the terraform IAM roles, for example https://github.com/cloudposse/terraform-aws-components/blob/main/modules/vpc/providers.tf#L17

Hi @Andriy Knysh (Cloud Posse) Thanks for sharing this information. Does this mean it’s mandatory to use account module and provision whole AWS Organization and AWS account, role etc.

no

you can use your own terraform modules

So if i already have AWS account how do it proceed in that case ?

Can i still be use account-map to store the information about account and roles ?

there are many diff things here. I’ll try to explain:

1. Atmos is not involved here, it’s all about terraform modules and components (root-modules)

1. If you are using Cloud Posse terraform components https://github.com/cloudposse/terraform-aws-components/tree/main/modules, then all of them use the account and account-map modules to provision the accounts and store info about accounts

1. The components use the information about the accounts and roles from the account-map component, for example https://github.com/cloudposse/terraform-aws-components/blob/main/modules/vpc/providers.tf

1. If you are using your own terraform modules/components, then it’s not required to use account-mao , but in this case you need to provide a Role for Terraform to assume to be able to access the resources in the account, as done here https://github.com/cloudposse/terraform-aws-components/blob/main/modules/vpc/providers.tf#L7

1. If you want to use the caller role (the role you assume then executing terraform plan/apply ) as the Terraform role, then you don’t need to specify a separate role for the aws provider, and the code becomes

   provider "aws" {
     region = var.region
   }
   

that’s what is shown in the Atmos quick-start https://atmos.tools/category/quick-start and https://github.com/cloudposse/atmos/blob/master/examples/quick-start/components/terraform/vpc/providers.tf

so here you need to decide what IAM roles you want to use and how to configure all of them. Do you want to use the caller role, or do you want to specify separate Terraform and Terraform backend roles to access the resources in different accounts and access the Tf backend

in Cloud Posse Terraform components, the combination of these components achieves that

https://github.com/cloudposse/terraform-aws-components/tree/main/modules/account https://github.com/cloudposse/terraform-aws-components/tree/main/modules/account-map https://github.com/cloudposse/terraform-aws-components/tree/main/modules/aws-teams https://github.com/cloudposse/terraform-aws-components/tree/main/modules/aws-team-roles

so each component then just gets the required IAM role for Terraform to assume when accessing the resourcs in the corresponding account (e.g. https://github.com/cloudposse/terraform-aws-components/blob/main/modules/ecr/providers.tf)

this is definitely a complicated topic. If you use or want to use the CloudPosse components, then yes, you should use the account-map component to store the info b/c all other components use it to get the IAM roles (they usually diff for all accounts, e.g. dev, staging, prod)

if you are uisng your own Terraform modules and components, then it’s not nesessary to use account-map, and as mentioned above, you can provide your own roles in

provider "aws" {
  region = var.region

  assume_role {
      role_arn = <role to access the AWS resources in the account>
    }
}

or, if you want to use the caller role (the role that YOU assume when executing terraform plan/apply), then the provider code becomes just

provider "aws" {
  region = var.region
}

see https://registry.terraform.io/providers/hashicorp/aws/latest/docs#authentication-and-configuration for more details

in other words, if you already have an account, and you want just to provision a VPC as shown in the Atmos quick start, and you want to use your caller role, then you don’t need the account- map module. This is simple and you can play with it, but obviously it’s not a production ready configuration (production ready configuration is all about roles, permissions and access control)

Thank you @Andriy Knysh (Cloud Posse) for this information now i got it. I will play with it more and hopefully i will go for production with roles and permissions

Do you have a article that can help with setup roles and permissions using account-map ?

no such public docs, but that’ involves not only the account-map , but other components like aws-teams and aws-team-roles if you are planning to use the CloudPose component https://github.com/cloudposse/terraform-aws-components/tree/main/modules. Or you can implement the roles yourself (starting with a simpler setup, e.g. diff terraform roles for dev, prod and staging)

@marcelo.eguino

i answered you in DM what could cause it

atmos validate stacks comes out clean

we are using the same repo

@jose.amengual are you still blocked?

no, this is related to my coworkers using a Mac m2 CPU

I meant if Andriy’s answer via DM solved the issue?

yes

FYI

in docker on a Mac M2/M1 that fixes that issue

nice, thank you, we need to document this

Docker recently published an update saying it resolved most Rosetta/x86 emulation issues. Hopefully this issue is included

oh sorry just noticed I was inside a atmos terraform shell and trying to run atmos against a different stack

ah that will do it

maybe the PS1 can be modified when in the atmos terraform shell so it’s more obvious that you’re in a subshell for a specific region dash component (stack)

https://github.com/cloudposse/atmos/issues/495

Removing import: [] from 02-atmos/stacks/catalog/example.yaml solved it

please use https://atmos.tools/category/quick-start

this is a supported doc, it gets reviewed and updated often (and wil be improved as Atmos evolves), and it points to the repo which you can start using

https://github.com/cloudposse/atmos/tree/master/examples/quick-start

We have different releases of infrastructure environments which means we’re not JUST doing the standard dev/test/prod

The current set up is something like this: atmos/components/terraform/{{release_version}}/vpc

Where release_version could be something like v1.0.0 or v2.0.0

The reason in doing this is so that it allows me to specify what version of the component I need in my atmos configuration.. like below:

import:
 - path: catalog/interfaces/components/vpc-interface
components:
 terraform:
   vpc:
     metadata:
       component: "{{ .release_version }}/vpc"
       inherits:
         - vpc-interface

The drawback is that I would have to duplicate files to across new versions and that would mean duplicating code and not keeping things DRY.

Is there a simple solution to this that I’m not seeing and do you all have any suggestions/ideas around that? Seems like things point to vendoring, but I’m not sure if that would suffice as it would require constant vendor pulls and actual copies of the components which need to be kept updated..

what you have described is a correct way of doing it (we do the same as well). There are a few notes:

• If you can add some flags to the component to make it support different functionality, that would be better since you have only one version of TF code to support. Then use the flag in vars to select the different functionality

• If you can’t modify the component, then yes, you can have multiple folders with different versions. You can use components/terraform/{{release_version}}/vpc or components/terraform/vpc/{{release_version}} or any other variants

• Or, you can use Atmos vendoring to vendor the core of the component, then add some mixins to add some custom files (the mixins can be remote or local). See https://atmos.tools/core-concepts/components/vendoring and https://atmos.tools/core-concepts/vendoring/. This way, you can vendor diff files into diff folders. This is similar to #2, but you don’t maintain all the component’s code, you maintain just mixins

if you are using 3rd-party components (e.g. https://github.com/cloudposse/terraform-aws-components/tree/main/modules), then #2 (which you explained) is the best way to do it (you just consider the diff versions of the component as diff components)

Ah I see. I like the possibility of flags to support different functionalities and even a slightly different variation of the folder structure. Also, it helps to know it’s a valid approach to this. This is all helpful thanks!

https://atmos.tools/cli/configuration#components

https://atmos.tools/quick-start/configure-cli

Yah, I saw that, but I was curious if it could be tuned in any way. How does it auto-generate the backend? Does it, or can it be told, to pick S3? Can a specific s3 bucket be specified while the key and everything else is dynamic?

ok, the whole thing consists of three steps (we’ll document it in https://atmos.tools/ in the net release):

1. Add auto_generate_backend_file: true to atmos.yaml in components.terraform section

1. Configure the backend in one of the _defaults.yaml manifests. You can configure it for the entire Org, or per OU/tenant, or per region, or per account. See https://github.com/cloudposse/atmos/blob/master/examples/tests/stacks/orgs/cp/_defaults.yaml#L7 as an example

1. (This is optional) For each component, you can add workspace_key_prefix similar to https://github.com/cloudposse/atmos/blob/master/examples/tests/stacks/catalog/terraform/vpc.yaml

note that this is optional - if you don’t add backend.s3.workspace_key_prefix to the component, the Atmos component name will be used automatically (which is this example is infra/vpc, but / will get replaced with -, so workspace_key_prefix will still be infra-vpc

we usually don’t specify workspace_key_prefix for each component and let Atmos to use the component name as workspace_key_prefix.

note that I mentioned that these sections

  backend_type: s3
  backend:
    s3:

Ahh .. thank you, this is exactly what I needed to know.

can be defined at any level: Org, OU, region, account (and even component). They participate in all the inheritance https://atmos.tools/core-concepts/components/inheritance

meaning that part of the config can be defined at Org level, part at the OU level, part at the account level, etc. - then all the pieces will be deep-merged together into the final backend config

i’m mentioning that b/c you will want to define different IAM roles to assume for each account (dev, staging, prod) in role_arn

so this part

terraform:
  backend_type: s3
  backend:
    s3:
      encrypt: true
      bucket: "cp-ue2-root-tfstate"
      key: "terraform.tfstate"
      dynamodb_table: "cp-ue2-root-tfstate-lock"
      acl: "bucket-owner-full-control"
      region: "us-east-2"

can be defined at the Org level

but this part

terraform:
  backend:
    s3:
      role_arn: xxxxxx

can be defined at the OU or account level with diff IAM roles

Atmos will deep-merge everything together and generate the final backend.tf.json file for each component taking the pieces from diff levels of the config

also, you might want to have diff S3 buckets for the backend for diff Orgs (if you have multiple Orgs), or different accounts or OUs

in which case, this part goes to the Org level (_defults.yaml)

terraform:
  backend_type: s3
  backend:
    s3:
      encrypt: true
      key: "terraform.tfstate"
      acl: "bucket-owner-full-control"
      region: "us-east-2"

and this part goes to the OU and/or account level

terraform:
  backend:
    s3:
      bucket: "xxxxxx"
      dynamodb_table: "xxxxxx"

so diff parts can be configured in diff manifests (Org, OU, region, account) depending on your needs

Wonderful. Thank you for the thorough explanation. This has been a huge help.

@Mark also note, that why auto-backend generation is a nice feature to have/use (it saves you from creating those [backend.tf](http://backend.tf) files in each component, it actually becomes a requirement when you want to provision multiple Atmos component instances using the same Terraform component (e.g. many VPCs into the same account/region with diff names and settings)

see https://atmos.tools/quick-start/final-notes

Does it auto generate the buckets as well?

if you create ` backend file manually, you’ll hardcode workspace_key_prefix for the component. But if you provision ultiple components using the same TF code, one of them will not work

Does it auto generate the buckets as well?

Yah, trying to sort of wrap my head around the total scope of how much it can do unattended. I am, in part, trying understand if there is some behind-the-scene magic for bootstrapping a bucket into IAC that is defined by the IAC for use by the IAC

buckets, no. Atmos nor TF does not generate anything in AWS, you need to provision it. See https://github.com/cloudposse/terraform-aws-components/tree/main/modules/tfstate-backend

https://github.com/cloudposse/terraform-aws-components/blob/main/modules/tfstate-backend/main.tf#L6

https://github.com/cloudposse/terraform-aws-tfstate-backend

tfstate-backend is another Atmos component that uses the terraform component

Interesting

you have all components in components/terraform including tfstate-backend - this is the code (logic) that can be deployed anywhere (any Org, any OU, any region, any account)

you have stacks - this is the configuration of the components for diff environments (regions, Orgs, OUs, accounts) - managed by Atmos

so all of that is about the separation of the code (logic) from the configuration, so the code is generic and can be deployed anywhere

in many cases, with enterprise-grade infras (multi-org, multi-tenant, multi-account, multi-region, multi-team), the configuration is much more complicated than the code. That’s what Atmos is trying to solve - to make the configuration manageable, reusable (by using https://atmos.tools/core-concepts/stacks/imports and https://atmos.tools/core-concepts/components/inheritance) and DRY, and to make the code completely generic

Yah, I am looking at solving some multi account/team/tenant issues, which is pretty much exactly how I found atmos. Trying to conceptualize a best-practices starting point to capture all of my goals. Quite a bit of research and thinking and drawing things out.

Alo, that PR safe-guards the change to only effect debian-based systems.

maybe try this example https://github.com/cloudposse/atmos/tree/master/examples/quick-start

the Dockerfile should work https://github.com/cloudposse/atmos/blob/master/examples/quick-start/Dockerfile

note that the example repo https://github.com/cloudposse/atmos/blob/master/examples/quick-start/ is explained in https://atmos.tools/category/quick-start

Yah, same problem

# Host filesystem device detection failed. Falling back to "path starts with /localhost".           

?

@Andriy Knysh (Cloud Posse) any suggestions?

i don’t now what the error is about. Try not to run geodesic directly, but instead copy this Dockerfile (https://github.com/cloudposse/atmos/blob/master/examples/quick-start/Dockerfile) and Makefile (https://github.com/cloudposse/atmos/blob/master/examples/quick-start/Makefile) and run make all

Yah, I have kinda of tried both, it breaks either way.

did you run git config --global --add safe.directory as the message above suggests?

yah, and that works, but it has to be done every time I jump into the container.

like, out of the box it doesn’t work, and if I exit the container and retart it then I have to re-run that config

which feels like something is broken

maybe something with your git config, see https://stackoverflow.com/questions/72978485/git-submodule-update-failed-with-fatal-detected-dubious-ownership-in-repositor

there are many suggestions in there

Yah, we don’t have any submodules or anything. This happens out of the box w/out me doing anything w/ the container.

In fact, this happens w/ atmos repo out-of-the-box.

and it doesn’t happen outside of the atmos container

you mean geodesic?

yah, that

@Jeremy G (Cloud Posse) do you have any insight on the error above?

so yah… did …

$ git clone <ssh://[email protected]/cloudposse/atmos.git> atmos.git
$ cd atmos.git/examples/quick-start
$ make all

when I start the same, I don’t see the messages

# BindFS mapping of /localhost.bindfs to /localhost enabled.
# Files created under /localhost will have UID:GID 1047026499:1047000513 on host.

something is wrong with permissions?

I am not certain there. This is default ubuntu install, nothing really special in the end.

https://github.com/cloudposse/geodesic/issues/594

I think that host filesystem device detection failed. is the only thing that stands out

Which sort of makes sense as this is a btrfs subvolume

on what OS are you running the container?

so your host OS is Ubuntu?

yes

I have pretty much only run Linux since 1993

hmm, we usually run on MacOS. Let’s ask @Jeremy G (Cloud Posse) again if he has anything to say about the issue

trying to figure out what is doing the rootfs probe

oh .. found that …

interesting

uhhhh .. funny

$ df --output=source /
Filesystem
/dev/nvme0n1p2

$ df --output=source /localhost
Filesystem
/localhost.bindfs

Which makes sense .. the startup scripts are kicking out the bindfs mount

does OSX not return the bind point? that seems an odd expectation

I am not super tickled with this test .. I have no way to manually specify same-a-root and bipass the check

Okay.. so if GEODESIC_LOCALHOST_DEVICE == disabled .. then we never check GEODESIC_LOCALHOST_MAPPED_DEVICE ?

that seems like a bug…

trying to understand why that test excludes a variable set in a different script

oh .. even weirder…

env | grep ^GEO
GEODESIC_LOCALHOST=/localhost.bindfs
GEODESIC_LOCALHOST_DEVICE=same-as-root
GEODESIC_PORT=49406
GEODESIC_WORKDIR=/localhost/Development/atmos/atmos.git/examples/quick-start
GEODESIC_AWS_HOME=/localhost/.aws
GEODESIC_SHELL=true
GEODESIC_LOCALHOST_MAPPED_DEVICE=/localhost.bindfs
GEODESIC_TF_PROMPT_ACTIVE=false
GEODESIC_HOST_CWD=/home/mark.ferrell/Development/atmos/atmos.git/examples/quick-start
GEODESIC_HOST_UID=1047026499
GEODESIC_DEV_VERSION=
GEODESIC_CONFIG_HOME=/localhost/.geodesic
GEODESIC_OS=debian
GEODESIC_HOST_GID=1047000513
GEODESIC_VERSION=2.8.0

oh .. nope .. that makes sense… disregard the previous

oh .. the container isn’t mapping the getent data from the host

found it

Sooo .. yah .. this breaks w/ anything using nis, nis+, ldap, sssd, activedirectory, etc..

/etc/profile.d/user.sh has a test that makes no sense

that id "${UISER}" is bad

that block will work if this is done as:

if [[ -n "${USER}" ]] && [[ -n "${USER_ID}" ]] && [[ -n "${GROUP_ID}" ]]; then  
  adduser -D -u ${USER_ID} -g ${GROUP_ID} -h ${HOME} ${USER} &>/dev/null  
fi  

or .. no .. hold it .. the test was right, but it isn’t being executed on my container?

oh funny .. the code doesn’t grab the group name… userful…

yah .. so that is a problem…

okay .. I think I have a fix for this .. but it requires patching geodesic and atmos

so … there are 2 bugs…

1. this particular use of adduser is not really portable
2. the code expects users to be using existing groups for some reason, so if the group doesn’t already exist then adduser fails (particularly for nis/nis+/ldap/ad/sssd groups)

if that’s a solution, we can update geodesic (@Jeremy G (Cloud Posse) is a maintainer). (not related to Atmos)

yah, I am trying to work through the build process so I can test this locally

for some odd reason running make build in the geodesic directory ends up installing packages for the wrong target..

oh .. I misread part of the output ..

I hate being sick

https://github.com/cloudposse/geodesic/pull/900

I am sort of wondering how this worked at all on a rather large stack of systems..

@Mark There are a few things going on here, none related to Atmos, but rather Geodesic, of which I am the lead maintainer.

First of all, the bindfs mount and uid and guid mappings are all workarounds for having Docker installed as a daemon running as root, which is not recommended, and has been discouraged for a long time. I think all of these problems (except for the AWS session manager plugin) will go away for you if you install Docker in “rootless” mode.

What I’d like to know is what user and group names you had that were giving you problems, so I can try to reproduce it.

@MarkYou point out

the container isn’t mapping the getent data from the host That is correct. It is not intended to.

Geodesic is 8 years old and still has a lot of outdated cruft that we leave around just in case someone is relying on it, as long as it isn’t interfering with anything. The user.sh script you updated was a hack for ansible that complained if id "${USER}" failed. It was written for Alpine and never updated for Debian (and never worked on Debian). Its point was to register the username as an alias for user root so that id "${USER}" would return 0 and not an error. In general it should not be important anymore. While your PR may get this working on Debian, it will break it on Alpine and I’d rather remove the whole thing. Why did you think this was needed? Is 1047026499:1047000513 the right UID:GID for your host?

As for why the “Host filesystem detection” fails, I’m at a loss. I’m guessing it has something to do with Ubuntu 22, which we have not tested yet. I cannot reproduce it on Ubuntu 20. From within Geodesic what does

df --output=source /localhost.bindfs

output?

Soo .. the problem here is that git is refusing to work because the host rootfs UID and GID file ownership is not being mapped into the container.

Updating the user.sh fixes the mapping and allows git to work from inside the container.

And .. bindfs is running as a user, not as root

I suspect the files system detection failed because the behavior you are expecting is not universal to all filesystems, it isn’t related to Ubuntu, or Linux actually.

but it is also a red herring. The larger issue here is that:

1. there was no user mapping created for the UID of the files which git was looking at. So git wont work sanely unless the UID:GID can be looked up.
2. the adduser command was invalid for debian based systems
3. the user.sh script wasn’t attempting to create group mappings

and yes, that UID:GID are the correct mappings

and yes, you will need to deal with getent if you want any sane software to work from w/in a container such that it can trust the UID:GID mappings

W/out the above changes, using the default atmos geodesc:

$ atmos
... clipped all the cruft

 ⧉  geodesic 
 ✗ . [none] (HOST) quick-start ⨠ git status
fatal: detected dubious ownership in repository at '/localhost/Development/atmos/atmos.git'
To add an exception for this directory, call:

	git config --global --add safe.directory /localhost/Development/atmos/atmos.git

Has anyone tried removing user.sh to see if git still works for them?

My expectation is that it shouldn’t work if user.sh is removed, and even if it was added a an ansible hack at some point, it has likely kept this working accidentally for some time.

But, whichever

@Mark Not to discount your problem, but to answer your question, we have many users successfully using Geodesic and Atmos and git on Ubuntu, WSL, and macOS, both ARM64 and AMD64 (Apple M*). All our engineers and many of our customers’ engineers use it every workday. Our standard operating procedure is to run Atmos from within Geodesic against a git repository hosted on Host system and mounted into the Docker container.

What your are experiencing is and issue with: • Running Docker as root • Requiring bindfs to do ownership translation between the owner in Geodesic (which must be root) and the owner of the files on the host system • A relatively new addition to git where it checks ownership of directories where it reads configuration • And an issue where we (I) used the wrong options to set up the bindfs mount. All the behavior makes sense to me except for why you see “Host filesystem device detection failed.” And since I can’t reproduce your problem exactly, I’m asking again, what do you get as the output of

df --output=source /localhost.bindfs

@Mark We have released Geodesic v2.8.1 which should fix this issue for you. LMK if you still have issues after updating.

Honestly, I am very thankful for the information you have provided and all of the feedback. Based on information you previously provided I did find information on installing the rootless docker packages for Ubuntu 22.04.

So far as the file_device() oddity, when I was stepping through the code I thought I found that the code was doing the following:

$ df --output=source /
Filesystem
/dev/nvme0n1p2

$ df --output=source /localhost
Filesystem
/localhost.bindfs

But after looking at my own log more I realized that it wasn’t the case.

My brain has been particularly broken this week. Anyway, my mis-identifying the second step lead me astray.

did you try the new geodesic release 2.8.1?

Not yet, been juggling a few things.

@Andriy Knysh (Cloud Posse)

all these Terraform components (Terraform root modules) are actually configured with Atmos (see the READMEs) https://github.com/cloudposse/terraform-aws-components/tree/main/modules

note that those are “real” production-ready components used for many deployments, and they are all configured with Atmos

adding something more complicated to https://atmos.tools/category/quick-start would make it more difficult for new people to understand the basics

I see there’s a lot more validate tests for atmos

I’ll take a look, it’s been a while since I’ve used atmos and I’m going to rebuild this site around it

@ballew what exactly are you referring to? Maybe we can provide you with more information

This stuff: https://github.com/cloudposse/atmos/blob/master/examples/quick-start/stacks/schemas/jsonschema/vpc/validate-vpc-component.json

if you are referring to https://github.com/cloudposse/atmos/tree/master/examples/tests, then yes, that’s for ALL Atmos tests, and there are much more to Atmos than explained in the Quick Start

Working off the quick-start there’s some stuff that doesn’t work out of the box, but it’s been a good way for me to figure out how to label things and separate out prod and dev

validation is explained in https://atmos.tools/quick-start/configure-validation

perfect, I’ll dig in from there

and in https://atmos.tools/core-concepts/components/validation

it’s not related to Atmos, it’s a terraform issue. Try to delete the .terraform folder and the lock file

and maybe try to delete HOME/.terraform.d folder where terraform stores all the cached providers

that did it! thank you Andriy

you can also use this command https://atmos.tools/cli/commands/terraform/clean (instead of going to the component folder and deleting the .terraform foder and the lock file manually)

could you share a transit gateway, instead of peering ?

We probably could, would that be the better option? I want to allow connections from a lambda function in one account to a VPC connected Aurora (RDS) cluster in another account.

It’s only 2 VPC’s so I think vpc peering should suffice for this use-case.

vpc peering does not have the same capabilities than a transit gateway

if is only two vpcs maybe it does not matter but if you think about maybe having 3 or 4 in the future then the TG is a better choice

price wise I don’t know if there is much difference

v1 is legacy aws accounts that were not created with atmos or cp refarch v2 is aws accounts created with atmos and cp refarch

If you created all your accounts using the cp/atmos method then you should use the v2 method

for multi-account and/or cross-region VPC peering, consider this module https://github.com/cloudposse/terraform-aws-vpc-peering-multi-account

this component https://github.com/cloudposse/terraform-aws-components/tree/main/modules/vpc-peering uses the module and can create cross-account and cross-region VPC peering

this is def needs to be improved and described in more details

I suppose v1 means you specify accepter_aws_assume_role_arn

and v2 means you specify accepter_stage_name and the role ARN is determined by

  accepter_aws_assume_role_arn = var.accepter_stage_name != null ? module.iam_roles.terraform_role_arns[var.accepter_stage_name] : var.accepter_aws_assume_role_arn

if you don’t use module.iam_roles (which uses the accoun-map component)

module "iam_roles" {
  source  = "../account-map/modules/iam-roles"
  context = module.this.context
}

then provide accepter_aws_assume_role_arn var

(you def don’t need a TGW when peering just 2 VPCs, especially if you don’t need transitive routing and don’t have CIDR overlapping)

How vpc may depend on account-map which depends on account, etc?

It is

https://atmos.tools/cli/commands/describe/dependents/

Also supported by https://atmos.tools/integrations/github-actions/affected-stacks

Very nice.

This would only work if depends_on is explicitly set for each component, right?

Is there a way to implicitly detect it by analyzing the terraform, perhaps by checking for usage of the remote state modules?

Or would it be better to generate the component default yaml with depends_on using a separate script to analyze the component tf code?

It requires explicit depends on