#atmos (2023-02)

show how do you use the remote-state module

context must be provided to it

context = module.this.context

see https://atmos.tools/core-concepts/components/remote-state

module "account_map" {
  source  = "cloudposse/stack-config/yaml//modules/remote-state"
  version = "1.3.1"

  component   = "account-map"
  privileged  = var.privileged
  tenant      = var.global_tenant_name
  environment = var.global_environment_name
  stage       = var.global_stage_name

  context = module.always.context
}

The remote state is the one defined in a vendor component, the child module of account-map

I have a remote_stack_backend configured in a _defaults.yaml (for now)

  remote_state_backend:
    s3:
      bucket: "pd-core-ue1-root-tfstate"
      dynamodb_table: "pd-core-ue1-root-tfstate-lock"
      key: "terraform.tfstate"
      role_arn: "arn:aws:iam::root_acc_id:role/seed_role"

Thought that was the sufficient condition to hook things up

are these set correctly?

tenant      = var.global_tenant_name
  environment = var.global_environment_name
  stage       = var.global_stage_name

roles-to-principles defaults them to core, gbl, and root — and I’m not deviating from the standard, so that should work

not easy to figure out what’s wrong, you can zip the project up and DM me

Seemed to have solved this, was a problem with the s3 remote state of the account-map component. Made a stub component to test remote state modules for various deployed components and each had a non-null outputs — except account-map

Complete delete and clean of the directory and redeploy fixed the problem, cheers for the help yesterday

super, glad you solved it. I looked at your code yesterday, but didn’t find anything obvious, that’s why i wanted to recheck it today

@Andriy Knysh (Cloud Posse) all atmos output should be to stderr IMO

I believe atmos output can be squelched with log levels, but we should still handle this use case as pipelining commands is common Unix behavior

all atmos output should be to stderr IMO

I think no, b/c we have many situations where we don’t want anything in stderr, e.g. in GH actions

and we even have a PR to disable stderr completely

https://github.com/cloudposse/atmos/pull/322

b/c we had issues with GH action seeing errors in stderr

Disabling stderr is a good feature, when we want to show only relevant information regarding a process, but process errors separately or even ignore them. But currently Atmos logs all service information to stdout as well as terraform output, and there is no ability to split terraform output from atmos output (without writing ugly wrapper to find a first figure bracket etc. ), am I right?

I see these kinds of entries in my file:

Found stack config files:
Variables for the component 'main' in the stack 'security':
Writing the variables to file:
Command info:
Executing command:

along with an output from terraform, when I redirect stdout to the file, but stderr is empty. So stderr disabling is not a magic bullet in this case

@Viacheslav atmos terraform ... currently does not support piping the terraform only output into a file b/c atmos always outputs some info. We need to add LOG levels with one LOG level to completely suppress any outputs from atmos

currently, you can do the following:

So stderr disabling is not a magic bullet in this case

yes, this is not for this case, this is mostly for GitHub actions to redirect stderr from other binaries if we don’t want to have errors in stderr in GH actions - I was trying to point out that we don’t need to redirect all atmos outputs to stderr, we need to introduce LOG levels

@Viacheslav for manual one-time execution, this should work for you

atmos terraform shell main -s security 
terraform output > file.json

it will not work in some cases in CICD. We’ll try to implement log levels in atmos soon

@Andriy Knysh (Cloud Posse) Thank you! atmos terraform shell is enough for testing purposes right now. So log levels would be a very expecting feature, we hope to see it in upcoming releases :)

I’m experiencing this problem today, hoping to use terraform output > out.json but it has atmos output. I have used --redirect-stderr but still have some atmos output in my out.json. I will try the shell command above.

Also Terragrunt outputs only to stderr but can be run in CI, I agree with @Erik Osterman (Cloud Posse) that it makes sense to send all atmos output to stderr

(it will not work in all cases, e.g. some GH actions)

I am getting a really strange error here

and when the process exited, I got kicked out into a subshell inside the component directory. @Andriy Knysh (Cloud Posse) do you see what I did wrong?

reordered the command to more closely match the above suggestion and still same error

those are two diff commands

first, you need to execute

atmos terraform shell <component> -s <stack>

then in the new shell you can execute any native TF command w/o using atmos syntax

Ohhh I see

https://atmos.tools/cli/commands/terraform/shell

@Viacheslav @kevcube please see this release https://github.com/cloudposse/atmos/releases/tag/v1.33.0

@Andriy Knysh (Cloud Posse) Thanks, looks good!

Sorry @Matthew Reggler (cc @Erik Osterman (Cloud Posse)), the published aws-sso component is old and has not been updated. The good news is that yes,

account_name = lookup(module.this.descriptors, "account_name", var.iam_primary_roles_stage_name)

is the way to move forward.

@Linda Pham (Cloud Posse) we should get this component updated this week then.

Heads up, some more updates went out to aws-sso

https://github.com/cloudposse/terraform-aws-components/pull/567

I think there may be more coming

Also, we created refarch so we can move these conversations out of atmos

I will say terragrunt is not needed in this case if you use atmos and I do not think it will even compatible

managing remote state with Atmos is trivial so no need for it

Here is the refarch repo structure for atmos

https://github.com/cloudposse/atmos/tree/master/examples/complete

And atmos.tools documentation.

The stacks/orgs contain the root stacks, stacks/catalogs contain per tf root dir yaml that gets turned into tfvars, and the components/terraform directory that houses all of the terraform root dirs

Pepe is right. Both atmos and terragrunt are terraform wrappers and result in a different structure of your code. I do not think it’s possible to combine them.

Why are you trying to combine them? What’s your use case?

There are other alternative layouts proposed here https://atmos.tools/core-concepts/stacks/catalogs

@Igor I’d be happy to do a screenshare and give you some more ideas

https://calendly.com/cloudposse

Great, this is what I was looking for! We are already using Terragrunt, and we like it. However, over time the directory structure got messy and inconsistent across teams/clouds. So we are in the process of refactoring it. I was hoping to introduce Atmos without removing Terragrunt as an additional layer.

It should be possible. That’s probably the right move as a first step to reduce risk.

Part of our motivation in writing atmos though was to ensure it works with vanilla terraform. I can see why you might want to keep using terragrunt if you’re using a lot of complex terragrunt.hcl files.

There’s a way to specify the terraform command which you would want to overwrite in the atmos.yaml

@Andriy Knysh (Cloud Posse) seems like we didn’t document it here: https://atmos.tools/cli/configuration#components

I believe you would add something like this to the atmos.yaml

components:
  terraform:
    command: /usr/bin/terragrunt

Also, since you are using terragrunt, you’ll probably want to disable the backend autogeneration in atmos

That would certainly simplify the migration! I would like to reduce the effort of initial refactoring by not having to refactor all Terragrunt code and just move the directories/update the links/etc.

Yes, agree

Keep us posted on your progress. If you run into any gotchas, let us know or open an issue. Chances are there’s an easy fix on our side.

I’d love to write up a page for “How to migrate from Terragrunt to Atmos”

Thank you Erik, I will pick a time slot for us to connect next week.

Igor, we need to write more documentation on different atmos topics. if you want us to review your work, you can add us to your repo or DM the repo structure

we designed infra using atmos for many different use cases, would help you with reviewing your design as well

Sounds good! I would love to see some documentation about how to migrate from Terragrunt to Atmos. Especially if it’s possible to keep the original Terragrunt code and even allow adding it. Over time DevOps engineers will get more comfortable with a new Atmos approach but we keeping the old approach (Terragrunt) would simplify the migration. Again, if it only makes sense since we don’t want to create a monster. Our life is already complicated by the fact we need to work across aws/gcp/azure.

let’s start with your repo I’m not super familiar with terragrant, will need something real to look at and start with, then we can write a doc on migrating from terragrant to atmos or how to use them together

also, just a few notes on multi-cloud, multi-org design (I’m planning to write a doc about that as well )

components folder:

separate the logic (code) by the cloud since the code is different

stacks folder:

then let’s take for example org1-core-us1-dev stack:

in the file stacks/orgs/org1/core/dev/us-eat-1.yaml, add this:

just an idea ^

this shows you how you can create different Atmos components and point them to the TF code for diff clouds

of cause, you can create diff orgs or diff tenants dedicated only to a specific cloud. For example, your org1 could work only with AWS, while org2 could work only with GCP, etc.

in which case, you would import the component and create Atmos components in the root stacks only for the specific cloud

similar with tenants inside one org

Yes that would be more common use case - tenants that are limited to one cloud. The main reason for multi-cloud on our side is because companies have a strong preference to working with a specific cloud provider. I also see it as an industry trend. So usually tenants are not replicated across clouds.

this is completely flexible and you can structure it in diff folders and sub-folders (both components and stacks folders)

anything can be in any subfolder at any level

just be consistent

we’ve accumulated a lot of patterns and best practices already, just need to document all of that

Yes the structure of stacks above looks great, it can consistently scale across many tenants/environments.

Just one more question - how do Terraform workspaces relate to this structure? Is it in separate folder per tenant or per environment?

I stand corrected, after reading the posts I think is definitely possible

how do Terraform workspaces relate to this structure? Is it in separate folder per tenant or per environment?

TF workspaces are done automatically by atmos taking into account the stack name (e.g. org1-core-ue1-dev ) and the Atmos component name (e.g. vpc or gcp-component-1)

note, that @Igor already has tfstate. so atmos handles workspaces, but won’t know what to do about the state you have managed by terragrunt. So either you’ll want to disable atmos usage of workspaces for state, or migrate the state. We do not have a process to migrate that state.

Igor let me know when you get to workspaces, I’ll show you how to override them per stack/component to keep the current ones

please show the Atmos component YAML definition in your stack

@Andriy Knysh (Cloud Posse) Sure, this is my mid.yaml stack file:

components:
  helmfile:
    helmfile: 
      vars:
        provider: azure

vars:
  stage: mid

also atmos.yaml contains this component configuration:

base_path: "."

components:
  terraform:
    base_path: "components/terraform"
    apply_auto_approve: false
    deploy_run_init: true
    init_run_reconfigure: true
    auto_generate_backend_file: true
  helmfile:
    base_path: "components/helmfile"
    kubeconfig_path: "~/.kube"
    use_eks: false
    cluster_name_pattern: "{stage}-cluster"

stacks:
  base_path: "stacks"
  included_paths:
    - "**/*"
  name_pattern: "{stage}"

workflows:
  base_path: "workflows"

logs:
  verbose: true
  colors: true

the name helmfile is not a good name for the component, try to name it something meaningful (e.g. what the component does)

try this:

metadata.component: attribute tells Atmos where to find the component’s folder. In this folder Atmos will generate the varfile

so if your helmfile component works from my-component/helmfile.d, you can point Atmos to it (note that for Helmfile to work, a helmfile.yaml should be in the same folder, or a similar file which Helmfile expects as the starting point)

@Viacheslav

@Andriy Knysh (Cloud Posse) mm, got it! Thanks :)

@Andriy Knysh (Cloud Posse) I have additional question - is it possible to use absolute path when run atmos helmfile apply ? So under the hood, instead of

helmfile -e asa -l app=kubeflow-core --state-values-file=somestatefile.yaml apply

be able to run:

helmfile -e asa -l app=kubeflow-core --state-values-file=${PWD}/somestatefile.yaml apply

An idea is to keep root of the repo as working directory, because in case of helmfile.d , helmfile goes to this folder and look for values there, instead of looking in the root folder. It is easy to fix by adding PWD prefix, but is there something that help to do it in Atmos?

not currently. atmos always uses the component’s folder to write files to and as the working dir