#atmos (2022-04)

2022-04-01

Josh Holloway avatar
Josh Holloway

So I’ve been thinking about how Atmos plays with larger infra deployments (read multiple accounts / envs). Typically for AWS, most people have several accounts across their org: • management account (e.g. root) • identity • audit • infrastructure ◦ dev ◦ prd • <org_unit> (e.g. SaaS) ◦ <team> (e.g. payments) ▪︎ dev ▪︎ prd So far, I’ve had the stacks/ dir mirror this layout and then been using tenant to ensure unique statefile paths (e.g. tenant: saas-payments ) e.g.

stacks/
  gbl-root.yaml
  gbl-identity.yaml
  gbl-audit.yaml
  infra/
    gbl-dev.yaml
    ue1-dev.yaml
    gbl-prd.yaml
  saas/
    payments/
      gbl-dev.yaml

But also wondered what others have done so far / might suggest.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

one thing noticeably absent is a catalog/ folder

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

this is where we centralize all the common patterns

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
import:
  - mgmt-uw2-globals
  - catalog/acm
  - catalog/vpc
  - catalog/aws-waf-acl
  - catalog/aws-backup/non-prod
  - catalog/eks/eks
  - catalog/eks/datadog-agent
  - catalog/eks/reloader
  - catalog/eks/ocean-controller
  - catalog/eks/cert-manager
  - catalog/eks/external-dns
  - catalog/eks/alb-controller
  - catalog/eks/alb-controller-ingress-group
  - catalog/eks/metrics-server
  - catalog/eks/aws-node-termination-handler
  - catalog/eks/sumologic
  - catalog/eks/sops-secrets-operator
  - catalog/documentdb/mgmt/sandbox
  - catalog/ses
  - catalog/dynamodb/mgmt/sandbox
  - catalog/s3/alb-access-logs
  - catalog/s3/cloudfront-logs
  - catalog/s3/s3-access-logs
  - catalog/elasticsearch/defaults
  - catalog/elasticache/elasticache-redis-defaults
  - catalog/msk/mgmt/sandbox
  - catalog/aurora/mgmt/sandbox
  - catalog/argocd/non-prod
  - catalog/okta-saml-apps/argocd/non-prod
  - catalog/spa-s3-cloudfront/mgmt/sandbox
  - catalog/bastion/bastion-ssm
  - catalog/maintenance-page
  - catalog/argo-workflows/non-prod


vars:
  stage: sandbox

terraform:
  vars: {}

helmfile:
  vars: {}

components:
  terraform:
    acm:
      vars:
        domain_name: uw2.sandbox.mgmt.acme.net
        zone_name: sandbox.mgmt.acme.net
    vpc:
      vars:
        cidr_block: 10.2.0.0/18
    eks:
      vars:
        # This cluster was created before `cloudposse/eks-cluster/aws` `v0.45.0`
        # See <https://github.com/cloudposse/terraform-aws-eks-cluster/blob/master/docs/migration-0.45.x%2B.md>
        backwards_compatibility_v0_45_0_enabled: true
        spotinst_oceans:
          main:
            max_group_size: 10
            min_group_size: 4
            desired_group_size: 5
        spotinst_ocean_vngs:
          fleet:
            max_group_size: 2
            min_group_size: 2
            instance_types: null
            preferred_spot_types: null
            security_groups: null
            subnet_ids: ['subnet-xxxxx', 'subnet-001955481398e8b77']
            kubelet_additional_options: --allowed-unsafe-sysctls=net.ipv6.conf.all.disable_ipv6,net.ipv6.conf.default.disable_ipv6
            kubernetes_labels:
              eks.acme.net/vng: fleet
            kubernetes_taints:
              net.ipv6.conf.all.disable_ipv6:
                value: 0
                effect: NoExecute
              net.ipv6.conf.default.disable_ipv6:
                value: 0
                effect: NoExecute
          fennec:
            max_group_size: 3
            min_group_size: 2
            instance_types: null
            preferred_spot_types: null
            security_groups: null
            subnet_ids: [ 'subnet-xxx', 'subnet-xxx' ]
            kubelet_additional_options: --allowed-unsafe-sysctls=net.ipv6.conf.all.disable_ipv6,net.ipv6.conf.default.disable_ipv6,net.ipv6.conf.all.forwarding,net.ipv6.conf.default.forwarding
workflows
      vars:
        template_referencing: ''
        workflow_namespaces:
          - argo-workflows-test
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

This is an example of how we import all the defaults we want, and then override what is different.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

WIth this pattern, you can easily define as many accounts and combinations of components as you want, just by using improts from the catalog.

Josh Holloway avatar
Josh Holloway

Oh yeah, sorry I should’ve included the catalog/ directory for completeness. But thanks, this is awesome.

Josh Holloway avatar
Josh Holloway

One thing I noticed with my pattern actually is that the remote-state module doesn’t work because the path to the stack_config_local_path is wrong when I’m working with a stack inside a sub directory.

2022-04-04

Release notes from atmos avatar
Release notes from atmos
05:44:21 PM

v1.4.1 what Add settings.spacelift.stack_name_pattern Fix parsing YAML config and detection of stacks when the stack name (on the command line) is the same as the YAML config file name in a subfolder why settings.spacelift.stack_name_pattern allows overriding Spacelift stack names. Supported tokens: {namespace}, {tenant}, {environment}, {stage}, {component} components: terraform: “test/test-component-override-2”: settings: spacelift: workspace_enabled: true #…

Release v1.4.1 · cloudposse/atmosattachment image

what Add settings.spacelift.stack_name_pattern Fix parsing YAML config and detection of stacks when the stack name (on the command line) is the same as the YAML config file name in a subfolder wh…

Release notes from atmos avatar
Release notes from atmos
02:44:15 AM

v1.4.2 what Add init_run_reconfigure CLI config Update stack_name_pattern Disable running terraform plan and terraform workspace on abstract components why init_run_reconfigure CLI config allows enabling/disabling the -reconfigure argument for terraform init when running it before running other terraform commands Don’t use the default stack_name_pattern because it used {tenant} which is not available for all clients Running terraform plan and terraform workspace on abstract components creates…

Release v1.4.2 · cloudposse/atmosattachment image

what Add init_run_reconfigure CLI config Update stack_name_pattern Disable running terraform plan and terraform workspace on abstract components why init_run_reconfigure CLI config allows enabli…

Release notes from atmos avatar
Release notes from atmos
03:04:15 AM

v1.4.2 what Add init_run_reconfigure CLI config Update stack_name_pattern Disable running terraform plan and terraform workspace on abstract components why init_run_reconfigure CLI config allows enabling/disabling the -reconfigure argument for terraform init when running it before running other terraform commands Don’t use the default stack_name_pattern because it used {tenant} which is not available for all clients Running terraform plan and terraform workspace on abstract components creates…

2022-04-05

2022-04-11

Zach Bridges avatar
Zach Bridges

hey yall! hope everyone is doing well. I have run into problems with atmos reading really large templates. i tried to read through the code to see if i could fix it but atmos v0.22 is written in variant and is not that amenable to being changed (the problem lies in a dependency’s dependency).

as a workaround I am wondering if there is an alternative in recent versions of atmos to the atmos v0.22 command atmos stack config --config-type all it looks like you can get the config per component but not per stack.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

we are working on new atmos commands (in the new atmos, not in the variant version) to show all the config for all stacks and components)

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

we’ll post here when it’s done

1
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

(we are not supporting the old atmos written in variant, if you are using it, please upgrade to the new one)

Zach Bridges avatar
Zach Bridges

thanks so much for the quick reply!

Release notes from atmos avatar
Release notes from atmos
02:04:18 PM

v1.4.3 what Add –dry-run command-line flag to all commands and workflows why Helps debugging atmos commands and workflows The –dry-run flag shows all the flow and commands without executing them and without writing files to the file system (e.g. varfiles and backend config are not written) The –dry-run flag shows all the workflow steps without executing them test atmos terraform plan test/test-component-override -s=tenant1-ue2-dev –dry-run

Variables for the component…

Release v1.4.3 · cloudposse/atmosattachment image

what Add –dry-run command-line flag to all commands and workflows why Helps debugging atmos commands and workflows The –dry-run flag shows all the flow and commands without executing them and …

2022-04-12

Release notes from atmos avatar
Release notes from atmos
08:24:18 PM

v1.4.4 what Add atmos describe stacks command Allow writing the result to a file by using –file command-line flag Allow formatting the result as YAML or JSON by using –format command-line flag Allow filtering of the result by using the command-line flags: stack, components-types, components, sections Available component sections: backend, backend_type, deps, env, inheritance, metadata, remote_state_backend, remote_state_backend_type, settings, vars why Command to show stack configs and all the…

Release v1.4.4 · cloudposse/atmosattachment image

what Add atmos describe stacks command Allow writing the result to a file by using –file command-line flag Allow formatting the result as YAML or JSON by using –format command-line flag Allow fi…

cool-doge1
1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@here this is awesome! you can now query your stack configurations

2
Release notes from atmos avatar
Release notes from atmos
09:04:18 PM

v1.4.4 what Add atmos describe stacks command Allow writing the result to a file by using –file command-line flag Allow formatting the result as YAML or JSON by using –format command-line flag Allow filtering of the result by using the command-line flags: stack, component-types, components, sections Available component sections: backend, backend_type, deps, env, inheritance, metadata, remote_state_backend, remote_state_backend_type, settings, vars why Command to show stack configs and all the…

Release v1.4.4 · cloudposse/atmosattachment image

what Add atmos describe stacks command Allow writing the result to a file by using –file command-line flag Allow formatting the result as YAML or JSON by using –format command-line flag Allow fi…

2022-04-13

Michael Dizon avatar
Michael Dizon

is there a way to create state information for resources (Like an aws org account) that already exist (and may have been created outside of TF)?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@Michael Dizon in atmos YAML config?

Michael Dizon avatar
Michael Dizon

i can just set the variables and it’ll just work?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

you can set any variables, if they are supported by your module

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

you can also use the static backend remote_state_backend

this1
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

where you just set the variables for the backend as if they were loaded from s3 backend

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yep, this static backend is probably what you want. We implemented it for this similar use-case where we couldn’t fully adopt an existing account into the organization, but we wanted to adopt it into our stack configurations.

Michael Dizon avatar
Michael Dizon

would i be able to migrate from the static backend to s3 (tfstate-backend)?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

you can always change the backend_type and backend settings for a component. If the state is in S3, change it to s3 and provide the bucket and DynamoDB config. If the state is not in s3 and you are using an already provisioned resource, use static backend

Michael Dizon avatar
Michael Dizon

awesome, i’ll give this a go in the morn

Michael Dizon avatar
Michael Dizon

circling back on this, i was able to use terraform import to achieve my goal. i may have miscommunicated my intended needs

1
Release notes from atmos avatar
Release notes from atmos
04:24:15 AM

v1.4.5 what Fix detection of component dependencies for imported YAML config files why If a file was imported and did not contain a vars section, global or related to the component, the imported file was not included in the component dependencies (deps section), and a Spacelift label for the file was not created (if the imported file changes, Spacelift would not notice the change and would not run the stack) Update the dependencies logic to check for these sections: backend, backend_type, env,…

Release v1.4.5 · cloudposse/atmosattachment image

what Fix detection of component dependencies for imported YAML config files why If a file was imported and did not contain a vars section, global or related to the component, the imported file w…

2022-04-14

Release notes from atmos avatar
Release notes from atmos
12:04:16 AM

v1.4.6 what If auto_generate_backend_file is true (we are auto-generating backend files), remove backend.tf.json when executing atmos terraform clean command why Useful when using different backends for the same component in different stacks

Release v1.4.6 · cloudposse/atmosattachment image

what If auto_generate_backend_file is true (we are auto-generating backend files), remove backend.tf.json when executing atmos terraform clean command why Useful when using different backends fo…

2022-04-15

Release notes from atmos avatar
Release notes from atmos
02:34:18 AM

v1.4.7 what Take into account init_run_reconfigure CLI config and –init-run-reconfigure command-line argument when running atmos terraform init … command why atmos terraform init must behave the same as all other commands that use init_run_reconfigure CLI config and –init-run-reconfigure command-line argument

Release v1.4.7 · cloudposse/atmos

what Take into account init_run_reconfigure CLI config and –init-run-reconfigure command-line argument when running atmos terraform init … command why atmos terraform init must behave the sam…

2022-04-16

2022-04-18

Release notes from atmos avatar
Release notes from atmos
03:14:16 PM

v1.4.8 what Add metadata.terraform_workspace_pattern why We already have metadata.terraform_workspace using which we can specify/override the terraform workspace for a component metadata.terraform_workspace_pattern introduces a pattern to override a terraform workspace for a component metadata: # Override Terraform workspace terraform_workspace: xxxxxxxxxxxxxxx terraform_workspace_pattern: “{tenant}-{environment}-{stage}-{component}”

The following tokens are supported…

Release v1.4.8 · cloudposse/atmosattachment image

what Add metadata.terraform_workspace_pattern why We already have metadata.terraform_workspace using which we can specify/override the terraform workspace for a component metadata.terraform_work…

2022-04-21

Release notes from atmos avatar
Release notes from atmos
10:54:16 PM

v1.4.9 what Add atmos aws commands Add atmos aws eks update-kubeconfig command why Execute aws CLI commands using atmos context (atmos.yaml CLI config and component/stack configurations) Downloading kubeconfig from an EKS cluster using `atmos Allow using this functionality in Terraform by implementing a new data source in the terraform-provider-utils provider (this will be added in a PR in <a…

Release v1.4.9 · cloudposse/atmosattachment image

what Add atmos aws commands Add atmos aws eks update-kubeconfig command why Execute aws CLI commands using atmos context (atmos.yaml CLI config and component/stack configurations) Downloading ku…

2022-04-22

Release notes from atmos avatar
Release notes from atmos
02:54:17 PM

v1.4.10 what If functions that are used by Terraform providers throw errors, print errors to std.Error why Terraform providers only see errors that are sent to std.Error

Release v1.4.10 · cloudposse/atmosattachment image

what If functions that are used by Terraform providers throw errors, print errors to std.Error why Terraform providers only see errors that are sent to std.Error

Matt Gowie avatar
Matt Gowie

Another Atmos in the terraform space — https://github.com/simplygenius/atmos

Just heard about it on a call with a company. Seems fairly dead and not all the interesting of a project, but it is funny that name would get chosen twice in the TF space.

simplygenius/atmos

Breathe easier with terraform. Cloud system architectures made easy

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yes, it appears to be on life support.

simplygenius/atmos

Breathe easier with terraform. Cloud system architectures made easy

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think the company behind it was acquired by CloudTruth

Matt Gowie avatar
Matt Gowie

Ah do you know the CloudTruth folks? They’re who mentioned it to me.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
How does terraforming work in the Alien universe?attachment image

Terraforming a planet and building better worlds is not as easy as it seems.

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)


“Automated Atmosphere Processor”

How does terraforming work in the Alien universe?attachment image

Terraforming a planet and building better worlds is not as easy as it seems.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Was called atmos for short

Andy Miguel (Cloud Posse) avatar
Andy Miguel (Cloud Posse)

I love part 2, Aliens. the rest of the series gets questionable from there.

Michael Dizon avatar
Michael Dizon

i remember reading something about nuking the martian poles to create at atmosphere

Michael Dizon avatar
Michael Dizon

atmos > nuke

Release notes from atmos avatar
Release notes from atmos
10:54:16 PM

v1.4.11 what Support dashes - in the tenant, environment and stage names In the examples, add a new stage test-1 and add tests for components, stacks, and spacelift to test having a dash in the stage name (the file name itself being without dashes) why The old atmos supported it (because it was filename-based, not logical stack name based) Some clients want to name tenants/environment/stages with dashes in the names (and some already have it, so we need to support that when converting from the old to…

Release v1.4.11 · cloudposse/atmosattachment image

what Support dashes - in the tenant, environment and stage names In the examples, add a new stage test-1 and add tests for components, stacks, and spacelift to test having a dash in the stage name…

2022-04-25

Release notes from atmos avatar
Release notes from atmos
05:14:22 PM

v1.4.3 what Add –dry-run command-line flag to all commands and workflows why Helps debugging atmos commands and workflows The –dry-run flag shows all the flow and commands without executing them and without writing files to the file system (e.g. varfiles and backend config are not written) The –dry-run flag shows all the workflow steps without executing them tests and examples Click to expandatmos terraform plan test/test-component-override -s=tenant1-ue2-dev –dry-run

Variables for the…

Release v1.4.3 · cloudposse/atmosattachment image

what Add –dry-run command-line flag to all commands and workflows why Helps debugging atmos commands and workflows The –dry-run flag shows all the flow and commands without executing them and …

Release notes from atmos avatar
Release notes from atmos
05:14:22 PM

v1.4.4 what Add atmos describe stacks command Allow writing the result to a file by using –file command-line flag Allow formatting the result as YAML or JSON by using –format command-line flag Allow filtering of the result by using the command-line flags: stack, component-types, components, sections Available component sections: backend, backend_type, deps, env, inheritance, metadata, remote_state_backend, remote_state_backend_type, settings, vars why Command to show stack configs and all the…

Release v1.4.4 · cloudposse/atmos

what Add atmos describe stacks command Allow writing the result to a file by using –file command-line flag Allow formatting the result as YAML or JSON by using –format command-line flag Allow fi…

2022-04-29

Chris Bloemker avatar
Chris Bloemker

Good morning all! We are trying to add a new mono repo for our SRE team to Spacelift with Atmos and it seems like Spacelift can’t find the stacks based on base_path

$ atmos describe config | jq '.Components.Terraform.base_path'
"/components/terraform"
$ atmos describe config | jq '.Stacks.base_path'
"/stacks"

Spacelift run error:

No stack config files found in the provided paths:
│ - /mnt/workspace/source/components/terraform/spacelift/stacks/**/*

Our directory structure is the same as the atmos example here where we have at the root of the repo:

components
stacks
atmos.yaml

Trying to see if there is another variable overriding stacks.base_path somewhere else.

RB (Ronak) (Cloud Posse) avatar
RB (Ronak) (Cloud Posse)

for spacelift, try setting ATMOS_BASE_PATH=/mnt/workspace/source

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

in .spacelift/config.yml in the repo, we have:

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
# <https://docs.spacelift.io/concepts/runtime-configuration>
version: "1"

stack_defaults:
  before_init:
    - spacelift-configure-paths
    - spacelift-git-use-https
    - spacelift-write-vars
    - spacelift-tf-workspace

  before_plan:
    - spacelift-configure-paths

  before_apply:
    - spacelift-configure-paths

  environment:
    AWS_SDK_LOAD_CONFIG: true
    AWS_CONFIG_FILE: /etc/aws-config/aws-config-cicd
    AWS_PROFILE: eg-gbl-identity
    ATMOS_BASE_PATH: /mnt/workspace/source

stacks:
  infrastructure:
    before_init: []
    before_plan: []
    before_apply: []
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

the important part is ATMOS_BASE_PATH: /mnt/workspace/source

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

also, in rootfs/usr/local/etc/atmos/atmos.yaml (which ends up in usr/local/etc/atmos/atmos.yamlin the Docker image that Spacelift is using to execute commands), we have:

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
# CLI config is loaded from the following locations (from lowest to highest priority):
# system dir (`/usr/local/etc/atmos` on Linux, `%LOCALAPPDATA%/atmos` on Windows)
# home dir (~/.atmos)
# current directory
# ENV vars
# Command-line arguments
#
# It supports POSIX-style Globs for file names/paths (double-star `**` is supported)
# <https://en.wikipedia.org/wiki/Glob_(programming)>

# Base path for components and stacks configurations.
# Can also be set using `ATMOS_BASE_PATH` ENV var, or `--base-path` command-line argument.
# Supports both absolute and relative paths.
# If not provided or is an empty string, `components.terraform.base_path`, `components.helmfile.base_path` and `stacks.base_path`
# are independent settings (supporting both absolute and relative paths).
# If `base_path` is provided, `components.terraform.base_path`, `components.helmfile.base_path` and `stacks.base_path`
# are considered paths relative to `base_path`.
base_path: ""

components:
  # Settings for all terraform components
  terraform:
    # Can also be set using `ATMOS_COMPONENTS_TERRAFORM_BASE_PATH` ENV var, or `--terraform-dir` command-line argument
    # Supports both absolute and relative paths
    base_path: "components/terraform"
    # Can also be set using `ATMOS_COMPONENTS_TERRAFORM_APPLY_AUTO_APPROVE` ENV var
    apply_auto_approve: false
    # Can also be set using `ATMOS_COMPONENTS_TERRAFORM_DEPLOY_RUN_INIT` ENV var, or `--deploy-run-init` command-line argument
    deploy_run_init: true
    # Can also be set using `ATMOS_COMPONENTS_TERRAFORM_INIT_RUN_RECONFIGURE` ENV var, or `--init-run-reconfigure` command-line argument
    init_run_reconfigure: false
    # Can also be set using `ATMOS_COMPONENTS_TERRAFORM_AUTO_GENERATE_BACKEND_FILE` ENV var, or `--auto-generate-backend-file` command-line argument
    auto_generate_backend_file: true

  # Settings for all helmfile components
  helmfile:
    # Can also be set using `ATMOS_COMPONENTS_HELMFILE_BASE_PATH` ENV var, or `--helmfile-dir` command-line argument
    # Supports both absolute and relative paths
    base_path: "components/helmfile"
    # Can also be set using `ATMOS_COMPONENTS_HELMFILE_KUBECONFIG_PATH` ENV var
    kubeconfig_path: "/dev/shm"
    # Can also be set using `ATMOS_COMPONENTS_HELMFILE_HELM_AWS_PROFILE_PATTERN` ENV var
    helm_aws_profile_pattern: "{namespace}-gbl-{stage}-helm"
    # Can also be set using `ATMOS_COMPONENTS_HELMFILE_CLUSTER_NAME_PATTERN` ENV var
    cluster_name_pattern: "{namespace}-{environment}-{stage}-eks-cluster"

# Settings for all stacks
stacks:
  # Can also be set using `ATMOS_STACKS_BASE_PATH` ENV var, or `--config-dir` and `--stacks-dir` command-line arguments
  # Supports both absolute and relative paths
  base_path: "stacks"
  # Can also be set using `ATMOS_STACKS_INCLUDED_PATHS` ENV var (comma-separated values string)
  included_paths:
    - "**/*"
  # Can also be set using `ATMOS_STACKS_EXCLUDED_PATHS` ENV var (comma-separated values string)
  excluded_paths:
    - "globals/**/*"
    - "**/*globals*"
    - "catalog/**/*"
    # exclude workflows
    - "workflows/**/*"
  # Can also be set using `ATMOS_STACKS_NAME_PATTERN` ENV var
  name_pattern: "{environment}-{stage}"

# <https://github.com/cloudposse/atmos/releases/tag/v1.4.0>
workflows:
  # Can also be set using `ATMOS_WORKFLOWS_BASE_PATH` ENV var, or `--workflows-dir` command-line arguments
  # Supports both absolute and relative paths
  base_path: "stacks/workflows"

logs:
  verbose: false
  colors: true
Chris Bloemker avatar
Chris Bloemker

oh interesting, I was missing the workflow base path var .spacelift/config.yml I was just using our platform_infrastructure repo as copy pasta but we aren’t using atmos latest there. Trying that now.

So essentially the rootfs atmos.yaml needs to be identical as the repo root atmos.yaml ?

Chris Bloemker avatar
Chris Bloemker

Now my error is Error: stack name pattern must be provided in 'stacks.name_pattern' config or 'ATMOS_STACKS_NAME_PATTERN' ENV variable In both of my atmos configs, stacks.name_pattern == "{tenant}-{environment}-{stage}" This error makes it seem like it’s not being set?

RB (Ronak) (Cloud Posse) avatar
RB (Ronak) (Cloud Posse)

what’s your atmos command that is failing

RB (Ronak) (Cloud Posse) avatar
RB (Ronak) (Cloud Posse)

are you using tenant

RB (Ronak) (Cloud Posse) avatar
RB (Ronak) (Cloud Posse)

if not, you can modify the pattern to only contain the environment and stage

Chris Bloemker avatar
Chris Bloemker

I’m not sure exactly which atmos command spacelift is doing here. This file stacks/globals/datadog_slo-globals-dev.yaml has the following vars.

vars:
  namespace: bd
  region: us-east-2
  tenant: datadog_slo
  environment: global
RB (Ronak) (Cloud Posse) avatar
RB (Ronak) (Cloud Posse)

oh so you are using tenant, then id keep the pattern the same.

from that config the stack name should be datadog_slo-global-<stage> but youre missing the stage name

Chris Bloemker avatar
Chris Bloemker

and that stage just needs to be defined in the that vars block?

1
RB (Ronak) (Cloud Posse) avatar
RB (Ronak) (Cloud Posse)

try this as an example

 vars:
   namespace: bd
   region: us-east-2
   tenant: datadog_slo
   environment: global
   stage: banana
RB (Ronak) (Cloud Posse) avatar
RB (Ronak) (Cloud Posse)

now the stack name will be datadog_slo-global-banana

RB (Ronak) (Cloud Posse) avatar
RB (Ronak) (Cloud Posse)

usually we use stage to be the account name

RB (Ronak) (Cloud Posse) avatar
RB (Ronak) (Cloud Posse)

so prod, dev, staging, sandbox, qa, or similar

Chris Bloemker avatar
Chris Bloemker

I’m taking over a project from someone that has been out and I don’t have much context on how these directory structures worked getting this new project going

1
Chris Bloemker avatar
Chris Bloemker

trying those vars now

1
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

If you are using geodesic, you don’t need atmos.yaml in the root of the repo. Both geodesic and Spacelift should take it from the rootfs

Chris Bloemker avatar
Chris Bloemker

We are using geodesic here. So I can remove the repo’s root atmost.yaml

Chris Bloemker avatar
Chris Bloemker

Hmm. still getting the same error

╷
│ Error: stack name pattern must be provided in 'stacks.name_pattern' config or 'ATMOS_STACKS_NAME_PATTERN' ENV variable
│ 
│   with module.spacelift.module.yaml_stack_config.data.utils_spacelift_stack_config.spacelift_stacks,
│   on .terraform/modules/spacelift.yaml_stack_config/modules/spacelift/main.tf line 1, in data "utils_spacelift_stack_config" "spacelift_stacks":
│    1: data "utils_spacelift_stack_config" "spacelift_stacks" {

This platform_sre is an Administrative stack if that gives more contexts, and currently the regular stacks are not shown.

RB (Ronak) (Cloud Posse) avatar
RB (Ronak) (Cloud Posse)

it would be easier to test this locally before running it in space lift

RB (Ronak) (Cloud Posse) avatar
RB (Ronak) (Cloud Posse)

get the atmos commands to work locally and then spacelift will be easier to troubleshoot

Chris Bloemker avatar
Chris Bloemker

Good point. I do recall this working locally in geodesic.. but seems my changes now it is not working

 √ . [bd-gbl-identity] / ⨠ atmos-local terraform plan -s datadog_slo-global-dev "datadog/sre-slo"
/localhost/git/bread/platform_sre /
Found ENV var ATMOS_STACKS_BASE_PATH=/localhost/git/bread/platform_sre/stacks

Could not find config for the component 'datadog/sre-slo' in the stack 'datadog_slo-global-dev'.
Check that all attributes in the stack name pattern '{tenant}-{environment}-{stage}' are defined in the stack config files.
Are the component and stack names correct? Did you forget an import?
Chris Bloemker avatar
Chris Bloemker

This is our directory tree of stacks

stacks
|-- datadog_slo
|   `-- global
|       `-- dev.yaml
|-- globals
|   |-- datadog_slo-globals-dev.yaml
|   `-- globals.yaml
`-- spacelift
    `-- global
        `-- dev.yaml
RB (Ronak) (Cloud Posse) avatar
RB (Ronak) (Cloud Posse)

the file structure doesn’t matter, it’s all about how the inputs in the yaml

RB (Ronak) (Cloud Posse) avatar
RB (Ronak) (Cloud Posse)

is your component datadog/sre-slo in the datadog_slo-global-dev stack ?

RB (Ronak) (Cloud Posse) avatar
RB (Ronak) (Cloud Posse)

can you revert your atmos.yaml file and get it to work again ?

Chris Bloemker avatar
Chris Bloemker

I reverted back on my branch that was working. and now I’m trying to do diffs and figure out what broke the local atmos run atmos-local terraform plan -s datadog_slo-global-dev "datadog/sre-slo" seems to be working now.

Chris Bloemker avatar
Chris Bloemker

my stacks/globals/datadog_slo-globals-dev.yaml is as follows:

vars:
  namespace: bd
  region: us-east-2
  tenant: datadog_slo
  environment: global

terraform:
  vars: {}

components:
  terraform:
    "datadog/sre-slo":
      vars:
        gh_org: getbread
Chris Bloemker avatar
Chris Bloemker

okay so I figured out what is breaking my local atmos.

when I add the following to stacks/spacelift/global/dev.yaml

vars:
  namespace: bd
  region: us-east-2
  tenant: spacelift
  environment: global
  stage: banana

terraform:
  vars: {}

I get this error. I think I am misunderstanding the relationship between the files/variables.

 √ . [bd-gbl-identity] / ⨠ atmos-local terraform plan -s datadog_slo-global-dev "datadog/sre-slo"
/localhost/git/bread/platform_sre /
Found ENV var ATMOS_STACKS_BASE_PATH=/localhost/git/bread/platform_sre/stacks

Could not find config for the component 'datadog/sre-slo' in the stack 'datadog_slo-global-dev'.
Check that all attributes in the stack name pattern '{tenant}-{environment}-{stage}' are defined in the stack config files.
Are the component and stack names correct? Did you forget an import?
RB (Ronak) (Cloud Posse) avatar
RB (Ronak) (Cloud Posse)

the file names are independent of the stacks themselves

RB (Ronak) (Cloud Posse) avatar
RB (Ronak) (Cloud Posse)

i was suggesting to append your vars with stage: whatever, but i think you created a new file side by side that may have duplicated the existing file which probably confused atmos

Chris Bloemker avatar
Chris Bloemker

Picking back up here on this issue. I have atmos-local running fine in geodesic. We are exec’d into the spacelift runner

ash-5.1$ pwd
/
bash-5.1$ find . -type f -executable | grep atmos
./mnt/workspace/source/atmos.yaml
./mnt/workspace/source/rootfs/etc/profile.d/atmos-local.sh
find: ./proc/tty/driver: Permission denied
./mnt/workspace/source/rootfs/usr/local/etc/atmos/atmos.yaml
find: ./root: Permission denied
bash-5.1$ cat /mnt/workspace/source/rootfs/etc/profile.d/atmos-local.sh
export LOCAL_PREFIX=/localhost${SRC_RELATIVE_PATH}
function atmos-local() {
  pushd $LOCAL_PREFIX
  ATMOS_STACKS_BASE_PATH=$LOCAL_PREFIX/stacks atmos [email protected]
  popd
}

Looks like it’s a wrapper here? I can’t find atmos binary anywhere in that image. How would we debug this trying to running atmos locally in the spacelift runner? All we can see is atmos is making its way into the provider here? Or is spacelift running a custom terraform command here?

	github.com/cloudposse/atmos v1.4.11
    keyboard_arrow_up