#atmos (2022-04)
2022-04-01
![Josh Holloway avatar](https://avatars.slack-edge.com/2022-03-30/3316196452386_d649b552e955a3153734_72.jpg)
So I’ve been thinking about how Atmos plays with larger infra deployments (read multiple accounts / envs).
Typically for AWS, most people have several accounts across their org:
• management account (e.g. root)
• identity
• audit
• infrastructure
◦ dev
◦ prd
• <org_unit> (e.g. SaaS)
◦ <team> (e.g. payments)
▪︎ dev
▪︎ prd
So far, I’ve had the stacks/
dir mirror this layout and then been using tenant
to ensure unique statefile paths (e.g. tenant: saas-payments
)
e.g.
stacks/
gbl-root.yaml
gbl-identity.yaml
gbl-audit.yaml
infra/
gbl-dev.yaml
ue1-dev.yaml
gbl-prd.yaml
saas/
payments/
gbl-dev.yaml
But also wondered what others have done so far / might suggest.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
one thing noticeably absent is a catalog/
folder
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
this is where we centralize all the common patterns
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
import:
- mgmt-uw2-globals
- catalog/acm
- catalog/vpc
- catalog/aws-waf-acl
- catalog/aws-backup/non-prod
- catalog/eks/eks
- catalog/eks/datadog-agent
- catalog/eks/reloader
- catalog/eks/ocean-controller
- catalog/eks/cert-manager
- catalog/eks/external-dns
- catalog/eks/alb-controller
- catalog/eks/alb-controller-ingress-group
- catalog/eks/metrics-server
- catalog/eks/aws-node-termination-handler
- catalog/eks/sumologic
- catalog/eks/sops-secrets-operator
- catalog/documentdb/mgmt/sandbox
- catalog/ses
- catalog/dynamodb/mgmt/sandbox
- catalog/s3/alb-access-logs
- catalog/s3/cloudfront-logs
- catalog/s3/s3-access-logs
- catalog/elasticsearch/defaults
- catalog/elasticache/elasticache-redis-defaults
- catalog/msk/mgmt/sandbox
- catalog/aurora/mgmt/sandbox
- catalog/argocd/non-prod
- catalog/okta-saml-apps/argocd/non-prod
- catalog/spa-s3-cloudfront/mgmt/sandbox
- catalog/bastion/bastion-ssm
- catalog/maintenance-page
- catalog/argo-workflows/non-prod
vars:
stage: sandbox
terraform:
vars: {}
helmfile:
vars: {}
components:
terraform:
acm:
vars:
domain_name: uw2.sandbox.mgmt.acme.net
zone_name: sandbox.mgmt.acme.net
vpc:
vars:
cidr_block: 10.2.0.0/18
eks:
vars:
# This cluster was created before `cloudposse/eks-cluster/aws` `v0.45.0`
# See <https://github.com/cloudposse/terraform-aws-eks-cluster/blob/master/docs/migration-0.45.x%2B.md>
backwards_compatibility_v0_45_0_enabled: true
spotinst_oceans:
main:
max_group_size: 10
min_group_size: 4
desired_group_size: 5
spotinst_ocean_vngs:
fleet:
max_group_size: 2
min_group_size: 2
instance_types: null
preferred_spot_types: null
security_groups: null
subnet_ids: ['subnet-xxxxx', 'subnet-001955481398e8b77']
kubelet_additional_options: --allowed-unsafe-sysctls=net.ipv6.conf.all.disable_ipv6,net.ipv6.conf.default.disable_ipv6
kubernetes_labels:
eks.acme.net/vng: fleet
kubernetes_taints:
net.ipv6.conf.all.disable_ipv6:
value: 0
effect: NoExecute
net.ipv6.conf.default.disable_ipv6:
value: 0
effect: NoExecute
fennec:
max_group_size: 3
min_group_size: 2
instance_types: null
preferred_spot_types: null
security_groups: null
subnet_ids: [ 'subnet-xxx', 'subnet-xxx' ]
kubelet_additional_options: --allowed-unsafe-sysctls=net.ipv6.conf.all.disable_ipv6,net.ipv6.conf.default.disable_ipv6,net.ipv6.conf.all.forwarding,net.ipv6.conf.default.forwarding
workflows
vars:
template_referencing: ''
workflow_namespaces:
- argo-workflows-test
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
This is an example of how we import all the defaults we want, and then override what is different.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
WIth this pattern, you can easily define as many accounts and combinations of components as you want, just by using improts from the catalog.
![Josh Holloway avatar](https://avatars.slack-edge.com/2022-03-30/3316196452386_d649b552e955a3153734_72.jpg)
Oh yeah, sorry I should’ve included the catalog/
directory for completeness. But thanks, this is awesome.
![Josh Holloway avatar](https://avatars.slack-edge.com/2022-03-30/3316196452386_d649b552e955a3153734_72.jpg)
One thing I noticed with my pattern actually is that the remote-state
module doesn’t work because the path to the stack_config_local_path
is wrong when I’m working with a stack inside a sub directory.
2022-04-04
![Release notes from atmos avatar](https://a.slack-edge.com/80588/img/services/rss_72.png)
v1.4.1 what Add settings.spacelift.stack_name_pattern Fix parsing YAML config and detection of stacks when the stack name (on the command line) is the same as the YAML config file name in a subfolder why settings.spacelift.stack_name_pattern allows overriding Spacelift stack names. Supported tokens: {namespace}, {tenant}, {environment}, {stage}, {component} components: terraform: “test/test-component-override-2”: settings: spacelift: workspace_enabled: true #…
what Add settings.spacelift.stack_name_pattern Fix parsing YAML config and detection of stacks when the stack name (on the command line) is the same as the YAML config file name in a subfolder wh…
![Release notes from atmos avatar](https://a.slack-edge.com/80588/img/services/rss_72.png)
v1.4.2 what Add init_run_reconfigure CLI config Update stack_name_pattern Disable running terraform plan and terraform workspace on abstract components why init_run_reconfigure CLI config allows enabling/disabling the -reconfigure argument for terraform init when running it before running other terraform commands Don’t use the default stack_name_pattern because it used {tenant} which is not available for all clients Running terraform plan and terraform workspace on abstract components creates…
what Add init_run_reconfigure CLI config Update stack_name_pattern Disable running terraform plan and terraform workspace on abstract components why init_run_reconfigure CLI config allows enabli…
![Release notes from atmos avatar](https://a.slack-edge.com/80588/img/services/rss_72.png)
v1.4.2 what Add init_run_reconfigure CLI config Update stack_name_pattern Disable running terraform plan and terraform workspace on abstract components why init_run_reconfigure CLI config allows enabling/disabling the -reconfigure argument for terraform init when running it before running other terraform commands Don’t use the default stack_name_pattern because it used {tenant} which is not available for all clients Running terraform plan and terraform workspace on abstract components creates…
2022-04-05
2022-04-11
![Zach Bridges avatar](https://avatars.slack-edge.com/2022-04-11/3376078893556_cdcada3ce1fe71328fc7_72.jpg)
hey yall! hope everyone is doing well. I have run into problems with atmos reading really large templates. i tried to read through the code to see if i could fix it but atmos v0.22 is written in variant and is not that amenable to being changed (the problem lies in a dependency’s dependency).
as a workaround I am wondering if there is an alternative in recent versions of atmos to the atmos v0.22 command atmos stack config --config-type all
it looks like you can get the config per component but not per stack.
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
we are working on new atmos commands (in the new atmos, not in the variant version) to show all the config for all stacks and components)
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
(we are not supporting the old atmos written in variant, if you are using it, please upgrade to the new one)
![Zach Bridges avatar](https://avatars.slack-edge.com/2022-04-11/3376078893556_cdcada3ce1fe71328fc7_72.jpg)
thanks so much for the quick reply!
![Release notes from atmos avatar](https://a.slack-edge.com/80588/img/services/rss_72.png)
v1.4.3 what Add –dry-run command-line flag to all commands and workflows why Helps debugging atmos commands and workflows The –dry-run flag shows all the flow and commands without executing them and without writing files to the file system (e.g. varfiles and backend config are not written) The –dry-run flag shows all the workflow steps without executing them test atmos terraform plan test/test-component-override -s=tenant1-ue2-dev –dry-run
Variables for the component…
what Add –dry-run command-line flag to all commands and workflows why Helps debugging atmos commands and workflows The –dry-run flag shows all the flow and commands without executing them and …
2022-04-12
![Release notes from atmos avatar](https://a.slack-edge.com/80588/img/services/rss_72.png)
v1.4.4 what Add atmos describe stacks command Allow writing the result to a file by using –file command-line flag Allow formatting the result as YAML or JSON by using –format command-line flag Allow filtering of the result by using the command-line flags: stack, components-types, components, sections Available component sections: backend, backend_type, deps, env, inheritance, metadata, remote_state_backend, remote_state_backend_type, settings, vars why Command to show stack configs and all the…
what Add atmos describe stacks command Allow writing the result to a file by using –file command-line flag Allow formatting the result as YAML or JSON by using –format command-line flag Allow fi…
![cool-doge](/assets/images/custom_emojis/cool-doge.gif)
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
@here this is awesome! you can now query your stack configurations
![Release notes from atmos avatar](https://a.slack-edge.com/80588/img/services/rss_72.png)
v1.4.4 what Add atmos describe stacks command Allow writing the result to a file by using –file command-line flag Allow formatting the result as YAML or JSON by using –format command-line flag Allow filtering of the result by using the command-line flags: stack, component-types, components, sections Available component sections: backend, backend_type, deps, env, inheritance, metadata, remote_state_backend, remote_state_backend_type, settings, vars why Command to show stack configs and all the…
what Add atmos describe stacks command Allow writing the result to a file by using –file command-line flag Allow formatting the result as YAML or JSON by using –format command-line flag Allow fi…
2022-04-13
![Michael Dizon avatar](https://avatars.slack-edge.com/2021-01-15/1664383757488_b5214d00b8fce4726a7c_72.jpg)
is there a way to create state information for resources (Like an aws org account) that already exist (and may have been created outside of TF)?
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
@Michael Dizon in atmos
YAML config?
![Michael Dizon avatar](https://avatars.slack-edge.com/2021-01-15/1664383757488_b5214d00b8fce4726a7c_72.jpg)
i can just set the variables and it’ll just work?
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
you can set any variables, if they are supported by your module
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
remote_state_backend:
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
where you just set the variables for the backend as if they were loaded from s3 backend
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Yep, this static backend is probably what you want. We implemented it for this similar use-case where we couldn’t fully adopt an existing account into the organization, but we wanted to adopt it into our stack configurations.
![Michael Dizon avatar](https://avatars.slack-edge.com/2021-01-15/1664383757488_b5214d00b8fce4726a7c_72.jpg)
would i be able to migrate from the static backend to s3 (tfstate-backend)?
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
you can always change the backend_type
and backend settings for a component. If the state is in S3, change it to s3 and provide the bucket and DynamoDB config. If the state is not in s3 and you are using an already provisioned resource, use static
backend
![Michael Dizon avatar](https://avatars.slack-edge.com/2021-01-15/1664383757488_b5214d00b8fce4726a7c_72.jpg)
awesome, i’ll give this a go in the morn
![Michael Dizon avatar](https://avatars.slack-edge.com/2021-01-15/1664383757488_b5214d00b8fce4726a7c_72.jpg)
circling back on this, i was able to use terraform import to achieve my goal. i may have miscommunicated my intended needs
![Release notes from atmos avatar](https://a.slack-edge.com/80588/img/services/rss_72.png)
v1.4.5 what Fix detection of component dependencies for imported YAML config files why If a file was imported and did not contain a vars section, global or related to the component, the imported file was not included in the component dependencies (deps section), and a Spacelift label for the file was not created (if the imported file changes, Spacelift would not notice the change and would not run the stack) Update the dependencies logic to check for these sections: backend, backend_type, env,…
what Fix detection of component dependencies for imported YAML config files why If a file was imported and did not contain a vars section, global or related to the component, the imported file w…
2022-04-14
![Release notes from atmos avatar](https://a.slack-edge.com/80588/img/services/rss_72.png)
v1.4.6 what If auto_generate_backend_file is true (we are auto-generating backend files), remove backend.tf.json when executing atmos terraform clean command why Useful when using different backends for the same component in different stacks
what If auto_generate_backend_file is true (we are auto-generating backend files), remove backend.tf.json when executing atmos terraform clean command why Useful when using different backends fo…
2022-04-15
![Release notes from atmos avatar](https://a.slack-edge.com/80588/img/services/rss_72.png)
v1.4.7 what Take into account init_run_reconfigure CLI config and –init-run-reconfigure command-line argument when running atmos terraform init … command why atmos terraform init must behave the same as all other commands that use init_run_reconfigure CLI config and –init-run-reconfigure command-line argument
what Take into account init_run_reconfigure CLI config and –init-run-reconfigure command-line argument when running atmos terraform init … command why atmos terraform init must behave the sam…
2022-04-16
2022-04-18
![Release notes from atmos avatar](https://a.slack-edge.com/80588/img/services/rss_72.png)
v1.4.8 what Add metadata.terraform_workspace_pattern why We already have metadata.terraform_workspace using which we can specify/override the terraform workspace for a component metadata.terraform_workspace_pattern introduces a pattern to override a terraform workspace for a component metadata: # Override Terraform workspace terraform_workspace: xxxxxxxxxxxxxxx terraform_workspace_pattern: “{tenant}-{environment}-{stage}-{component}”
The following tokens are supported…
what Add metadata.terraform_workspace_pattern why We already have metadata.terraform_workspace using which we can specify/override the terraform workspace for a component metadata.terraform_work…
2022-04-21
![Release notes from atmos avatar](https://a.slack-edge.com/80588/img/services/rss_72.png)
v1.4.9 what Add atmos aws commands Add atmos aws eks update-kubeconfig command why Execute aws CLI commands using atmos context (atmos.yaml CLI config and component/stack configurations) Downloading kubeconfig from an EKS cluster using `atmos Allow using this functionality in Terraform by implementing a new data source in the terraform-provider-utils provider (this will be added in a PR in <a…
what Add atmos aws commands Add atmos aws eks update-kubeconfig command why Execute aws CLI commands using atmos context (atmos.yaml CLI config and component/stack configurations) Downloading ku…
2022-04-22
![Release notes from atmos avatar](https://a.slack-edge.com/80588/img/services/rss_72.png)
v1.4.10 what If functions that are used by Terraform providers throw errors, print errors to std.Error why Terraform providers only see errors that are sent to std.Error
what If functions that are used by Terraform providers throw errors, print errors to std.Error why Terraform providers only see errors that are sent to std.Error
![Matt Gowie avatar](https://avatars.slack-edge.com/2023-02-06/4762019351860_44dadfaff89f62cba646_72.jpg)
Another Atmos in the terraform space — https://github.com/simplygenius/atmos
Just heard about it on a call with a company. Seems fairly dead and not all the interesting of a project, but it is funny that name would get chosen twice in the TF space.
Breathe easier with terraform. Cloud system architectures made easy
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Yes, it appears to be on life support.
Breathe easier with terraform. Cloud system architectures made easy
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
I think the company behind it was acquired by CloudTruth
![Matt Gowie avatar](https://avatars.slack-edge.com/2023-02-06/4762019351860_44dadfaff89f62cba646_72.jpg)
Ah do you know the CloudTruth folks? They’re who mentioned it to me.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Why we chose atmos name: https://www.space.com/terraforming-in-alien-universe
![attachment image](https://cdn.mos.cms.futurecdn.net/Et3vfWu2eTVocjMiaKuLt3-1200-80.jpg)
Terraforming a planet and building better worlds is not as easy as it seems.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
“Automated Atmosphere Processor”
![attachment image](https://cdn.mos.cms.futurecdn.net/Et3vfWu2eTVocjMiaKuLt3-1200-80.jpg)
Terraforming a planet and building better worlds is not as easy as it seems.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Was called atmos
for short
![Andy Miguel (Cloud Posse) avatar](https://avatars.slack-edge.com/2021-01-31/1681606086343_27574601efa96f8283e4_72.png)
I love part 2, Aliens. the rest of the series gets questionable from there.
![Michael Dizon avatar](https://avatars.slack-edge.com/2021-01-15/1664383757488_b5214d00b8fce4726a7c_72.jpg)
i remember reading something about nuking the martian poles to create at atmosphere
![Michael Dizon avatar](https://avatars.slack-edge.com/2021-01-15/1664383757488_b5214d00b8fce4726a7c_72.jpg)
atmos > nuke
![Release notes from atmos avatar](https://a.slack-edge.com/80588/img/services/rss_72.png)
v1.4.11 what Support dashes - in the tenant, environment and stage names In the examples, add a new stage test-1 and add tests for components, stacks, and spacelift to test having a dash in the stage name (the file name itself being without dashes) why The old atmos supported it (because it was filename-based, not logical stack name based) Some clients want to name tenants/environment/stages with dashes in the names (and some already have it, so we need to support that when converting from the old to…
what Support dashes - in the tenant, environment and stage names In the examples, add a new stage test-1 and add tests for components, stacks, and spacelift to test having a dash in the stage name…
2022-04-25
![Release notes from atmos avatar](https://a.slack-edge.com/80588/img/services/rss_72.png)
v1.4.3 what Add –dry-run command-line flag to all commands and workflows why Helps debugging atmos commands and workflows The –dry-run flag shows all the flow and commands without executing them and without writing files to the file system (e.g. varfiles and backend config are not written) The –dry-run flag shows all the workflow steps without executing them tests and examples Click to expandatmos terraform plan test/test-component-override -s=tenant1-ue2-dev –dry-run
Variables for the…
what Add –dry-run command-line flag to all commands and workflows why Helps debugging atmos commands and workflows The –dry-run flag shows all the flow and commands without executing them and …
![Release notes from atmos avatar](https://a.slack-edge.com/80588/img/services/rss_72.png)
v1.4.4 what Add atmos describe stacks command Allow writing the result to a file by using –file command-line flag Allow formatting the result as YAML or JSON by using –format command-line flag Allow filtering of the result by using the command-line flags: stack, component-types, components, sections Available component sections: backend, backend_type, deps, env, inheritance, metadata, remote_state_backend, remote_state_backend_type, settings, vars why Command to show stack configs and all the…
what Add atmos describe stacks command Allow writing the result to a file by using –file command-line flag Allow formatting the result as YAML or JSON by using –format command-line flag Allow fi…
2022-04-29
![Chris Bloemker avatar](https://secure.gravatar.com/avatar/c00b92218bf66c3f79dc5cd9b2eca831.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0024-72.png)
Good morning all! We are trying to add a new mono repo for our SRE team to Spacelift with Atmos and it seems like Spacelift can’t find the stacks based on base_path
$ atmos describe config | jq '.Components.Terraform.base_path'
"/components/terraform"
$ atmos describe config | jq '.Stacks.base_path'
"/stacks"
Spacelift run error:
No stack config files found in the provided paths:
│ - /mnt/workspace/source/components/terraform/spacelift/stacks/**/*
Our directory structure is the same as the atmos example here where we have at the root of the repo:
components
stacks
atmos.yaml
Trying to see if there is another variable overriding stacks.base_path
somewhere else.
![RB avatar](https://avatars.slack-edge.com/2020-02-26/958727689603_86844033e59114029b3c_72.png)
for spacelift, try setting ATMOS_BASE_PATH=/mnt/workspace/source
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
in .spacelift/config.yml
in the repo, we have:
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
# <https://docs.spacelift.io/concepts/runtime-configuration>
version: "1"
stack_defaults:
before_init:
- spacelift-configure-paths
- spacelift-git-use-https
- spacelift-write-vars
- spacelift-tf-workspace
before_plan:
- spacelift-configure-paths
before_apply:
- spacelift-configure-paths
environment:
AWS_SDK_LOAD_CONFIG: true
AWS_CONFIG_FILE: /etc/aws-config/aws-config-cicd
AWS_PROFILE: eg-gbl-identity
ATMOS_BASE_PATH: /mnt/workspace/source
stacks:
infrastructure:
before_init: []
before_plan: []
before_apply: []
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
the important part is ATMOS_BASE_PATH: /mnt/workspace/source
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
also, in rootfs/usr/local/etc/atmos/atmos.yaml
(which ends up in usr/local/etc/atmos/atmos.yaml
in the Docker image that Spacelift is using to execute commands), we have:
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
# CLI config is loaded from the following locations (from lowest to highest priority):
# system dir (`/usr/local/etc/atmos` on Linux, `%LOCALAPPDATA%/atmos` on Windows)
# home dir (~/.atmos)
# current directory
# ENV vars
# Command-line arguments
#
# It supports POSIX-style Globs for file names/paths (double-star `**` is supported)
# <https://en.wikipedia.org/wiki/Glob_(programming)>
# Base path for components and stacks configurations.
# Can also be set using `ATMOS_BASE_PATH` ENV var, or `--base-path` command-line argument.
# Supports both absolute and relative paths.
# If not provided or is an empty string, `components.terraform.base_path`, `components.helmfile.base_path` and `stacks.base_path`
# are independent settings (supporting both absolute and relative paths).
# If `base_path` is provided, `components.terraform.base_path`, `components.helmfile.base_path` and `stacks.base_path`
# are considered paths relative to `base_path`.
base_path: ""
components:
# Settings for all terraform components
terraform:
# Can also be set using `ATMOS_COMPONENTS_TERRAFORM_BASE_PATH` ENV var, or `--terraform-dir` command-line argument
# Supports both absolute and relative paths
base_path: "components/terraform"
# Can also be set using `ATMOS_COMPONENTS_TERRAFORM_APPLY_AUTO_APPROVE` ENV var
apply_auto_approve: false
# Can also be set using `ATMOS_COMPONENTS_TERRAFORM_DEPLOY_RUN_INIT` ENV var, or `--deploy-run-init` command-line argument
deploy_run_init: true
# Can also be set using `ATMOS_COMPONENTS_TERRAFORM_INIT_RUN_RECONFIGURE` ENV var, or `--init-run-reconfigure` command-line argument
init_run_reconfigure: false
# Can also be set using `ATMOS_COMPONENTS_TERRAFORM_AUTO_GENERATE_BACKEND_FILE` ENV var, or `--auto-generate-backend-file` command-line argument
auto_generate_backend_file: true
# Settings for all helmfile components
helmfile:
# Can also be set using `ATMOS_COMPONENTS_HELMFILE_BASE_PATH` ENV var, or `--helmfile-dir` command-line argument
# Supports both absolute and relative paths
base_path: "components/helmfile"
# Can also be set using `ATMOS_COMPONENTS_HELMFILE_KUBECONFIG_PATH` ENV var
kubeconfig_path: "/dev/shm"
# Can also be set using `ATMOS_COMPONENTS_HELMFILE_HELM_AWS_PROFILE_PATTERN` ENV var
helm_aws_profile_pattern: "{namespace}-gbl-{stage}-helm"
# Can also be set using `ATMOS_COMPONENTS_HELMFILE_CLUSTER_NAME_PATTERN` ENV var
cluster_name_pattern: "{namespace}-{environment}-{stage}-eks-cluster"
# Settings for all stacks
stacks:
# Can also be set using `ATMOS_STACKS_BASE_PATH` ENV var, or `--config-dir` and `--stacks-dir` command-line arguments
# Supports both absolute and relative paths
base_path: "stacks"
# Can also be set using `ATMOS_STACKS_INCLUDED_PATHS` ENV var (comma-separated values string)
included_paths:
- "**/*"
# Can also be set using `ATMOS_STACKS_EXCLUDED_PATHS` ENV var (comma-separated values string)
excluded_paths:
- "globals/**/*"
- "**/*globals*"
- "catalog/**/*"
# exclude workflows
- "workflows/**/*"
# Can also be set using `ATMOS_STACKS_NAME_PATTERN` ENV var
name_pattern: "{environment}-{stage}"
# <https://github.com/cloudposse/atmos/releases/tag/v1.4.0>
workflows:
# Can also be set using `ATMOS_WORKFLOWS_BASE_PATH` ENV var, or `--workflows-dir` command-line arguments
# Supports both absolute and relative paths
base_path: "stacks/workflows"
logs:
verbose: false
colors: true
![Chris Bloemker avatar](https://secure.gravatar.com/avatar/c00b92218bf66c3f79dc5cd9b2eca831.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0024-72.png)
oh interesting, I was missing the workflow base path var .spacelift/config.yml
I was just using our platform_infrastructure
repo as copy pasta but we aren’t using atmos latest there. Trying that now.
So essentially the rootfs atmos.yaml
needs to be identical as the repo root atmos.yaml ?
![Chris Bloemker avatar](https://secure.gravatar.com/avatar/c00b92218bf66c3f79dc5cd9b2eca831.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0024-72.png)
Now my error is Error: stack name pattern must be provided in 'stacks.name_pattern' config or 'ATMOS_STACKS_NAME_PATTERN' ENV variable
In both of my atmos configs, stacks.name_pattern
== "{tenant}-{environment}-{stage}"
This error makes it seem like it’s not being set?
![RB avatar](https://avatars.slack-edge.com/2020-02-26/958727689603_86844033e59114029b3c_72.png)
what’s your atmos command that is failing
![RB avatar](https://avatars.slack-edge.com/2020-02-26/958727689603_86844033e59114029b3c_72.png)
are you using tenant
![RB avatar](https://avatars.slack-edge.com/2020-02-26/958727689603_86844033e59114029b3c_72.png)
if not, you can modify the pattern to only contain the environment and stage
![Chris Bloemker avatar](https://secure.gravatar.com/avatar/c00b92218bf66c3f79dc5cd9b2eca831.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0024-72.png)
I’m not sure exactly which atmos command spacelift is doing here. This file stacks/globals/datadog_slo-globals-dev.yaml
has the following vars.
vars:
namespace: bd
region: us-east-2
tenant: datadog_slo
environment: global
![RB avatar](https://avatars.slack-edge.com/2020-02-26/958727689603_86844033e59114029b3c_72.png)
oh so you are using tenant, then id keep the pattern the same.
from that config the stack name should be datadog_slo-global-<stage>
but youre missing the stage name
![Chris Bloemker avatar](https://secure.gravatar.com/avatar/c00b92218bf66c3f79dc5cd9b2eca831.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0024-72.png)
![RB avatar](https://avatars.slack-edge.com/2020-02-26/958727689603_86844033e59114029b3c_72.png)
try this as an example
vars:
namespace: bd
region: us-east-2
tenant: datadog_slo
environment: global
stage: banana
![RB avatar](https://avatars.slack-edge.com/2020-02-26/958727689603_86844033e59114029b3c_72.png)
now the stack name will be datadog_slo-global-banana
![RB avatar](https://avatars.slack-edge.com/2020-02-26/958727689603_86844033e59114029b3c_72.png)
usually we use stage to be the account name
![RB avatar](https://avatars.slack-edge.com/2020-02-26/958727689603_86844033e59114029b3c_72.png)
so prod, dev, staging, sandbox, qa, or similar
![Chris Bloemker avatar](https://secure.gravatar.com/avatar/c00b92218bf66c3f79dc5cd9b2eca831.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0024-72.png)
I’m taking over a project from someone that has been out and I don’t have much context on how these directory structures worked getting this new project going
![Chris Bloemker avatar](https://secure.gravatar.com/avatar/c00b92218bf66c3f79dc5cd9b2eca831.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0024-72.png)
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
If you are using geodesic, you don’t need atmos.yaml in the root of the repo. Both geodesic and Spacelift should take it from the rootfs
![Chris Bloemker avatar](https://secure.gravatar.com/avatar/c00b92218bf66c3f79dc5cd9b2eca831.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0024-72.png)
We are using geodesic here. So I can remove the repo’s root atmost.yaml
![Chris Bloemker avatar](https://secure.gravatar.com/avatar/c00b92218bf66c3f79dc5cd9b2eca831.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0024-72.png)
Hmm. still getting the same error
╷
│ Error: stack name pattern must be provided in 'stacks.name_pattern' config or 'ATMOS_STACKS_NAME_PATTERN' ENV variable
│
│ with module.spacelift.module.yaml_stack_config.data.utils_spacelift_stack_config.spacelift_stacks,
│ on .terraform/modules/spacelift.yaml_stack_config/modules/spacelift/main.tf line 1, in data "utils_spacelift_stack_config" "spacelift_stacks":
│ 1: data "utils_spacelift_stack_config" "spacelift_stacks" {
This platform_sre
is an Administrative stack if that gives more contexts, and currently the regular stacks are not shown.
![RB avatar](https://avatars.slack-edge.com/2020-02-26/958727689603_86844033e59114029b3c_72.png)
it would be easier to test this locally before running it in space lift
![RB avatar](https://avatars.slack-edge.com/2020-02-26/958727689603_86844033e59114029b3c_72.png)
get the atmos commands to work locally and then spacelift will be easier to troubleshoot
![Chris Bloemker avatar](https://secure.gravatar.com/avatar/c00b92218bf66c3f79dc5cd9b2eca831.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0024-72.png)
Good point. I do recall this working locally in geodesic.. but seems my changes now it is not working
√ . [bd-gbl-identity] / ⨠ atmos-local terraform plan -s datadog_slo-global-dev "datadog/sre-slo"
/localhost/git/bread/platform_sre /
Found ENV var ATMOS_STACKS_BASE_PATH=/localhost/git/bread/platform_sre/stacks
Could not find config for the component 'datadog/sre-slo' in the stack 'datadog_slo-global-dev'.
Check that all attributes in the stack name pattern '{tenant}-{environment}-{stage}' are defined in the stack config files.
Are the component and stack names correct? Did you forget an import?
![Chris Bloemker avatar](https://secure.gravatar.com/avatar/c00b92218bf66c3f79dc5cd9b2eca831.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0024-72.png)
This is our directory tree of stacks
stacks
|-- datadog_slo
| `-- global
| `-- dev.yaml
|-- globals
| |-- datadog_slo-globals-dev.yaml
| `-- globals.yaml
`-- spacelift
`-- global
`-- dev.yaml
![RB avatar](https://avatars.slack-edge.com/2020-02-26/958727689603_86844033e59114029b3c_72.png)
the file structure doesn’t matter, it’s all about how the inputs in the yaml
![RB avatar](https://avatars.slack-edge.com/2020-02-26/958727689603_86844033e59114029b3c_72.png)
is your component datadog/sre-slo
in the datadog_slo-global-dev
stack ?
![RB avatar](https://avatars.slack-edge.com/2020-02-26/958727689603_86844033e59114029b3c_72.png)
can you revert your atmos.yaml file and get it to work again ?
![Chris Bloemker avatar](https://secure.gravatar.com/avatar/c00b92218bf66c3f79dc5cd9b2eca831.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0024-72.png)
I reverted back on my branch that was working. and now I’m trying to do diffs and figure out what broke the local atmos run
atmos-local terraform plan -s datadog_slo-global-dev "datadog/sre-slo"
seems to be working now.
![Chris Bloemker avatar](https://secure.gravatar.com/avatar/c00b92218bf66c3f79dc5cd9b2eca831.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0024-72.png)
my stacks/globals/datadog_slo-globals-dev.yaml
is as follows:
vars:
namespace: bd
region: us-east-2
tenant: datadog_slo
environment: global
terraform:
vars: {}
components:
terraform:
"datadog/sre-slo":
vars:
gh_org: getbread
![Chris Bloemker avatar](https://secure.gravatar.com/avatar/c00b92218bf66c3f79dc5cd9b2eca831.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0024-72.png)
okay so I figured out what is breaking my local atmos.
when I add the following to stacks/spacelift/global/dev.yaml
vars:
namespace: bd
region: us-east-2
tenant: spacelift
environment: global
stage: banana
terraform:
vars: {}
I get this error. I think I am misunderstanding the relationship between the files/variables.
√ . [bd-gbl-identity] / ⨠ atmos-local terraform plan -s datadog_slo-global-dev "datadog/sre-slo"
/localhost/git/bread/platform_sre /
Found ENV var ATMOS_STACKS_BASE_PATH=/localhost/git/bread/platform_sre/stacks
Could not find config for the component 'datadog/sre-slo' in the stack 'datadog_slo-global-dev'.
Check that all attributes in the stack name pattern '{tenant}-{environment}-{stage}' are defined in the stack config files.
Are the component and stack names correct? Did you forget an import?
![RB avatar](https://avatars.slack-edge.com/2020-02-26/958727689603_86844033e59114029b3c_72.png)
the file names are independent of the stacks themselves
![RB avatar](https://avatars.slack-edge.com/2020-02-26/958727689603_86844033e59114029b3c_72.png)
i was suggesting to append your vars
with stage: whatever
, but i think you created a new file side by side that may have duplicated the existing file which probably confused atmos
![Chris Bloemker avatar](https://secure.gravatar.com/avatar/c00b92218bf66c3f79dc5cd9b2eca831.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0024-72.png)
Picking back up here on this issue. I have atmos-local
running fine in geodesic. We are exec’d into the spacelift runner
ash-5.1$ pwd
/
bash-5.1$ find . -type f -executable | grep atmos
./mnt/workspace/source/atmos.yaml
./mnt/workspace/source/rootfs/etc/profile.d/atmos-local.sh
find: ./proc/tty/driver: Permission denied
./mnt/workspace/source/rootfs/usr/local/etc/atmos/atmos.yaml
find: ./root: Permission denied
bash-5.1$ cat /mnt/workspace/source/rootfs/etc/profile.d/atmos-local.sh
export LOCAL_PREFIX=/localhost${SRC_RELATIVE_PATH}
function atmos-local() {
pushd $LOCAL_PREFIX
ATMOS_STACKS_BASE_PATH=$LOCAL_PREFIX/stacks atmos $@
popd
}
Looks like it’s a wrapper here? I can’t find atmos binary anywhere in that image. How would we debug this trying to running atmos locally in the spacelift runner? All we can see is atmos is making its way into the provider here? Or is spacelift running a custom terraform command here?
github.com/cloudposse/atmos v1.4.11