#atmos (2024-11)
2024-11-01
with the atmos github actions I can plan/apply no problem but now I want to destroy and I do not have a count = module.this.enabled ? 1 : 0
, how do you guys destroy? I will like to be able to find the delete yaml from the stack and destroy that one if possible
Interesting. So we are working on supporting an enabled flag for components.
what
why
references
• DEV-2150
The scope is not currently to destroy. However, it might be worth considering that.
In the current implementation, enabled would make the component “invisible”
an alternative to commenting it out.
I can see a component as having 3 states
• enabled
• disabled
• destroyed
I think if the components are commented out or deleted from the stack file, describe affected
should know what to do with it or output a destroyed
flag or something like that for another job to use that matrix to destroy it
Ok, that makes sense.
So enabled = true/false affects the visibility, but removal from the configuration is visible via atmos describe affected
@Andriy Knysh (Cloud Posse) what happens today in describe affected if the component is removed? is that surfaced?
I renamed a component from pepetest
to pepetest1
, describe affected saw the new pepetest1
component got deployed, but the old one is still there in the cloud environment
the key is do we have the information in describe affected
JSON output
The right behavior might not be implemented, but maybe we have the data there to act on
if a component is removed, Atmos does not see it (it will not consider it affected) - this is the current implementation. This is b/c Atmos compares the current branch with a remote branch/tag/sha - if the current branch does not have the component, then it’s “not affected”.
Setting enabled: false
is not “removal”, so describe affected
sees that
but with that, you need a two-step approach, one to say enable: false and then another PR to remove the yaml from the stack file
yes
i would say we didn’t consider a complete component removal with describe affected - we need to revisit this
2024-11-02
Improve terraform
and helmfile
help. Enable Go
templating in the command
field. Clean Terraform workspace before executing terraform init
@aknysh (#759)
what
• Improve terraform
and helmfile
help
• Enable Go
templating in the command
field of stack config
• Clean Terraform workspace before executing terraform init
why
• Improve the help messages. When a user executes atmos terraform --help
or atmos helmfile --help
(or help for a subcommand), print a message describing the command and how to execute the terraform
and helmfile
help command
atmos terraform –help
• Enable Go templating in the command
stack config in addition to the already supported sections.
You can now use Go
templates in the following Atmos sections to refer to values in the same or other sections:
• vars
• settings
• env
• providers
• overrides
• backend
• backend_type
• component
• metadata.component
• command
Enabling Go
templates in the command
section allows specifying different Terraform/OpenTofu/Helmfile versions per component/stack, and get the value from different Atmos sections or from external data sources
• Clean Terraform workspace before executing terraform init
. When using multiple backends for the same component (e.g. separate backends per tenant or account), and if an Atmos command was executed that selected a Terraform workspace, Terraform will prompt the user to select one of the following workspaces:
- default
-
The prompt forces the user to always make a selection (which is error-prone), and also makes it complicated when running on CI/CD. The PR adds the logic that deletes the `.terraform/environment` file from the component directory before executing `terraform init`. The `.terraform/environment` file contains the name of the currently selected workspace, helping Terraform identify the active workspace context for managing your infrastructure. We delete the file before executing `terraform init` to prevent the Terraform prompt asking to select the default or the previously used workspace.
@jose.amengual another one for you
Improve terraform
and helmfile
help. Enable Go
templating in the command
field. Clean Terraform workspace before executing terraform init
@aknysh (#759)
what
• Improve terraform
and helmfile
help
• Enable Go
templating in the command
field of stack config
• Clean Terraform workspace before executing terraform init
why
• Improve the help messages. When a user executes atmos terraform --help
or atmos helmfile --help
(or help for a subcommand), print a message describing the command and how to execute the terraform
and helmfile
help command
atmos terraform –help
• Enable Go templating in the command
stack config in addition to the already supported sections.
You can now use Go
templates in the following Atmos sections to refer to values in the same or other sections:
• vars
• settings
• env
• providers
• overrides
• backend
• backend_type
• component
• metadata.component
• command
Enabling Go
templates in the command
section allows specifying different Terraform/OpenTofu/Helmfile versions per component/stack, and get the value from different Atmos sections or from external data sources
• Clean Terraform workspace before executing terraform init
. When using multiple backends for the same component (e.g. separate backends per tenant or account), and if an Atmos command was executed that selected a Terraform workspace, Terraform will prompt the user to select one of the following workspaces:
- default
-
The prompt forces the user to always make a selection (which is error-prone), and also makes it complicated when running on CI/CD. The PR adds the logic that deletes the `.terraform/environment` file from the component directory before executing `terraform init`. The `.terraform/environment` file contains the name of the currently selected workspace, helping Terraform identify the active workspace context for managing your infrastructure. We delete the file before executing `terraform init` to prevent the Terraform prompt asking to select the default or the previously used workspace.
2024-11-03
hey folks, for some reason atmos generate wrong terraform workspace name, i have a component named nats
and stack named dev
and instead of dev-nats
the workspace is simply dev
what could cause that ?
Something is wrong with your name_pattern or name_template
i didn’t change those.
Share your atmos config, if you can
Use the atmos.yaml
to configure where Atmos will discover stack configurations.
base_path: .
components:
terraform:
command: tofu
base_path: components/terraform
apply_auto_approve: false
deploy_run_init: true
init_run_reconfigure: true
auto_generate_backend_file: true
stacks:
base_path: stacks
included_paths:
- "deploy/**/*"
excluded_paths:
- "**/_defaults.yaml"
name_pattern: "{stage}"
workflows:
base_path: stacks/workflows
templates:
settings:
enabled: true
sprig:
enabled: true
logs:
file: /dev/stderr
level: Info
Ok, I think I initially misunderstood.
Are you setting workspace key prefix anywhere?
nope
@Andriy Knysh (Cloud Posse) any ideas
is the workspaces are port of the terraform state right?
i’ve tried to clean all tf files and deleted this workspace, but still got back named as dev
so maybe the default workspace holds some information?
Yes, in atmos we use one workspace for each instance of a component deployed.
Have you updated to the latest atmos? We just fixed a problem related to workspaces and changing backends
yes, i’m using the very latest
Workspace "dev" doesn't exist.
You can create this workspace with the "new" subcommand
or include the "-or-create" flag with the "select" subcommand.
Created and switched to workspace "dev"!
but i bet it’s atmos what is switching to the workspace so the name dev
comes from atmos not from the state
Yes, atmos dynamically computes the workspace name and switches to it
i see.
vars:
stage: dev
import:
- deploy/_defaults
- catalog/do/project
- catalog/do/doks
- catalog/nats
components:
terraform:
project:
vars:
name: "platform-{{ .stack }}"
environment: Development
cluster:
vars:
name: doks-cluster-1
project: '{{ (atmos.Component "project" .stack).outputs.id }}'
nats:
vars:
kube_host: '{{ (atmos.Component "cluster" .stack).outputs.kube_host }}'
kube_token: '{{ (atmos.Component "cluster" .stack).outputs.kube_token }}'
kube_cert: '{{ (atmos.Component "cluster" .stack).outputs.kube_cert }}'
this is my dev stack file
and interestingly for the project and the cluster the names are generated correctly
❯ tofu -chdir=components/terraform/nats workspace list
default
* dev
dev-cluster
dev-project
i have a component named nats
and stack named dev
and instead of dev-nats
the workspace is simply dev
this is the correct behavior for Atmos components that don’t inherit from other components
in this case, the TF workspace is simply the stack name
only if you have a derived component (inherited from a base component), then TF workspace will be <stack>+<component>
hmm. what you mean by “inherit” the do cluster and project name is correct and didn’t inherit from anything. or i miss something here.
components:
terraform:
nats:
metadata:
component: nats
vars:
...
vs
components:
terraform:
cluster:
metadata:
component: do/doks
regarding inheritance, please see https://atmos.tools/core-concepts/stacks/inheritance/
Inheritance provides a template-free way to customize Stack configurations. When combined with imports, it provides the ability to combine multiple configurations through ordered deep-merging of configurations. Inheritance is how you manage configuration variations, without resorting to templating.
ok i read that before. but i didn’t us that in any catalog so far.
in the two examples above, both nats
and cluster
Atmos components do not inherit from any other Atmos components, so the TF workspaces for both of them will be dev
so what you have is 100% correct, the workspaces for these two components are just dev
- the stack name
❯ tofu -chdir=components/terraform/do/doks workspace list
default
dev
* dev-cluster
dev-project
maybe those were generated wrongly because i made a lot changes back and forth since.
yes, looks like it
anyhow i’ fine with a single dev workspace named after the stack. as long as the states are correctly separated.
i’m not fully familiar with tf workspaces, but that means if all these components are in the same workspace they are sharing the state or not ?
each component has workspace_key_prefix
- it’s usually generated by Atmos, but you can override it per component
workspace_key_prefix
is, if you look at the backend s3 bucket, the top-level folder
so each component will have it’s own top-level folder in the bucket, and each stack, in a separate TF worksapce, will have its own subfolder in the folder
so yes, each component state is separated from any other component state (diff folders in the state bucket)
note that it’s still in the same backend (same S3 bucket). If you want to separate backends (e.g. per tenant/OU, per account, etc.), you need to create and configure multiple backends
sure. but i don’t see workspace_key_prefix
generated anywhere.
{
"terraform": {
"backend": {
"gcs": {
"bucket": "mw-tf-state",
"encryption_key": "...",
"prefix": "platform/infra"
}
}
}
}
it’s gcs, not s3 actually.
_defaults.yaml
:
terraform:
backend_type: gcs
backend:
gcs:
bucket: mw-tf-state
prefix: platform/infra
encryption_key: '{{ env "GCS_ENCRYPTION_KEY" }}'
GCP has prefix
which is the same
Configure Terraform Backends.
If the prefix is not specified for a component, Atmos will use the component name (my-component in the example above) to auto-generate the prefix. In the component name, all occurrences of / (slash) will be replaced with - (dash).
you don’t need to hardcode it here
backend:
gcs:
bucket: mw-tf-state
prefix: platform/infra
in this case, all components will use the same prefix
hmm ok, let me check that
please review the doc, if you don’t specify prefix
, Atmos will auto-generate it
thank you!
so if i understand it correctly, without setting the prefix atmos will generate it and because of that my component states will end up in separate folders in the gcs bucket, so even they share a workspace states are separated.
good to know that.:)
only problem is that i prefer to store them in some folder instead of the root of the bucket but i can leave with that
note that you can specify the prefix per component, in which case Atmos will just use it
i can review your config, let me know
it’s fine. this bucket is solely for tf states so it’s ok even in the root. i prefer to leave it to atmos to generate.
on a different topic while we chat.. any chance to add support for command like this in atmos.yaml
:
components:
terraform:
command: xy command -- tofu
so support command with double dash
it would be perfect that way i could load secrets as env vars. and i won’t need custom commands.
i don’t remember if command: xy command -- tofu
is supported now (need to look at the code). Did you test it?
yes. i just tested unfortunately it’s not working.
atmos terraform plan cluster --stack dev
template: all-atmos-sections:100:35: executing "all-atmos-sections" at <atmos.Component>: error calling Component: exec: "op run --no-masking --env-file=.env -- tofu": executable file not found in $PATH
I have done that with asdf and it works
ok, we’ll create a task for this, thank you
did you try to quote it?
thanks a lot!!
@Erik Osterman (Cloud Posse) do you have an example maybe with asdf?
i’ve just tried with quote but not working.
trying with a small shell script:
command: ./optofu.sh
but still not working
i’ve created a PR: https://github.com/cloudposse/atmos/pull/762
what
Add support for shell commands.
why
To support complex commands, for example:
components: terraform: command: op run –no-masking –env-file=.env – tofu
thanks, i’ll review it today. Did you test it?
roughly. is there any go test(s) i can run?
strange but for some reason it’s not working with my atmos config. it’s working fine with atmos.yaml in the repository. i will dig into it.
problem is that when the template executed it uses tfexec
ah, yes, tfexec doesn’t understand those commands, it needs terraform
2024-11-04
possible to organize a few smaller components into one catalog?
Of course… this is what we frequently do
you can create a catalog stack file for a solution.
E.g. here’s how we do “EKS” and all related components
ok. is there any related example in the repo?
Here you can see we created a default cluster config, that imports a bunch of other componetns
Those could be inline, but we chose to import them
2024-11-05
whats the best way to share vars between some components but not all of them?
if i place in the stack yaml vars section, i got warnings from the components which are not using them.
Warning: Value for undeclared variable
yes, please don’t use globals
there are a few ways to share vars b/w components
e.g. create a base abstract component (with the default values) and inherit it in the other components
see this doc for ref https://atmos.tools/design-patterns/abstract-component
Abstract Component Atmos Design Pattern
And multiple inheritance
Inheritance provides a template-free way to customize Stack configurations. When combined with imports, it provides the ability to combine multiple configurations through ordered deep-merging of configurations. Inheritance is how you manage configuration variations, without resorting to templating.
2024-11-06
Hey, when using helmfile with atmos it requires to have helm_aws_profile_pattern
and cluster_name_pattern
set. Is there a way to use a name_template
like in the stack configuration?
It’s actually not required and we improved the demo and examples here
This uses k3s
But if I want to use it with eks I have to set it, no? Your examples all set use_eks
to false
Or other question: if I use use_eks: false
how does atmos now to which kubernetes to deploy to? Will it use my current context? Can I dynamically change the context in atmos in the stack configuration?
that’s only if you want the automatic kubeconfig creation
use_eks
could probably be better named. Its true, it’s if you use eks, but it’s not required to use EKS.
In the end, all you need is a kubeconfig. With use_eks
set to false, it just means you need to manage the kubeconfig yourself.
Atmos may need a RAG application for QA
Would love a bot that could auto answer with links to threads
and TL;DR
exactly, RAG can do with its metadata
and need an custom integration with slack, there should be existing SaaS service on the market to do the similar thing
I’m looking into RAG recently, and it is not hard to write one
And want to train it on atmos docs
and examples
Have you also checked out danswer?
just heard of it, saw some similar projects, will take a look into it
after a quick review, danswer uses langchain and fast api, should be a reliable project to be used
this looks a good use case, https://docs.danswer.dev/more/use_cases/support
Help your customer support team instantly answer any question across your entire product.
dived into the project and gave it a test, seems it is not easy to make it fully up, e.g. I tried with a public #atoms web page, but QA failed, I used local LLM so I guess it may work with public LLM service. side note: the project has a big vision to be a platform, so the codes are abstracted very well.
On what way did the QA fail?
You had it index the atmos slack channel in the public web archive?
api_server-1 | Traceback (most recent call last):
api_server-1 | File "/app/danswer/chat/process_message.py", line 731, in stream_chat_message_objects
api_server-1 | for packet in answer.processed_streamed_output:
api_server-1 | File "/app/danswer/llm/answering/answer.py", line 280, in processed_streamed_output
api_server-1 | for processed_packet in self._get_response([llm_call]):
api_server-1 | File "/app/danswer/llm/answering/answer.py", line 245, in _get_response
api_server-1 | yield from response_handler_manager.handle_llm_response(stream)
api_server-1 | File "/app/danswer/llm/answering/llm_response_handler.py", line 69, in handle_llm_response
api_server-1 | for message in stream:
api_server-1 | File "/app/danswer/llm/chat_llm.py", line 386, in _stream_implementation
api_server-1 | for part in response:
api_server-1 | File "/usr/local/lib/python3.11/site-packages/litellm/llms/ollama.py", line 427, in ollama_completion_stream
api_server-1 | raise e
api_server-1 | File "/usr/local/lib/python3.11/site-packages/litellm/llms/ollama.py", line 403, in ollama_completion_stream
api_server-1 | response_content = "".join(content_chunks)
api_server-1 | ^^^^^^^^^^^^^^^^^^^^^^^
api_server-1 | TypeError: sequence item 19: expected str instance, NoneType found
I used one of archived page, https://archive.sweetops.com/atmos/2024/08/
SweetOps Slack archive of #atmos for August, 2024.
should be related to ollama python lib
litellm’s ollama lib
Aha understand.. so got stuck on even just indexing one page
indexing is ok, should be in the retrieve answer part
side note: found this one, https://ollama.com/blog/continue-code-assistant looks useful for code refactoring
An entirely open-source AI code assistant inside your editor
Good morning gents, just checking in here - is this the module usually used to automate remote backend stand-up - https://github.com/cloudposse/terraform-aws-tfstate-backend
Terraform module that provision an S3 bucket to store the terraform.tfstate
file and a DynamoDB table to lock the state file to prevent concurrent modifications and state corruption.
@Erik Osterman (Cloud Posse)
Terraform module that provision an S3 bucket to store the terraform.tfstate
file and a DynamoDB table to lock the state file to prevent concurrent modifications and state corruption.
Yes, but in the context of atmos
, we use our component
ty thats perfect, never got a chance to walk through the process using automation. appreciate the quick response.
coming back here this is cool, sorry i was stuck in my head in the chicken/egg scenario of the backend and the cold start stuff helped alot.
Glad to hear!
I should have also shared https://docs.cloudposse.com/layers/accounts/initialize-tfstate/
Follow these steps to configure and initialize the Terraform state backend using Atmos, ensuring proper setup of the infrastructure components and state management.
ooo thank you.
I will say I think our internal Atmos is on an older ver
I’m kind of at a design decision with regards to a second updated atmos for my new region + new backend, idk yet
Hello, I just found this project and trying to plan how im going to design things. Would like input on how atmos required git repos structured in respect to monorepo vs multirepo? I intend to create modules in 1 repo and create environments in another. Also curious about submitting to terraform/opentudfu registry and I think they require 1 repo per module or something. Would appreciate input from others with experience!
Kudos.
Hey @Derrick Hammer great to hear you’re checking it out.
We have an example mono repo structured here: https://github.com/cloudposse-examples/infra-demo-atmos-pro/tree/main
I intend to create modules in 1 repo and create environments in another
Perfect, checkout vendoring:
https://atmos.tools/core-concepts/vendor/
Use Atmos vendoring to make copies of 3rd-party components, stacks, and other artifacts in your own repo.
apiVersion: atmos/v1
kind: AtmosVendorConfig
metadata:
name: demo-vendoring
description: Atmos vendoring manifest for Atmos demo component library
spec:
# Import other vendor manifests, if necessary
imports: []
sources:
- component: "github/stargazers"
source: "github.com/cloudposse/atmos.git//examples/demo-library/{{ .Component }}?ref={{.Version}}"
version: "main"
targets:
- "components/terraform/{{ .Component }}/{{.Version}}"
included_paths:
- "**/*.tf"
- "**/*.tfvars"
- "**/*.md"
tags:
- demo
- github
- component: "weather"
source: "github.com/cloudposse/atmos.git//examples/demo-library/{{ .Component }}?ref={{.Version}}"
version: "main"
targets:
- "components/terraform/{{ .Component }}/{{.Version}}"
tags:
- demo
- component: "ipinfo"
source: "github.com/cloudposse/atmos.git//examples/demo-library/{{ .Component }}?ref={{.Version}}"
version: "main"
targets:
- "components/terraform/{{ .Component }}/{{.Version}}"
tags:
- demo
We also provide a well-maintained reference architecture for AWS (which is how Cloud Posse makes money). The docs are public here: https://docs.cloudposse.com
The turnkey architecture for AWS, Datadog & GitHub Actions to get up and running quickly using the Atmos open source framework.
Oh, we also have some design patterns that might be helpful.
https://atmos.tools/design-patterns/organizational-structure-configuration
Organizational Structure Configuration Atmos Design Pattern
2024-11-07
Hello, atmos describe affected
always output the component that changed , or that is a somewhat recent change?
Not sure the question
The whole point is to output the components and stacks that changed
(so it’s always done that)
It also conveys what triggered the change as some metadata in the JSON
ok, for some weird reason I thought it was only yaml updates to stacks
it has been a coincidence that I have always done stack.yaml changes together with component changes
and since now is the first time I use describe affected
on a pipeline, I just realized yesterday that a component change will show as a change
the command checks a lot of things: atmos stack for the component, terraform code in the component folder, and terraform modules in any other folder that the current component uses in terraform (atmos detects it by using terraform metadata about the component)
ahhh interesting
also, it checks the depends_on attributes in the stacks, if a component depends on an external file or folder, and the file or folder changes, the component will be considered affected
Add .coderabbit.yaml for CodeRabbit integration configuration settings @osterman (#758)
what
• Add coderrabit config • Tune the prompt • Enable linear integration
why
• Want to work towards a config that is less noisy (although this is probably not the PR that solves that)
Enhancements
handle invalid command error @pkbhowmick (#766)
what
• Improved error handling for command arguments, providing clearer feedback when invalid commands are used • Enhanced logging to include a list of available commands when an error occurs due to invalid arguments.
why
• Better user experience
working example
Before:
Screenshot 2024-11-08 at 1 56 30 AM
After fix:
2024-11-08
Skip component if metadata.enabled
is set to false
@pkbhowmick (#756)
what
• Skip component if metadata.enabled is set to false
• Added documentation on using the metadata.enabled
parameter to conditionally exclude components in deployment
why
• Allow disabling Atmos components from being processed and provisioned by setting metadata.enabled
to false
in the stack manifest w/o affecting/changing/disabling the Terraform components (e.g. w/o setting the enabled
variable to false
)
demo
@Andriy Knysh (Cloud Posse) if we set the metadata.enabled
to false
will it also be ignore on CI/CD? or do we have to have both settings.github.actions_enabled
to false
?
Skip component if metadata.enabled
is set to false
@pkbhowmick (#756)
what
• Skip component if metadata.enabled is set to false
• Added documentation on using the metadata.enabled
parameter to conditionally exclude components in deployment
why
• Allow disabling Atmos components from being processed and provisioned by setting metadata.enabled
to false
in the stack manifest w/o affecting/changing/disabling the Terraform components (e.g. w/o setting the enabled
variable to false
)
demo
related to the question: describe affected
will show it as change?
i guess the second question relates to the first one
@Pulak Kanti Bhowmick can you please confirm what atmos describe affected
will return if a component is disabled using metadata.enabled: false
if I add anything to metadata
like sdfsfdsafdsdf: ff
it shows on atmos describe affected
output
using atmos latest
Hi @Andriy Knysh (Cloud Posse), let me check and get back here
it should be affected because it’s changed
it’s up to the caller to decide what to do with the disabled component
@jose.amengual I believe you were asking for this functionality (metadata.enabled
) ^
2024-11-10
Wrapper for long lines in help @Cerebrovinny (#770)
what
• Implemented a new terminal-aware text wrapping system for CLI help output • Added responsive width handling based on terminal size with fallback values • Introduced custom usage template handling for consistent help text formatting • Created dedicated terminal writer component for automatic text wrapping
why
• Improves readability of CLI help text by ensuring content fits within terminal width • Provides better user experience with dynamic text wrapping based on terminal size • Standardizes help text formatting across all commands • Fixes potential issues with text overflow in narrow terminal windows
references
Before:
Screenshot 2024-11-09 at 16 15 26
Screenshot 2024-11-09 at 16 31 54
Hi, I’ve been reading up on your documentation on Atmos and the reference architecture for the past 1 or 2 weeks and some things are not clicking in my head. I hope you can give me some pointers in the right direction because it is not easy to find this on the Internet.
For AWS, I see references in the mixins to regions, tenants and stages but the sample info only gives me a name, but I’m not seeing how it relates to let’s say an Account ID. This for example:
vars:
stage: sandbox
# Other defaults for the `sandbox` stage/account
Am I overlooking some documentation part where I can see what can be a default for OUs/Accounts? There should be some relationship to AWS terminology right?
I hope you can give me a hint. Thanks in advance!
For additional context, do you mean these docs:
doc.cloudposse.com (cloud posse’s reference architecture)
Or https://atmos.tools?
That will help me address any confusion.
Hi, also there I see not the relationship made between let’s say a stage and the account ID.
It’s one of those pieces that would make Atmos click in my head. Architecture wise I can relate Catalogs/mixins/stacks etc but somewhere down the line you can’t apply to a ‘sandbox’ as long that ‘sandbox’ has an account id…right?
So just to disambiguate some things, in cloudposse’s refarch (docs.cloudposse.com), we by convention tie a stage (dev, staging, production) to an AWS Account.
In our refarch, we have something called the account-map
https://docs.cloudposse.com/components/library/aws/account-map/
This component is responsible for provisioning information only: it simply populates Terraform state with data (account
This is what handles that mapping
This is what allows us to refer to everything by name, instead of thinking of accounts account IDs
But it’s also what can make using the Cloud Posse refarch harder with brownfield environments https://atmos.tools/core-concepts/components/terraform/brownfield/
There are some considerations you should be aware of when adopting Atmos in a brownfield environment. Atmos works best when you adopt the Atmos mindset.
All that said, I want to point out that these are not Atmos conventions, these are Cloud Posse conventions in our reference architecture for AWS that uses Atmos
The account-map returns an IAM role that is used to access a given account
the IAM role will have the AWS account ID information
(that’s the missing gap, I believe in the understanding)
Thanks for your answers.
I saw on GH a file (see below) and it gives me some direction. But maybe it’s me and then I think “what are all the possible defaults?
ah..that last one.
# Global variables used for account maps, role maps and other global values
vars:
account_map:
dev: 222222222222
staging: 333333333333
automation: 111111111111
prod: 444444444444
Yes, good question. I’ll answer that in a sec.
(Also, in a future release of our refarch, we intend to move away from the account-map convention to make it easier to use in brownfield environments)
Ah…that’s where my head indeed is: How on earth would I use this in brownfield situations? We have a lot of those; Customers that tried first themselves and then start looking for a partner.
Understandable… so what we have as our refarch was born from 99% of our engagements which are what we call “cold starts”, so brownfields considerations we seldom a concern. As we now reach a much wider audience, that’s coming up more and more often.
We’re doing some groundwork changes first, related to https://github.com/cloudposse/terraform-aws-components/issues/1177
Hello, Cloud Posse Community!
We’re excited to announce that starting on November 12, 2024, we will begin migrating each component in the cloudposse/terraform-aws-components
repository to individual repositories under a new GitHub organization. This change aims to improve the stability, maintainability, and usability of our components.
Why This Migration?
Our goal is to make each component easier to use, contribute to, and maintain. This migration will allow us to:
• Leverage terratest
automation for better testing.
• Implement semantic versioning to clearly communicate updates and breaking changes.
• Improve PR review times and accelerate community contributions.
• Enable Dependabot automation for dependency management.
• And much more!
What to Expect Starting November 12, 2024
• Migration Timeline: The migration will begin on November 12 and is anticipated to finish by the end of the following week.
• Code Freeze: Starting on November 12, this repository will be set to read-only mode, marking the beginning of a code freeze. No new pull requests or issues will be accepted here after that date.
• New Contribution Workflow: After the migration, all contributions should be directed to the new individual component repositories.
• Updated Documentation: To support this transition, we are updating our documentation and cloudposse-component
updater.
• Future Archiving: In approximately six months, we plan to archive this repository and transfer it to the cloudposse-archives
organization.
Frequently Asked Questions
• Does this affect Terraform modules? No, only the terraform-aws-components
repository is affected. Our Terraform modules will remain where they are.
We are committed to making this transition as seamless as possible. If you have any questions or concerns, please feel free to post them in this issue. Your feedback is important to us, and we appreciate your support as we embark on this new chapter!
Thank you,
The Cloud Posse Team
And once we have that in place, we can start making more breaking changes to our components to support brownfield - while keeping our ecosystem stable.
Sounds really promising! I’m going to do some dishes over here (CET) and then just start with this afterwards.
Thanks for all your time!
@Andriy Knysh (Cloud Posse) @Jeremy G (Cloud Posse) where do we have a full definition of the static backend
I need to add it here so we can document it.
There are some considerations you should be aware of when adopting Atmos in a brownfield environment. Atmos works best when you adopt the Atmos mindset.
I don’t think we have a full definition anywhere. In part, that is because it is too simple. See the examples here. You just set the remote_state_backend_type
to static
and set the outputs as a map under remote_state_backend.static
. Although the example shows setting backend_type: static
, that is a bit misleading because you cannot run terraform plan
etc. with a static backend.
Also, what about a static account-map? @NotWillFarrell is working in a brownfield
My recommendation for brownfield is to use a static backend for account
to create the list of accounts, and use the real account-map
. Account map is an information collector and processor, it it does not manage any cloud resources. If account-map
contains information for accounts you are not using, that should not cause problems.
The example @NotWillFarrell cited:
# Global variables used for account maps, role maps and other global values
vars:
account_map:
dev: 222222222222
staging: 333333333333
automation: 111111111111
prod: 444444444444
is from before account-map
got that information from account
. Convert that to a static backend for account
and I think account-map
will work fine. If not, we should fix account-map
so that it does.
here is an example of the static
backend for account
components:
terraform:
account:
backend:
s3:
role_arn: null
vars:
enabled: false
# Use `static` remote state to configure the attributes (outputs) for the existing organization, OUs and accounts
remote_state_backend_type: static
remote_state_backend:
static:
account_arns:
- "arn:aws:organizations::xxxxxxxxxxx:account/o-xxxxxxxxxxx/xxxxxxxxxxx"
- "arn:aws:organizations::xxxxxxxxxxx:account/o-xxxxxxxxxxx/xxxxxxxxxxx"
- "arn:aws:organizations::xxxxxxxxxxx:account/o-xxxxxxxxxxx/xxxxxxxxxxx"
account_ids:
- "xxxxxxxxxxx"
- "xxxxxxxxxxx"
- "xxxxxxxxxxx"
account_info_map: {}
account_names_account_arns: {}
account_names_account_ids: {}
organization_arn: "arn:aws:organizations::xxxxxxxxxxx:organization/o-xxxxxxxxxxx"
organization_id: "o-xxxxxxxxxxx"
all the outputs returned from the account
component https://github.com/cloudposse/terraform-aws-components/blob/main/modules/account/outputs.tf. can be added to the static backend and then used in other components as if they were returned from the account
remote state
output "account_arns" {
value = local.all_account_arns
description = "List of account ARNs (excluding root account)"
}
output "account_ids" {
value = local.all_account_ids
description = "List of account IDs (excluding root account)"
}
output "organizational_unit_arns" {
value = local.organizational_unit_arns
description = "List of Organizational Unit ARNs"
}
output "organizational_unit_ids" {
value = local.organizational_unit_ids
description = "List of Organizational Unit IDs"
}
output "account_info_map" {
value = local.account_info_map
description = <<-EOT
Map of account names to
eks: boolean, account hosts at least one EKS cluster
id: account id (number)
stage: (optional) the account "stage"
tenant: (optional) the account "tenant"
EOT
}
output "account_names_account_arns" {
value = local.account_names_account_arns
description = "Map of account names to account ARNs (excluding root account)"
}
output "account_names_account_ids" {
value = local.account_names_account_ids
description = "Map of account names to account IDs (excluding root account)"
}
output "organizational_unit_names_organizational_unit_arns" {
value = local.organizational_unit_names_organizational_unit_arns
description = "Map of Organizational Unit names to Organizational Unit ARNs"
}
output "organizational_unit_names_organizational_unit_ids" {
value = local.organizational_unit_names_organizational_unit_ids
description = "Map of Organizational Unit names to Organizational Unit IDs"
}
output "organization_id" {
value = local.organization_id
description = "Organization ID"
}
output "organization_arn" {
value = local.organization_arn
description = "Organization ARN"
}
output "organization_master_account_id" {
value = local.organization_master_account_id
description = "Organization master account ID"
}
output "organization_master_account_arn" {
value = local.organization_master_account_arn
description = "Organization master account ARN"
}
output "organization_master_account_email" {
value = local.organization_master_account_email
description = "Organization master account email"
}
output "eks_accounts" {
value = local.eks_account_names
description = "List of EKS accounts"
}
output "non_eks_accounts" {
value = local.non_eks_account_names
description = "List of non EKS accounts"
}
output "organization_scp_id" {
value = join("", module.organization_service_control_policies.*.organizations_policy_id)
description = "Organization Service Control Policy ID"
}
output "organization_scp_arn" {
value = join("", module.organization_service_control_policies.*.organizations_policy_arn)
description = "Organization Service Control Policy ARN"
}
output "account_names_account_scp_ids" {
value = local.account_names_account_scp_ids
description = "Map of account names to SCP IDs for accounts with SCPs"
}
output "account_names_account_scp_arns" {
value = local.account_names_account_scp_arns
description = "Map of account names to SCP ARNs for accounts with SCPs"
}
output "organizational_unit_names_organizational_unit_scp_ids" {
value = local.organizational_unit_names_organizational_unit_scp_ids
description = "Map of OU names to SCP IDs"
}
output "organizational_unit_names_organizational_unit_scp_arns" {
value = local.organizational_unit_names_organizational_unit_scp_arns
description = "Map of OU names to SCP ARNs"
}
The only outputs of account
used by account-map
are
• eks_accounts
• non_eks_accounts
• account_info_map
The first 2 are just lists of account names, indicating which accounts are expected to have EKS deployments and which are not. Every account should be in one of those lists.
account_info_map
is a map of account name to various pieces of information. In the real account
output, there is more information, but I think you only need to fill in:
• eks: boolean, account hosts at least one EKS cluster
• id: account id (number)
• stage: the account “stage”
• tenant: (optional) the account “tenant”, or null
if you are not using tenant
Example:
account_info_map:
artifacts:
eks: false
id: "123456789012"
stage: "artifacts"
tenant: null
dev:
eks: true
id: "210987654321"
stage: "dev"
tenant: null
The other outputs of account
are used by components that manage the accounts and organizations, but you can skip all that if you already that set up.
cc: @Erik Osterman (Cloud Posse)
2024-11-11
feat: additional atmos docs
parameters for specifying width, using auto-styling and color profile, and preserving new lines @RoseSecurity (#757)
what
• Add an additional atmos docs
flag for specifying the width of markdown output
• Utilizing auto-styling based on light or dark mode preferences instead of hardcoding to dark
• Preserving new lines with rendered markdown
why
• Enhance the user experience for interacting with documentation. The width
parameter is useful for users who prefer seeing wider output for Terraform docs-generated tables and is defined in the atmos.yaml
:
settings: docs: max-width: 200
references
Change PS1 to show that Atmos is in the atmos terraform shell
mode @pkbhowmick (#761)
what
• Change PS1 to show that Atmos is in the atmos terraform shell
mode
• Customized command prompt for the interactive shell with the addition of the “atmos>” prefix
• Enhanced shell behavior by removing the unnecessary -l
flag for non-Windows systems and implementing a fallback to sh
if bash
is unavailable.
• Improved handling for the /bin/zsh
shell with additional flags
why
• Improve user experience
test
@RB this closes something you asked for a long time ago
Change PS1 to show that Atmos is in the atmos terraform shell
mode @pkbhowmick (#761)
what
• Change PS1 to show that Atmos is in the atmos terraform shell
mode
• Customized command prompt for the interactive shell with the addition of the “atmos>” prefix
• Enhanced shell behavior by removing the unnecessary -l
flag for non-Windows systems and implementing a fallback to sh
if bash
is unavailable.
• Improved handling for the /bin/zsh
shell with additional flags
why
• Improve user experience
test
I saw that! Thank you very much
2024-11-12
does anybody have thoughts on how to manage ECS image tags? we are used to creating different image tags whenever something is ready to be tested or to go to production on each project’s pipeline. So let’s say I have an application that just generated a new docker tag corresponding to some new changes, and that this tag is now saved in ECR, how can I reflect this image tag in my atmos/infrastructure repository? at first I thought about the app opening a PR to atmos, changing the line that corresponds to the image, but I’m unsure if this is the best way. How do you typically deal with this?
Our ECS components solve this using SSM parameters
We use the ecspresso deployment tool for Amazon ECS to manage ECS services using a code-driven approach, alongside reusable GitHub Action workflows. This setup allows tasks to be defined with Terraform within the infrastructure repository, and task definitions to reside alongside the application code. Ecspresso provides extensive configuration options via YAML, JSON, and Jsonnet, and includes plugins for enhanced functionality such as Terraform state lookups.
I’ll take a look. Thank you!
Clean Terraform workspace before executing terraform init
in the atmos.Component
template function @aknysh (#775)
what
• Clean Terraform workspace before executing terraform init
in the atmos.Component
template function
why
When using multiple backends for the same component (e.g. separate backends per tenant or account), and if an Atmos command was executed that selected a Terraform workspace, Terraform will prompt the user to select one of the following workspaces:
- default
-
The prompt forces the user to always make a selection (which is error-prone), and also makes it complicated when running on CI/CD.
This PR adds the logic that deletes the .terraform/environment
file from the component directory before executing terraform init
when executing the atmos.Component
template function. It allows executing the atmos.Component
function for a component in different Terraform workspaces without Terraform asking to select a workspace. The .terraform/environment
file contains the name of the currently selected workspace, helping Terraform identify the active workspace context for managing your infrastructure.
Hi,
i’ve found, that the validate output differs between 1.98 and 1.99 (and ever since). I don’t know if i am doing anything wrong or if its a bug:
old behavior (1.98.0)
❯ atmos validate component wms-base -s wms-xe02-sandbox
component 'wms-base' in stack 'wms-xe02-sandbox' validated successfully
new behavior (1.99.0)
❯ atmos validate component wms-base -s wms-xe02-sandbox
'atmos' supports native ' wms-base' command with all the options, arguments and flags.
In addition, 'component' and 'stack' are required in order to generate variables for the component in the stack.
atmos wms-base <component> -s <stack> [options]
atmos wms-base <component> --stack <stack> [options]
component 'wms-base' in stack 'wms-xe02-sandbox' validated successfully
this is a regression from the many new PRs, we’ll fix it, thank you @Stephan Helas
2024-11-13
Is there already prior art using github actions and atmos to use a readonly for the plan and an admin role for the apply ?
That way if the plan was compromised without the ability to apply, no one could do any funny business with a local exec data source
@jose.amengual where did we leave off with this?
Waiting for @Andriy Knysh (Cloud Posse) to have some time to go over the failing tests on my PR
in the convo we have in the other slack
Quick update: @Igor Rodionov will review the GH action PR (prob tomorrow), and @Andriy Knysh (Cloud Posse) will check what @jose.amengual said about Atmos (not related to the GH action)
Explore this post and more from the Terraform community
Add support for vendor path setting in atmos.yaml
@Cerebrovinny (#737)
what
• Add support for vendor path setting in atmos.yaml
• Add support for vendor files under folders or multiple vendor files to be processed in lexicographic order
why
• Users now should be able to use new variable vendor in atmos.yaml
and process different vendor files at different locations
2024-11-14
I need to share my local submodules/child modules with every component that I will define in atmos. I’d like to move to atmos, but I have a codebase that I need to migrate to atmos with many modules. At this point I cannot afford to rewrite each small module into a component, but I’d like to move my root modules into atmos components as a starting point. I’d like to build my components out of the existing modules. Any advice how to approach?
@toka please show the current file system layout
you can use your TF modules as components (there is nothing special about components)
if you put your modules into components/terraform
and define Atmos stacks in stacks
, it should work.
Atmos will generate the backend and varfile for the modules and execute Terraform
Yes, “components” are really just a philosophy. You don’t need to rewrite anything.
…it won’t be worse off than it is now, but it won’t benefit from the way of thinking about architectures as made up of components.
Also, components typically will comprise a solution, so they shouldn’t be as small as a small module.
Atmos can change how you think about the Terraform code you write to build your infrastructure.
Learn the opinionated “Best Practices” for using Components with Atmos
2024-11-15
Hi all, if you folks have a second, i have a couple questions on the component migration. Very excited for the component testing
https://github.com/cloudposse/terraform-aws-components/issues/1177#issuecomment-2474148290
Ah thanks, I will respond later today
@nitrocode thanks for raising these questions.
- Will all the same components be available after the migration?
• Yes, we’re migrating the components as is, bit-for-bit, to facilitate a switch. However, we anticipate promptly doing major releases on many of the components after that point, to introduce new functionality and improve brownfield infra (a large driver for the initiative). Those breaking changes will likely require local changes in your configurations. That won’t happen immediately, but is the reason we’re doing all this.
- Will components still be open source?
Yes, we have no plans to change license.
- What will the new organization be named?
Looked through the source code of the component updater and saw this https://github.com/cloudposse-terraform-components
Correct, you found it!
https://github.com/cloudposse-terraform-components
As part of our transition to GitHub Enterprise (GHE), we are reorganizing our open-source projects into more purpose-built organizations. This allows us to better manage repository rulesets, GitHub Apps, and other configurations specific to each organization’s purpose. This approach enhances our security posture and improves discoverability. Additionally, keeping our components separate from less opinionated child modules avoids confusion and ensures clearer organization.
- Will the component updater be updated to allow overriding the above org to use different sources if needed?
The component updater uses the sources as define in the vendor and component manifests. Thus, that’s supported today.
One thing we’ve added to the component updater to make this switch less painful, is the ability for it to rewrite the sources to their new homes. So if it see’s references to components in cloudposse/terraform-aws-components
it will rewrite those to the new locations.
- Would you folks consider opening up the codeowners for components once they are all in their own repositories like you folks do with terraform modules ?
Yes, so we’ll be able to accept more contributions of components and delegate ownership of components with this move. Note, CODEOWNERS
only works with paid GitHub seats, so I think we’ll continue to look for solutions that work better for non-org members, such as “allow lists” that we’ve implemented elsewhere.
hey folks, is there a way to generate kubernetes provider blocks for different cloud providers? scenario: stack-1 is ecs, stack-2 is gke, and i’d like to use the same components for kubernetes resources.
i can output host, cert and token from the cluster components, however i’d like to configure kubernetes provider with oauth2 access token. using google_client_config
data source for example.
Are you familiar with the atmos provider generation?
Configure and override Terraform Providers.
@Kalman Speier the question is, how do you handle the auth token. In terraform, you can get it from a secret storage (SSM/ASM, GCP vault, etc.), and send it as an input to the other resources/modules
the token should def not be hardcoded in Atmos stacks
I guess you are asking about how to do it in Terraform (get the token from data sources and then use it as input to other modules, per cloud)
what i’d like to achieve is simply like this: https://github.com/hashicorp/terraform-provider-kubernetes/blob/main/_examples/gke/main.tf
so, the kubernetes provider is configured outside of the component which is responsible to deploy k8s resources.
if the cluster outputs the token and i set as a variable to my k8s component that works fine, however these tokens are expiring, hence i’d like to use a provider specific datasource but i couldn’t place that in the component obviously.
-
create a cluster on gke for example
-
run cloud specific kubernetes provider configuration ``` data “google_client_config” “default” {}
provider “kubernetes” { token = data.google_client_config.default.access_token … } ```
- deploy my k8s resources using the configured provider without the component knowing about which cloud provider in use
the #2 needs to run each time #3 is running, because outputs from state isn’t working as tokens expiring.
i would put all those providers and the data sources in separate files per cloud (obe for AWS/ECS, one for GKE) Then add a variable defining the cloud
variable "cloud" {
description = "Cloud provider"
}
then in each provider "kubernetes"
, use count
depending on the cloud
variable
then in [main.tf](http://main.tf)
, use https://developer.hashicorp.com/terraform/language/functions/coalesce to read the token from all the data sources
The coalesce function takes any number of arguments and returns the first one that isn’t null nor empty.
similar to
token = coalesce(data.google_client_config.default.xxxx, data.xxxx.xxxx.xxxx)
too bad that the provider
block still does not support count
and for_each
(people have been asking for that for many years).
So oyu will need to make sure your TF code has access to all clouds at once for the provider
blokcs to work and not throw errors
https://support.hashicorp.com/hc/en-us/articles/6304194229267-Using-count-or-for-each-in-Provider-Configuration#<i class="em em-~"</i>text=While%20a%20longtime%20requested%20feature,provider%20configuration%20block%20in%20Terraform>.
Current Status While a longtime requested feature in Terraform, it is not possible to use count or for_each in the provider configuration block in Terraform. Background Much of the reasoning be…
i would create a common TF module to deal with the clusters, then a few root modules (components) for the specific clouds
the common module would accept the token as a variable
the parent/root modules would read the token using the corresponding cloud provider and provide it in the variable to the child module
and if you are using Atmos, it’s easy to configure your components pointing to the diff root modules per cloud
components:
terraform:
ecs-component:
metadata:
component:
ecs/xxxxx. # Point to the Terraform component (root module)
gke-component:
metadata:
component:
gke/xxxxx. # Point to the Terraform component (root module)
ecs/xxxxx
terraform component uses a data source to read the token from SSM/ASM
gke/xxxxx
terraform component uses data "google_client_config" "default" {}
to read the token from GCP
ok, thanks a lot, i will think about these.
tha main point is to create a common TF child module (with almost all the code except the code to read the token from the data sources), then reuse it in the root modules using diff providers
Hey guys I was looking for a way to read secrets from 1password and saw this PR: https://github.com/cloudposse/atmos/pull/762 What are the plans for this?
This will actually solved a loot of issues and simplify my work on some projects hehe
what
Integrate vals as a template function.
why
Loading configuration values and secrets from external sources, supporting various backends.
Summary by CodeRabbit
• New Features
• Introduced the atmos.Vals
template function for loading configuration values and secrets from external sources.
• Added a logging mechanism for improved tracking of value operations.
• Updates
• Updated various dependencies to newer versions, enhancing compatibility with cloud services and improving overall performance.
• Documentation
• Added comprehensive documentation for the atmos.Vals
template function, including usage examples and security best practices.
@Erik Osterman (Cloud Posse) any updates on this?
what
Integrate vals as a template function.
why
Loading configuration values and secrets from external sources, supporting various backends.
Summary by CodeRabbit
• New Features
• Introduced the atmos.Vals
template function for loading configuration values and secrets from external sources.
• Added a logging mechanism for improved tracking of value operations.
• Updates
• Updated various dependencies to newer versions, enhancing compatibility with cloud services and improving overall performance.
• Documentation
• Added comprehensive documentation for the atmos.Vals
template function, including usage examples and security best practices.
Our plan is leaning more towards implementing a pluggable way of storing/retrieving values from directly within Atmos, but supporting many of these same backends.
I’ll have a more conclusive answer by the end of this week.
oh nice thanks!
Update gettings started, add $schema directive at the top of files @osterman (#769)
what
• Remove unimplemented commands • $schema directive at the top of files
why
• Not everyone will have $schema validation enabled by default in their editor
Enhance WriteToFileAsJSON
with pretty-printing support @RoseSecurity (#783)
what
• Used the ConvertToJSON
utility with json.MarshalIndent
to produce formatted JSON
• Indentation is set to two spaces (“ “) for consistent readability
why
• This PR improves the WriteToFileAsJSON
function by introducing pretty-printing for JSON outputs. Previously, the function serialized JSON using a compact format, which could make the resulting files harder to read. With this change, all JSON written by this function will now be formatted with indentation, making it easier for developers and users to inspect and debug the generated files
• This specifically addresses #778 , which previously rendered auto-generated backends as:
{ “terraform”: { “backend”: { “s3”: { “acl”: “bucket-owner-full-control”, “bucket”: “my-tfstate-bucket”, “dynamodb_table”: “some-dynamo-table”, “encrypt”: true, “key”: “terraform.tfstate”, “profile”: “main”, “region”: “us-west-2”, “workspace_key_prefix”: “something” } } } }
With this addition, the output appears as:
{ “terraform”: { “backend”: { “s3”: { “acl”: “bucket-owner-full-control”, “bucket”: “my-tfstate-bucket”, “dynamodb_table”: “some-dynamo-table”, “encrypt”: true, “key”: “terraform.tfstate”, “profile”: “main”, “region”: “us-west-2”, “workspace_key_prefix”: “something” } } } }
references
• Stack Overflow • Closes #778
2024-11-16
Add support for custom atmos terraform shell
prompt @pkbhowmick (#786)
what
• Add support for custom atmos terraform shell
prompt
• Allow specifying custom prompt for atmos terraform shell
command in atmos.yaml
. Supports Go
templates
why
• Improve user experience • Make the prompt customizable
Working demo
With custom prompt:
Screenshot 2024-11-16 at 11 20 14 PM
Without custom prompt:
@Michael this should restore the behavior in #geodesic
Add support for custom atmos terraform shell
prompt @pkbhowmick (#786)
what
• Add support for custom atmos terraform shell
prompt
• Allow specifying custom prompt for atmos terraform shell
command in atmos.yaml
. Supports Go
templates
why
• Improve user experience • Make the prompt customizable
Working demo
With custom prompt:
Screenshot 2024-11-16 at 11 20 14 PM
Without custom prompt:
We disable the prompt formatting by default, and instead allow it to be customized in atmos.yaml
Awesome stuff, thank you for such a quick turnaround on it! Excited to try it out
2024-11-18
Hey guys, thanks for awesome project! Trying to recreate multi workspace project in terraform.io with atmos.
My use case is provisioning pretty same infra for multiple tenants. Need your advise on how to proper organise variables.
Each component in tenant share a list of variables like project_id
and region
which I put to mixin with the same name as tenant.
Then for each component I’m passing project_number
with atmos.Component
(tenants are named as pokemons):
deploy/bulbasaur-stg.yaml
vars:
tenant: bulbasaur
stage: stg
import:
- path: "deploy/_defaults.yaml.tmpl"
context:
stack: bulbasaur
components:
terraform:
tenant:
vars:
foo: bar
db:
vars:
project_number: '{{ (atmos.Component "tenant" .stack).outputs.project_number }}' <-- this I also want to DRY somehow
cloudrun:
vars:
project_number: '{{ (atmos.Component "tenant" .stack).outputs.project_number }}'
jobs:
vars:
project_number: '{{ (atmos.Component "tenant" .stack).outputs.project_number }}'
All good for now, then for cloudrun
and jobs
components I need same list of ENV variables that are used to provision docker image.
The problem here is that I want to use previously defined project_id
and pokemon_name
in templating of those envs…
I tried to create mixin and thought that it can be templated like that:
mixins/tenants/bulbasaur-stg.yaml
vars:
region: europe-west3
env_vars:
TENANT: '{{ .vars.tenant }}'
DATABASE_USER: 'user@{{ .vars.project_id }}.iam'
BIGQUERY_PROJECTID: '{{ .vars.project_id }}'
...
deploy/_defaults.yaml.tmpl
import:
- mixins/tenants/{{ .stack }}
terraform:
backend_type: gcs
backend:
gcs:
bucket: "tf-state"
It is not working giving me <no-value>
for TENANT
Clearly I’m doing it wrong. Should I create a component that just outputs env_vars
instead and pass it to cloudrun and jobs?
P.S. I have name_pattern: "{tenant}-{stage}"
in atmos.yaml
I’ve not yet had a chance to read through the entire message
Note, you will likely need to commit the varfiles for it, for it to work with TFC
Also, dynamic backend generation will not work well, if you use multiple backends for the same component (E.g. by region)
Opentofu is considering deprecating workspaces
https://github.com/opentofu/opentofu/issues/2160
Is it possible to use atmos without workspaces and instead use unique keys per stack instead of unique workspaces per stack ?
Currently, Atmos relies on the concept of workspaces for managing unique configurations and state per stack. However, with the introduction of OpenTofu’s Early Evaluation feature, there is potential to move away from workspaces and instead use unique keys per stack to manage state more flexibly.
While this functionality is not natively supported yet, I believe it could be feasible to implement. By leveraging Early Evaluation, we could dynamically configure the backend state storage, using variables to differentiate each stack’s state, rather than depending on separate workspaces. This approach would allow us to specify unique keys based on the stack name or environment and ensure proper isolation of state per stack.
In essence, although Atmos doesn’t currently support a workspace-less setup, utilizing unique keys per stack with Early Evaluation could be a similar concept and an effective alternative worth exploring.
Therefore, I don’t anticipate that the CloudPosse team will find it impossible to adapt to the deprecation of workspaces. They are an incredibly talented group, and I’m confident in their ability to develop a robust solution or an alternative approach. With their expertise, I believe they will be able to leverage features like Early Evaluation effectively to maintain or even improve the functionality of Atmos without relying on workspaces.
Actually, I think it’s already supported, depending on your backend. With S3, it’s just a different path. Since the backends are entirely configurable in Atmos, I think it’s just about configuring the right path to match the workspace path, and dropping the workspace parameter.
We might need to add a parameter in atmos to disable workspace operations, but the lift on that is trivial.
We talk about the file structure of the S3 backend here https://docs.cloudposse.com/layers/accounts/tutorials/terraform-s3-state/
Understand the structure of a Terraform S3 state backend
So with that it should be as simple as updating the key to the fully qualified path to the workspace tfstate file
https://developer.hashicorp.com/terraform/language/backend/s3
Terraform can store state remotely in S3 and lock that state with DynamoDB.
Thanks, ill review!
Yes i think the workspace option would be needed.
You’re right, all the other stuff is there to override the backend key per component using some yaml magic (use stack name as the unique key, as you said)
Yes i think the workspace option would be needed.
Just to confirm, an option to disable usage of workspace operations?
:rocket: Enhancements
Handle empty stack YAML file configurations @haitham911 (#791)
what
• Handle empty stack YAML file configurations
why
• atmos validate stacks
should not error on empty stack manifest files
2024-11-19
For root modules that do not use the terraform-null-label module (e.g., modules from terraform-aws-modules instead of CloudPosse), I find it challenging to maintain a consistent naming convention for resources. Specifically, I use a mix of CloudPosse-provided modules and other third-party modules as needed, but ensuring uniformity in the naming and tagging of provisioned resources (not just the stack’s name_pattern, but the actual resource names) is difficult.
I’ve tried using the Component Vendor’s Mixin feature to blend context.tf, but it proved to be inconvenient.
Does anyone have ideas or alternative methods for achieving a uniform naming and tagging convention across all resources? Any suggestions would be greatly appreciated!
what I do is save the other third-party module in another directory and use that with CloudPosse context.tf file to create the naming.
For example:
components/vendor/aws/vpc
-> AWS Module
compoents/aws/vpc
-> custom module using the vendor aws vpc with CloudPosse [context.tf](http://context.tf)
file for naming and enable
ENV
and I will use the compoents/aws/vpc
module in catalog
@Miguel Zablah Thanks! I understand, but to help, could you give me a simple example? I get the general picture, but the ‘custom module using the vendor aws vpc with CloudPosse context.tf
file for naming and enable
ENV’ part doesn’t come to mind specifically.
If what you mean by custom module is ‘combine the newly attached context.tf with the “aws/vpc” component in the vendor directory to create a new Root Module (Component)’, this seems like it would be complicated to configure and maintain the component every time. Am I not understanding this correctly?
not really bc both are going to be manage by atmos vendor file, so using this example I use this two modules: https://github.com/cloudposse/terraform-null-label https://github.com/terraform-aws-modules/terraform-aws-vpc
so I add both of them to the Atmos vendor file and in a different directory on components in this example it will be something like this:
components/terraform/vendor/cloudposse/tf-null-label
-> https://github.com/cloudposse/terraform-null-label
components/terraform/vendor/aws/vpc
-> https://github.com/terraform-aws-modules/terraform-aws-vpc
than in I will create a new root module with this modules here:
components/terraform/aws/vpc
where I will have a this files:
[main.tf](http://main.tf)
-> to call the components/terraform/vendor/aws/vpc
[context.tf](http://context.tf)
-> to call the components/terraform/vendor/cloudposse/tf-null-label
so essentially what I do is copy there context file but reference the module internally kind of what they do in this example: https://github.com/cloudposse/terraform-null-label/blob/main/examples/autoscalinggroup/context.tf
and then I can use module.this.id
for naming, module.this.tag
for tagging and module.this.enabled
to enable/disable the module like CloudPosse dose on there modules all of this I will use it on [main.tf](http://main.tf)
when calling the VPC
module
hopefully this explains it a bit better
@Miguel Zablah Thanks for the detailed explanation, I understood it perfectly. My only further question is, so what I need to do is to actually create a components/terraform/vendor/aws/vpc
module block in main.tf
in the components/terraform/vendor/aws/vpc
directory and declare the required variables one by one inside the module block so that they are assignable (ex: azs, cidr, private_subnes, etc…)?
so this is how the root component will look like:
components/terraform/aws/vpc
:
• [main.tf](http://main.tf)
-> here you will put the module block calling the components/terraform/vendor/aws/vpc
• [context.tf](http://context.tf)
-> use this example but reference your vendor module components/terraform/vendor/cloudposse/tf-null-label
• [outputs.tf](http://outputs.tf)
-> you will expose the vpc module outputs (you can use a loop for this)
• [variables.tf](http://variables.tf)
-> this will be almost the same as the aws-vpc module but removing name and stuff that is manage by [context.tf](http://context.tf)
example [main.tf](http://main.tf)
:
module "vpc" {
count = module.this.enabled ? 1 : 0
source = "../../vendor/aws/vpc"
name = module.this.id
...
}
after this is setup correctly made you can use it as normally on your atmos catalog and what not
# DO NOT COPY THIS FILE
#
# This is a specially modified version of this file, since it is used to test
# the unpublished version of this module. Normally you should use a
# copy of the file as explained below.
#
# ONLY EDIT THIS FILE IN github.com/cloudposse/terraform-null-label
# All other instances of this file should be a copy of that one
#
#
# Copy this file from <https://github.com/cloudposse/terraform-null-label/blob/master/exports/context.tf>
# and then place it in your Terraform module to automatically get
# Cloud Posse's standard configuration inputs suitable for passing
# to Cloud Posse modules.
#
# Modules should access the whole context as `module.this.context`
# to get the input variables with nulls for defaults,
# for example `context = module.this.context`,
# and access individual variables as `module.this.<var>`,
# with final values filled in.
#
# For example, when using defaults, `module.this.context.delimiter`
# will be null, and `module.this.delimiter` will be `-` (hyphen).
#
module "this" {
source = "../.."
enabled = var.enabled
namespace = var.namespace
environment = var.environment
stage = var.stage
name = var.name
delimiter = var.delimiter
attributes = var.attributes
tags = var.tags
additional_tag_map = var.additional_tag_map
label_order = var.label_order
regex_replace_chars = var.regex_replace_chars
id_length_limit = var.id_length_limit
context = var.context
}
# Copy contents of cloudposse/terraform-null-label/variables.tf here
variable "context" {
type = object({
enabled = bool
namespace = string
environment = string
stage = string
name = string
delimiter = string
attributes = list(string)
tags = map(string)
additional_tag_map = map(string)
regex_replace_chars = string
label_order = list(string)
id_length_limit = number
label_key_case = string
label_value_case = string
})
default = {
enabled = true
namespace = null
environment = null
stage = null
name = null
delimiter = null
attributes = []
tags = {}
additional_tag_map = {}
regex_replace_chars = null
label_order = []
id_length_limit = null
label_key_case = null
label_value_case = null
}
description = <<-EOT
Single object for setting entire context at once.
See description of individual variables for details.
Leave string and numeric variables as `null` to use default value.
Individual variable settings (non-null) override settings in context object,
except for attributes, tags, and additional_tag_map, which are merged.
EOT
validation {
condition = var.context["label_key_case"] == null ? true : contains(["lower", "title", "upper"], var.context["label_key_case"])
error_message = "Allowed values: `lower`, `title`, `upper`."
}
validation {
condition = var.context["label_value_case"] == null ? true : contains(["lower", "title", "upper", "none"], var.context["label_value_case"])
error_message = "Allowed values: `lower`, `title`, `upper`, `none`."
}
}
variable "enabled" {
type = bool
default = null
description = "Set to false to prevent the module from creating any resources"
}
variable "namespace" {
type = string
default = null
description = "Namespace, which could be your organization name or abbreviation, e.g. 'eg' or 'cp'"
}
variable "environment" {
type = string
default = null
description = "Environment, e.g. 'uw2', 'us-west-2', OR 'prod', 'staging', 'dev', 'UAT'"
}
variable "stage" {
type = string
default = null
description = "Stage, e.g. 'prod', 'staging', 'dev', OR 'source', 'build', 'test', 'deploy', 'release'"
}
variable "name" {
type = string
default = null
description = "Solution name, e.g. 'app' or 'jenkins'"
}
variable "delimiter" {
type = string
default = null
description = <<-EOT
Delimiter to be used between `namespace`, `environment`, `stage`, `name` and `attributes`.
Defaults to `-` (hyphen). Set to `""` to use no delimiter at all.
EOT
}
variable "attributes" {
type = list(string)
default = []
description = "Additional attributes (e.g. `1`)"
}
variable "tags" {
type = map(string)
default = {}
description = "Additional tags (e.g. `map('BusinessUnit','XYZ')`"
}
variable "additional_tag_map" {
type = map(string)
default = {}
description = "Additional tags for appending to tags_as_list_of_maps. Not added to `tags`."
}
variable "label_order" {
type = list(string)
default = null
description = <<-EOT
The naming order of the id output and Name tag.
Defaults to ["namespace", "environment", "stage", "name", "attributes"].
You can omit any of the 5 elements, but at least one must be present.
EOT
}
variable "regex_replace_chars" {
type = string
default = null
description = <<-EOT
Regex to replace chars with empty string in `namespace`, `environment`, `stage` and `name`.
If not set, `"/[^a-zA-Z0-9-]/"` is used to remove all characters other than hyphens, letters and digits.
EOT
}
variable "id_length_limit" {
type = number
default = null
description = <<-EOT
Limit `id` to this many characters.
Set to `0` for unlimited length.
Set to `null` for default, which is `0`.
Does not affect `id_full`.
EOT
}
variable "label_key_case" {
type = string
default = null
description = <<-EOT
The letter case of label keys (`tag` names) (i.e. `name`, `namespace`, `environment`, `stage`, `attributes`) to use in `tags`.
Possible values: `lower`, `title`, `upper`.
Default value: `title`.
EOT
validation {
condition = var.label_key_case == null ? true : contains(["lower", "title", "upper"], var.label_key_case)
error_message = "Allowed values: `lower`, `title`, `upper`."
}
}
variable "label_value_case" {
type = string
default = null
description = <<-EOT
The letter case of output label values (also used in `tags` and `id`).
Possible values: `lower`, `title`, `upper` and `none` (no transformation).
Default value: `lower`.
EOT
validation {
condition = var.label_value_case == null ? true : contains(["lower", "title", "upper", "none"], var.label_value_case)
error_message = "Allowed values: `lower`, `title`, `upper`, `none`."
}
}
#### End of copy of cloudposse/terraform-null-label/variables.tf
btw this just how I do it there might be a better way to do this haha
Enhancements
Set Default Schema to Remote Schema @haitham911 (#777)
what
• Set Default Validation Schema to Remote Schema
why
• We should set the default schema to the remote atmos schema so that atmos validate work even if the user does not configure a validation schema
:wave: We’re experimenting with Atmos and seeing a strange behavior with templating:
$ atmos describe stacks --process-templates | grep Component
vpc_id: '{{ (atmos.Component "vpc" .stack).outputs.vpc_id }}'
vpc_id: '{{ (atmos.Component "vpc" .stack).outputs.vpc_id }}'
vpc_id: '{{ (atmos.Component "vpc" .stack).outputs.vpc_id }}'
It seems like templates just…aren’t being processed and I’m not really sure how to debug this…
The docs imply this should “just work”. I’m clearly missing something obvious, and would love some help.
(Atmos 1.107.1 on darwin/arm64
)
Some more context:
components:
terraform:
vpc:
vars:
enabled: true
name: "compute"
ipv4_primary_cidr_block: "10.1.0.0/16"
vpc_flow_logs_enabled: false
nat_gateway_enabled: true
public_subnets_enabled: true
vpc-flow-logs-bucket:
vars:
name: "vpc-flow-logs"
internal-domain-and-cert:
settings:
depends_on:
1:
component: vpc
vars:
create_wildcard_cert: true
vpc_id: '{{ (atmos.Component "vpc" .stack).outputs.vpc_id }}'
By default, templating is disabled. We originally implemented this as an escape hatch, but it’s become very popular.
It is (theoretically) super useful!
Ooo…I probably just want inheritance, huh?
So at Cloud Posse, in our refarch, we almost never use templating, and instead use inheritance and imports 99% of the time.
However, we acknowledge the usefulness of the template functions. We’re also working on improvements, which involve moving towards what YAML calls “explicit types”, which are basically first-class functions in YAML.
Also, since you’re using atmos.Component
make sure to read https://atmos.tools/core-concepts/stacks/templates/functions/atmos.Component#handling-outputs-containing-maps-or-lists
Read the remote state or configuration of any Atmos component
The docs imply this should “just work”. I’m clearly missing something obvious, and would love some help.
Yes, that might be the case. We’re working on 2 things.
- Warning if you’re using templates and have it disabled
- Improving the docs to call it out
If you could share a screenshot or link to the page/chapter that you encountered it, I’ll fix it
Jumps right out at you in the docs. Root page and all…
Yup, that could definitely need some TLC
Thanks!!
I appreciate the context, but do you have any tips on how I can reference the vpc_id
from vpc
in internal-domain-and-cert
in my example above? All the links you’ve passed along seem to talk about sharing data between stacks, and I just need to pass between components in a single stack here.
This, no?
Share data between loosely-coupled components in Atmos
Yep. That is what isn’t working.
But do you have templating enabled in atmos.yaml
That looks like what I was missing!
Awesome. Thanks, Eric.
Definitely just saw that root page before I read more deeply about templating.
I’m going to update “Share Data Between Components”, to reference back to Template Configurations, to help avoid this snafu
and call out that templating needs to be enabled.
That’s great. The docs are generally pretty robust, which may have lulled me into a false sense of “well…if they say this just works…”
Hi, I’m assessing Atmos in a use case that needs to provision a set of infrastructure components that are deployed separately (their own TF root module), in different AWS accounts. I also have to use Atlantis to apply changes. To simplify if I have three components C1, C2 and C3. C2 and C3 depend on the result of applying C1. I want to orchestrate the plan/apply flow of C1, C2 and C3 in that order. With Atmos, do I need to define a Workflow and how would it be used from Atlantis?
A workflow is one way to do that. We haven’t looked into solving ordered dependencies in a way that provides first-class support for Atlantis. But to your point, you could create a custom workflow in atmos that you call via Atmos. The issue is you really want to review/approve each individual component’s plan. So for that, a more elaborate approach is necessary.
@jose.amengual is one of the #atlantis maintainers and might have some other ideas
Thanks for the reply. I was initially confused with Atmos stacks thinking that I could deploy in an ordered way the components listed in it. But reading the documentation I realised is probably a workflow. Another option I’ve been thinking is to do this via CI/CD and build the dependencies in each stage of the CI/CD pipeline. So, what is atmos solution to be able to orchestrate multiple Terraform root modules that depend on each other?
Atlantis supports dependencies
we could potentially ask CloudPosse to add component dependencies to the automatic workflow generation that atmos can create
That would be great. We’re not using Atlantis dependencies at the moment. It would be good to find a away to orchestrate the plan/apply flow of multiple components (TF root modules) that depend on each other.
Oh that’s interesting, I think I forgot about that
Since we already represent dependencies in stack configs, it’s maybe a simple mapping
@karel_alfonso did you see the atmos Atlantis generation?
It’s templatized so it might already be possible
Atmos natively supports Atlantis for Terraform Pull Request Automation.
https://www.runatlantis.io/docs/repo-level-atlantis-yaml.html#example-using-all-keys (depends-on)
depends_on:
- project-1
Atlantis: Terraform Pull Request Automation
@karel_alfonso did you see the atmos Atlantis generation?
No, haven’t looked into that yet. Will the generation take into account the dependencies in the stack config?
But the entire Atlantis config is one big template
And atmos stacks define dependencies
The templates have the full context of the stack configs, so it should be possible to define the Atlantis dependencies
The way to think about it is you are transforming the shape of one YAML configuration to the shape of another
Atmos supports configuring the relationships between components in the same or different stacks. You can define dependencies between components to ensure that components are deployed in the correct order.
Look at the example here: https://atmos.tools/integrations/atlantis
Atmos natively supports Atlantis for Terraform Pull Request Automation.
that section, I believe is not free form Erik
What do you mean by free form?
I think if you add a line that is not predefined, atmos does not add it to the generated atlantis,yaml
Aha
Ok, that is plausible- but and easy change
yes, I remember Andry adding a few lined when I was testing the integration pretty quickly
Ok, so @karel_alfonso if you end up pursuing this route, we may need to make some tweaks, but I don’t think it would be anything radical.
Atmos supports configuring the relationships between components in the same or different stacks. You can define dependencies between components to ensure that components are deployed in the correct order.
Is there a way to apply an entire stack with its dependencies using atmos CLI (without Atlantis) for demonstration purposes? I want to show to my team that a tool like atmos can help organise a large Terraform codebase with multiple tenants and AWS accounts. I can then look into Atlantis
All examples I’ve seen deploy a specific component in a stack
A concrete example of what I want to achieve is that I have a TF component that provisions a Kafka cluster. Whenever I change the number of brokers I want to deploy/update two other components that require the bootstrap servers returned after applying the first component. I thought/wished that atmos could help achieve something like that
In the CLI we have not implemented that. The main reason is that it’s dangeous to “apply all”, however, we do have it on our roadmap but no ETA.
However, it is possible using custom commands. Others have done the same.
Atmos can be easily extended to support any number of custom CLI commands. Custom commands are exposed through the atmos CLI when you run atmos help. It’s a great way to centralize the way operational tools are run in order to improve DX.
excellent! thanks so much for the help and support. I’ll proceed with a demo I’m preparing using custom commands and later on will look into the integration with Atlantis
Identify drift and create GitHub Issues for remediation
Drift detection will pick up the changes in the scenario you described, but the way it works is not based on dependencies
It work based on replanning all components
Oh, hadn’t seen that feature. It’s something we can run in a scheduled CI/CD job
Exactly!
Ultimately, I want to prove that we don’t need TACOS, just Terraform, best practices framework and existing tools to improve all the issues you’ve listed that teams run into when scaling TF to a large org.
Nothing helpful to add here, but want to just say thank you for asking the question and folks for answering swiftly. I was about to go on a rabbit hole reading atmos docs for the same exact use case (just no atlantis requirement). @karel_alfonso Curious on what you come up with.
Before reading this, I was also under the impression atmos can deploy all components in the stack in order of dependency. I’ll start investigating custom command and play around with our use case as well
Seeing HCP Terraform stacks (and deferred changes/plan capability) got me wanting to see what is available out there outside of the usual TACOS. Saw Terragrunt Stacks RFC, exciting, but timing of its release is unknown
you can create an atmos workflow to deploy components in order using the CLI
2024-11-20
Hi, i’m at the stage of initialising the tfstate-backend in a brownfield environment context.
I was successful in creating the s3 and dynamodb part and migrating all the workspace to s3.
However, i hit a wall when it comes to the process of enabling the access_roles_enabled flag.
The following is the error i recieved
Error:
│ Could not find the component 'account-map' in the stack 'cs-core-gbl-root'.
│ Check that all the context variables are correctly defined in the stack manifests.
│ Are the component and stack names correct? Did you forget an import?
My namespace is cs however the stack name -core-gbl-root is foreign to me as i’ve not declare any of that. Not sure how that came about.
This was the stack yaml i’m using to deploy the tfstate-backend. I used the output of the iam role created by the access_role flag and pass it into the role_arn prior to turning on the access_role_enable flag.
Is there some sort of mapping i have miss configured ?
tfstate-backend:
backend:
s3:
role_arn: null
vars:
access_roles_enabled: true # Set to false initially, and only used for cold start.
enable_server_side_encryption: true
enabled: true
force_destroy: false
name: terraformstate
prevent_unencrypted_uploads: true
label_order: ["namespace", "tenant", "environment", "stage", "name"]
access_roles:
default: &tfstate-access-template
write_enabled: true
allowed_roles: {}
denied_roles: {}
allowed_permission_sets: {}
denied_permission_sets: {}
allowed_principal_arns: [
"arn:aws:iam::XXXXXXXX:role/XXXXXXXX"
]
denied_principal_arns: []
tags:
component: "tfstate-backend"
expense-class: "storage"
My folder structure is stacks/orgs/cs/xxx/dev/ap-southeast-2/tfstate-backend.yaml and my stack name is cs-xxx-apse2-dev.
@Dan Miller (Cloud Posse) @Ben Smith (Cloud Posse) maybe an easy one for you
your backend configuration is likely configured at a higher level so that it can be reused. It’s most likely under stacks/orgs/cs/_defaults.yaml
you can also always check the final result of stack configuration with atmos. For example,
atmos describe component tfstate-backend -s your-stack-name
Use this command to describe the complete configuration for an Atmos component in an Atmos stack.