#atmos (2022-11)
2022-11-01
@Dan Miller (Cloud Posse) has joined the channel
2022-11-02
v1.11.0 what Add disabled field to component validation steps Add verbose field to custom command steps Expose component config in custom commands why Add disabled field to component validation steps to be able to to skip the validation step. disabled is set to false by default and the validation step is allowed if disabled attribute is not specified. This is useful when we add component validation in settings.validation at environment/tenant/stage/base component level, but we want to disable…
v1.11.0 what Add disabled field to component validation steps Add verbose field to custom command steps Expose component config in custom commands why Add disabled field to component validation steps to be able to to skip the validation step. disabled is set to false by default and the validation step is allowed if disabled attribute is not specified. This is useful when we add component validation in settings.validation at environment/tenant/stage/base component level, but we want to disable…
2022-11-03
v1.11.1 what Update custom commands Refactor custom commands code why When parsing and executing CLI custom commands, clone the command struct into a local variable because of the automatic closure in the Run function of the Cobra command (the last processed command config affects all the previous ones). Cloning will make a closure over the local variable which is different in each iteration (we cloned just Go slices before, but we need to clone Go maps as well, so we clone the whole struct instead)
v1.11.1 what Update custom commands Refactor custom commands code why When parsing and executing CLI custom commands, clone the command struct into a local variable because of the automatic closure in the Run function of the Cobra command (the last processed command config affects all the previous ones). Cloning will make a closure over the local variable which is different in each iteration (we cloned just Go slices before, but we need to clone Go maps as well, so we clone the whole struct instead)
v1.11.2 what Update parsing command-line arguments why After refactoring in atmos v1.10.x, the command-line arguments were parsed incorrectly. For example, in the following command atmos terraform plan -s –auto-generate-backend-file=true
the argument –auto-generate-backend-file=true was lost and never applied.
v1.11.2 what Update parsing command-line arguments why After refactoring in atmos v1.10.x, the command-line arguments were parsed incorrectly. For example, in the following command atmos terraform plan -s –auto-generate-backend-file=true
the argument –auto-generate-backend-file=true was lost and never applied.
2022-11-04
v1.11.3 what Fix atmos vendor pull command why The function os.IsNotExist has probably changed in some latest Go versions After calling os.Stat, don’t check for “path does not exist” errors using os.IsNotExist, instead check for all errors This fixes the error: atmos vendor pull -c acm
Pulling sources for the component ‘acm’ from
‘/localhost/infra/components/terraform/acm/github.com/cloudposse/terraform-aws-components.git/modules/acm?ref=1.86.0’
and writing to…
v1.11.3 what Fix atmos vendor pull command why The function os.IsNotExist has probably changed in some latest Go versions After calling os.Stat, don’t check for “path does not exist” errors using os.IsNotExist, instead check for all errors This fixes the error: atmos vendor pull -c acm
Pulling sources for the component ‘acm’ from
‘/localhost/infra/components/terraform/acm/github.com/cloudposse/terraform-aws-components.git/modules/acm?ref=1.86.0’
and writing to…
2022-11-05
v1.12.0 what Update atmos custom commands Update atmos terraform generate backends command why Update atmos custom commands to allow the command’s ENV var values to have access to {{ .ComponentConfig.xxx.yyy.zzz }} Go template variables. If a custom command defines component_config section with component and stack, atmos generates the config for the component in the stack and makes it available in {{ .ComponentConfig.xxx.yyy.zzz }} Go template variables, exposing all the component sections. This is…
v1.12.1 what Update atmos atlantis generate repo-config command why If an atmos component is derived from a base component, terraform workspace is calculated differently: for a regular atmos component, terraform workspace is the stack name (e.g. tenant1-ue2-prod) for a derived atmos component, terraform workspace is stack name + component name (e.g. tenant1-ue2-prod-)
For example, for the following vpc-2 atmos component vpc-2: metadata: component: infra/vpc terraform workspace in the…
v1.12.1 what Update atmos atlantis generate repo-config command why If an atmos component is derived from a base component, terraform workspace is calculated differently: for a regular atmos component, terraform workspace is the stack name (e.g. tenant1-ue2-prod) for a derived atmos component, terraform workspace is stack name + component name (e.g. tenant1-ue2-prod-)
For example, for the following vpc-2 atmos component vpc-2: metadata: component: infra/vpc terraform workspace in the…
v1.12.2 what Add –format=backend-config to atmos terraform generate backends command why Terraform supports -backend-config argument, which required the backend config files to have just the backend attributes without terraform.backend header references https://developer.hashicorp.com/terraform/language/settings/backends/configuration#file test atmos terraform generate backends…
v1.12.2 what Add –format=backend-config to atmos terraform generate backends command why Terraform supports -backend-config argument, which required the backend config files to have just the backend attributes without terraform.backend header references https://developer.hashicorp.com/terraform/language/settings/backends/configuration#file test atmos terraform generate backends…
2022-11-09
@Andriy Knysh (Cloud Posse), in office hours today, @Joe Caulfield brought up an edge case that prevents atlantis from detecting any changes. When there’s a backend-only change, there’s no .tf or .tfvar file that changes, and it’s non-trivial to know which yaml files should be added as dependencies for the atlantis project. After some ideation, we came up with a solution to address this situation and any other similar one. The root issue, is if we change anything about the configration as it relates to that component instance, (e.g. changed atmos args in the atmos.yaml), we should be triggering a plan. I think the solution is very easy:
• take the deep-merged output from atmos describe component and take an MD5 checksum of it.
• Write the checksum as a comment at the top of the .tfvar file that is generated.
• This way, any time any configuration that affects this component is changed, the checksum will change triggering a changed file.
cc @jose.amengual
I will like to understand this more
is this running atmos with atlantis on the atlantis image?
@jose.amengual are you using atmos to generate the backends?
…if the backends are not always the same for the component depending on the stack, then the backend.tf.json cannot be committed.
yes I do
but if the file is not committed, then you cannot detect changes when only the backend changes, and no .tf or varfiles change
I have a s3 backend per stack
I commit the files
with my github action
# Generate tfvars files always
- name: Generate TF var files and backend configs
if: github.event.pull_request.state == 'open' || ${{ github.event.issue.pull_request }}
shell: bash
run: |
wget -q <https://github.com/cloudposse/atmos/releases/download/v${ATMOS_VERSION}/atmos_${ATMOS_VERSION}_linux_amd64> && \
mv atmos_${ATMOS_VERSION}_linux_amd64 /usr/local/bin/atmos && \
chmod +x /usr/local/bin/atmos
atmos terraform generate varfiles --file-template={component-path}/varfiles/{namespace}-{environment}-{component}.tfvars.json
atmos terraform generate backends --format=backend-config --file-template={component-path}/backends/{namespace}-{environment}-{component}.backend
# Commit vars changes (if any) to the PR branch
- name: Commit changes to the PR branch
if: github.event.pull_request.state == 'open' || ${{ github.event.issue.pull_request }}
shell: bash
run: |
untracked=$(git ls-files --others --exclude-standard)
changes_detected=$(git diff --name-only)
if [ -n "$untracked" ] || [ -n "$changes_detected" ]; then
git config --global user.name github-actions
git config --global user.email [email protected]
git add -A *
git commit -m "Commiting generated autogenerated var files"
git push
fi
atmos.yaml snip :
# Workflow templates
# <https://www.runatlantis.io/docs/custom-workflows.html#custom-init-plan-apply-commands>
# <https://www.runatlantis.io/docs/custom-workflows.html#custom-run-command>
# Select a template by using the `--workflow-template <workflow_template>` command-line argument in `atmos atlantis generate repo-config` command
workflow_templates:
workflow-1:
plan:
steps:
- run: cp backends/backend.tf .
- run: terraform init --reconfigure -input=false -backend-config=backends/$PROJECT_NAME.backend
# When using workspaces, you need to select the workspace using the $WORKSPACE environment variable
- run: terraform workspace select $WORKSPACE || terraform workspace new $WORKSPACE
# You must output the plan using `-out $PLANFILE` because Atlantis expects plans to be in a specific location
- run: terraform plan -input=false -refresh -out $PLANFILE -var-file varfiles/$PROJECT_NAME.tfvars.json
apply:
steps:
- run: terraform apply $PLANFILE
the autogenerated backend files that atmos created are in .gitignore
PePe is using this new functionality to generate the backend config files per component per stack
is still hot out of the oven feature
I am using a very similar process in my project that I’m testing with atlantis. The only difference is that I am generating the backend files when atlantis runs but I plan on changing that to commit the .tfbackend files along with the varfiles I am already committing. My initial question I asked that brought this up was if there are any plans to introduce similar functionality to the terraform init command run when running terraform commands through atmos normally. The problem I have run into is that when deploying the same component to multiple stacks since the workspaces are different in each stack terraform init will prompt to select a workspace since the workspace previously selected will not exist in the current stack. My current workaround is to modify the .terraform/environment file but I like the suggestion to use environment variables instead. My though was that having a similar option to the atmos terraform generate backends command to provide the ability to include a partial backend config file in the component then when running terraform command through atmos it could create the backend using the backend-config format instead of the entire backend as json.
yes, that can (and will) be implemented: 1) in atmos.yaml we’ll add a flag to specify which backend type to generate; 2) then in all related atmos commands (that use/generate backend files) we’ll check the type, geberate backend file(s) and use -backend-file flag when needed
Awesome, that is exactly what I was hoping to hear!! I’ll be keeping an eye on the release notes
in case you missed it :
atmos terraform generate backends --format=backend-config --file-template={component-path}/backends/{namespace}-{environment}-{component}.backend
that will generate a backend-config format
but the dummy backend.tf is blank
terraform {backend s3 {}}
yes correct
we want to generate backends in diff formats in atmos terraform plan/apply on the fly
ahhhhhhhh I c ok
If anyone is trying to run Atmos with Atlantis hit me up , in todays OfficeHours there were a few questions about it and I have answers for all!! and the cool think Atmos already supports all those cases
@jose.amengual we just merged the rough draft of the docs for https://atmos.tools
2022-11-10
2022-11-11
v1.13.0 what && why New docs website https://atmos.tools
As the Cloud Posse team is updating their documentation and tutorials for using SweetOps, I thought it might be useful to develop a sandbox/lab environment for folks to learn some of the fundamentals. We internally use the Instruqt e-learning platform which is convenient for building out demos/labs/training environments.
https://play.instruqt.com/riverpointtechnology/invite/9hknxtho0prk
Feel free to check out the invitation above and walk through the labs that mirror the Cloud Posse tutorials. Like most of you I learn by doing so being able to spin up an on demand environment is very helpful (AWS Account, Code Editor & Lab Guides included). There is a cost associated with us hosting these labs, which I am happy to currently cover, but may need to rate limit. Let me know if they are helpful.
As you can see we have a number of labs that I would like to continue to build out including integrations with Spacelift, Atlantis and TFC.
thanks @gabe, this sounds interesting, we’ll take a look
wow, that’s rad
@Erik Osterman (Cloud Posse) - did this lab environment provision for you? 2 mins is avg time for lab to spin up
I wasn’t able to get back to it
v1.13.0 what && why New docs website https://atmos.tools
v1.13.1 what Allow Helmfile component without using EKS. why Because atmos could be used without using EKS.
v1.13.1 what Allow Helmfile component without using EKS. why Because atmos could be used without using EKS.
2022-11-14
v1.13.2 what Update docs website and atmos help commands why Better looking website Improve the website color theme Update/improve CLI/Configuration and CLI/Commands sections (https://atmos.tools/category/commands)
v1.13.2 what Update docs website and atmos help commands why Better looking website Improve the website color theme Update/improve CLI/Configuration and CLI/Commands sections (https://atmos.tools/category/commands)
v1.13.3 what Documentation website sections: tutorials glossary component library mixins imports
why Consolidate atmos documentation under the new docs site
2022-11-15
@here we’re actively working on documentation and have pushed a whole new site to https://atmos.tools
Please let us know what you want to see…
Documentation Updates Reorganize CLI commands & subcommands Improve the introduction Describe the AWS integrations
v1.13.4 Reorganize CLI commands & subcommands Improve the introduction Describe the AWS integrations
2022-11-16
v1.13.5 what Standardize CLI commands formatting Provide a general command usage page that embeds the auto-generated index Add “Component Inheritance” section to the docs why It was not clear what each command should be used for without reading further down the page Having one separate index page and an index page of subcommand led to information being sprawled across two pages, rather than having a single usage page that shows general command usage and all available subcommands “Component…
v1.13.5 what Standardize CLI commands formatting Provide a general command usage page that embeds the auto-generated index Add “Component Inheritance” section to the docs why It was not clear what each command should be used for without reading further down the page Having one separate index page and an index page of subcommand led to information being sprawled across two pages, rather than having a single usage page that shows general command usage and all available subcommands “Component…
v1.14.0 what add shell completion script generation for bash, zsh, fish and powershell users can now run atmos completion [bash|zsh|fish|powershell] and atmos will output the shell completion script for that shell why so users can run atmos [tab][tab] and get command auto-completion
v1.14.0 what add shell completion script generation for bash, zsh, fish and powershell users can now run atmos completion [bash|zsh|fish|powershell] and atmos will output the shell completion script for that shell why so users can run atmos [tab][tab] and get command auto-completion
2022-11-17
Love seeing all the work on Atmos docs. Feel like it has the chance to get some serious legs under it now that there is a strong starting point that is in one place! Well done to ya’ll Gonna spin through the new docs at some point over the coming week or so and I’ll provide feedback when I do!
Thanks man! Let me know where you think we should focus next.
@Andriy Knysh (Cloud Posse) is working on the QuickStart
yes, we have some things to add yet 9which we know about), we’ll add it in the coming days/weeks. Please let us know what info you want to see in the docs
2022-11-21
v1.15.0 what & why Update atmos “deps” calculation. In some cases, when more than one different YAML stack config files imported the same import (b/c some components in both derived from the same base component), then all of those stack config files were added to the component’s “deps” list (resulting in Spacelift stacks having unrelated dependency labels which would cause unnecessary stack triggering) Document atmos completion CLI command Add “Component-Oriented Programming” page to document…
Is this coming? https://atmos.tools/quick-start/create-atmos-stacks
Yes! @Andriy Knysh (Cloud Posse) is planning to finish that this week
Tag me when you do I was going to write a python wrapper to deploy my topology but i’m really excited to use atmos and share it with my team. Struggle bussing through the docs!
Can atmos workflows be used to deploy multiple terraform configurations? For example, I have broken down my deployment into services: vpc, eks, rds, etc. Currently, this requires cd’ing to each directory and running plan then apply to bring up each service. Could atmos workflows be leveraged to string these together?
yes
you can use the deploy command in atmos to do a plan and apply in one go
or do sequencial run of plan and apply etc
The command subcommand under workflows i thought would only be terraform commands as shown here - https://github.com/cloudposse/atmos/blob/master/examples/complete/stacks/workflows/workflow1.yaml
how can i use the atmos deploy
is there an example or tutorial you can link? this is my first time using atmos. heard about it on the Office Hour last week
I thought you were talking about terraform
i am talking about terraform
terraform doesn’t have the deploy command, atmos does. i didn’t knw you could provide atmos workflow with atmos commands. that is what i was saying
ahhh ok, yes you can
the deploy command is the same as: terrafrom plan && terraform apply -auto-aprove
that is what basically it does
# Atmos workflows for managing EKS clusters
workflows:
deploy-basics:
description: Deploy the basic infrastructure for an EKS cluster
steps:
- command: terraform deploy eks/karpenter
- command: terraform deploy eks/karpenter-provisioner
- command: terraform deploy eks/cert-manager
- command: terraform deploy eks/alb-controller
- command: terraform deploy eks/external-dns
- command: terraform deploy eks/reloader
- command: terraform deploy eks/echo-server
command can be of type atmos (can execute any atmos command, this is the default) or shell (can execute any shell command)
OK, i see what you’re saying now. I was confused when you used terraform deploy since that’s not a terraform command. Like how you have it in “# Atmos workflows for managing EKS clusters”. Maybe you have an alias for it or something.
I now have the workflow I want but I am stuck on how to build a stack. I’ll waiti for the docs to be updates on that unless there is an example somewhere? I have all my variables in a .tfvars file and am passing that to terraform but atmos won’t let me run without the stack
workflows:
# Can also be set using 'ATMOS_WORKFLOWS_BASE_PATH' ENV var, or '--workflows-dir' command-line arguments
# Supports both absolute and relative paths
vpc-and-s3:
description: |
Deploy the VPC and S3 buckets to test atmos
steps:
- command: terraform plan aws-networking -var-file="UKDev_vpn.tfvars"
- command: terraform plan s3-buckets
this is in my atmos.yaml
Do i have to redefine the values in my tfvars into the stack or does atmos support referencing the vars file?
Can the stack by empty like it is here?
atmos does not reference anything in tfvar files, it’s terraform thing
the stacks can be empty (if you have all vars in varfiles, it will work, but it’s not what atmos is for)
here is an example on how to create first stacks (using catalog and import) https://atmos.tools/core-concepts/components/inheritance
Executing command:
/opt/homebrew/bin/terraform plan -var-file example-aws-networking.terraform.tfvars.json -out example-aws-networking.planfile -var-file='UKDev_vpn.tfvars'
╷
│ Error: Failed to read variables file
│
│ Given variables file 'UKDev_vpn.tfvars' does not exist.
workflows:
# Can also be set using 'ATMOS_WORKFLOWS_BASE_PATH' ENV var, or '--workflows-dir' command-line arguments
# Supports both absolute and relative paths
vpc-and-s3:
description: Deploy the VPC and S3 buckets to test atmos
steps:
- command: terraform plan aws-networking -var-file='UKDev_vpn.tfvars'
- command: terraform plan s3-buckets
thanks, i am copying the stack from the tutorial
I’ll move the var file to atmos config in the future, i’m first confirming that this will work for what the project needs.
Disregard my last comments, I see that i can simply past the content of my .tvars file under the vars section in the components.