#atmos (2022-09)

2022-09-07

Release notes from atmos avatar
Release notes from atmos
03:54:37 PM

v1.5.0 what Add support for custom integrations in atmos.yaml Add Atlantis support (Atlantis is an integration) Add atmos terraform generate varfiles and atmos atlantis generate repo-config CLI commands why Support Atlantis Generate the varfiles for all components in all stacks (this is used in Atlantis repo config, and will be used to detect drifts in variables to simplify triggering Spacelift stacks) Automatically generate Atlantis repo config file atlantis.yaml. Using the config, project and…

Release v1.5.0 · cloudposse/atmosattachment image

what Add support for custom integrations in atmos.yaml Add Atlantis support (Atlantis is an integration) Add atmos terraform generate varfiles and atmos atlantis generate repo-config CLI commands …

Release notes from atmos avatar
Release notes from atmos
04:14:41 PM

v1.5.0 what Add support for custom integrations in atmos.yaml Add Atlantis support (Atlantis is an integration) Add atmos terraform generate varfiles and atmos atlantis generate repo-config CLI commands why Support Atlantis Generate the varfiles for all components in all stacks (this is used in Atlantis repo config, and will be used to detect drifts in variables to simplify triggering Spacelift stacks) Automatically generate Atlantis repo config file atlantis.yaml. Using the config, project and…

2022-09-09

Matt Gowie avatar
Matt Gowie

Hey folks — I posed this question to my team the other day and would like to see if there was more specific or correct nomenclature that I’m not aware of.
We should discuss what to call an instance of a component in a stack file. My thoughts on the nomenclature right now is:
1. stack file / stack (lowercase) — The files and atmos stacks e.g. gbl-root, ue1-automation, etc.
2. Spacelift Stack / Stack (uppercase) — Spacelift’s term for an instance of a component — This has its own state and is planned / applied. e.g. ue1-audit-cloudtrail-bucket
3. Component instance (This is what I would like to have a better term for, but don’t) — The configuration in our stack files that allows us to create an instance of a component with atmos and is the local version of a “Stack”.
Any thoughts?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@Matt Gowie thanks, those are good questions

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

TF components in components/terraform folder are called terraform components (or helmfile components).

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

in stacks, we have a few things:

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
  1. Stack config files (e.g. ue1-dev.yaml ) where the top-level stacks are defined
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
  1. Atmos stacks - the logical stacks (not related to stack config files). The logical stack names are always derived from the context (e.g. tenant-ue2-prod) and they are not the config files. That’s why you can name your stack config files anything and put them in any folder or subfolder at any level, atmos just cares about the logical names. In short, the stack config files are for humans to organize the config and use the file and folder names that people understand. The logical stack names (derived from the context) are for atmos
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
  1. Atmos components - that’s what you define in the YAML stack config files and point them to the “physical” terraform/helmfile components using metadata.component: xxx attribute
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
  1. Atmos components can be of a few types: real and abstract. The abstract components (defined by metadata.type: abstract) are just containers with some default values, they can’t be provisioned. The real components get provisioned. Real components can inherit from abstract components (and other real atmos components). Another important thing about atmos components is that they allow you to provision more than one same TF component with diff name in the same stack
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
  1. Which bring us to the concept of atmos component inheritance. An atmos component (defined in the YAML files) can be a base component for other atmos components, which we call derived components. Derived components can inherit from other atmos components (abstract and real) using metadat.inherits: [xxx, yyy]
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
  1. Spacelift stacks are diff concept. They are a Cartesian product of all (real) atmos components and all atmos stacks (logical names derived from the context), meaning we generate as many Spacelift stacks as we have atmos components multiplied by atmos stacks, e.g. for a component vpc in stacks ue2-dev and ue2-prod, we generate the Spacelift stacks ue2-dev-vpc and ue2-prod-vpc etc.
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

so yes, many definitions are reused or overridden. But to summarize, we have:

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

• atmos stack config files (top-level stacks and catalog) • atmos stacks (derived from the context). Each atmos stack is a collection of atmos components provisioned into a particular environment (region/account/stage). Each atmos stack can be defined in one or many atmos stack config files (meaning you can define some components in one stack config file, and other components in other stack config files for the same logical stack (region/env/stage) • terraform and helmfile components (in components folder) - this is the TF/helmfile code

• atmos components in YAML config files (where you define vars and other config for TF/helmfile components) - and they can be real/abstract, base/derived

• Spacelift stacks (in short, each Spacelift stack is an atmos component in an atmos stack)

• Atlantis projects - similar to Spacelift stacks, each atlantis project is an atmos component in an atmos stack

RB (Ronak) (Cloud Posse) avatar
RB (Ronak) (Cloud Posse)

Here’s my take on it (ignore me if I’m confusing things, please let me know if I’m correct)

# catalog/s3-bucket/defaults.yaml

components:
  terraform:
    # abstract component
    s3-bucket/defaults:
      metadata:
        type: abstract

# catalog/s3-bucket/flavor.yaml
import:
  - catalog/s3-bucket/defaults

components:
  terraform:
    # real component
    s3-bucket/flavor:
      metadata:
        inherits:
        - s3-bucket/defaults

then there is a difference between a real component based on an abstract and a real component based on nothing. The former real is non-derived whereas the latter real is derived. `I’ve been using

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

yes, but let’s keep in mind that abstract/real and base/derived are diff concepts and not related to each other except that you usually define an abstract component to be a base and inherit from it later in other components (so it’s just a collection of some default values)

RB (Ronak) (Cloud Posse) avatar
RB (Ronak) (Cloud Posse)

you cannot have a real abstract, but you can have a real base (or non-derived) and a real derived

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

(let’s not complicate things even more it’s already a lot of diff thing here to name )

Matt Gowie avatar
Matt Gowie

@Andriy Knysh (Cloud Posse) awesome stuff — so what I have been referring to as a “component instance”, you refer to as an “atmos component”, which I can understand.

I really like the term “logical stack” as a differentiator against stack files because I think that is one of the more confusing things to understand when coming into atmos: logical stacks are derived from context and not from file names. And really, the logical stacks are more important because that is actually what you need to reference and what gets used in your workspace name and similar.

Anyway, thanks for braindumping all of the above. We should sticky this thread and as you folks are writing out atmos documentation, you should copy + pasta a good bit of the above.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

any atmos component can be a base for any other atmos component (it’s inheritance chain). Making a component abstract just prevents it from being deployed (atmos will complain if you try to deploy an abstract component). Both base and derived components can be made abstract, which just means you don’t want them to be provisioned and will derive from them in other atmos components

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

that’s some OOP here

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

and we have not even started on Component-Oriented-Programming (COP) yet

1
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

but thanks you @Matt Gowie for bringing up the naming issue. Yes, we have called all of that diff names (mostly using components and stacks), but components and stacks are completely diff overloaded things and we have many of them - TF components, atmos components, stack config files, stacks, Spacelift stacks - all diff things, and having a common dictionary for all of that stuff is important otherwise people will not understand each other

1
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

if you have ideas on how to better name these things, let us know

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

(e.g. component instance is a good name for atmos component since in OOP they call it class instance or object . E.g. in C# they don’t call an object created from a C# class a C# object)

Matt Gowie avatar
Matt Gowie

I’ll keep it in mind. But yeah component instance is the only one I felt was my own term and I didn’t know what you folks called that internally. I think instance follows the OOP terminology, but I also get what you would call them atmos components as that makes a bit of sense too.

Release notes from atmos avatar
Release notes from atmos
01:34:35 AM

v1.5.1 what Adds a –skip-init flag which allows skipping terraform init why This can help speed up workflows in the case that the user knows their last command successfully ran terraform init and they do not need to run init again.

Release v1.5.1 · cloudposse/atmosattachment image

what Adds a –skip-init flag which allows skipping terraform init why This can help speed up workflows in the case that the user knows their last command successfully ran terraform init and they…

Release notes from atmos avatar
Release notes from atmos
01:54:34 AM

v1.5.1 what Adds a –skip-init flag which allows skipping terraform init why This can help speed up workflows in the case that the user knows their last command successfully ran terraform init and they do not need to run init again.

2022-09-10

Release notes from atmos avatar
Release notes from atmos
08:54:35 PM

v1.6.0 what Update atmos atlantis generate repo-config command Support native HCL output format in atmos terraform generate varfiles command why Do not iterate over Go maps in atmos atlantis generate repo-config command. Go iterates over maps in a non-deterministic order resulting in constant drift in the final atlantis.yaml file. Instead, get the map keys, sort them, and iterate over the sorted keys Support native HCL output format in atmos terraform generate varfiles command - when ejecting from…

Release v1.6.0 · cloudposse/atmosattachment image

what Update atmos atlantis generate repo-config command Support native HCL output format in atmos terraform generate varfiles command why Do not iterate over Go maps in atmos atlantis generate r…

party_parrot1
Release notes from atmos avatar
Release notes from atmos
09:14:36 PM

v1.6.0 what Update atmos atlantis generate repo-config command Support native HCL output format in atmos terraform generate varfiles command why Do not iterate over Go maps in atmos atlantis generate repo-config command. Go iterates over maps in a non-deterministic order resulting in constant drift in the final atlantis.yaml file. Instead, get the map keys, sort them, and iterate over the sorted keys Support native HCL output format in atmos terraform generate varfiles command - when ejecting from…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Andriy Knysh (Cloud Posse) do you know why it keeps posting 2 updates for every release?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

it’s prob RSS config, we need to check it

2022-09-11

Release notes from atmos avatar
Release notes from atmos
05:34:36 PM

v1.7.0 what Add atmos terraform generate backends command why Generate terraform backend configs for all terraform components Supported formats HCL and JSON A GitHub Action that generates all .tfvar files and backend.tf files so that projects can be used with conventional terraform GitOps tools like atlantis, Terraform Cloud, et al. test

# hcl is default, no need to specify it atmos terraform generate backends –format=hcl

Writing backend config for the terraform component ‘test/test-component’ to…

Release v1.7.0 · cloudposse/atmosattachment image

what Add atmos terraform generate backends command why Generate terraform backend configs for all terraform components Supported formats HCL and JSON A GitHub Action that generates all .tfvar fi…

jose.amengual avatar
jose.amengual

if you keep up the pace you will be in version 100.0.0 in no time

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

it’s still a long way to go to 100

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

All because of you @jose.amengual

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

jose.amengual avatar
jose.amengual

lol, I asked for one thing…..

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(so the back story here, is to support atlantis, everything should be committed, which means all the varfiles and all the backends. that’s what we were generating on the fly.)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

So in this model, either with precommit hooks or something like a GitHub Action workflow, generate the backends and varfiles with atmos, then atlanits will work very well.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

In fact, so will TFC, Spacelift, Env0, etc because it’s all vanilla HCL.

jose.amengual avatar
jose.amengual

@RB (Ronak) (Cloud Posse) was asking about this

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
jose.amengual avatar
jose.amengual

Atlantis currently can read on the fly Atlantis yaml files

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

the reason for committing the varfiles and backends is to detect “affected files” in the PR

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Hence, they cannot be generated on the fly

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(is my understanding)

jose.amengual avatar
jose.amengual

but I’m. not sure if it run the plans correctly, that is what I need to test next

jose.amengual avatar
jose.amengual

it was never my intention to commit the files back

Release notes from atmos avatar
Release notes from atmos
05:54:36 PM

v1.7.0 what Add atmos terraform generate backends command why Generate terraform backend configs for all terraform components Supported formats HCL and JSON A GitHub Action that generates all .tfvar files and backend.tf files so that projects can be used with conventional terraform GitOps tools like atlantis, Terraform Cloud, et al. test

# hcl is default, no need to specify it atmos terraform generate backends –format=hcl

Writing backend config for the terraform component ‘test/test-component’ to…

2022-09-13

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@jose.amengual any opinion on using App Runner for a simpler hosting of Atlantis?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Also, we’re thinking maybe it should just be a super simple cloudformation template

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

that way you can bootstrap everything even without a terraform backend,.

jose.amengual avatar
jose.amengual

mmmmm

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Imagine then in a control tower organization, this could be a part of the baseline deployed.

jose.amengual avatar
jose.amengual

only apprunner apps can be part of that?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

No, but it basically reduces the entire problem set down to 1-2 resources

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

No need for ALB, target groups, ecs cluster, ecs tasks, and two dozen more resources.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Less resources, less things that can go wrong.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Less things that cost money

jose.amengual avatar
jose.amengual

mmm I will take a look at apprunner

jose.amengual avatar
jose.amengual

I never used it

jose.amengual avatar
jose.amengual

2 vCPU 4GB max for app runner

jose.amengual avatar
jose.amengual

that is no much

jose.amengual avatar
jose.amengual

for a small atlantis that could work but for a bigger deployment/infra repo it could be too small

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Oh interesting

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Surprised it is that low

jose.amengual avatar
jose.amengual

I have never used it

Release notes from atmos avatar
Release notes from atmos
05:54:36 PM

v1.7.1 what Fixes an issue where ATMOS_CLI_CONFIG_PATH points to a non directory and results in a panic. why This provides proper messaging and gracefully fails. The error in question here that was getting skipped over by os.IsNotExist(err) was the following: stat /usr/local/etc/atmos/atmos.yaml/atmos.yaml: not a directory

Release v1.7.1 · cloudposse/atmosattachment image

what Fixes an issue where ATMOS_CLI_CONFIG_PATH points to a non directory and results in a panic. why This provides proper messaging and gracefully fails. The error in question here that was get…

cool-doge1

2022-09-14

Release notes from atmos avatar
Release notes from atmos
05:14:37 PM

v1.8.0 what Remove the default hardcoded CLI config Update TF workspace calculation for Spacelift stacks why Make atmos.yaml CLI config always required. Remove the default hardcoded CLI config b/c it had some default values which are not applicable for all use cases. Instead, throw error if atmos.yaml is not found export ATMOS_LOGS_VERBOSE=true atmos describe component test/test-component-override -s tenant1-ue2-dev

Found ENV var ATMOS_LOGS_VERBOSE=true

Searching, processing and merging atmos CLI…

Release v1.8.0 · cloudposse/atmosattachment image

what Remove the default hardcoded CLI config Update TF workspace calculation for Spacelift stacks why Make atmos.yaml CLI config always required. Remove the default hardcoded CLI config b/c it h…

    keyboard_arrow_up