#atmos (2024-07)

2024-07-01

Miguel Zablah avatar
Miguel Zablah

Hi, we are staring to switch to use Atmos instead of a custom terragrunt config to manage terraform since it resolves a lot of custom stuff we need to do so thanks for this awesome tool I just have 2 questions.

  1. When creating the initial S3 backend and DynamoDB do you guys use the tfstate-backend module outside of atmos or create this manually? I was following the advance tutorial and it looks good but I don’t see any docs around this although maybe I miss something
  2. Is there a global env to get the git commit number so I can set it as a tag?
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yes, so in this case I recommend a hierarchical backend architecture.

We use the following workflow to initialize the root tfstate backend.

init/tfstate:
    description: Provision Terraform State Backend for initial deployment.
    steps:
      - command: terraform deploy tfstate-backend -var=access_roles_enabled=false --stack core-use1-root --auto-generate-backend-file=false
      - command: until aws s3 ls acme-core-use1-root-tfstate; do sleep 5; done
        type: shell
      - command: terraform deploy tfstate-backend -var=access_roles_enabled=false --stack core-use1-root --init-run-reconfigure=false
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Then you can initialize any number of additional backends, with the state of each of those backends the root backend.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

global env to get the git commit number so I can set it as a tag? In the stack configurations, you can use any Sprig or Gomplate function.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Sprig Function Documentation

Useful template functions for Go templates.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Configuration - gomplate documentation

In addition to command-line arguments, gomplate supports the use of configuration files to control its behaviour. Using a file for configuration can be useful especially when rendering templates that use multiple datasources, plugins, nested templates, etc… In situations where teams share templates, it can be helpful to commit config files into the team’s source control system. By default, gomplate will look for a file .gomplate.yaml in the current working directory, but this path can be altered with the –config command-line argument, or the GOMPLATE_CONFIG environment variable.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

So if you are running in a GHA, you can read the environment variables using the env function

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Or if. you want to read it from the .git/HEAD file you can use a file data source

Miguel Zablah avatar
Miguel Zablah

Ah I see and then do you just download the tfstate-backend as a vendor but not actually use it or even define it in stack/catalog? or you will still do this to move the state of this later to the bucket it created?

Miguel Zablah avatar
Miguel Zablah

also thanks for the info on the template I will look into this so I can add this it looks like this will do the trick

Miguel Zablah avatar
Miguel Zablah

@Erik Osterman (Cloud Posse) the cmd on the workflow:

- command: until aws s3 ls acme-core-use1-root-tfstate; do sleep 5; done
        type: shell

dose not work for me so I change it to the default wait from aws cli:

- command: aws s3api wait bucket-exists --bucket acme-core-use1-root-tfstate

is there a way to also use template here? like if I want to use profile on this cmd?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Atmos Custom Commands | atmos

Atmos can be easily extended to support any number of custom CLI commands. Custom commands are exposed through the atmos CLI when you run atmos help. It’s a great way to centralize the way operational tools are run in order to improve DX.

Miguel Zablah avatar
Miguel Zablah

I see I will try it out thanks!

2024-07-02

2024-07-03

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@here

For a limited time, we’re offering the complete Cloud Posse reference architecture as a perk for a monthly GitHub Enterprise Sponsorship.

Please feel free to contact me if you’d like to learn more.

This is ideal for savvy teams who want a head start. The feedback we’ve received is excellent.

1
sweetops1

2024-07-04

2024-07-05

2024-07-08

Miguel Zablah avatar
Miguel Zablah

dose atmos has a validation to check version?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
atmos version | atmos

Use this command to get the Atmos CLI version

Miguel Zablah avatar
Miguel Zablah

I mean to set it like a .terraform-version

Miguel Zablah avatar
Miguel Zablah

or to validate that we are using the correct atmos version

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

do you want to validate it from the command-line, or from a CI/CD?

Miguel Zablah avatar
Miguel Zablah

I will like to validate this in command-line for CI/CD I guess I can just create a workflow to run but a team mate ask if we have a way to set the atmos version like we do terraform with like a file or something like that

Gabriela Campana (Cloud Posse) avatar
Gabriela Campana (Cloud Posse)

@Andriy Knysh (Cloud Posse)

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@Miguel Zablah i guess we need more details on what you wnat to do here. By validating Atmos version, do you mean you want to install a particular Atmos version in your CI/CD workflow?

burnzy avatar

Sounds like maybe you’re looking for a version manager, like https://github.com/tofuutils/tenv

tofuutils/tenv

OpenTofu / Terraform / Terragrunt and Atmos version manager

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

oh, it’s a nice tool, thanks @burnzy

1
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@Miguel Zablah if you were asking about version management, tenv looks very nice for that

Miguel Zablah avatar
Miguel Zablah

Oh yeah this sounds like what I was looking for

Miguel Zablah avatar
Miguel Zablah

nice tool!! thanks @burnzy and @Andriy Knysh (Cloud Posse)! party_parrot

1

2024-07-09

Brett L avatar
Brett L

Is there some examples of Atmos that are not reference architecture? I’m trying to find if it’s a good fit for our use case, which is simply managing Datadog resources

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Have you taken a look at some of these examples: https://github.com/cloudposse/atmos

cloudposse/atmos

Terraform Orchestration Tool for DevOps. Keep environment configuration DRY with hierarchical imports of configurations, inheritance, and WAY more. Native support for Terraform and Helmfile.

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

2024-07-10

Stephan Helas avatar
Stephan Helas

Hello,

would it be possible to make the interactive mode optional? like using a parameter to start / env variable to prevent it?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

what Atmos command do you want to prevent the interactive mode for?

almost all commands don’t use UX, it’s just a few will open a TUI when called w/o parameters, e.g.

atmos atmos workflow

w/o parameters, they will not execute at all, hence we start an interactive TUI for the user to select the parameters for the commands

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Also we have an open task to fix using interactive mode for things like help. It was not supposed to work that way

Stephan Helas avatar
Stephan Helas

i mean basically if you just type $>atmos

Stephan Helas avatar
Stephan Helas

its different behavior to almost (aptitude comes to mind) all other cmd line utilities.

Stephan Helas avatar
Stephan Helas

just an idea. not too important

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yes, agree - it should not load interactive mode.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(I also find it very annoying when it does that)

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

we can add a setting to atmos.yaml to control that

1
Marat Bakeev avatar
Marat Bakeev

I like the interactive mode -_- Keep it at least as an option pls

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yes definitely not going away, and we want to invest more into it

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

It’s just that things like help menus gain nothing by being interactive. And when mistyping a command it goes into an interactive menu, rather than just emit the help option, which is awkward.

1
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

we will overhaul atmos help and add atmos man command soon

2024-07-11

github3 avatar
github3
03:16:19 PM

Add Atmos Pro integration to atmos.yaml. Add caching to atmos.Component template function. Implement atmos.GomplateDatasource template function @aknysh (#647) ## what

• Add Atmos Pro integration to atmos.yaml • Add caching to atmos.Component template function • Implement atmos.GomplateDatasource template function • Update docs • atmos.Componentatmos.GomplateDatasource

why

• Add Atmos Pro integration to atmos.yaml. This is in addition to the functionality added in Add –upload flag to atmos describe affected command. If the Atmos Pro configuration is present in the [integrations.pro](http://integrations.pro) section in atmos.yaml, it will be added in the config section when executing the atmos describe affected --upload=true command for further processing on the server

    {
         "base_sha": "6746ba4df9e87690c33297fe740011e5ccefc1f9",
         "head_sha": "5360d911d9bac669095eee1ca1888c3ef5291084",
         "owner": "cloudposse",
         "repo": "atmos",
         "config": {
           "timeout": 3,
            "events": [
               "pull_request": [
                  {
                    "on": ["open", "synchronize", "reopen"],
                    "workflow": "atmos-plan.yml",
                    "dispatch_only_top_level_stacks": true
                  },               
                  {
                    "on": ["merged"],
                    "workflow": "atmos-apply.yaml",
                  },               
                ],
                "release": [
                ]
             ]
         }
         "stacks": [
            {
              "component": "vpc",
              "component_type": "terraform",
              "component_path": "components/terraform/vpc",
              "stack": "plat-ue2-dev",
              "stack_slug": "plat-ue2-dev-vpc",
              "affected": "stack.vars",
              "included_in_dependents": false,
              "dependents": []
            }
        ]
     }
    

Add caching to atmos.Component template function

Atmos caches (in memory) the results of `atmos.Component` template function execution. If you call the function for the same component in a stack more than once, the first call will produce the result and cache it, and all the consecutive calls will just use the cached data. This is useful when you use the `atmos.Component` function for the same component in a stack in multiple places in Atmos stack manifests. It will speed up the function execution and stack processing.

For example:
    components:
      terraform:
        test2:
          vars:
            tags:
              test: '{{ (atmos.Component "test" .stack).outputs.id }}'
              test2: '{{ (atmos.Component "test" .stack).outputs.id }}'
              test3: '{{ (atmos.Component "test" .stack).outputs.id }}'
    
In the example, the `test2` Atmos component uses the outputs (remote state) of the `test` Atmos component from the same stack. The template function `{{ atmos.Component "test" .stack }}` is executed three times (once for each tag).

After the first execution, Atmos caches the result in memory (all the component sections, including the `outputs`), and reuses it in the next two calls to the function. The caching makes the stack processing about three times faster in this particular example. In a production environment where many components are used, the speedup can be even more significant.

Implement atmos.GomplateDatasource template function

The `atmos.GomplateDatasource` template function wraps the [Gomplate Datasources](https:/./atmos.tools/core-concepts/stacks/templates/datasources) and caches the results, allowing executing the same datasource many times without calling the external endpoint multiple times. It speeds up the datasource execution and stack processing, and can eliminate other issues with calling an external endpoint, e.g. timeouts and rate limiting.

*Usage*
      {{ (atmos.GomplateDatasource "<alias>").<attribute> }}
    
*Caching the result of `atmos.GomplateDatasource` function*

Atmos caches (in memory) the results of `atmos.GomplateDatasource` template function execution. If you execute the function for the same datasource alias more than once, the first execution will call the external endpoint, produce the result and cache it. All the consecutive calls will just use the cached data. This is useful when you use the `atmos.GomplateDatasource` function for the same datasource alias in multiple places in Atmos stack manifests. It will speed up the function execution and stack processing.

For example:
    settings:
      templates:
        settings:
          gomplate:
            timeout: 5
            datasources:
              ip:
                url: "<https://api.ipify.org?format=json>"
                headers:
                  accept:
                    - "application/json"
    components:
      terraform:
        test:
          vars:
            tags:
              test1: '{{ (datasource "ip").ip }}'
              test2: '{{ (atmos.GomplateDatasource "ip").ip }}'
              test3: '{{ (atmos.GomplateDatasource "ip").ip }}'
              test4: '{{ (atmos.GomplateDatasource "ip").ip }}'
    
In the example, we define a `gomplate` datasource `ip` and specify an external endpoint in the `url` parameter.

We use the [Gomplate `datasource`](https://docs.gomplate.ca/datasources/) function in the tag `test1`, and the `atmos.GomplateDatasource` wrapper for the same datasource alias `ip` in the other tags. The `atmos.GomplateDatasource` wrapper will call the same external endpoint, but will cache the result and reuse it between the datasource invocations.

When processing the component `test` from the above example, Atmos does the following:

• Executes the `{{ (datasource "ip").ip }}` template. It calls the external endpoint using the HTTP protocol and assign the `ip` attribute from the result to the tag `test1`
• Executes the `{{ (atmos.GomplateDatasource "ip").ip }}` template. It calls the external endpoint again, caches the result in memory, and assigns the `ip` attribute from the result to the tag `test2`
• Executes the `{{ (atmos.GomplateDatasource "ip").ip }}` two more times for the tags `test3` and `test4`. It detects that the result for the same datasource alias `ip` is already presend in the memory cache and reuses it without calling the external endpoint two more times

The datasource result caching makes the stack processing much faster and significantly reduces the load on external endpoints, preventing such issues as timeouts and rate limiting.
1
burnzy avatar

can confirm the speed-up is significant with the atmos.Component caching now, thanks @Andriy Knysh (Cloud Posse)!

Add Atmos Pro integration to atmos.yaml. Add caching to atmos.Component template function. Implement atmos.GomplateDatasource template function @aknysh (#647) ## what

• Add Atmos Pro integration to atmos.yaml • Add caching to atmos.Component template function • Implement atmos.GomplateDatasource template function • Update docs • atmos.Componentatmos.GomplateDatasource

why

• Add Atmos Pro integration to atmos.yaml. This is in addition to the functionality added in Add –upload flag to atmos describe affected command. If the Atmos Pro configuration is present in the [integrations.pro](http://integrations.pro) section in atmos.yaml, it will be added in the config section when executing the atmos describe affected --upload=true command for further processing on the server

    {
         "base_sha": "6746ba4df9e87690c33297fe740011e5ccefc1f9",
         "head_sha": "5360d911d9bac669095eee1ca1888c3ef5291084",
         "owner": "cloudposse",
         "repo": "atmos",
         "config": {
           "timeout": 3,
            "events": [
               "pull_request": [
                  {
                    "on": ["open", "synchronize", "reopen"],
                    "workflow": "atmos-plan.yml",
                    "dispatch_only_top_level_stacks": true
                  },               
                  {
                    "on": ["merged"],
                    "workflow": "atmos-apply.yaml",
                  },               
                ],
                "release": [
                ]
             ]
         }
         "stacks": [
            {
              "component": "vpc",
              "component_type": "terraform",
              "component_path": "components/terraform/vpc",
              "stack": "plat-ue2-dev",
              "stack_slug": "plat-ue2-dev-vpc",
              "affected": "stack.vars",
              "included_in_dependents": false,
              "dependents": []
            }
        ]
     }
    

Add caching to atmos.Component template function

Atmos caches (in memory) the results of `atmos.Component` template function execution. If you call the function for the same component in a stack more than once, the first call will produce the result and cache it, and all the consecutive calls will just use the cached data. This is useful when you use the `atmos.Component` function for the same component in a stack in multiple places in Atmos stack manifests. It will speed up the function execution and stack processing.

For example:
    components:
      terraform:
        test2:
          vars:
            tags:
              test: '{{ (atmos.Component "test" .stack).outputs.id }}'
              test2: '{{ (atmos.Component "test" .stack).outputs.id }}'
              test3: '{{ (atmos.Component "test" .stack).outputs.id }}'
    
In the example, the `test2` Atmos component uses the outputs (remote state) of the `test` Atmos component from the same stack. The template function `{{ atmos.Component "test" .stack }}` is executed three times (once for each tag).

After the first execution, Atmos caches the result in memory (all the component sections, including the `outputs`), and reuses it in the next two calls to the function. The caching makes the stack processing about three times faster in this particular example. In a production environment where many components are used, the speedup can be even more significant.

Implement atmos.GomplateDatasource template function

The `atmos.GomplateDatasource` template function wraps the [Gomplate Datasources](https:/./atmos.tools/core-concepts/stacks/templates/datasources) and caches the results, allowing executing the same datasource many times without calling the external endpoint multiple times. It speeds up the datasource execution and stack processing, and can eliminate other issues with calling an external endpoint, e.g. timeouts and rate limiting.

*Usage*
      {{ (atmos.GomplateDatasource "<alias>").<attribute> }}
    
*Caching the result of `atmos.GomplateDatasource` function*

Atmos caches (in memory) the results of `atmos.GomplateDatasource` template function execution. If you execute the function for the same datasource alias more than once, the first execution will call the external endpoint, produce the result and cache it. All the consecutive calls will just use the cached data. This is useful when you use the `atmos.GomplateDatasource` function for the same datasource alias in multiple places in Atmos stack manifests. It will speed up the function execution and stack processing.

For example:
    settings:
      templates:
        settings:
          gomplate:
            timeout: 5
            datasources:
              ip:
                url: "<https://api.ipify.org?format=json>"
                headers:
                  accept:
                    - "application/json"
    components:
      terraform:
        test:
          vars:
            tags:
              test1: '{{ (datasource "ip").ip }}'
              test2: '{{ (atmos.GomplateDatasource "ip").ip }}'
              test3: '{{ (atmos.GomplateDatasource "ip").ip }}'
              test4: '{{ (atmos.GomplateDatasource "ip").ip }}'
    
In the example, we define a `gomplate` datasource `ip` and specify an external endpoint in the `url` parameter.

We use the [Gomplate `datasource`](https://docs.gomplate.ca/datasources/) function in the tag `test1`, and the `atmos.GomplateDatasource` wrapper for the same datasource alias `ip` in the other tags. The `atmos.GomplateDatasource` wrapper will call the same external endpoint, but will cache the result and reuse it between the datasource invocations.

When processing the component `test` from the above example, Atmos does the following:

• Executes the `{{ (datasource "ip").ip }}` template. It calls the external endpoint using the HTTP protocol and assign the `ip` attribute from the result to the tag `test1`
• Executes the `{{ (atmos.GomplateDatasource "ip").ip }}` template. It calls the external endpoint again, caches the result in memory, and assigns the `ip` attribute from the result to the tag `test2`
• Executes the `{{ (atmos.GomplateDatasource "ip").ip }}` two more times for the tags `test3` and `test4`. It detects that the result for the same datasource alias `ip` is already presend in the memory cache and reuses it without calling the external endpoint two more times

The datasource result caching makes the stack processing much faster and significantly reduces the load on external endpoints, preventing such issues as timeouts and rate limiting.
2
1
RB avatar

What is atmos pro ?

Gabriela Campana (Cloud Posse) avatar
Gabriela Campana (Cloud Posse)

@Erik Osterman (Cloud Posse) @Matt Calhoun

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We’re working on something that enhances GitHub Actions for continuous delivery

RB avatar

Does this mean there will be a free version and a pay version ?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

It doesn’t change anything about how it works today. We are building new features that are not possible today in GHA, and incorporating those into the Atmos Pro GitHub App.

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

The GitHub App will be a paid offering.

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

For now, we’re working on enterprise=level features that work in concert with GitHub Enterprise. We’ll have more details in the coming months.

RB avatar

cool, looking forward to it

thanks for sharing!

2024-07-12

2024-07-14

2024-07-15

Bradley A. Wangia, Ph.D. avatar
Bradley A. Wangia, Ph.D.
HCSEC-2024-13 - HashiCorp go-getter Vulnerable to Code Execution On Git Update Via Git Config Manipulation

Bulletin ID: HCSEC-2024-13 Affected Products / Versions: go-getter up to 1.7.4; fixed in go-getter 1.7.5. Publication Date: June 24, 2024 Summary HashiCorp’s go-getter library can be coerced into executing Git update on an existing maliciously modified Git Configuration, potentially leading to arbitrary code execution. This vulnerability, CVE-2024-6257, was fixed in go-getter 1.7.5. Background HashiCorp’s go-getter is a library for Go for downloading files or directories from various sourc…

#23 Update module github.com/gruntwork-io/terratest to v0.47.0

Mend Renovate

This PR contains the following updates:

PackageChangeAgeAdoptionPassingConfidence
[github.com/gruntwork-io/terratest](http://github.com/gruntwork-io/terratest)v0.46.14 -> v0.47.0[age](https://camo.githubusercontent.com/bb114c1831edc6a0d99b0b3edbe4172aa347d2f51b53e6b17e2726dd2a744126/68747470733a2f2f646576656c6f7065722e6d656e642e696f2f6170692f6d632f6261646765732f6167652f676f2f6769746875622e636f6d2532666772756e74776f726b2d696f2532667465727261746573742f76302e34372e303f736c696d3d74727565)[adoption](https://camo.githubusercontent.com/f367be4b1f1b478b9154f6926856fbfb26b603af2e0503360818e6a82ebf14ad/68747470733a2f2f646576656c6f7065722e6d656e642e696f2f6170692f6d632f6261646765732f61646f7074696f6e2f676f2f6769746875622e636f6d2532666772756e74776f726b2d696f2532667465727261746573742f76302e34372e303f736c696d3d74727565)[passing](https://camo.githubusercontent.com/62db3770e02eaae00480aa3c33c1174babbf2773045d19ab9bc5932d38255aac/68747470733a2f2f646576656c6f7065722e6d656e642e696f2f6170692f6d632f6261646765732f636f6d7061746962696c6974792f676f2f6769746875622e636f6d2532666772756e74776f726b2d696f2532667465727261746573742f76302e34362e31342f76302e34372e303f736c696d3d74727565)[confidence](https://camo.githubusercontent.com/86d44af64fd50ec6cf096484828d03e4c02e0ddeacc6b06e0b7f08bbbecacd2c/68747470733a2f2f646576656c6f7065722e6d656e642e696f2f6170692f6d632f6261646765732f636f6e666964656e63652f676f2f6769746875622e636f6d2532666772756e74776f726b2d696f2532667465727261746573742f76302e34362e31342f76302e34372e303f736c696d3d74727565)

Release Notes

gruntwork-io/terratest (github.com/gruntwork-io/terratest)

v0.47.0

Compare Source

Modules affected

packerawshelmazure

Description

• Added support for Packer 1.10 • Fixed error checking in Azure • Fixed tests and documentation for helm, aws, packer modules

Related links

https://github.com/gruntwork-io/terratest/pull/1395https://github.com/gruntwork-io/terratest/pull/1419

v0.46.16

Compare Source

Description

• Updated [github.com/hashicorp/go-getter](http://github.com/hashicorp/go-getter) from 1.7.4 to 1.7.5.

Related links

https://github.com/gruntwork-io/terratest/pull/1415

v0.46.15

Compare Source

Description

• Updated golang.org/x/net from 0.17.0 to 0.23.0.

Related links

https://github.com/gruntwork-io/terratest/pull/1402


Configuration

Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

Automerge: Disabled by config. Please merge this manually once you are satisfied.

Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

Ignore: Close this PR and you won’t be reminded about this update again.


• If you want to rebase/retry this PR, check this box


This PR has been generated by Mend Renovate. View repository job log here.

1
Gabriela Campana (Cloud Posse) avatar
Gabriela Campana (Cloud Posse)

@Matt Calhoun @Igor Rodionov @Jeremy G (Cloud Posse)

HCSEC-2024-13 - HashiCorp go-getter Vulnerable to Code Execution On Git Update Via Git Config Manipulation

Bulletin ID: HCSEC-2024-13 Affected Products / Versions: go-getter up to 1.7.4; fixed in go-getter 1.7.5. Publication Date: June 24, 2024 Summary HashiCorp’s go-getter library can be coerced into executing Git update on an existing maliciously modified Git Configuration, potentially leading to arbitrary code execution. This vulnerability, CVE-2024-6257, was fixed in go-getter 1.7.5. Background HashiCorp’s go-getter is a library for Go for downloading files or directories from various sourc…

#23 Update module github.com/gruntwork-io/terratest to v0.47.0

Mend Renovate

This PR contains the following updates:

PackageChangeAgeAdoptionPassingConfidence
[github.com/gruntwork-io/terratest](http://github.com/gruntwork-io/terratest)v0.46.14 -> v0.47.0[age](https://camo.githubusercontent.com/bb114c1831edc6a0d99b0b3edbe4172aa347d2f51b53e6b17e2726dd2a744126/68747470733a2f2f646576656c6f7065722e6d656e642e696f2f6170692f6d632f6261646765732f6167652f676f2f6769746875622e636f6d2532666772756e74776f726b2d696f2532667465727261746573742f76302e34372e303f736c696d3d74727565)[adoption](https://camo.githubusercontent.com/f367be4b1f1b478b9154f6926856fbfb26b603af2e0503360818e6a82ebf14ad/68747470733a2f2f646576656c6f7065722e6d656e642e696f2f6170692f6d632f6261646765732f61646f7074696f6e2f676f2f6769746875622e636f6d2532666772756e74776f726b2d696f2532667465727261746573742f76302e34372e303f736c696d3d74727565)[passing](https://camo.githubusercontent.com/62db3770e02eaae00480aa3c33c1174babbf2773045d19ab9bc5932d38255aac/68747470733a2f2f646576656c6f7065722e6d656e642e696f2f6170692f6d632f6261646765732f636f6d7061746962696c6974792f676f2f6769746875622e636f6d2532666772756e74776f726b2d696f2532667465727261746573742f76302e34362e31342f76302e34372e303f736c696d3d74727565)[confidence](https://camo.githubusercontent.com/86d44af64fd50ec6cf096484828d03e4c02e0ddeacc6b06e0b7f08bbbecacd2c/68747470733a2f2f646576656c6f7065722e6d656e642e696f2f6170692f6d632f6261646765732f636f6e666964656e63652f676f2f6769746875622e636f6d2532666772756e74776f726b2d696f2532667465727261746573742f76302e34362e31342f76302e34372e303f736c696d3d74727565)

Release Notes

gruntwork-io/terratest (github.com/gruntwork-io/terratest)

v0.47.0

Compare Source

Modules affected

packerawshelmazure

Description

• Added support for Packer 1.10 • Fixed error checking in Azure • Fixed tests and documentation for helm, aws, packer modules

Related links

https://github.com/gruntwork-io/terratest/pull/1395https://github.com/gruntwork-io/terratest/pull/1419

v0.46.16

Compare Source

Description

• Updated [github.com/hashicorp/go-getter](http://github.com/hashicorp/go-getter) from 1.7.4 to 1.7.5.

Related links

https://github.com/gruntwork-io/terratest/pull/1415

v0.46.15

Compare Source

Description

• Updated golang.org/x/net from 0.17.0 to 0.23.0.

Related links

https://github.com/gruntwork-io/terratest/pull/1402


Configuration

Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

Automerge: Disabled by config. Please merge this manually once you are satisfied.

Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

Ignore: Close this PR and you won’t be reminded about this update again.


• If you want to rebase/retry this PR, check this box


This PR has been generated by Mend Renovate. View repository job log here.

Gabriela Campana (Cloud Posse) avatar
Gabriela Campana (Cloud Posse)

@Andriy Knysh (Cloud Posse)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Since the tests passed, I just released it as a patch release. @Matt Calhoun is out today.

ameer avatar

how would one go about running a terraform import state under atmos? something like? atmos terraform "import resource resource-id" component --s stack (this didn’t work)

1
Gabriela Campana (Cloud Posse) avatar
Gabriela Campana (Cloud Posse)

@Andriy Knysh (Cloud Posse)

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

atmos terraform import <component> -s <stack> ADDRESS ID

1
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@ameer

1
ameer avatar

thank you!

2024-07-16

Miguel Zablah avatar
Miguel Zablah

dose atmos.Component allows for a dependency of list or only strings?

I try to do this:

vpc_cidr_blocks: '{{ (atmos.Component "vpc" .stack).outputs.vpc_private_cidr_blocks }}'

but I get this:

vpc_cidr_blocks: "[10.100.0.0/24 10.100.1.0/24]"

I will like to have this as a normal list not a string:

vpc_cidr_blocks: [10.0.0.0/24, 10.0.1.0/24]

2024-07-17

Patrick McDonald avatar
Patrick McDonald

Hello, I’m using atmos.Component in my stack manifest file and I’m trying to fetch a list of subnet IDs. I can’t seem to find the correct syntax.

This:

subnet_ids: |
  {{- range (atmos.Component "vpc" .stack).outputs.private_subnet_ids }}
    - {{ . }}
  {{- end }}

produces this in the tfvars.json file:

"subnet_ids": "- subnet-0x8ae0b62b933ab72\n- subnet-08551c9314d89f695\n- subnet-04f5a5e154ec39aa3\n",

This:

subnet_ids:
  '{{- range (atmos.Component "vpc" .stack).outputs.private_subnet_ids }}
    - {{ . }}
  {{- end }}'

produces this in the tfvars.json file:

"subnet_ids": " - subnet-0f8ae0b634933ab72 - subnet-08551c9314d89f695 - subnet-0423a4e154ec39aa3",

And this:

subnet_ids: '{{ (atmos.Component "vpc" .stack).outputs.private_subnet_ids | toJson}}'

produces this in the tfvars.json file:

"subnet_ids": "[\"subnet-0f8ae2362b933ab72\",\"subnet-08551c9314d89f695\",\"subnet-0423a4e154ec39aa3\"]",

Terraform output:

private_subnet_ids = [
  "subnet-0f8a23b62b933ab72",
  "subnet-0855123314d89f695",
  "subnet-04f5a2354ec39aa3",
]

What’s the correct way to fetch the list of subnet IDs?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

this is not the first time people asked the same question. It’s not about “the correct way to fetch the list of subnet IDs”, it’s how to use lists data type, Go templates and YAML to correctly construct valid YAML files (since by uisng Go templates, you are just manipulating text files). We’ll test that and add some docs in the next Atmos release

1
Patrick McDonald avatar
Patrick McDonald

So what I’m trying to accomplish won’t work?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

it will work, let us find the correct way and we’ll show it here

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Patrick McDonald are you familiar with Helm by any chance?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

With the information shared above I’m trying to move up a level with processing a datasource. If I start with a datasource of:

tags:
  - tag1:foo
  - tag2:bar
  - tag3:123

and a template of:

components:
  terraform:
    monitors:
      vars:
        datasource_import: {{ (datasource "global_tags").tags }}
        test:
          {{- range (datasource "global_tags").tags }}
          - {{ . }}
          {{- end }}

I can run gomplate -d global_tags.yaml -f _template.tmpl and get an output of:

components:
  terraform:
    monitors:
      vars:
        datasource_import: [tag1:foo tag2:bar tag3:123]
        test:
          - tag1:foo
          - tag2:bar
          - tag3:123

I’m having trouble translating that to use in an Atmos stack however. I can do a stack such as:

import:
  - catalog/rick/*.yaml

vars:
  environment: test
  stage: rick

components:
  terraform:
    monitors:
      vars:
        datasource: '{{ (datasource "global_tags").tags }}'

And get the output:

vars:
  datasource: '[tag1:foo tag2:bar tag3:123]'

But without putting that in quotes I get errors and I cannot do range over the datasource to get a list.

Any guidance on how using that datasource might work?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Hi, I’m testing out atmos v1.81 template functions release: How do I get array element for this one?

- '{{ (atmos.Component "aws-vpc" .stack).outputs.private_subnets }}'
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Andriy Knysh (Cloud Posse) I think we need to add a chapter about this, since it’s very easily misunderstood

1
Patrick McDonald avatar
Patrick McDonald

are single quotes required? Its treating the list as a string.

subnet_ids: '{{ toJson (atmos.Component "vpc" .stack).outputs.private_subnet_ids }}'

subnet_ids: '["subnet-0f8ae0b62b933ab72","subnet-08551c9314d89f695","subnet-04f5a4e154ec39aa3"]'
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

so we have two diff things here:

  1. The quotes are not needed for Go templates (if you use the quotes, the result will be quoted as well)
  2. But w/o using the quotes, it’s not a valid YAML It’s a combination of Go templates and YAML. We’ll check how to do it correctly and let you know
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

( I realize I didn’t read your post carefully enough, @Patrick McDonald)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)


But w/o using the quotes, it’s not a valid YAML
Well, that’s only because validation is happening before rendering instead of after rendering

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think we have a task to fix that, and move validation after rendering

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

it’s not about validation (we are not validating anything yet), the YAML Go lib we are using does not like that YAML. We are reading the files as YAML before rendering the templates

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

And we need to read the files as YAML to be able to parse them and get all the sections in order to get the final values for all the sections which are actually used in the Go templates.

So we read the files, parse the YAML, deep-merge all the sections, then use the final values as the context for all the Go templates.

It’s not the other way around - we don’t read the files and process the templates b/c the context for the templates are the final mvalues from all the files and sections.

It’s reading all the files first, then deep-merging all the sections, then finding the final values for the component in the stack which are the context for all the Go templates.

If we did evaluate the Go templates before reading the files as YAML and processing the stacks, then context would have to be provided separately (we don’t know how, and it would not work anyway b/c we want to use the final values for the component in the stack, which we can know only after processing everything).

we’ll look into that

Miguel Zablah avatar
Miguel Zablah

@Patrick McDonald did you found a way to do this?

Miguel Zablah avatar
Miguel Zablah

Okay I think I got it this works for me:

vpc_private_subnets: ['{{ range $index, $element := (atmos.Component "vpc" .stack).outputs.vpc_private_subnets }}{{ if $index }}, {{ end }}{{ $element }}{{ end }}']
2

2024-07-18

github3 avatar
github3
01:58:43 PM

Update atmos describe affected and atmos terraform commands @aknysh (#654) ## what • Update atmos describe affected command • Update atmos terraform command • Allow Gomplate, Sprig and Atmos template functions in imports in Atmos stack manifests

why

• The atmos describe affected command had an issue with calculating the included_in_dependents field for all combination of the affected components with their dependencies. Now it’s correctly calculated for all affected • In atmos describe affected command, if the Git config core.untrackedCache is enabled, it breaks the command execution. We disable this option if it is set • The atmos terraform command now respects the TF_WORKSPACE environment variable. If the environment variable is set by the caller, Atmos will not calculate and set a Terraform workspace for the component in the stack, but instead will let Terraform use the workspace provided in the TF_WORKSPACE environment variable • Allow Gomplate, Sprig and Atmos template functions in imports in Atmos stack manifests. All functions are allowed now in Atmos stacks manifests and in the import templates

Andrew Chemis avatar
Andrew Chemis

Hi all - In my atmos.yaml file, where I call the terraform_workspace_pattern, I want to force all values to be lowercase. Is that possible?

terraform:
  metadata:
    terraform_workspace_pattern: "{tenant}-{environment}-{stage}-ws"
1
Gabriela Campana (Cloud Posse) avatar
Gabriela Campana (Cloud Posse)

@Andriy Knysh (Cloud Posse)

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@Andrew Chemis terraform_workspace_pattern attribute is not in atmos.yaml, it should be defined in component stack manifests in metadata section, e.g.

components:
  terraform:
    my-component:
      vars: {}
      env: {}
      metadata:
        # Override Terraform workspace
        # Note that by default, Terraform workspace is generated from the context, e.g. `<environment>-<stage>`
        terraform_workspace_pattern: "{tenant}-{environment}-{stage}-{component}"
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)


I want to force all values to be lowercase. Is that possible?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

if you want to force it at the component level, you can use Go templates and Gomplate functions https://docs.gomplate.ca/functions/strings/

strings functions - gomplate documentation

strings.Abbrev Abbreviates a string using … (ellipses). Takes an optional offset from the beginning of the string, and a maximum final width (including added ellipses). Also see strings.Trunc. Added in gomplate v2.6.0 Usage strings.Abbrev [offset] width inputinput | strings.Abbrev [offset] widthArguments name description offset (optional) offset from the start of the string. Must be 4 or greater for ellipses to be added. Defaults to 0 width (required) the desired maximum final width of the string, including ellipses input (required) the input string to abbreviate Examples $ gomplate -i ‘{{ “foobarbazquxquux” | strings.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

for example:

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
components:
  terraform:
    my-component:
      vars: {}
      env: {}
      metadata:
        # Override Terraform workspace
        terraform_workspace_pattern: '{{ strings.ToLower "{tenant}-{environment}-{stage}-{component}" }}'
1
1
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

refer to templating doc for more details https://atmos.tools/core-concepts/stacks/templates/

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

let me know if you have questions or need more help on that

Andrew Chemis avatar
Andrew Chemis

Ah, I was in my root manifest file and said the wrong thing . Exactly what I was looking for - thanks!

1
Andrew Chemis avatar
Andrew Chemis

@Andriy Knysh (Cloud Posse) hmm just now trying this, getting the error: Error: Error loading state: Failed to retrieve workspace {{ strings.ToLower "SharedServices-uw2-prod-vpc-ws" }}: invalid value for workspace

In the new backend config its creating

   "workspaces": {
   "name": "{{ strings.ToLower \"SharedServices-uw2-prod-vpc-ws\" }}"
}

I have it enabled in my atmos.yaml

templates:
  settings:
    enabled: true
    sprig:
      enabled: false
    gomplate:
      enabled: true
      timeout: 5

and in my stack manifest:

components:
  terraform:
    vpc:
      metadata:
        terraform_workspace_pattern: '{{ strings.ToLower "{tenant}-{environment}-{stage}-{component}-ws" }}'

Any thoughts? Thanks again!

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@Andrew Chemis I’m sorry I showed you that, but Atmos does not support templates in terraform_workspace_pattern, it currently supports templates in these sections https://atmos.tools/core-concepts/stacks/templates/#atmos-sections-supporting-go-templates

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

templates are not supported in the entire metadata section for other reasons (related to inheritance). We will add templates to terraform_workspace_pattern in the next Atmos release

1
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

why do you have the tenant SharedServices in upper-case? Can you make it lower-case?

Andrew Chemis avatar
Andrew Chemis

brownfield deployment that uses control tower and account factory, so a few accounts don’t match naming convention. Ill just adjust the casing

Andrew Chemis avatar
Andrew Chemis

@Andriy Knysh (Cloud Posse) Sorry for repeated annoyances :sweat_smile:. When I try this config,

terraform:
  backend_type: cloud
  backend:
    cloud:
      organization: "ac_atmos_testing"
      hostname: "app.terraform.io"
      workspaces:
        name: "{terraform_workspace}-ws"

with a -ws at the end, I get the issue Workspace "sharedservices-uw2-prod-ws" doesn't exist.

I think this isnt working because the stack manifest is trying to create the workspace using a different naming convention. I dont want to change how im naming my stacks to include ws at the end.

I know i’m not using Atmos properly, but, when I change workspaces:name to anything except {terraform_workspaces} it simply errors out.

workspaces:
  name: "null"

Workspace "sharedservices-uw2-prod" doesn't exist.

So the only way to customize a workspace name is a component level override? End of the day, what im trying to do is set a default workspace convention that differs from the stack name. In my current environment, I will never reuse components in my organization. Adding {component} to my workspace is noise and complicates permission management. But, then for a weird reason we want to append -ws to the end of the workspace name.

Desired stack name: {tenant}-{environment}-{stage} desired workspace name: {tenant}-{environment}-{stage}-ws

In our env, {tenant} is basically the account name, so were also probably null-label wrong

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)


with a -ws at the end, I get the issue Workspace "sharedservices-uw2-prod-ws" doesn't exist.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

that’s because the selected TF workspace is what’s replaced in the {terraform_workspace } token

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

to understand how Atmos ctreates TF workspaces for components, please refer to this doc https://atmos.tools/core-concepts/components/terraform/workspaces/

Terraform Workspaces | atmos

Terraform Workspaces.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

whatever workspace is calculated for a component in a stack (automatically by using the stack name, or by overriding it in metadata.terraform_worspace_pattern), the token {terraform_workspace} gets replaced with it

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

in the above case,

      workspaces:
        name: "{terraform_workspace}-ws"
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

is not the same as the selected TF workspace (it adds -ws )

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

execute atmos describe component <component> -s <stack> to see what TF workspace is used, it should be in the workspace field in the output

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

in any case, this will be always wrong

      workspaces:
        name: "{terraform_workspace}-ws"
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

b/c the selected TF workspace will be inserted instead of the {terraform_workspace} token. Adding -ws will make the workspace in TF Cloud different from the selected TF workspace

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

use this instead

terraform:
  backend_type: cloud
  backend:
    cloud:
      organization: "ac_atmos_testing"
      hostname: "app.terraform.io"
      workspaces:
        name: "{terraform_workspace}"
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

and then you can add -ws at the end for a component:

components:
  terraform:
    vpc:
      metadata:
        terraform_workspace_pattern: '{tenant}-{environment}-{stage}-ws'

2024-07-19

Brett L avatar
Brett L

Does atmos have features for importing existing resources / moving state?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

It does not. The back story is how you work with state largely depends on the backend (e.g. TFE, AWS/S3, GCP, Azure, etc). It’s hard to do a generic implementation.

Brett L avatar
Brett L

got it

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

but it supports all standard Terraform commands for state management atmos terraform state list/mv <component> -s <stack>

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

also terraform import

atmos terraform import <component> -s <stack> ADDRESS ID 
RB avatar

The import or move blocks should work too…

Actually come to think of it, if you added those blocks, it would only work for a single workspace and not all workspaces.


Like andriy said, you can use the atmos terraform import command. If you have multiple imports, you can also use the shell to run the raw commands

atmos terraform shell <component> -s <stack>
terraform import ADDRESS ID
1

2024-07-20

    keyboard_arrow_up