#atmos-dev (2024-11)

what

why

references

• DEV-2150

Summary by CodeRabbit

• New Features
• Enhanced component processing logic to include checks for whether components are enabled, improving metadata handling.
• Introduced a new field to track the enabled state of components, optimizing performance and resource management.
• Updated processing methods to skip disabled components, refining the control flow and enhancing efficiency. • Bug Fixes
• Adjusted control flow in component processing to ensure only enabled components are processed, reducing potential errors.

what

why

working demo

image

references

• DEV-2150

Summary by CodeRabbit

Summary by CodeRabbit

• New Features
• Enhanced component processing logic to include checks for whether components are enabled, improving metadata handling.
• Introduced a new field to track the enabled state of components, optimizing performance and resource management.
• Updated processing methods to skip disabled components, refining the control flow and enhancing efficiency.
• Added documentation on using the metadata.enabled parameter to conditionally include components in deployment. • Bug Fixes
• Adjusted control flow in component processing to ensure only enabled components are processed, reducing potential errors.

what

• Added new support for vendor files under folders or multiple vendor files to be processed in Lexicographic order • Refactored logic that does check the presence of vendor file

why

• Users now should be able to use new variable vendor in atmos.yaml and process different locations and amounts of vendor files.

references

#DEV-2378

CleanShot 2024-10-22 at 23 34 50@2x

New tests after general refactoring base path
CleanShot 2024-10-30 at 17 02 50

Summary by CodeRabbit

Release Notes

• New Features
• Introduced a vendor section in the atmos.yaml configuration, allowing users to specify a base_path for vendor files.
• Added support for the new command-line flag --vendor-base-path to enhance configuration options.
• Enhanced documentation to clarify new configuration options and loading behavior. • Configuration Updates
• New YAML files for development, production, and staging environments with structured variable management.
• Added new vendor configuration files to demonstrate sourcing components from GitHub. • Documentation
• Updated CLI documentation to include new parameters and configuration options for better usability.
• Expanded details on the loading order of configuration files, logging behavior, and added sections for aliases and templates.

what

This PR changes the default behavior of atmos describe stacks it filter empty stacks by default unless user pass the flag --include-empty-stacks

why

This was causing stacks with empty results or no components/imports components to be displayed.

Test Results

Screenshot 2024-11-05 at 11 36 39
Screenshot 2024-11-05 at 11 36 53

references

DEV-1880

Summary by CodeRabbit

• New Features
• Introduced a new command-line flag --include-empty-stacks to include stacks without components in the output.
• Enhanced stack processing logic to support filtering based on the new flag. • Bug Fixes
• Improved error handling and messaging for Git reference issues. • Tests
• Added new test cases to validate the behavior of the ExecuteDescribeStacks function with empty stacks.

What

• Added improved workflow failure handling with clear error messages • Added --from-step flag to resume workflows from specific steps • Added documentation and help text for workflow failure handling • Added example workflow demonstrating failure scenarios

Why

• Makes it easier to debug and fix workflow failures by showing: • Which step failed • The exact command that failed • How to resume from the failed step • Saves time by allowing users to resume workflows from failed steps instead of restarting from beginning • Improves user experience with clear guidance on how to recover from failures

Example workflow demonstrating the feature:

workflows:

test-1: description: “Test workflow with intentionally failing steps” steps: - name: “step-1” type: shell command: “echo ‘This step will succeed’ && exit 0”

  - name: "step-2"
    type: shell
    command: "echo 'This step will fail' && exit 1"
  
  - name: "step-3"
    type: shell
    command: "echo 'This step should not execute'"

Tests

Last test after requested changes @osterman @aknysh
When step-2 fails, users will see:
CleanShot 2024-11-08 at 20 00 10

references

DEV-2717

Summary by CodeRabbit

• New Features
• Added detailed help information for workflow commands, including failure handling and resume functionality.
• Enhanced error messages during workflow execution to provide clearer guidance on failures and resuming workflows. • Documentation
• Updated documentation to include a new section on “Workflow Failure Handling and Resume.”
• Clarified the “Stack Precedence” section to better explain stack definitions within workflows.

Description

The output of cosign verify --help is unreadable due to line wrapping.

image

We solved this in kubectl by creating a custom tab printer.

kubernetes/kubernetes#104736

What type of PR is this?

/kind design

What this PR does / why we need it:

Make the help flags of kubectl more readable / user friendly

Which issue(s) this PR fixes:

Fixes kubernetes/kubectl#1069

Special notes for your reviewer:

Before,

Apply a configuration to a resource by file name or stdin. The resource name must be specified. This
resource will be created if it doesn't exist yet. To use 'apply', always create the resource
initially with either 'apply' or 'create --save-config'.

 JSON and YAML formats are accepted.

 Alpha Disclaimer: the --prune functionality is not yet complete. Do not use unless you are aware of
what the current state is. See <https://issues.k8s.io/34274>.

Examples:
  # Apply the configuration in pod.json to a pod
  kubectl apply -f ./pod.json
  
  # Apply resources from a directory containing kustomization.yaml - e.g. dir/kustomization.yaml
  kubectl apply -k dir/
  
  # Apply the JSON passed into stdin to a pod
  cat pod.json | kubectl apply -f -
  
  # Note: --prune is still in Alpha
  # Apply the configuration in manifest.yaml that matches label app=nginx and delete all other
resources that are not in the file and match label app=nginx
  kubectl apply --prune -f manifest.yaml -l app=nginx
  
  # Apply the configuration in manifest.yaml and delete all the other config maps that are not in
the file
  kubectl apply --prune -f manifest.yaml --all --prune-whitelist=core/v1/ConfigMap

Available Commands:
  edit-last-applied Edit latest last-applied-configuration annotations of a resource/object
  set-last-applied  Set the last-applied-configuration annotation on a live object to match the
contents of a file
  view-last-applied View the latest last-applied-configuration annotations of a resource/object

Options:
      --all=false: Select all resources in the namespace of the specified resource types.
      --allow-missing-template-keys=true: If true, ignore any errors in templates when a field or
map key is missing in the template. Only applies to golang and jsonpath output formats.
      --cascade='background': Must be "background", "orphan", or "foreground". Selects the deletion
cascading strategy for the dependents (e.g. Pods created by a ReplicationController). Defaults to
background.
      --dry-run='none': Must be "none", "server", or "client". If client strategy, only print the
object that would be sent, without sending it. If server strategy, submit server-side request
without persisting the resource.
      --field-manager='kubectl-client-side-apply': Name of the manager used to track field
ownership.
  -f, --filename=[]: that contains the configuration to apply
      --force=false: If true, immediately remove resources from API and bypass graceful deletion.
Note that immediate deletion of some resources may result in inconsistency or data loss and requires
confirmation.
      --force-conflicts=false: If true, server-side apply will force the changes against conflicts.
      --grace-period=-1: Period of time in seconds given to the resource to terminate gracefully.
Ignored if negative. Set to 1 for immediate shutdown. Can only be set to 0 when --force is true
(force deletion).
  -k, --kustomize='': Process a kustomization directory. This flag can't be used together with -f or
-R.
      --openapi-patch=true: If true, use openapi to calculate diff when the openapi presents and the
resource can be found in the openapi spec. Otherwise, fall back to use baked-in types.
  -o, --output='': Output format. One of:
json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-as-json|jsonpath-file.
      --overwrite=true: Automatically resolve conflicts between the modified and live configuration
by using values from the modified configuration
      --prune=false: Automatically delete resource objects, including the uninitialized ones, that
do not appear in the configs and are created by either apply or create --save-config. Should be used
with either -l or --all.
      --prune-whitelist=[]: Overwrite the default whitelist with <group/version/kind> for --prune
  -R, --recursive=false: Process the directory used in -f, --filename recursively. Useful when you
want to manage related manifests organized within the same directory.
  -l, --selector='': Selector (label query) to filter on, supports '=', '==', and '!='.(e.g. -l
key1=value1,key2=value2)
      --server-side=false: If true, apply runs in the server instead of the client.
      --show-managed-fields=false: If true, keep the managedFields when printing objects in JSON or
YAML format.
      --template='': Template string or path to template file to use when -o=go-template,
-o=go-template-file. The template format is golang templates
[<http://golang.org/pkg/text/template/#pkg-overview>].
      --timeout=0s: The length of time to wait before giving up on a delete, zero means determine a
timeout from the size of the object
      --validate=true: If true, use a schema to validate the input before sending it
      --wait=false: If true, wait for resources to be gone before returning. This waits for
finalizers.

Usage:
  kubectl apply (-f FILENAME | -k DIRECTORY) [options]

Use "kubectl <command> --help" for more information about a given command.
Use "kubectl options" for a list of global command-line options (applies to all commands).

After,

``` Apply a configuration to a resource by file name or stdin. The resource name must be specified. This resource will be created if it doesn’t exist yet. To use ‘apply’, always create the resource initially with either ‘apply’ or ‘create –save-config’.

JSON and YAML formats are accepted.

Alpha Disclaimer: the –prune functionality is not yet complete. Do not use unless you are aware of what the current state is. See https://issues.k8s.io/34274.

Examples: # Apply the configuration in pod.json to a pod kubectl apply -f ./pod.json

# Apply resources from a directory containing kustomization.yaml - e.g. dir/kustomization.yaml kubectl apply -k dir/

# Apply the JSON passed into stdin to a pod cat pod.json | kubectl apply -f -

# Note: –prune is still in Alpha # Apply the configuration in manifest.yaml that matches label app=nginx and delete all other resources that are not in the file and match label app=nginx kubectl apply –prune -f manifest.yaml -l app=nginx

# Apply the configuration in manifest.yaml and delete all the other config maps that are not in the file kubectl apply –prune -f manifest.yaml –all –prune-whitelist=core/v1/ConfigMap

Available Commands: edit-last-applied Edit latest last-applied-configuration annotations of a resource/object set-last-applied Set the last-applied-configuration annotation on a live object to match the contents of a file view-last-applied View the latest last-applied-configuration annotations of a resource/object

Options: –all=false: Select all resources in the namespace of the specified resource types.

--allow-missing-template-keys=true:
If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to
golang and jsonpath output formats.

--cascade='background':
Must be "background", "orphan", or "foreground". Selects the deletion cascading strategy for the dependents
(e.g. Pods created by a ReplicationController). Defaults to background.

--dry-run='none':
Must be "none", "server", or "client". If client strategy, only print the object that would be sent, without
sending it. If server strategy, submit server-side request without persisting the resource.

--field-manager='kubectl-client-side-apply':
Name of the manager used to track field ownership.

-f, --filename=[]:
that contains the configuration to apply

--force=false:
If true, immediately remove resources from API and bypass graceful deletion. Note that immediate deletion of
some resources may result in inconsist…

what

why

references

• DEV-1899

Summary by CodeRabbit

Summary by CodeRabbit

• New Features
• Customized command prompt for the interactive shell with the addition of the “atmos>” prefix. • Bug Fixes
• Enhanced shell behavior by removing the unnecessary -l flag for non-Windows systems and implementing a fallback to sh if bash is unavailable.
• Improved handling for the /bin/zsh shell with additional flags.

Closes #495

what

• New Feature interactive package manager for vendor pull
interactive shell for atmos vendor pull –everything
Support –everything flag
for test non TTY mod run atmos vendor pull –everything </dev/null |& cat > atmos_vendor_pull.log
2024-11-09 1502_32-vendor_model go - atmos WSL Ubuntu - Visual Studio Code
2024-11-09 1502_51-vendor_model go - atmos WSL Ubuntu - Visual Studio Code

why

build an interface for a package manager using bubbletea

references

https://linear.app/cloudposse/issue/DEV-270/atmos-vendor-pull-ux
https://linear.app/cloudposse/issue/DEV-2716/support-atmos-vendor-pull-everything

Summary by CodeRabbit

Summary by CodeRabbit

• New Features
• Enhanced user interface for vendor commands utilizing the Bubble Tea framework.
• New --everything flag for the vendorPullCmd command, allowing users to vendor all components in one go.
• Improved functionality for managing vendor package installations, including enhanced error handling and progress feedback.
• Updated configuration files and documentation to utilize the new --everything option in vendor commands. • Bug Fixes
• Refined validation logic for command-line flags to prevent conflicts. • Refactor
• Modularized code structure for better readability and maintainability.
• Improved error handling and logging for vendor operations. • Documentation
• Updated internal documentation to reflect changes in methods and functionalities.
• Revised CLI Commands Cheat Sheet and other documentation to include the new --everything flag for vendor commands.

Describe the Bug

After updating from v1.105.0 to v1.106.0, using a component.yaml file in the component directories no longer works as it returns an error if the vendor file or directory does not exist. Appears to be related to this change: https://github.com/cloudposse/atmos/pull/737/files#diff-35bbd3669178afb0acaaebfb4ec924ec99189338a9bc05232b4f92a53f261518L83

Expected Behavior

Running atmos vendor pull for a component containing a component.yaml file should result in the component being successfully vendored with specified mixins added.

Component config file

image

Expected command output

image

Steps to Reproduce

Running the same command using v1.106.0 results in an error due to the vendor file and/or directory not existing.

image

Screenshots

No response

Environment

• OS: WLS2/Debian12 using Geodesic 2.8.7 & Atmos 1.106.0

Additional Context

No response

Use Atmos vendoring to make copies of 3rd-party components, stacks, and other artifacts in your own repo.

Describe the Bug

After updating from v1.105.0 to v1.106.0, using a component.yaml file in the component directories no longer works as it returns an error if the vendor file or directory does not exist. Appears to be related to this change: https://github.com/cloudposse/atmos/pull/737/files#diff-35bbd3669178afb0acaaebfb4ec924ec99189338a9bc05232b4f92a53f261518L83

Expected Behavior

Running atmos vendor pull for a component containing a component.yaml file should result in the component being successfully vendored with specified mixins added.

Component config file

image

Expected command output

image

Steps to Reproduce

Running the same command using v1.106.0 results in an error due to the vendor file and/or directory not existing.

image

Screenshots

No response

Environment

• OS: WLS2/Debian12 using Geodesic 2.8.7 & Atmos 1.106.0

Additional Context

No response

what

Added directory creation logic before copying vendored files
Fixed the “no such file or directory” error when running atmos vendor pull
Ensures target directories are created before attempting to vendor components

why

Previously, the vendor command would fail if target directories didn’t exist
Directory creation is a prerequisite for vendoring components

references

#780

Logic design

Summary by CodeRabbit

• Bug Fixes
• Improved error handling for vendor configuration file checks, preventing unnecessary error messages when the file does not exist.
• Streamlined the logic for processing vendor configuration, enhancing clarity and robustness.

@Cerebrovinny I do not think this is the fix based on the description of the problem. Bear in mind we have two types of vendor manifests.

https://atmos.tools/core-concepts/vendor/

The bug report concerns the component manifest file. That means that any logic relating to the vendor manifests file should not inhibit the component manifest from functioning.

To me, it sounds like the regression is that when vendoring a component manifest, it is erroring if the vendor manifests file path does not exist. However, that should not be required. We should instead log to warning that the path does not exist and continue processing.

We should not create the vendor configuration path if it does not exist.

Describe the Bug

While trying to understand why I have to set both ATMOS_BASE_PATH and ATMOS_CLI_CONFIG_PATH and if one of them used the other as a default I stumbled over this discrepancy in the “Learn Atmos” section:

The explanation of base_path at https://atmos.tools/core-concepts/projects/configuration/#path-configuration says

We set it to an empty string because we’ve decided to use the ENV var ATMOS_BASE_PATH to point to the absolute path of the root of the repo

But the example further up at https://atmos.tools/core-concepts/projects/configuration/#atmos-cli-configuration-schema actually sets it to the value ./ (which is clearly not an empty string).

And if I interpret https://atmos.tools/cli/configuration/#base-path correctly an empty string would indeed have a different behaviour than that . value.

Expected Behavior

I read the documentation and shouldn’t have been confused.

Steps to Reproduce

Read the documentation :-)

Screenshots

No response

Environment

No response

Additional Context

No response

what

• Used the ConvertToJSON utility with json.MarshalIndent to produce formatted JSON • Indentation is set to two spaces (“ “) for consistent readability

why

• This PR improves the WriteToFileAsJSON function by introducing pretty-printing for JSON outputs. Previously, the function serialized JSON using a compact format, which could make the resulting files harder to read. With this change, all JSON written by this function will now be formatted with indentation, making it easier for developers and users to inspect and debug the generated files • This specifically addresses #778 , which previously rendered auto-generated backends as:

{ “terraform”: { “backend”: { “s3”: { “acl”: “bucket-owner-full-control”, “bucket”: “my-tfstate-bucket”, “dynamodb_table”: “some-dynamo-table”, “encrypt”: true, “key”: “terraform.tfstate”, “profile”: “main”, “region”: “us-west-2”, “workspace_key_prefix”: “something” } } } }

With this addition, the output appears as:

{ “terraform”: { “backend”: { “s3”: { “acl”: “bucket-owner-full-control”, “bucket”: “my-tfstate-bucket”, “dynamodb_table”: “some-dynamo-table”, “encrypt”: true, “key”: “terraform.tfstate”, “profile”: “main”, “region”: “us-west-2”, “workspace_key_prefix”: “something” } } } }

references

• Stack Overflow • Closes #778

Summary by CodeRabbit

Summary by CodeRabbit

• New Features
• Enhanced JSON output formatting with indented structure for better readability. • Refactor
• Simplified variable declarations for consistency in JSON handling functions.

what

why

Working demo

With custom prompt:

Screenshot 2024-11-16 at 11 20 14 PM

Without custom prompt:

image

Closes #779 (disable new prompt by default until we make it work better with geodesic)

references

Summary by CodeRabbit

Summary by CodeRabbit

• New Features
• Introduced a configurable shell prompt for Terraform commands, allowing users to customize their command-line interface.
• Added a new ShellConfig type to support the new shell prompt configuration. • Bug Fixes
• Improved flexibility in shell prompt settings by allowing dynamic configuration based on user-defined CLI settings.

what

• Explain when to display help vs usage, and their respective behaviors • Explain when to use various log levels

why

• Improve code consistency as the dev team expands

what

• Creates an atmos list command for listing stacks and components

atmos_list

• Incorporates custom list commands into Atmos • Updates documentation and website • Removes atmos.yaml references to custom list command

why

• While the custom Atmos commands for listing stacks and components are great, incorporating the command into Atmos is far more efficient and parallelized, achieving similar or better results in 0.741 seconds compared to 8.131 seconds for the custom command

references

• Atmos Custom Command Docs

Summary by CodeRabbit

Release Notes

• New Features
• Introduced a list command with subcommands for listing Atmos stacks and components.
• Enhanced command functionality with options to filter by stack and component type. • Documentation
• Added comprehensive documentation for the new list, list stacks, and list components commands, including usage examples and flags.
• Updated existing documentation to clarify command usage and improve user guidance. • Bug Fixes
• Improved error handling and output formatting for commands related to listing stacks and components.

Describe the Bug

We bumped from 1.90.0 to 1.105.0. Our CI uses

atmos validate component <component> --stack <stack>

With 1.105.0 (and 1.106.0) we now get the commands usage printed instead/additionally:

atmos validate component "vpc/foo" --stack bar

Atmos supports native '' commands with all the options, arguments and flags.

In addition, 'component' and 'stack' are required in order to generate variables for the component in the stack.

atmos  <subcommand> <component> -s <stack> [options]
atmos  <subcommand> <component> --stack <stack> [options]

For more details, execute ' --help'

component 'vpc/foo' in stack 'bar' validated successfully

So it looks like some validation still took place and it was successfull.

Expected Behavior

Exit code 0 on successfull validation, some other number otherwise. No usage gets printed to the screen.

Steps to Reproduce

Run atmos validate component <component> --stack <stack> as documented via --help.

Screenshots

No response

Environment

• OS: Linux • atmos Version: 1.105.0 • terraform Version: 1.8.5

Additional Context

No response

what

Fixed the regression where the vendor command would error if the vendor manifests file path did not exist.

why

Previously, the atmos vendor pull command would fail with a “no such file or directory” error when the vendor manifests file path was missing.

references

Closes #780

Logic design

Summary by CodeRabbit

Summary by CodeRabbit

• Bug Fixes
• Improved error handling for vendor configuration file checks, preventing unnecessary error messages when the file does not exist.
• Enhanced handling of empty YAML configuration files, ensuring graceful processing without errors.
• Streamlined handling of the Atmos JSON Schema manifest, providing a default when none is specified and improving clarity of error messages.