#atmos-dev (2024-11)

Discuss atmos core development (golang)

2024-11-04

2024-11-05

Pulak Kanti Bhowmick avatar
Pulak Kanti Bhowmick

Hi team, this PR is good to review. Thank you!

https://github.com/cloudposse/atmos/pull/756

#756 skip component if metadata.enabled is set to false

what

why

references

DEV-2150

Summary by CodeRabbit

New Features
• Enhanced component processing logic to include checks for whether components are enabled, improving metadata handling.
• Introduced a new field to track the enabled state of components, optimizing performance and resource management.
• Updated processing methods to skip disabled components, refining the control flow and enhancing efficiency. • Bug Fixes
• Adjusted control flow in component processing to ensure only enabled components are processed, reducing potential errors.

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Andriy Knysh (Cloud Posse) LGTM, ready for your review

#756 skip component if metadata.enabled is set to false

what

why

references

DEV-2150

Summary by CodeRabbit

New Features
• Enhanced component processing logic to include checks for whether components are enabled, improving metadata handling.
• Introduced a new field to track the enabled state of components, optimizing performance and resource management.
• Updated processing methods to skip disabled components, refining the control flow and enhancing efficiency. • Bug Fixes
• Adjusted control flow in component processing to ensure only enabled components are processed, reducing potential errors.

2024-11-06

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

if you addressed PR comments, please ping me here for re-review (i don’t get notifications from GitHub when you mark a PR as ready for review)

1
Pulak Kanti Bhowmick avatar
Pulak Kanti Bhowmick

Yes, I have addressed your comments. Please review again https://github.com/cloudposse/atmos/pull/756

#756 skip component if metadata.enabled is set to false

what

why

working demo

image

references

DEV-2150

Summary by CodeRabbit

Summary by CodeRabbit

New Features
• Enhanced component processing logic to include checks for whether components are enabled, improving metadata handling.
• Introduced a new field to track the enabled state of components, optimizing performance and resource management.
• Updated processing methods to skip disabled components, refining the control flow and enhancing efficiency.
• Added documentation on using the metadata.enabled parameter to conditionally include components in deployment. • Bug Fixes
• Adjusted control flow in component processing to ensure only enabled components are processed, reducing potential errors.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@Pulak Kanti Bhowmick please address the comments in this PR

Pulak Kanti Bhowmick avatar
Pulak Kanti Bhowmick

Yes, apology for my late reply. Checking now

2024-11-07

Pulak Kanti Bhowmick avatar
Pulak Kanti Bhowmick

Hi Team, this PR is good to review: https://github.com/cloudposse/atmos/pull/766

Thank you so much!

1
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

thanks @Pulak Kanti Bhowmick, approved and merged

Vinny avatar

@Andriy Knysh (Cloud Posse) this is good for review https://github.com/cloudposse/atmos/pull/767

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

and the docs are updated, nice thanks

1
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@Vinny the PR looks good, please address the comments

1

2024-11-08

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Andriy Knysh (Cloud Posse) i think this is ready for final review https://github.com/cloudposse/atmos/pull/756

#756 skip component if metadata.enabled is set to false

what

why

working demo

image

references

DEV-2150

Summary by CodeRabbit

Summary by CodeRabbit

New Features
• Enhanced component processing logic to include checks for whether components are enabled, improving metadata handling.
• Introduced a new field to track the enabled state of components, optimizing performance and resource management.
• Updated processing methods to skip disabled components, refining the control flow and enhancing efficiency.
• Added documentation on using the metadata.enabled parameter to conditionally include components in deployment. • Bug Fixes
• Adjusted control flow in component processing to ensure only enabled components are processed, reducing potential errors.

1
1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Vinny please address outstanding comments in PRs so we can get them merged

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
#737 Feat: Vendor path setting in yaml support

what

• Added new support for vendor files under folders or multiple vendor files to be processed in Lexicographic order • Refactored logic that does check the presence of vendor file

why

• Users now should be able to use new variable vendor in atmos.yaml and process different locations and amounts of vendor files.

references

#DEV-2378

CleanShot 2024-10-22 at 23 34 50@2x

New tests after general refactoring base path
CleanShot 2024-10-30 at 17 02 50

Summary by CodeRabbit

Release Notes

New Features
• Introduced a vendor section in the atmos.yaml configuration, allowing users to specify a base_path for vendor files.
• Added support for the new command-line flag --vendor-base-path to enhance configuration options.
• Enhanced documentation to clarify new configuration options and loading behavior. • Configuration Updates
• New YAML files for development, production, and staging environments with structured variable management.
• Added new vendor configuration files to demonstrate sourcing components from GitHub. • Documentation
• Updated CLI documentation to include new parameters and configuration options for better usability.
• Expanded details on the loading order of configuration files, logging behavior, and added sections for aliases and templates.

Vinny avatar

ok will get this done today also

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Are you stuck on https://github.com/cloudposse/atmos/pull/764? The tests are still failing

#764 Filter out empty results from describe stacks

what

This PR changes the default behavior of atmos describe stacks it filter empty stacks by default unless user pass the flag --include-empty-stacks

why

This was causing stacks with empty results or no components/imports components to be displayed.

Test Results

Screenshot 2024-11-05 at 11 36 39
Screenshot 2024-11-05 at 11 36 53

references

DEV-1880

Summary by CodeRabbit

New Features
• Introduced a new command-line flag --include-empty-stacks to include stacks without components in the output.
• Enhanced stack processing logic to support filtering based on the new flag. • Bug Fixes
• Improved error handling and messaging for Git reference issues. • Tests
• Added new test cases to validate the behavior of the ExecuteDescribeStacks function with empty stacks.

Vinny avatar

good to go https://github.com/cloudposse/atmos/pull/767 added a test image also cc: @Andriy Knysh (Cloud Posse)

#767 Show failed command after atmos workflow failure

What

• Added improved workflow failure handling with clear error messages • Added --from-step flag to resume workflows from specific steps • Added documentation and help text for workflow failure handling • Added example workflow demonstrating failure scenarios

Why

• Makes it easier to debug and fix workflow failures by showing: • Which step failed • The exact command that failed • How to resume from the failed step • Saves time by allowing users to resume workflows from failed steps instead of restarting from beginning • Improves user experience with clear guidance on how to recover from failures

Example workflow demonstrating the feature:

workflows:

test-1: description: “Test workflow with intentionally failing steps” steps: - name: “step-1” type: shell command: “echo ‘This step will succeed’ && exit 0”

  - name: "step-2"
    type: shell
    command: "echo 'This step will fail' && exit 1"
  
  - name: "step-3"
    type: shell
    command: "echo 'This step should not execute'"

Tests

Last test after requested changes @osterman @aknysh
When step-2 fails, users will see:
CleanShot 2024-11-08 at 20 00 10

references

DEV-2717

Summary by CodeRabbit

New Features
• Added detailed help information for workflow commands, including failure handling and resume functionality.
• Enhanced error messages during workflow execution to provide clearer guidance on failures and resuming workflows. • Documentation
• Updated documentation to include a new section on “Workflow Failure Handling and Resume.”
• Clarified the “Stack Precedence” section to better explain stack definitions within workflows.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Left some more feedback

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Also, the test image has the wrong text “Failed command:” instead of “Command failed:”

Vinny avatar

this is a pain to fix found another more samples but the Flags: part is still not good https://linear.app/cloudposse/issue/DEV-335/atmos-needs-to-wrap-long-lines-in-help

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Need more specific details to be able to advise

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Did you explore how they fixed it here with cobra? https://github.com/sigstore/cosign/issues/2887

#2887 Cleanup cosign help text with a custom printerattachment image

Description

The output of cosign verify --help is unreadable due to line wrapping.

image

We solved this in kubectl by creating a custom tab printer.

kubernetes/kubernetes#104736

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
#104736 Beautify kubectl help flag commands

What type of PR is this?

/kind design

What this PR does / why we need it:

Make the help flags of kubectl more readable / user friendly

Which issue(s) this PR fixes:

Fixes kubernetes/kubectl#1069

Special notes for your reviewer:

Before,

Apply a configuration to a resource by file name or stdin. The resource name must be specified. This
resource will be created if it doesn't exist yet. To use 'apply', always create the resource
initially with either 'apply' or 'create --save-config'.

 JSON and YAML formats are accepted.

 Alpha Disclaimer: the --prune functionality is not yet complete. Do not use unless you are aware of
what the current state is. See <https://issues.k8s.io/34274>.

Examples:
  # Apply the configuration in pod.json to a pod
  kubectl apply -f ./pod.json
  
  # Apply resources from a directory containing kustomization.yaml - e.g. dir/kustomization.yaml
  kubectl apply -k dir/
  
  # Apply the JSON passed into stdin to a pod
  cat pod.json | kubectl apply -f -
  
  # Note: --prune is still in Alpha
  # Apply the configuration in manifest.yaml that matches label app=nginx and delete all other
resources that are not in the file and match label app=nginx
  kubectl apply --prune -f manifest.yaml -l app=nginx
  
  # Apply the configuration in manifest.yaml and delete all the other config maps that are not in
the file
  kubectl apply --prune -f manifest.yaml --all --prune-whitelist=core/v1/ConfigMap

Available Commands:
  edit-last-applied Edit latest last-applied-configuration annotations of a resource/object
  set-last-applied  Set the last-applied-configuration annotation on a live object to match the
contents of a file
  view-last-applied View the latest last-applied-configuration annotations of a resource/object

Options:
      --all=false: Select all resources in the namespace of the specified resource types.
      --allow-missing-template-keys=true: If true, ignore any errors in templates when a field or
map key is missing in the template. Only applies to golang and jsonpath output formats.
      --cascade='background': Must be "background", "orphan", or "foreground". Selects the deletion
cascading strategy for the dependents (e.g. Pods created by a ReplicationController). Defaults to
background.
      --dry-run='none': Must be "none", "server", or "client". If client strategy, only print the
object that would be sent, without sending it. If server strategy, submit server-side request
without persisting the resource.
      --field-manager='kubectl-client-side-apply': Name of the manager used to track field
ownership.
  -f, --filename=[]: that contains the configuration to apply
      --force=false: If true, immediately remove resources from API and bypass graceful deletion.
Note that immediate deletion of some resources may result in inconsistency or data loss and requires
confirmation.
      --force-conflicts=false: If true, server-side apply will force the changes against conflicts.
      --grace-period=-1: Period of time in seconds given to the resource to terminate gracefully.
Ignored if negative. Set to 1 for immediate shutdown. Can only be set to 0 when --force is true
(force deletion).
  -k, --kustomize='': Process a kustomization directory. This flag can't be used together with -f or
-R.
      --openapi-patch=true: If true, use openapi to calculate diff when the openapi presents and the
resource can be found in the openapi spec. Otherwise, fall back to use baked-in types.
  -o, --output='': Output format. One of:
json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-as-json|jsonpath-file.
      --overwrite=true: Automatically resolve conflicts between the modified and live configuration
by using values from the modified configuration
      --prune=false: Automatically delete resource objects, including the uninitialized ones, that
do not appear in the configs and are created by either apply or create --save-config. Should be used
with either -l or --all.
      --prune-whitelist=[]: Overwrite the default whitelist with <group/version/kind> for --prune
  -R, --recursive=false: Process the directory used in -f, --filename recursively. Useful when you
want to manage related manifests organized within the same directory.
  -l, --selector='': Selector (label query) to filter on, supports '=', '==', and '!='.(e.g. -l
key1=value1,key2=value2)
      --server-side=false: If true, apply runs in the server instead of the client.
      --show-managed-fields=false: If true, keep the managedFields when printing objects in JSON or
YAML format.
      --template='': Template string or path to template file to use when -o=go-template,
-o=go-template-file. The template format is golang templates
[<http://golang.org/pkg/text/template/#pkg-overview>].
      --timeout=0s: The length of time to wait before giving up on a delete, zero means determine a
timeout from the size of the object
      --validate=true: If true, use a schema to validate the input before sending it
      --wait=false: If true, wait for resources to be gone before returning. This waits for
finalizers.

Usage:
  kubectl apply (-f FILENAME | -k DIRECTORY) [options]

Use "kubectl <command> --help" for more information about a given command.
Use "kubectl options" for a list of global command-line options (applies to all commands).


After,

``` Apply a configuration to a resource by file name or stdin. The resource name must be specified. This resource will be created if it doesn’t exist yet. To use ‘apply’, always create the resource initially with either ‘apply’ or ‘create –save-config’.

JSON and YAML formats are accepted.

Alpha Disclaimer: the –prune functionality is not yet complete. Do not use unless you are aware of what the current state is. See https://issues.k8s.io/34274.

Examples: # Apply the configuration in pod.json to a pod kubectl apply -f ./pod.json

# Apply resources from a directory containing kustomization.yaml - e.g. dir/kustomization.yaml kubectl apply -k dir/

# Apply the JSON passed into stdin to a pod cat pod.json | kubectl apply -f -

# Note: –prune is still in Alpha # Apply the configuration in manifest.yaml that matches label app=nginx and delete all other resources that are not in the file and match label app=nginx kubectl apply –prune -f manifest.yaml -l app=nginx

# Apply the configuration in manifest.yaml and delete all the other config maps that are not in the file kubectl apply –prune -f manifest.yaml –all –prune-whitelist=core/v1/ConfigMap

Available Commands: edit-last-applied Edit latest last-applied-configuration annotations of a resource/object set-last-applied Set the last-applied-configuration annotation on a live object to match the contents of a file view-last-applied View the latest last-applied-configuration annotations of a resource/object

Options: –all=false: Select all resources in the namespace of the specified resource types.

--allow-missing-template-keys=true:
If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to
golang and jsonpath output formats.

--cascade='background':
Must be "background", "orphan", or "foreground". Selects the deletion cascading strategy for the dependents
(e.g. Pods created by a ReplicationController). Defaults to background.

--dry-run='none':
Must be "none", "server", or "client". If client strategy, only print the object that would be sent, without
sending it. If server strategy, submit server-side request without persisting the resource.

--field-manager='kubectl-client-side-apply':
Name of the manager used to track field ownership.

-f, --filename=[]:
that contains the configuration to apply

--force=false:
If true, immediately remove resources from API and bypass graceful deletion. Note that immediate deletion of
some resources may result in inconsist…
Vinny avatar

yes, also with k8 options

Vinny avatar

I will figure out and let you know

1
Vinny avatar

Ha resolved

1
Vinny avatar

what a boring issue

Vinny avatar

will polish this up and send at morning I guess

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Haha, sorry for the boring issue Very excited though to polish up the fundamental UI

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Help was miserable

Vinny avatar

do you guys prefer that I leave the comments open and you close after checking the changes or should I resolve it once I send the fixes to clean up the PR?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

please leave them open

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Haha ok, do as @Andriy Knysh (Cloud Posse) asked. I contradicted this on my comments to you

1
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

that’s ok, either way is good

Vinny avatar

yep I generally keep it open, but I am looking to reduce the load in you guys and take care of more chores, but feel free to close it when you want, I will put a when I get it done

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

yes, is perfect

1

2024-11-09

Vinny avatar

good to review @Andriy Knysh (Cloud Posse) https://github.com/cloudposse/atmos/pull/770 no docs to review only design changes

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

One more very minor nit pick

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

There is too much white space between the arg and type

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Maybe put them in one column or don’t make columns equal width

Vinny avatar

also think this is better if we don’t aim to make columns equal width I can remove the padding

1
Vinny avatar

let me know and I can send the changes

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

That looks better!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Shoot that over and let’s get this merged

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Cc @Andriy Knysh (Cloud Posse)

1
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@Vinny looks like coderabbit made some useful suggestions, can you please review them

Vinny avatar

done @Andriy Knysh (Cloud Posse)

1
1

2024-11-10

Pulak Kanti Bhowmick avatar
Pulak Kanti Bhowmick

Hi Team, this PR is good to review now. Thank you https://github.com/cloudposse/atmos/pull/761

#761 change PS1 to show that atmos is in the `atmos terraform shell` mode

what

why

references

DEV-1899

Summary by CodeRabbit

Summary by CodeRabbit

New Features
• Customized command prompt for the interactive shell with the addition of the “atmos>” prefix. • Bug Fixes
• Enhanced shell behavior by removing the unnecessary -l flag for non-Windows systems and implementing a fallback to sh if bash is unavailable.
• Improved handling for the /bin/zsh shell with additional flags.

Closes #495

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We need to make this a little bit more robust. See comments.

#761 change PS1 to show that atmos is in the `atmos terraform shell` mode

what

why

references

DEV-1899

Summary by CodeRabbit

Summary by CodeRabbit

New Features
• Customized command prompt for the interactive shell with the addition of the “atmos>” prefix. • Bug Fixes
• Enhanced shell behavior by removing the unnecessary -l flag for non-Windows systems and implementing a fallback to sh if bash is unavailable.
• Improved handling for the /bin/zsh shell with additional flags.

Closes #495

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@haitham911eg looks like this is almost ready. I think there’s one outstanding comment. https://github.com/cloudposse/atmos/pull/727/files

2024-11-11

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Andriy Knysh (Cloud Posse) this is ready for final review: https://github.com/cloudposse/atmos/pull/757 cc @Michael

#757 feat: additional `atmos docs` parameters for specifying width, using auto-styling and color profile, and preserving new lines

what

atmos_docs

• Add an additional atmos docs flag for specifying the width of markdown output • Utilizing auto-styling based on light or dark mode preferences instead of hardcoding to dark • Preserving new lines with rendered markdown

why

• Enhance the user experience for interacting with documentation. The width parameter is useful for users who prefer seeing wider output for Terraform docs-generated tables and is defined in the atmos.yaml:

settings: docs: max-width: 200

future development

• The goal of this PR is to continue to iterate on the command with future improvements including text searching and pagination

references

glow docs

Summary by CodeRabbit

New Features
• Introduced a new command option to customize the maximum width for documentation output, enhancing readability.
• Added a new configuration field for documentation settings within the CLI configuration. • Bug Fixes
• Improved error handling for component path checks in documentation commands. • Documentation
• Enhanced documentation rendering with improved styling and word wrapping capabilities.

1
1
1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Andriy Knysh (Cloud Posse) this is ready for final review (works for me) https://github.com/cloudposse/atmos/pull/761

#761 change PS1 to show that atmos is in the `atmos terraform shell` mode

what

why

image

references

DEV-1899New Features
• Customized command prompt for the interactive shell with the addition of the “atmos>” prefix. • Bug Fixes
• Enhanced shell behavior by removing the unnecessary -l flag for non-Windows systems and implementing a fallback to sh if bash is unavailable.
• Improved handling for the /bin/zsh shell with additional flags.

Closes #495

1
1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Andriy Knysh (Cloud Posse) I think this is also ready for final review cc @Vinny https://github.com/cloudposse/atmos/pull/737

#737 Feat: Vendor path setting in yaml support

what

• Added new support for vendor files under folders or multiple vendor files to be processed in Lexicographic order • Refactored logic that does check the presence of vendor file

why

• Users now should be able to use new variable vendor in atmos.yaml and process different locations and amounts of vendor files.

references

#DEV-2378

CleanShot 2024-10-22 at 23 34 50@2x

New tests after general refactoring base path
CleanShot 2024-10-30 at 17 02 50

Summary by CodeRabbit

Release Notes

New Features
• Introduced a vendor section in the atmos.yaml configuration, allowing users to specify a base_path for vendor files.
• Added support for the new command-line flag --vendor-base-path to enhance configuration options.
• Enhanced documentation to clarify new configuration options and loading behavior. • Configuration Updates
• New YAML files for development, production, and staging environments with structured variable management.
• Added new vendor configuration files to demonstrate sourcing components from GitHub. • Documentation
• Updated CLI documentation to include new parameters and configuration options for better usability.
• Expanded details on the loading order of configuration files, logging behavior, and added sections for aliases and templates.

1
1

2024-11-13

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Andriy Knysh (Cloud Posse) I think this is ready for your review. @haitham911eg and me have tagged you in some comments.

https://github.com/cloudposse/atmos/pull/768

#768 package manager for vendor pull

what

New Feature interactive package manager for vendor pull
interactive shell for atmos vendor pull –everything
Support –everything flag
for test non TTY mod run atmos vendor pull –everything </dev/null |& cat > atmos_vendor_pull.log
2024-11-09 1502_32-vendor_model go - atmos WSL Ubuntu - Visual Studio Code
2024-11-09 1502_51-vendor_model go - atmos WSL Ubuntu - Visual Studio Code

why

build an interface for a package manager using bubbletea

references

https://linear.app/cloudposse/issue/DEV-270/atmos-vendor-pull-ux
https://linear.app/cloudposse/issue/DEV-2716/support-atmos-vendor-pull-everything

Summary by CodeRabbit

Summary by CodeRabbit

New Features
• Enhanced user interface for vendor commands utilizing the Bubble Tea framework.
• New --everything flag for the vendorPullCmd command, allowing users to vendor all components in one go.
• Improved functionality for managing vendor package installations, including enhanced error handling and progress feedback.
• Updated configuration files and documentation to utilize the new --everything option in vendor commands. • Bug Fixes
• Refined validation logic for command-line flags to prevent conflicts. • Refactor
• Modularized code structure for better readability and maintainability.
• Improved error handling and logging for vendor operations. • Documentation
• Updated internal documentation to reflect changes in methods and functionalities.
• Revised CLI Commands Cheat Sheet and other documentation to include the new --everything flag for vendor commands.

1
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@haitham911eg please resolve the conflicts https://github.com/cloudposse/atmos/pull/768

#768 package manager for vendor pull

what

New Feature interactive package manager for vendor pull
interactive shell for atmos vendor pull –everything
Support –everything flag
for test non TTY mod run atmos vendor pull –everything </dev/null |& cat > atmos_vendor_pull.log
2024-11-09 1502_32-vendor_model go - atmos WSL Ubuntu - Visual Studio Code
2024-11-09 1502_51-vendor_model go - atmos WSL Ubuntu - Visual Studio Code

why

build an interface for a package manager using bubbletea

references

https://linear.app/cloudposse/issue/DEV-270/atmos-vendor-pull-ux
https://linear.app/cloudposse/issue/DEV-2716/support-atmos-vendor-pull-everything

Summary by CodeRabbit

Summary by CodeRabbit

New Features
• Enhanced user interface for vendor commands utilizing the Bubble Tea framework.
• New --everything flag for the vendorPullCmd command, allowing users to vendor all components in one go.
• Improved functionality for managing vendor package installations, including enhanced error handling and progress feedback.
• Updated configuration files and documentation to utilize the new --everything option in vendor commands. • Bug Fixes
• Refined validation logic for command-line flags to prevent conflicts. • Refactor
• Modularized code structure for better readability and maintainability.
• Improved error handling and logging for vendor operations. • Documentation
• Updated internal documentation to reflect changes in methods and functionalities.
• Revised CLI Commands Cheat Sheet and other documentation to include the new --everything flag for vendor commands.

haitham911eg avatar
haitham911eg

done

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
1
Vinny avatar

@Andriy Knysh (Cloud Posse) all is resolved here ready to merge https://github.com/cloudposse/atmos/pull/737

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@Vinny please review this issue https://github.com/cloudposse/atmos/issues/780

#780 Running `atmos vendor pull` using component.yaml failing

Describe the Bug

After updating from v1.105.0 to v1.106.0, using a component.yaml file in the component directories no longer works as it returns an error if the vendor file or directory does not exist. Appears to be related to this change: https://github.com/cloudposse/atmos/pull/737/files#diff-35bbd3669178afb0acaaebfb4ec924ec99189338a9bc05232b4f92a53f261518L83

Expected Behavior

Running atmos vendor pull for a component containing a component.yaml file should result in the component being successfully vendored with specified mixins added.

Component config file

image

Expected command output

image

Steps to Reproduce

Running the same command using v1.106.0 results in an error due to the vendor file and/or directory not existing.

image

Screenshots

No response

Environment

• OS: WLS2/Debian12 using Geodesic 2.8.7 & Atmos 1.106.0

Additional Context

No response

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

it broke after we merged the PR

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

note that Atmos supports vendor.yaml and component.yaml (in the component directory)

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

looks like the component.yaml functionality is broken

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
Vendoring | atmos

Use Atmos vendoring to make copies of 3rd-party components, stacks, and other artifacts in your own repo.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@Vinny did it address the issue described in https://github.com/cloudposse/atmos/issues/780 ?

both vendor.yaml and component.yaml (from the component’s folder) must work. Note that 80% of the Atmos users still use component.yaml to vendor components, we need to maintain this functionality and fix any issues

#780 Running `atmos vendor pull` using component.yaml failing

Describe the Bug

After updating from v1.105.0 to v1.106.0, using a component.yaml file in the component directories no longer works as it returns an error if the vendor file or directory does not exist. Appears to be related to this change: https://github.com/cloudposse/atmos/pull/737/files#diff-35bbd3669178afb0acaaebfb4ec924ec99189338a9bc05232b4f92a53f261518L83

Expected Behavior

Running atmos vendor pull for a component containing a component.yaml file should result in the component being successfully vendored with specified mixins added.

Component config file

image

Expected command output

image

Steps to Reproduce

Running the same command using v1.106.0 results in an error due to the vendor file and/or directory not existing.

image

Screenshots

No response

Environment

• OS: WLS2/Debian12 using Geodesic 2.8.7 & Atmos 1.106.0

Additional Context

No response

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

we have the test for component.yaml in

pkg/vender/component_vendor_test.go
examples/tests/components/terraform/infra/vpc-flow-logs-bucket/component.yaml
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

the logic of the atmos vendor pull command was as the following (and we need to maintain it):

• Check if vendor.yaml file exists (after your changes, check it in multiple places defined in atmos.yaml • If vendor.yaml exists and a component is not passed in atmos vendor pull, use vendor.yaml to pull all components • If vendor.yaml does not exist and a component is not passed in, show an error • If component is passed in, and vendor.yaml exists, use it to vendor the component • Otherwise, search for component.yaml in the component’s dir • If component.yaml exists, vendor the component

• Otherwise, show an error

Vinny avatar

ok this has to be reworked then

Vinny avatar

this exactly logic has been applied now

Vinny avatar
#782 Fix vendor pull directory creation issue

what

Added directory creation logic before copying vendored files
Fixed the “no such file or directory” error when running atmos vendor pull
Ensures target directories are created before attempting to vendor components

why

Previously, the vendor command would fail if target directories didn’t exist
Directory creation is a prerequisite for vendoring components

references

#780

Logic design

Summary by CodeRabbit

Bug Fixes
• Improved error handling for vendor configuration file checks, preventing unnecessary error messages when the file does not exist.
• Streamlined the logic for processing vendor configuration, enhancing clarity and robustness.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

thanks @Vinny, will review

Vinny avatar

good to review @Andriy Knysh (Cloud Posse) https://github.com/cloudposse/atmos/pull/764

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@Vinny please address the comments

2024-11-14

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Comment on #782 Fix vendor pull directory creation issue

@Cerebrovinny I do not think this is the fix based on the description of the problem. Bear in mind we have two types of vendor manifests.

https://atmos.tools/core-concepts/vendor/

The bug report concerns the component manifest file. That means that any logic relating to the vendor manifests file should not inhibit the component manifest from functioning.

To me, it sounds like the regression is that when vendoring a component manifest, it is erroring if the vendor manifests file path does not exist. However, that should not be required. We should instead log to warning that the path does not exist and continue processing.

We should not create the vendor configuration path if it does not exist.

Vinny avatar

ok I got the point

Comment on #782 Fix vendor pull directory creation issue

@Cerebrovinny I do not think this is the fix based on the description of the problem. Bear in mind we have two types of vendor manifests.

https://atmos.tools/core-concepts/vendor/

The bug report concerns the component manifest file. That means that any logic relating to the vendor manifests file should not inhibit the component manifest from functioning.

To me, it sounds like the regression is that when vendoring a component manifest, it is erroring if the vendor manifests file path does not exist. However, that should not be required. We should instead log to warning that the path does not exist and continue processing.

We should not create the vendor configuration path if it does not exist.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
#706 Wrong documentation for `base_path` in "Learn Atmos" section

Describe the Bug

While trying to understand why I have to set both ATMOS_BASE_PATH and ATMOS_CLI_CONFIG_PATH and if one of them used the other as a default I stumbled over this discrepancy in the “Learn Atmos” section:

The explanation of base_path at https://atmos.tools/core-concepts/projects/configuration/#path-configuration says

We set it to an empty string because we’ve decided to use the ENV var ATMOS_BASE_PATH to point to the absolute path of the root of the repo

But the example further up at https://atmos.tools/core-concepts/projects/configuration/#atmos-cli-configuration-schema actually sets it to the value ./ (which is clearly not an empty string).

And if I interpret https://atmos.tools/cli/configuration/#base-path correctly an empty string would indeed have a different behaviour than that . value.

Expected Behavior

I read the documentation and shouldn’t have been confused.

Steps to Reproduce

Read the documentation :-)

Screenshots

No response

Environment

No response

Additional Context

No response

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

yes, looks good to me

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

in the atmos.yaml in the root of the repo, we pointed it to (before the changes)

base_path: "./examples/quick-start-advanced"
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

we were able to run Atmos commands from the root of the repo

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

This is not about documenting the behavior in cloudposse/atmos/atmos.yaml

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

now we are pointing it to / - maybe we should update it to

base_path: "./examples/quick-start-simple"
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

This is about documenting how Atmos uses base_path

1

2024-11-15

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Andriy Knysh (Cloud Posse) can you take a look at this https://github.com/cloudposse/atmos/issues/781#issuecomment-2478506957

Comment on #781 validate command prints usage when used like documented?

@aknysh not able to found the regression checking this https://raw.githubusercontent.com/cloudposse/atmos/b53d696375a6716b2fd0bc024fb1861b46c78bb5/internal/exec/validate_stacks.go

maybe you can point out the root cause for this? also silence usage from cobra can be a viable option on this case

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Vinny make sure when referencing issues to say “Closes #123” so that when the PR is merged, the issue will get closed automatically

1
Vinny avatar

good point, I have few questions about that, should I create the branchs with the linear ticket number is fine? should I mention linear numbers in description?

Vinny avatar

If I’m working on something that exists in Linear but not in GitHub issues, should I open a ticket in GitHub so the community is aware?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

It’s okay to use the linear issue in the branch names. I would avoid it in the PR description.

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Since this is all open source, those are not available to most to view.

Vinny avatar

yeah I stop using it since I saw you removed one there

Michael avatar
Michael

Took a stab at prettying up the JSON indentation for generated files such as backend.tf.json (https://github.com/cloudposse/atmos/pull/783). Feel free to rework or disregard if this doesn’t meet the need! I tested it across five or so of our stacks

#783 feat: enhance `WriteToFileAsJSON` with pretty-printing support

what

• Used the ConvertToJSON utility with json.MarshalIndent to produce formatted JSON • Indentation is set to two spaces (“ “) for consistent readability

why

• This PR improves the WriteToFileAsJSON function by introducing pretty-printing for JSON outputs. Previously, the function serialized JSON using a compact format, which could make the resulting files harder to read. With this change, all JSON written by this function will now be formatted with indentation, making it easier for developers and users to inspect and debug the generated files • This specifically addresses #778 , which previously rendered auto-generated backends as:

{ “terraform”: { “backend”: { “s3”: { “acl”: “bucket-owner-full-control”, “bucket”: “my-tfstate-bucket”, “dynamodb_table”: “some-dynamo-table”, “encrypt”: true, “key”: “terraform.tfstate”, “profile”: “main”, “region”: “us-west-2”, “workspace_key_prefix”: “something” } } } }

With this addition, the output appears as:

{ “terraform”: { “backend”: { “s3”: { “acl”: “bucket-owner-full-control”, “bucket”: “my-tfstate-bucket”, “dynamodb_table”: “some-dynamo-table”, “encrypt”: true, “key”: “terraform.tfstate”, “profile”: “main”, “region”: “us-west-2”, “workspace_key_prefix”: “something” } } } }

references

Stack Overflow • Closes #778

Summary by CodeRabbit

Summary by CodeRabbit

New Features
• Enhanced JSON output formatting with indented structure for better readability. • Refactor
• Simplified variable declarations for consistency in JSON handling functions.

1
1
1
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

approved and merged, thank you @Michael

#783 feat: enhance `WriteToFileAsJSON` with pretty-printing support

what

• Used the ConvertToJSON utility with json.MarshalIndent to produce formatted JSON • Indentation is set to two spaces (“ “) for consistent readability

why

• This PR improves the WriteToFileAsJSON function by introducing pretty-printing for JSON outputs. Previously, the function serialized JSON using a compact format, which could make the resulting files harder to read. With this change, all JSON written by this function will now be formatted with indentation, making it easier for developers and users to inspect and debug the generated files • This specifically addresses #778 , which previously rendered auto-generated backends as:

{ “terraform”: { “backend”: { “s3”: { “acl”: “bucket-owner-full-control”, “bucket”: “my-tfstate-bucket”, “dynamodb_table”: “some-dynamo-table”, “encrypt”: true, “key”: “terraform.tfstate”, “profile”: “main”, “region”: “us-west-2”, “workspace_key_prefix”: “something” } } } }

With this addition, the output appears as:

{ “terraform”: { “backend”: { “s3”: { “acl”: “bucket-owner-full-control”, “bucket”: “my-tfstate-bucket”, “dynamodb_table”: “some-dynamo-table”, “encrypt”: true, “key”: “terraform.tfstate”, “profile”: “main”, “region”: “us-west-2”, “workspace_key_prefix”: “something” } } } }

references

Stack Overflow • Closes #778

Summary by CodeRabbit

Summary by CodeRabbit

New Features
• Enhanced JSON output formatting with indented structure for better readability. • Refactor
• Simplified variable declarations for consistency in JSON handling functions.

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Thanks @Michael!

2024-11-16

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Andriy Knysh (Cloud Posse) this should fix the regression raised by @Michael by disabling the custom prompt by default and enhancing it so it can be customized via the atmos config.

https://github.com/cloudposse/atmos/pull/786

#786 add support for disabling custom terraform shell prompt

what

why

Working demo

With custom prompt:

Screenshot 2024-11-16 at 11 20 14 PM

Without custom prompt:

image

Closes #779 (disable new prompt by default until we make it work better with geodesic)

references

Summary by CodeRabbit

Summary by CodeRabbit

New Features
• Introduced a configurable shell prompt for Terraform commands, allowing users to customize their command-line interface.
• Added a new ShellConfig type to support the new shell prompt configuration. • Bug Fixes
• Improved flexibility in shell prompt settings by allowing dynamic configuration based on user-defined CLI settings.

1
1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I haven’t had a chance to test it yet

#786 add support for disabling custom terraform shell prompt

what

why

Working demo

With custom prompt:

Screenshot 2024-11-16 at 11 20 14 PM

Without custom prompt:

image

Closes #779 (disable new prompt by default until we make it work better with geodesic)

references

Summary by CodeRabbit

Summary by CodeRabbit

New Features
• Introduced a configurable shell prompt for Terraform commands, allowing users to customize their command-line interface.
• Added a new ShellConfig type to support the new shell prompt configuration. • Bug Fixes
• Improved flexibility in shell prompt settings by allowing dynamic configuration based on user-defined CLI settings.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Cc @Pulak Kanti Bhowmick

Vinny avatar

Just one question here: https://linear.app/cloudposse/issue/DEV-2713/automatic-templated-file-search-for-atmos-imports. It looks like the .tmpl files take precedence over the .yaml, .yml extensions is that right?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

all imports are processed in the order they are defined regardless of the file extensions

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

the order of imports is important because of the inheritance and deep merging

2024-11-18

2024-11-19

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I’ve started documenting some dev conventions for atmos that I feel we struggle with. Note that these are the intents, but they are not necessarily how Atmos works today. We need everyone’s help to conform to these conventions and update atmos as we encounter inconsistencies with these conventions.

https://github.com/cloudposse/atmos/pull/793

#793 Document Atmos Development Conventions

what

• Explain when to display help vs usage, and their respective behaviors • Explain when to use various log levels

why

• Improve code consistency as the dev team expands

2
Vinny avatar

I think this is very needed, as I am confuse myself sometimes

#793 Document Atmos Development Conventions

what

• Explain when to display help vs usage, and their respective behaviors • Explain when to use various log levels

why

• Improve code consistency as the dev team expands

Vinny avatar

this is good to final review @Andriy Knysh (Cloud Posse) https://github.com/cloudposse/atmos/pull/764

1
Vinny avatar

this is good to start review @Andriy Knysh (Cloud Posse) https://github.com/cloudposse/atmos/pull/795

1

2024-11-20

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Michael I am extremely excited for the list commands you implemented! https://github.com/cloudposse/atmos/pull/797

#797 feat: create `atmos list` command for listing stacks and components

what

• Creates an atmos list command for listing stacks and components

atmos_list

• Incorporates custom list commands into Atmos • Updates documentation and website • Removes atmos.yaml references to custom list command

why

• While the custom Atmos commands for listing stacks and components are great, incorporating the command into Atmos is far more efficient and parallelized, achieving similar or better results in 0.741 seconds compared to 8.131 seconds for the custom command

references

Atmos Custom Command Docs

Summary by CodeRabbit

Release Notes

New Features
• Introduced a list command with subcommands for listing Atmos stacks and components.
• Enhanced command functionality with options to filter by stack and component type. • Documentation
• Added comprehensive documentation for the new list, list stacks, and list components commands, including usage examples and flags.
• Updated existing documentation to clarify command usage and improve user guidance. • Bug Fixes
• Improved error handling and output formatting for commands related to listing stacks and components.

1
Michael avatar
Michael

Thank you! I am going to move some of the logic into pkg/utils if that’s okay to simplify the command logic!

#797 feat: create `atmos list` command for listing stacks and components

what

• Creates an atmos list command for listing stacks and components

atmos_list

• Incorporates custom list commands into Atmos • Updates documentation and website • Removes atmos.yaml references to custom list command

why

• While the custom Atmos commands for listing stacks and components are great, incorporating the command into Atmos is far more efficient and parallelized, achieving similar or better results in 0.741 seconds compared to 8.131 seconds for the custom command

references

Atmos Custom Command Docs

Summary by CodeRabbit

Release Notes

New Features
• Introduced a list command with subcommands for listing Atmos stacks and components.
• Enhanced command functionality with options to filter by stack and component type. • Documentation
• Added comprehensive documentation for the new list, list stacks, and list components commands, including usage examples and flags.
• Updated existing documentation to clarify command usage and improve user guidance. • Bug Fixes
• Improved error handling and output formatting for commands related to listing stacks and components.

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I’m always sensitive what we can ask for from outside contributors, but since I was just working on defining the requirements for a separate atmos list vendor command, thought I would share what I was thinking

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

So for the list vendor command, wanted to do something like this in atmos.yaml

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think the list stacks and list components could benefit from the same configurability

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

And then using lipgloss for display

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Just throwing this out there in the event I successfully nerd snipe you into implementing it! But if not, this is still a step up.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

E.g.

stacks:
  list:
    columns:
      - name: Stack
        value: '{{ .Stack }}'
      - name: Tenant
        value: '{{ .vars.tenant }}' 
      - name: Environment
        value: '{{ .vars.environment }}'
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Stack           Tenant        Environment
plat-ue2-dev    plat          ue2
Michael avatar
Michael

That’s fancy! Would it be useful to include an option for plain or pretty printing? I was also thinking this function could be leveraged for autocompletion in the future, such as with atmos describe component <tab>, where autocompletion is based on the output of listing stacks and components

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Yes, I think if there’s no TTY, it would degrade to plain output, so standard pipelining works well (e.g. atmos list stacksgrep foo)
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Re: tab completion, yes, we have to improve that!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

It should work like you describe

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

for components and –stack

Michael avatar
Michael

Sounds good, I will periodically work on those additions today

1
    keyboard_arrow_up