#atmos-dev (2024-11)
Discuss atmos core development (golang)
2024-11-04
2024-11-05
Hi team, this PR is good to review. Thank you!
what
why
references
• DEV-2150
Summary by CodeRabbit
• New Features
• Enhanced component processing logic to include checks for whether components are enabled, improving metadata handling.
• Introduced a new field to track the enabled state of components, optimizing performance and resource management.
• Updated processing methods to skip disabled components, refining the control flow and enhancing efficiency.
• Bug Fixes
• Adjusted control flow in component processing to ensure only enabled components are processed, reducing potential errors.
@Andriy Knysh (Cloud Posse) LGTM, ready for your review
what
why
references
• DEV-2150
Summary by CodeRabbit
• New Features
• Enhanced component processing logic to include checks for whether components are enabled, improving metadata handling.
• Introduced a new field to track the enabled state of components, optimizing performance and resource management.
• Updated processing methods to skip disabled components, refining the control flow and enhancing efficiency.
• Bug Fixes
• Adjusted control flow in component processing to ensure only enabled components are processed, reducing potential errors.
2024-11-06
if you addressed PR comments, please ping me here for re-review (i don’t get notifications from GitHub when you mark a PR as ready for review)
Yes, I have addressed your comments. Please review again https://github.com/cloudposse/atmos/pull/756
what
why
working demo
references
• DEV-2150
Summary by CodeRabbit
Summary by CodeRabbit
• New Features
• Enhanced component processing logic to include checks for whether components are enabled, improving metadata handling.
• Introduced a new field to track the enabled state of components, optimizing performance and resource management.
• Updated processing methods to skip disabled components, refining the control flow and enhancing efficiency.
• Added documentation on using the metadata.enabled
parameter to conditionally include components in deployment.
• Bug Fixes
• Adjusted control flow in component processing to ensure only enabled components are processed, reducing potential errors.
@Pulak Kanti Bhowmick please address the comments in this PR
Yes, apology for my late reply. Checking now
2024-11-07
Hi Team, this PR is good to review: https://github.com/cloudposse/atmos/pull/766
Thank you so much!
thanks @Pulak Kanti Bhowmick, approved and merged
@Andriy Knysh (Cloud Posse) this is good for review https://github.com/cloudposse/atmos/pull/767
2024-11-08
@Andriy Knysh (Cloud Posse) i think this is ready for final review https://github.com/cloudposse/atmos/pull/756
what
why
working demo
references
• DEV-2150
Summary by CodeRabbit
Summary by CodeRabbit
• New Features
• Enhanced component processing logic to include checks for whether components are enabled, improving metadata handling.
• Introduced a new field to track the enabled state of components, optimizing performance and resource management.
• Updated processing methods to skip disabled components, refining the control flow and enhancing efficiency.
• Added documentation on using the metadata.enabled
parameter to conditionally include components in deployment.
• Bug Fixes
• Adjusted control flow in component processing to ensure only enabled components are processed, reducing potential errors.
@Vinny please address outstanding comments in PRs so we can get them merged
what
• Added new support for vendor files under folders or multiple vendor files to be processed in Lexicographic order • Refactored logic that does check the presence of vendor file
why
• Users now should be able to use new variable vendor in atmos.yaml and process different locations and amounts of vendor files.
references
#DEV-2378
CleanShot 2024-10-22 at 23 34 50@2x
New tests after general refactoring base path
CleanShot 2024-10-30 at 17 02 50
Summary by CodeRabbit
Release Notes
• New Features
• Introduced a vendor
section in the atmos.yaml
configuration, allowing users to specify a base_path
for vendor files.
• Added support for the new command-line flag --vendor-base-path
to enhance configuration options.
• Enhanced documentation to clarify new configuration options and loading behavior.
• Configuration Updates
• New YAML files for development, production, and staging environments with structured variable management.
• Added new vendor configuration files to demonstrate sourcing components from GitHub.
• Documentation
• Updated CLI documentation to include new parameters and configuration options for better usability.
• Expanded details on the loading order of configuration files, logging behavior, and added sections for aliases and templates.
ok will get this done today also
Are you stuck on https://github.com/cloudposse/atmos/pull/764? The tests are still failing
what
This PR changes the default behavior of atmos describe stacks
it filter empty stacks by default unless user pass the flag --include-empty-stacks
why
This was causing stacks with empty results or no components/imports components to be displayed.
Test Results
Screenshot 2024-11-05 at 11 36 39
Screenshot 2024-11-05 at 11 36 53
references
DEV-1880
Summary by CodeRabbit
• New Features
• Introduced a new command-line flag --include-empty-stacks
to include stacks without components in the output.
• Enhanced stack processing logic to support filtering based on the new flag.
• Bug Fixes
• Improved error handling and messaging for Git reference issues.
• Tests
• Added new test cases to validate the behavior of the ExecuteDescribeStacks
function with empty stacks.
good to go https://github.com/cloudposse/atmos/pull/767 added a test image also cc: @Andriy Knysh (Cloud Posse)
What
• Added improved workflow failure handling with clear error messages
• Added --from-step
flag to resume workflows from specific steps
• Added documentation and help text for workflow failure handling
• Added example workflow demonstrating failure scenarios
Why
• Makes it easier to debug and fix workflow failures by showing: • Which step failed • The exact command that failed • How to resume from the failed step • Saves time by allowing users to resume workflows from failed steps instead of restarting from beginning • Improves user experience with clear guidance on how to recover from failures
Example workflow demonstrating the feature:
workflows:
test-1: description: “Test workflow with intentionally failing steps” steps: - name: “step-1” type: shell command: “echo ‘This step will succeed’ && exit 0”
- name: "step-2"
type: shell
command: "echo 'This step will fail' && exit 1"
- name: "step-3"
type: shell
command: "echo 'This step should not execute'"
Tests
Last test after requested changes @osterman @aknysh
When step-2 fails, users will see:
CleanShot 2024-11-08 at 20 00 10
references
DEV-2717
Summary by CodeRabbit
• New Features
• Added detailed help information for workflow commands, including failure handling and resume functionality.
• Enhanced error messages during workflow execution to provide clearer guidance on failures and resuming workflows.
• Documentation
• Updated documentation to include a new section on “Workflow Failure Handling and Resume.”
• Clarified the “Stack Precedence” section to better explain stack definitions within workflows.
Left some more feedback
Also, the test image has the wrong text “Failed command:” instead of “Command failed:”
this is a pain to fix found another more samples but the Flags: part is still not good https://linear.app/cloudposse/issue/DEV-335/atmos-needs-to-wrap-long-lines-in-help
Need more specific details to be able to advise
Did you explore how they fixed it here with cobra? https://github.com/sigstore/cosign/issues/2887
Description
The output of cosign verify --help
is unreadable due to line wrapping.
We solved this in kubectl by creating a custom tab printer.
And kubectl which also uses cobra https://github.com/kubernetes/kubernetes/pull/104736
What type of PR is this?
/kind design
What this PR does / why we need it:
Make the help flags of kubectl more readable / user friendly
Which issue(s) this PR fixes:
Fixes kubernetes/kubectl#1069
Special notes for your reviewer:
Before,
Apply a configuration to a resource by file name or stdin. The resource name must be specified. This
resource will be created if it doesn't exist yet. To use 'apply', always create the resource
initially with either 'apply' or 'create --save-config'.
JSON and YAML formats are accepted.
Alpha Disclaimer: the --prune functionality is not yet complete. Do not use unless you are aware of
what the current state is. See <https://issues.k8s.io/34274>.
Examples:
# Apply the configuration in pod.json to a pod
kubectl apply -f ./pod.json
# Apply resources from a directory containing kustomization.yaml - e.g. dir/kustomization.yaml
kubectl apply -k dir/
# Apply the JSON passed into stdin to a pod
cat pod.json | kubectl apply -f -
# Note: --prune is still in Alpha
# Apply the configuration in manifest.yaml that matches label app=nginx and delete all other
resources that are not in the file and match label app=nginx
kubectl apply --prune -f manifest.yaml -l app=nginx
# Apply the configuration in manifest.yaml and delete all the other config maps that are not in
the file
kubectl apply --prune -f manifest.yaml --all --prune-whitelist=core/v1/ConfigMap
Available Commands:
edit-last-applied Edit latest last-applied-configuration annotations of a resource/object
set-last-applied Set the last-applied-configuration annotation on a live object to match the
contents of a file
view-last-applied View the latest last-applied-configuration annotations of a resource/object
Options:
--all=false: Select all resources in the namespace of the specified resource types.
--allow-missing-template-keys=true: If true, ignore any errors in templates when a field or
map key is missing in the template. Only applies to golang and jsonpath output formats.
--cascade='background': Must be "background", "orphan", or "foreground". Selects the deletion
cascading strategy for the dependents (e.g. Pods created by a ReplicationController). Defaults to
background.
--dry-run='none': Must be "none", "server", or "client". If client strategy, only print the
object that would be sent, without sending it. If server strategy, submit server-side request
without persisting the resource.
--field-manager='kubectl-client-side-apply': Name of the manager used to track field
ownership.
-f, --filename=[]: that contains the configuration to apply
--force=false: If true, immediately remove resources from API and bypass graceful deletion.
Note that immediate deletion of some resources may result in inconsistency or data loss and requires
confirmation.
--force-conflicts=false: If true, server-side apply will force the changes against conflicts.
--grace-period=-1: Period of time in seconds given to the resource to terminate gracefully.
Ignored if negative. Set to 1 for immediate shutdown. Can only be set to 0 when --force is true
(force deletion).
-k, --kustomize='': Process a kustomization directory. This flag can't be used together with -f or
-R.
--openapi-patch=true: If true, use openapi to calculate diff when the openapi presents and the
resource can be found in the openapi spec. Otherwise, fall back to use baked-in types.
-o, --output='': Output format. One of:
json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-as-json|jsonpath-file.
--overwrite=true: Automatically resolve conflicts between the modified and live configuration
by using values from the modified configuration
--prune=false: Automatically delete resource objects, including the uninitialized ones, that
do not appear in the configs and are created by either apply or create --save-config. Should be used
with either -l or --all.
--prune-whitelist=[]: Overwrite the default whitelist with <group/version/kind> for --prune
-R, --recursive=false: Process the directory used in -f, --filename recursively. Useful when you
want to manage related manifests organized within the same directory.
-l, --selector='': Selector (label query) to filter on, supports '=', '==', and '!='.(e.g. -l
key1=value1,key2=value2)
--server-side=false: If true, apply runs in the server instead of the client.
--show-managed-fields=false: If true, keep the managedFields when printing objects in JSON or
YAML format.
--template='': Template string or path to template file to use when -o=go-template,
-o=go-template-file. The template format is golang templates
[<http://golang.org/pkg/text/template/#pkg-overview>].
--timeout=0s: The length of time to wait before giving up on a delete, zero means determine a
timeout from the size of the object
--validate=true: If true, use a schema to validate the input before sending it
--wait=false: If true, wait for resources to be gone before returning. This waits for
finalizers.
Usage:
kubectl apply (-f FILENAME | -k DIRECTORY) [options]
Use "kubectl <command> --help" for more information about a given command.
Use "kubectl options" for a list of global command-line options (applies to all commands).
After,
``` Apply a configuration to a resource by file name or stdin. The resource name must be specified. This resource will be created if it doesn’t exist yet. To use ‘apply’, always create the resource initially with either ‘apply’ or ‘create –save-config’.
JSON and YAML formats are accepted.
Alpha Disclaimer: the –prune functionality is not yet complete. Do not use unless you are aware of what the current state is. See https://issues.k8s.io/34274.
Examples: # Apply the configuration in pod.json to a pod kubectl apply -f ./pod.json
# Apply resources from a directory containing kustomization.yaml - e.g. dir/kustomization.yaml kubectl apply -k dir/
# Apply the JSON passed into stdin to a pod cat pod.json | kubectl apply -f -
# Note: –prune is still in Alpha # Apply the configuration in manifest.yaml that matches label app=nginx and delete all other resources that are not in the file and match label app=nginx kubectl apply –prune -f manifest.yaml -l app=nginx
# Apply the configuration in manifest.yaml and delete all the other config maps that are not in the file kubectl apply –prune -f manifest.yaml –all –prune-whitelist=core/v1/ConfigMap
Available Commands: edit-last-applied Edit latest last-applied-configuration annotations of a resource/object set-last-applied Set the last-applied-configuration annotation on a live object to match the contents of a file view-last-applied View the latest last-applied-configuration annotations of a resource/object
Options: –all=false: Select all resources in the namespace of the specified resource types.
--allow-missing-template-keys=true:
If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to
golang and jsonpath output formats.
--cascade='background':
Must be "background", "orphan", or "foreground". Selects the deletion cascading strategy for the dependents
(e.g. Pods created by a ReplicationController). Defaults to background.
--dry-run='none':
Must be "none", "server", or "client". If client strategy, only print the object that would be sent, without
sending it. If server strategy, submit server-side request without persisting the resource.
--field-manager='kubectl-client-side-apply':
Name of the manager used to track field ownership.
-f, --filename=[]:
that contains the configuration to apply
--force=false:
If true, immediately remove resources from API and bypass graceful deletion. Note that immediate deletion of
some resources may result in inconsist…
yes, also with k8 options
what a boring issue
will polish this up and send at morning I guess
Haha, sorry for the boring issue Very excited though to polish up the fundamental UI
Help was miserable
do you guys prefer that I leave the comments open and you close after checking the changes or should I resolve it once I send the fixes to clean up the PR?
Haha ok, do as @Andriy Knysh (Cloud Posse) asked. I contradicted this on my comments to you
that’s ok, either way is good
yep I generally keep it open, but I am looking to reduce the load in you guys and take care of more chores, but feel free to close it when you want, I will put a when I get it done
2024-11-09
good to review @Andriy Knysh (Cloud Posse) https://github.com/cloudposse/atmos/pull/770 no docs to review only design changes
One more very minor nit pick
There is too much white space between the arg and type
Maybe put them in one column or don’t make columns equal width
also think this is better if we don’t aim to make columns equal width I can remove the padding
let me know and I can send the changes
That looks better!
Shoot that over and let’s get this merged
@Vinny looks like coderabbit made some useful suggestions, can you please review them
2024-11-10
good again for review https://github.com/cloudposse/atmos/pull/767
Hi Team, this PR is good to review now. Thank you https://github.com/cloudposse/atmos/pull/761
what
why
references
• DEV-1899
Summary by CodeRabbit
Summary by CodeRabbit
• New Features
• Customized command prompt for the interactive shell with the addition of the “atmos>” prefix.
• Bug Fixes
• Enhanced shell behavior by removing the unnecessary -l
flag for non-Windows systems and implementing a fallback to sh
if bash
is unavailable.
• Improved handling for the /bin/zsh
shell with additional flags.
Closes #495
We need to make this a little bit more robust. See comments.
what
why
references
• DEV-1899
Summary by CodeRabbit
Summary by CodeRabbit
• New Features
• Customized command prompt for the interactive shell with the addition of the “atmos>” prefix.
• Bug Fixes
• Enhanced shell behavior by removing the unnecessary -l
flag for non-Windows systems and implementing a fallback to sh
if bash
is unavailable.
• Improved handling for the /bin/zsh
shell with additional flags.
Closes #495
@haitham911eg looks like this is almost ready. I think there’s one outstanding comment. https://github.com/cloudposse/atmos/pull/727/files
2024-11-11
@Andriy Knysh (Cloud Posse) this is ready for final review: https://github.com/cloudposse/atmos/pull/757 cc @Michael
what
• Add an additional atmos docs
flag for specifying the width of markdown output
• Utilizing auto-styling based on light or dark mode preferences instead of hardcoding to dark
• Preserving new lines with rendered markdown
why
• Enhance the user experience for interacting with documentation. The width
parameter is useful for users who prefer seeing wider output for Terraform docs-generated tables and is defined in the atmos.yaml
:
settings: docs: max-width: 200
future development
• The goal of this PR is to continue to iterate on the command with future improvements including text searching and pagination
references
Summary by CodeRabbit
• New Features
• Introduced a new command option to customize the maximum width for documentation output, enhancing readability.
• Added a new configuration field for documentation settings within the CLI configuration.
• Bug Fixes
• Improved error handling for component path checks in documentation commands.
• Documentation
• Enhanced documentation rendering with improved styling and word wrapping capabilities.
@Andriy Knysh (Cloud Posse) this is ready for final review (works for me) https://github.com/cloudposse/atmos/pull/761
what
why
references
• DEV-1899
• New Features
• Customized command prompt for the interactive shell with the addition of the “atmos>” prefix.
• Bug Fixes
• Enhanced shell behavior by removing the unnecessary -l
flag for non-Windows systems and implementing a fallback to sh
if bash
is unavailable.
• Improved handling for the /bin/zsh
shell with additional flags.
Closes #495
@Andriy Knysh (Cloud Posse) I think this is also ready for final review cc @Vinny https://github.com/cloudposse/atmos/pull/737
what
• Added new support for vendor files under folders or multiple vendor files to be processed in Lexicographic order • Refactored logic that does check the presence of vendor file
why
• Users now should be able to use new variable vendor in atmos.yaml and process different locations and amounts of vendor files.
references
#DEV-2378
CleanShot 2024-10-22 at 23 34 50@2x
New tests after general refactoring base path
CleanShot 2024-10-30 at 17 02 50
Summary by CodeRabbit
Release Notes
• New Features
• Introduced a vendor
section in the atmos.yaml
configuration, allowing users to specify a base_path
for vendor files.
• Added support for the new command-line flag --vendor-base-path
to enhance configuration options.
• Enhanced documentation to clarify new configuration options and loading behavior.
• Configuration Updates
• New YAML files for development, production, and staging environments with structured variable management.
• Added new vendor configuration files to demonstrate sourcing components from GitHub.
• Documentation
• Updated CLI documentation to include new parameters and configuration options for better usability.
• Expanded details on the loading order of configuration files, logging behavior, and added sections for aliases and templates.
2024-11-13
@Andriy Knysh (Cloud Posse) I think this is ready for your review. @haitham911eg and me have tagged you in some comments.
what
• New Feature interactive package manager for vendor pull
interactive shell for atmos vendor pull –everything
Support –everything flag
for test non TTY mod run atmos vendor pull –everything </dev/null |& cat > atmos_vendor_pull.log
2024-11-09 1502_32-vendor_model go - atmos WSL Ubuntu - Visual Studio Code
2024-11-09 1502_51-vendor_model go - atmos WSL Ubuntu - Visual Studio Code
why
build an interface for a package manager using bubbletea
references
https://linear.app/cloudposse/issue/DEV-270/atmos-vendor-pull-ux
https://linear.app/cloudposse/issue/DEV-2716/support-atmos-vendor-pull-everything
Summary by CodeRabbit
Summary by CodeRabbit
• New Features
• Enhanced user interface for vendor commands utilizing the Bubble Tea framework.
• New --everything
flag for the vendorPullCmd
command, allowing users to vendor all components in one go.
• Improved functionality for managing vendor package installations, including enhanced error handling and progress feedback.
• Updated configuration files and documentation to utilize the new --everything
option in vendor commands.
• Bug Fixes
• Refined validation logic for command-line flags to prevent conflicts.
• Refactor
• Modularized code structure for better readability and maintainability.
• Improved error handling and logging for vendor operations.
• Documentation
• Updated internal documentation to reflect changes in methods and functionalities.
• Revised CLI Commands Cheat Sheet and other documentation to include the new --everything
flag for vendor commands.
@haitham911eg please resolve the conflicts https://github.com/cloudposse/atmos/pull/768
what
• New Feature interactive package manager for vendor pull
interactive shell for atmos vendor pull –everything
Support –everything flag
for test non TTY mod run atmos vendor pull –everything </dev/null |& cat > atmos_vendor_pull.log
2024-11-09 1502_32-vendor_model go - atmos WSL Ubuntu - Visual Studio Code
2024-11-09 1502_51-vendor_model go - atmos WSL Ubuntu - Visual Studio Code
why
build an interface for a package manager using bubbletea
references
https://linear.app/cloudposse/issue/DEV-270/atmos-vendor-pull-ux
https://linear.app/cloudposse/issue/DEV-2716/support-atmos-vendor-pull-everything
Summary by CodeRabbit
Summary by CodeRabbit
• New Features
• Enhanced user interface for vendor commands utilizing the Bubble Tea framework.
• New --everything
flag for the vendorPullCmd
command, allowing users to vendor all components in one go.
• Improved functionality for managing vendor package installations, including enhanced error handling and progress feedback.
• Updated configuration files and documentation to utilize the new --everything
option in vendor commands.
• Bug Fixes
• Refined validation logic for command-line flags to prevent conflicts.
• Refactor
• Modularized code structure for better readability and maintainability.
• Improved error handling and logging for vendor operations.
• Documentation
• Updated internal documentation to reflect changes in methods and functionalities.
• Revised CLI Commands Cheat Sheet and other documentation to include the new --everything
flag for vendor commands.
@Andriy Knysh (Cloud Posse) https://github.com/cloudposse/atmos/pull/762#discussion_r1838875067
@Andriy Knysh (Cloud Posse) all is resolved here ready to merge https://github.com/cloudposse/atmos/pull/737
@Vinny please review this issue https://github.com/cloudposse/atmos/issues/780
Describe the Bug
After updating from v1.105.0
to v1.106.0
, using a component.yaml file in the component directories no longer works as it returns an error if the vendor file or directory does not exist. Appears to be related to this change: https://github.com/cloudposse/atmos/pull/737/files#diff-35bbd3669178afb0acaaebfb4ec924ec99189338a9bc05232b4f92a53f261518L83
Expected Behavior
Running atmos vendor pull for a component containing a component.yaml file should result in the component being successfully vendored with specified mixins added.
Component config file
Expected command output
Steps to Reproduce
Running the same command using v1.106.0 results in an error due to the vendor file and/or directory not existing.
Screenshots
No response
Environment
• OS: WLS2/Debian12 using Geodesic 2.8.7 & Atmos 1.106.0
Additional Context
No response
it broke after we merged the PR
note that Atmos supports vendor.yaml
and component.yaml
(in the component directory)
looks like the component.yaml
functionality is broken
Use Atmos vendoring to make copies of 3rd-party components, stacks, and other artifacts in your own repo.
ready to review https://github.com/cloudposse/atmos/pull/782
@Vinny did it address the issue described in https://github.com/cloudposse/atmos/issues/780 ?
both vendor.yaml
and component.yaml
(from the component’s folder) must work. Note that 80% of the Atmos users still use component.yaml
to vendor components, we need to maintain this functionality and fix any issues
Describe the Bug
After updating from v1.105.0
to v1.106.0
, using a component.yaml file in the component directories no longer works as it returns an error if the vendor file or directory does not exist. Appears to be related to this change: https://github.com/cloudposse/atmos/pull/737/files#diff-35bbd3669178afb0acaaebfb4ec924ec99189338a9bc05232b4f92a53f261518L83
Expected Behavior
Running atmos vendor pull for a component containing a component.yaml file should result in the component being successfully vendored with specified mixins added.
Component config file
Expected command output
Steps to Reproduce
Running the same command using v1.106.0 results in an error due to the vendor file and/or directory not existing.
Screenshots
No response
Environment
• OS: WLS2/Debian12 using Geodesic 2.8.7 & Atmos 1.106.0
Additional Context
No response
we have the test for component.yaml
in
pkg/vender/component_vendor_test.go
examples/tests/components/terraform/infra/vpc-flow-logs-bucket/component.yaml
the logic of the atmos vendor pull
command was as the following (and we need to maintain it):
• Check if vendor.yaml
file exists (after your changes, check it in multiple places defined in atmos.yaml
• If vendor.yaml
exists and a component is not passed in atmos vendor pull
, use vendor.yaml
to pull all components
• If vendor.yaml
does not exist and a component is not passed in, show an error
• If component is passed in, and vendor.yaml
exists, use it to vendor the component
• Otherwise, search for component.yaml
in the component’s dir
• If component.yaml
exists, vendor the component
• Otherwise, show an error
ok this has to be reworked then
this exactly logic has been applied now
what
Added directory creation logic before copying vendored files
Fixed the “no such file or directory” error when running atmos vendor pull
Ensures target directories are created before attempting to vendor components
why
Previously, the vendor command would fail if target directories didn’t exist
Directory creation is a prerequisite for vendoring components
references
Summary by CodeRabbit
• Bug Fixes
• Improved error handling for vendor configuration file checks, preventing unnecessary error messages when the file does not exist.
• Streamlined the logic for processing vendor configuration, enhancing clarity and robustness.
thanks @Vinny, will review
good to review @Andriy Knysh (Cloud Posse) https://github.com/cloudposse/atmos/pull/764
@Vinny please address the comments
2024-11-14
@Cerebrovinny I do not think this is the fix based on the description of the problem. Bear in mind we have two types of vendor manifests.
https://atmos.tools/core-concepts/vendor/
The bug report concerns the component manifest file. That means that any logic relating to the vendor manifests file should not inhibit the component manifest from functioning.
To me, it sounds like the regression is that when vendoring a component manifest, it is erroring if the vendor manifests file path does not exist. However, that should not be required. We should instead log to warning that the path does not exist and continue processing.
We should not create the vendor configuration path if it does not exist.
ok I got the point
@Cerebrovinny I do not think this is the fix based on the description of the problem. Bear in mind we have two types of vendor manifests.
https://atmos.tools/core-concepts/vendor/
The bug report concerns the component manifest file. That means that any logic relating to the vendor manifests file should not inhibit the component manifest from functioning.
To me, it sounds like the regression is that when vendoring a component manifest, it is erroring if the vendor manifests file path does not exist. However, that should not be required. We should instead log to warning that the path does not exist and continue processing.
We should not create the vendor configuration path if it does not exist.
@Andriy Knysh (Cloud Posse) is this the correct explanation? https://github.com/cloudposse/atmos/pull/769/commits/d2cc5ec78569db7024f0813a351b72a0f6050768
Describe the Bug
While trying to understand why I have to set both ATMOS_BASE_PATH
and ATMOS_CLI_CONFIG_PATH
and if one of them used the other as a default I stumbled over this discrepancy in the “Learn Atmos” section:
The explanation of base_path
at https://atmos.tools/core-concepts/projects/configuration/#path-configuration says
We set it to an empty string because we’ve decided to use the ENV var ATMOS_BASE_PATH to point to the absolute path of the root of the repo
But the example further up at https://atmos.tools/core-concepts/projects/configuration/#atmos-cli-configuration-schema actually sets it to the value ./
(which is clearly not an empty string).
And if I interpret https://atmos.tools/cli/configuration/#base-path correctly an empty string would indeed have a different behaviour than that .
value.
Expected Behavior
I read the documentation and shouldn’t have been confused.
Steps to Reproduce
Read the documentation :-)
Screenshots
No response
Environment
No response
Additional Context
No response
yes, looks good to me
in the atmos.yaml
in the root of the repo, we pointed it to (before the changes)
base_path: "./examples/quick-start-advanced"
we were able to run Atmos commands from the root of the repo
This is not about documenting the behavior in cloudposse/atmos/atmos.yaml
now we are pointing it to /
- maybe we should update it to
base_path: "./examples/quick-start-simple"
2024-11-15
@Andriy Knysh (Cloud Posse) can you take a look at this https://github.com/cloudposse/atmos/issues/781#issuecomment-2478506957
@aknysh not able to found the regression checking this https://raw.githubusercontent.com/cloudposse/atmos/b53d696375a6716b2fd0bc024fb1861b46c78bb5/internal/exec/validate_stacks.go
maybe you can point out the root cause for this? also silence usage from cobra can be a viable option on this case
@Vinny make sure when referencing issues to say “Closes #123” so that when the PR is merged, the issue will get closed automatically
good point, I have few questions about that, should I create the branchs with the linear ticket number is fine? should I mention linear numbers in description?
If I’m working on something that exists in Linear but not in GitHub issues, should I open a ticket in GitHub so the community is aware?
It’s okay to use the linear issue in the branch names. I would avoid it in the PR description.
Since this is all open source, those are not available to most to view.
yeah I stop using it since I saw you removed one there
Took a stab at prettying up the JSON indentation for generated files such as backend.tf.json
(https://github.com/cloudposse/atmos/pull/783). Feel free to rework or disregard if this doesn’t meet the need! I tested it across five or so of our stacks
what
• Used the ConvertToJSON
utility with json.MarshalIndent
to produce formatted JSON
• Indentation is set to two spaces (“ “) for consistent readability
why
• This PR improves the WriteToFileAsJSON
function by introducing pretty-printing for JSON outputs. Previously, the function serialized JSON using a compact format, which could make the resulting files harder to read. With this change, all JSON written by this function will now be formatted with indentation, making it easier for developers and users to inspect and debug the generated files
• This specifically addresses #778 , which previously rendered auto-generated backends as:
{ “terraform”: { “backend”: { “s3”: { “acl”: “bucket-owner-full-control”, “bucket”: “my-tfstate-bucket”, “dynamodb_table”: “some-dynamo-table”, “encrypt”: true, “key”: “terraform.tfstate”, “profile”: “main”, “region”: “us-west-2”, “workspace_key_prefix”: “something” } } } }
With this addition, the output appears as:
{ “terraform”: { “backend”: { “s3”: { “acl”: “bucket-owner-full-control”, “bucket”: “my-tfstate-bucket”, “dynamodb_table”: “some-dynamo-table”, “encrypt”: true, “key”: “terraform.tfstate”, “profile”: “main”, “region”: “us-west-2”, “workspace_key_prefix”: “something” } } } }
references
• Stack Overflow • Closes #778
Summary by CodeRabbit
Summary by CodeRabbit
• New Features
• Enhanced JSON output formatting with indented structure for better readability.
• Refactor
• Simplified variable declarations for consistency in JSON handling functions.
approved and merged, thank you @Michael
what
• Used the ConvertToJSON
utility with json.MarshalIndent
to produce formatted JSON
• Indentation is set to two spaces (“ “) for consistent readability
why
• This PR improves the WriteToFileAsJSON
function by introducing pretty-printing for JSON outputs. Previously, the function serialized JSON using a compact format, which could make the resulting files harder to read. With this change, all JSON written by this function will now be formatted with indentation, making it easier for developers and users to inspect and debug the generated files
• This specifically addresses #778 , which previously rendered auto-generated backends as:
{ “terraform”: { “backend”: { “s3”: { “acl”: “bucket-owner-full-control”, “bucket”: “my-tfstate-bucket”, “dynamodb_table”: “some-dynamo-table”, “encrypt”: true, “key”: “terraform.tfstate”, “profile”: “main”, “region”: “us-west-2”, “workspace_key_prefix”: “something” } } } }
With this addition, the output appears as:
{ “terraform”: { “backend”: { “s3”: { “acl”: “bucket-owner-full-control”, “bucket”: “my-tfstate-bucket”, “dynamodb_table”: “some-dynamo-table”, “encrypt”: true, “key”: “terraform.tfstate”, “profile”: “main”, “region”: “us-west-2”, “workspace_key_prefix”: “something” } } } }
references
• Stack Overflow • Closes #778
Summary by CodeRabbit
Summary by CodeRabbit
• New Features
• Enhanced JSON output formatting with indented structure for better readability.
• Refactor
• Simplified variable declarations for consistency in JSON handling functions.
Thanks @Michael!
2024-11-16
@Andriy Knysh (Cloud Posse) this should fix the regression raised by @Michael by disabling the custom prompt by default and enhancing it so it can be customized via the atmos config.
what
why
Working demo
With custom prompt:
Screenshot 2024-11-16 at 11 20 14 PM
Without custom prompt:
Closes #779 (disable new prompt by default until we make it work better with geodesic)
references
Summary by CodeRabbit
Summary by CodeRabbit
• New Features
• Introduced a configurable shell prompt for Terraform commands, allowing users to customize their command-line interface.
• Added a new ShellConfig
type to support the new shell prompt configuration.
• Bug Fixes
• Improved flexibility in shell prompt settings by allowing dynamic configuration based on user-defined CLI settings.
I haven’t had a chance to test it yet
what
why
Working demo
With custom prompt:
Screenshot 2024-11-16 at 11 20 14 PM
Without custom prompt:
Closes #779 (disable new prompt by default until we make it work better with geodesic)
references
Summary by CodeRabbit
Summary by CodeRabbit
• New Features
• Introduced a configurable shell prompt for Terraform commands, allowing users to customize their command-line interface.
• Added a new ShellConfig
type to support the new shell prompt configuration.
• Bug Fixes
• Improved flexibility in shell prompt settings by allowing dynamic configuration based on user-defined CLI settings.
Cc @Pulak Kanti Bhowmick
Just one question here: https://linear.app/cloudposse/issue/DEV-2713/automatic-templated-file-search-for-atmos-imports. It looks like the .tmpl files take precedence over the .yaml, .yml extensions is that right?
all imports are processed in the order they are defined regardless of the file extensions
the order of imports is important because of the inheritance and deep merging
2024-11-18
2024-11-19
I’ve started documenting some dev conventions for atmos that I feel we struggle with. Note that these are the intents, but they are not necessarily how Atmos
works today. We need everyone’s help to conform to these conventions and update atmos as we encounter inconsistencies with these conventions.
what
• Explain when to display help vs usage, and their respective behaviors • Explain when to use various log levels
why
• Improve code consistency as the dev team expands
I think this is very needed, as I am confuse myself sometimes
what
• Explain when to display help vs usage, and their respective behaviors • Explain when to use various log levels
why
• Improve code consistency as the dev team expands
this is good to final review @Andriy Knysh (Cloud Posse) https://github.com/cloudposse/atmos/pull/764
this is good to start review @Andriy Knysh (Cloud Posse) https://github.com/cloudposse/atmos/pull/795
2024-11-20
@Michael I am extremely excited for the list commands you implemented! https://github.com/cloudposse/atmos/pull/797
what
• Creates an atmos list
command for listing stacks and components
• Incorporates custom list
commands into Atmos
• Updates documentation and website
• Removes atmos.yaml
references to custom list command
why
• While the custom Atmos commands for listing stacks and components are great, incorporating the command into Atmos is far more efficient and parallelized, achieving similar or better results in 0.741 seconds compared to 8.131 seconds for the custom command
references
Summary by CodeRabbit
Release Notes
• New Features
• Introduced a list
command with subcommands for listing Atmos stacks and components.
• Enhanced command functionality with options to filter by stack and component type.
• Documentation
• Added comprehensive documentation for the new list
, list stacks
, and list components
commands, including usage examples and flags.
• Updated existing documentation to clarify command usage and improve user guidance.
• Bug Fixes
• Improved error handling and output formatting for commands related to listing stacks and components.
Thank you! I am going to move some of the logic into pkg/utils if that’s okay to simplify the command logic!
what
• Creates an atmos list
command for listing stacks and components
• Incorporates custom list
commands into Atmos
• Updates documentation and website
• Removes atmos.yaml
references to custom list command
why
• While the custom Atmos commands for listing stacks and components are great, incorporating the command into Atmos is far more efficient and parallelized, achieving similar or better results in 0.741 seconds compared to 8.131 seconds for the custom command
references
Summary by CodeRabbit
Release Notes
• New Features
• Introduced a list
command with subcommands for listing Atmos stacks and components.
• Enhanced command functionality with options to filter by stack and component type.
• Documentation
• Added comprehensive documentation for the new list
, list stacks
, and list components
commands, including usage examples and flags.
• Updated existing documentation to clarify command usage and improve user guidance.
• Bug Fixes
• Improved error handling and output formatting for commands related to listing stacks and components.
I’m always sensitive what we can ask for from outside contributors, but since I was just working on defining the requirements for a separate atmos list vendor
command, thought I would share what I was thinking
So for the list vendor command, wanted to do something like this in atmos.yaml
I think the list stacks
and list components
could benefit from the same configurability
And then using lipgloss for display
Just throwing this out there in the event I successfully nerd snipe you into implementing it! But if not, this is still a step up.
E.g.
stacks:
list:
columns:
- name: Stack
value: '{{ .Stack }}'
- name: Tenant
value: '{{ .vars.tenant }}'
- name: Environment
value: '{{ .vars.environment }}'
Stack Tenant Environment
plat-ue2-dev plat ue2
That’s fancy! Would it be useful to include an option for plain or pretty printing? I was also thinking this function could be leveraged for autocompletion in the future, such as with atmos describe component <tab>
, where autocompletion is based on the output of listing stacks and components
Yes, I think if there’s no TTY, it would degrade to plain output, so standard pipelining works well (e.g. atmos list stacks | grep foo) |
Re: tab completion, yes, we have to improve that!
It should work like you describe
for components and –stack
@Erik Osterman (Cloud Posse) From a user perspective, we utilize the custom command every day because it’s hard to keep track of the amount of stacks and components we have. The current pain point with the command is that it takes about 8 seconds to complete with our current setup. The general workflow when using the command is:
√ . [devops] (HOST) infra ⨠ atmos list stacks -c eks/cluster
plat-ue2-dev
√ . [devops] (HOST) infra ⨠ atmos terraform apply eks/cluster -s plat-ue2-dev
I can see having the pretty-print output as beneficial, but being able to quickly copy into the next command makes for a streamlined workflow. Is your vision to have it default to a “plain” output if not specified in atmos.yaml
?
No, so this is very common in command line tools. If you run, for example:
cat /etc/passwd | more | grep root
The more
detects there’s no TTY and does nothing
If you use bcat
and display a file on the command line, it’s pretty printed
If you run bcat | pbcopy
then it’s not pretty printed
That said, not asking/suggesting that you have to implement this I can add it to our backlog. But we will implement this, as we’re trying to make a push to polish off atmos so it continues to operate like a modern TUI CLI
(and, I’m not opposed to a atmos.yaml
flag to disable TUI; but regardless, we’ll need to degrade TUIs with or without that flag)
2024-11-21
Hello @Andriy Knysh (Cloud Posse) this is good to start review https://github.com/cloudposse/atmos/pull/798
Can you double check the code rabbit comments
yes please hold on a bit on this
good to go now
This might be related: DEV-2746
Describe the Bug
We bumped from 1.90.0 to 1.105.0. Our CI uses
atmos validate component <component> --stack <stack>
With 1.105.0 (and 1.106.0) we now get the commands usage printed instead/additionally:
atmos validate component "vpc/foo" --stack bar
Atmos supports native '' commands with all the options, arguments and flags.
In addition, 'component' and 'stack' are required in order to generate variables for the component in the stack.
atmos <subcommand> <component> -s <stack> [options]
atmos <subcommand> <component> --stack <stack> [options]
For more details, execute ' --help'
component 'vpc/foo' in stack 'bar' validated successfully
So it looks like some validation still took place and it was successfull.
Expected Behavior
Exit code 0 on successfull validation, some other number otherwise. No usage gets printed to the screen.
Steps to Reproduce
Run atmos validate component <component> --stack <stack>
as documented via --help
.
Screenshots
No response
Environment
• OS: Linux • atmos Version: 1.105.0 • terraform Version: 1.8.5
Additional Context
No response
I am sending a fix for validations when we execute –help commands it should stop output that error msgs like requesting stacks folder
so, it sounds like it should fix this case as well?
it’s another problem I will send in other PR so it won’t affect this terraform fixes
cool
@Andriy Knysh (Cloud Posse) this looks ready for review https://github.com/cloudposse/atmos/pull/782
what
Fixed the regression where the vendor command would error if the vendor manifests file path did not exist.
why
Previously, the atmos vendor pull command would fail with a “no such file or directory” error when the vendor manifests file path was missing.
references
Closes #780
Summary by CodeRabbit
Summary by CodeRabbit
• Bug Fixes
• Improved error handling for vendor configuration file checks, preventing unnecessary error messages when the file does not exist.
• Enhanced handling of empty YAML configuration files, ensuring graceful processing without errors.
• Streamlined handling of the Atmos JSON Schema manifest, providing a default when none is specified and improving clarity of error messages.
@Vinny would prefer screenshots, before/after to showcase this working.
what
Fixed the regression where the vendor command would error if the vendor manifests file path did not exist.
why
Previously, the atmos vendor pull command would fail with a “no such file or directory” error when the vendor manifests file path was missing.
references
Closes #780
Summary by CodeRabbit
Summary by CodeRabbit
• Bug Fixes
• Improved error handling for vendor configuration file checks, preventing unnecessary error messages when the file does not exist.
• Enhanced handling of empty YAML configuration files, ensuring graceful processing without errors.
• Streamlined handling of the Atmos JSON Schema manifest, providing a default when none is specified and improving clarity of error messages.
One thing I forgot to ask: it seems that running help without the flag doesn’t display the interactive menu from Cobra. Is that intentional?
it is, however, it’s not what I would prefer.
let me talk about it with our team tomorrow
I think by default, we should show help, and have a subcommand for the interactive mode.
e.g.
atmos menu
ok I got the point
2024-11-22
this is ready for review again @Andriy Knysh (Cloud Posse) https://github.com/cloudposse/atmos/pull/798
2024-11-23
2024-11-24
Just made some more documentation updates and additional error handling for the list commands! Feel free to review whenever you get the chance! https://github.com/cloudposse/atmos/pull/797
what
• Creates an atmos list
commands for listing stacks and components
• Incorporates custom list
commands into Atmos
• Updates documentation and website
• Removes atmos.yaml
references to custom list command
why
• While the custom Atmos commands for listing stacks and components are great, incorporating the command into Atmos is far more efficient and parallelized, achieving similar or better results in 0.741 seconds compared to 8.131 seconds for the custom command
testing
• Listing all stacks in quick-start-advanced
❯ atmos list stacks plat-ue2-dev plat-ue2-prod plat-ue2-staging plat-uw2-dev plat-uw2-prod plat-uw2-staging
• Listing all stacks by component
❯ ./atmos list stacks -c vpc-flow-logs-bucket plat-ue2-dev plat-ue2-prod plat-ue2-staging plat-uw2-dev plat-uw2-prod plat-uw2-staging
• Listing stacks for non-existent component
❯ ./atmos list stacks -c test No stacks found for component ‘test’
• Listing all components
❯ ./atmos list components vpc vpc-flow-logs-bucket
• Listing components by stack
❯ ./atmos list components -s plat-ue2-prod vpc vpc-flow-logs-bucket
• Listing components by invalid stack
❯ ./atmos list components -s invalid-stack Error: stack ‘invalid-stack’ not found
references
Summary by CodeRabbit
Release Notes
• New Features
• Introduced a list
command with subcommands for listing Atmos stacks and components.
• Enhanced command functionality with options to filter by stack and component type.
• Documentation
• Added comprehensive documentation for the new list
, list stacks
, and list components
commands, including usage examples and flags.
• Updated existing documentation to clarify command usage and improve user guidance.
• Bug Fixes
• Improved error handling and output formatting for commands related to listing stacks and components.
2024-11-28
https://github.com/cloudposse/atmos/pull/795 good for review again @Andriy Knysh (Cloud Posse), once we get into a good state with the changes I will publish final evidences before we get ready to merge
good for final review @Andriy Knysh (Cloud Posse) https://github.com/cloudposse/atmos/pull/798
2024-11-29
very good for final review https://github.com/cloudposse/atmos/pull/782 @Andriy Knysh (Cloud Posse)
@Andriy Knysh (Cloud Posse) hi, have pushed a PR, would you please take a look/possibly approve in case no changes are required? https://github.com/cloudposse/atmos/pull/809
what
This pull request enhances the atmos CLI by adding a bordered update notification message at the bottom of the help output for all commands, including atmos –help and subcommands like atmos terraform –help. The update notification informs users if a newer version of atmos is available
Befroe
CleanShot 2024-11-20 at 17 00 54 (1)
After
final version
00005
why
• Prompts users to upgrade • The bordered message is more noticeable
references
Linear issue https://linear.app/cloudposse/issue/DEV-2766/atmos-help-should-show-latest-version
Summary by CodeRabbit
Summary by CodeRabbit
• New Features
• Enhanced help functionality now includes upgrade notifications for the latest release.
• Improved formatting for upgrade messages with added borders and resource links.
• New function to check for the latest version and notify users if an upgrade is available.
• Bug Fixes
• Resolved potential infinite recursion in the help command by preserving the original help function.
• Documentation
• Updated user notifications to provide timely information about available upgrades.
thank you all for the PRs. I’ll review all of them after I finish some other tasks (prob tonight or tomorrow morning)
@Andriy Knysh (Cloud Posse) hi, just in case have tagged your name in the PR. Would appreciate if you could please take a look when you have a minute.
@Aaron Bidderman let’s reimplement this using lipgloss/bubbletea