#atmos (2023-08)
2023-08-01
![John avatar](https://secure.gravatar.com/avatar/46258afadda3a3087ca98cc494c13ca0.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0006-72.png)
Does anyone has atmos
running in gitlab cicd? I m trying to set-up gitlab CiCd using atmos in order to apply my stacks and wondering if someone has it working with some basic example. Thank you
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
(we have examples for GHA, but not GitLab)
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
GitHub Actions
![Matt Gowie avatar](https://avatars.slack-edge.com/2023-02-06/4762019351860_44dadfaff89f62cba646_72.jpg)
Is there JSON Schema files somehow for Atmos stacks/component.yaml/atmos.yaml?
![Brian avatar](https://secure.gravatar.com/avatar/22efa2a78045ecdab74494db85b41ba7.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0021-72.png)
This is a good question. I would like to know as well. I have detailed schemas for majority of my components, but only a “good enough one” for stack and workflow files. @Matt Gowie, this is my current one if you want a starting point. I have another one that extends that one with several conditional blocks (see below) that will reference JSONSchema for a component used by atmos validate
too.
{
"$schema": "<https://json-schema.org/draft/2020-12/schema>",
"$id": "tf-component-stacks",
"title": "component-stack",
"type": "object",
"properties": {
"import": {
"type": "array",
"items": {
"type": "string"
}
},
"components": {
"type": "object",
"properties": {
"helmfile": {},
"terraform": {
"type": "object",
"patternProperties": {
".+": {
"allOf": [
{
"if": {
"properties": {
"component": {
"enum": [
"account-monitors"
]
}
},
"required": [
"component"
]
},
"then": {
"$ref": "account-monitors.json",
"additionalProperties": false
}
},
{
"if": {
"properties": {
"component": {
"enum": [
"account-settings"
]
}
},
"required": [
"component"
]
},
"then": {
"$ref": "account-settings.json",
"additionalProperties": false
}
},
// ...more if-then block for components with scheams
// below is a generic schema for components without scheams
{
"if": {
"patternProperties": {
".+": {}
}
},
"then": {
"properties": {
"remote_state_backend": {},
"remote_state_backend_type": {},
"settings": {},
"backend": {},
"backend_type": {},
"deps": {},
"command": {},
"env": {},
"metadata": {},
"workspace": {},
"inheritance": {},
"sources": {},
"component": {
"type": "string"
},
"vars": {
"type": "object",
"properties": {
"enabled": {
"type": "boolean"
},
"environment": {
"type": "string"
},
"label_key_case": {
"type": "string"
},
"label_value_case": {
"type": "string"
},
"localstack_enabled": {
"type": "boolean"
},
"namespace": {
"type": "string"
},
"region": {
"type": "string",
"enum": [
"us-east-1"
]
},
"sandbox_mode_enabled": {
"type": "boolean"
},
"stage": {
"type": "string"
},
"tags": {
"type": "object",
"required": [
"env"
],
"patternProperties": {
"[a-zA-Z0-9_-]+": {
"type": "string"
}
}
},
"name": {
"type": "string",
"pattern": ".+"
}
}
}
},
"additionalProperties": false
}
}
]
}
}
}
},
"additionalProperties": false
},
"env": {
"type": "object",
"patternProperties": {
".+": {
"type": [
"boolean",
"number",
"string"
]
}
}
},
"terraform": {
"properties": {
"remote_state_backend": {},
"remote_state_backend_type": {},
"settings": {},
"backend": {},
"backend_type": {},
"deps": {},
"command": {},
"env": {},
"metadata": {},
"workspace": {},
"inheritance": {},
"sources": {},
"s3": {}
},
"additionalProperties": false
},
"vars": {
"type": "object",
"properties": {
"environment": {
"type": "string"
},
"label_key_case": {
"type": "string"
},
"label_value_case": {
"type": "string"
},
"namespace": {
"type": "string"
},
"stage": {
"type": "string"
},
"tags": {
"type": "object",
"patternProperties": {
".+": {
"type": [
"boolean",
"number",
"string"
]
}
}
}
}
}
},
"additionalProperties": false
}
![Matt Gowie avatar](https://avatars.slack-edge.com/2023-02-06/4762019351860_44dadfaff89f62cba646_72.jpg)
Ah good stuff, will check that out.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
@Andriy Knysh (Cloud Posse)
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
we have a task to add schema for all Atmos configs, and execute it in atmos validate stacks
![Matt Gowie avatar](https://avatars.slack-edge.com/2023-02-06/4762019351860_44dadfaff89f62cba646_72.jpg)
Is that a GH issue? Got something to track?
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
in the schema, we will not be checking each component’s variables as in the example above - this is not practical and not scalable. We’ll be checking the general structure and all the sections like vars
, settings
, env
, etc.
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
to validate component’s vars, we already have atmos validate component
with OPA policies and JSONSchema
![Brian avatar](https://secure.gravatar.com/avatar/22efa2a78045ecdab74494db85b41ba7.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0021-72.png)
I use the atmos validate componenet
to check the component vars too, but use the schema above for dev-time type-hints.
![Matt Gowie avatar](https://avatars.slack-edge.com/2023-02-06/4762019351860_44dadfaff89f62cba646_72.jpg)
Yeah those serve different purposes and I think that makes total sense
2023-08-02
2023-08-07
![Release notes from atmos avatar](https://a.slack-edge.com/80588/img/services/rss_72.png)
v1.43.0 what Add the –skip-lock-file flag to the atmos terraform clean command why For users who want to keep a stable lock file but otherwise want to clean the component directory
what Add the –skip-lock-file flag to the atmos terraform clean command why For users who want to keep a stable lock file but otherwise want to clean the component directory
![Release notes from atmos avatar](https://a.slack-edge.com/80588/img/services/rss_72.png)
v1.43.0 what Add the –skip-lock-file flag to the atmos terraform clean command why For users who want to keep a stable lock file but otherwise want to clean the component directory
2023-08-11
![johncblandii avatar](https://avatars.slack-edge.com/2020-04-14/1062347993890_6fd142c15ffef426eeba_72.png)
@Andriy Knysh (Cloud Posse) is there a way to use a glob and control the order?
- path: mixins/services/**/*-[default][dev]*
context:
stage: dev
environment: ue1
basically, we need default
imported before dev
.
we’re trying to accomplish importing separate files to enable stage-specific imports to override defaults.
ex:
• dev imports default then dev
• staging importing default then staging
A straight import with -dev
in the filename will error if there are no matches for that particular import.
![johncblandii avatar](https://avatars.slack-edge.com/2020-04-14/1062347993890_6fd142c15ffef426eeba_72.png)
So largely looking for guidance as I update our architecture for our stage-specific updates
![johncblandii avatar](https://avatars.slack-edge.com/2020-04-14/1062347993890_6fd142c15ffef426eeba_72.png)
and it looks like this is going to cause problems with that regardless so we likely need a sortable filename that ensures our defaults are always first
https://github.com/cloudposse/atmos/blob/master/pkg/stack/stack_processor.go#L86
sort.Strings(uniqueImports)
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
imports are processed in the order they are defined (the above code is not related to how imports are processed, it’s to output the imports
field in atmos describe component
command)
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
when using Globs, we don’t control how the files are processed since we are using a 3rd-party lib for that
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
so to control the imports, you have to use two diff imports, one for the defaults, and the other for de/prod/staging etc. settings
![johncblandii avatar](https://avatars.slack-edge.com/2020-04-14/1062347993890_6fd142c15ffef426eeba_72.png)
so how can i use 2 separate without it erroring if a file of that pattern doesn’t exist?
![johncblandii avatar](https://avatars.slack-edge.com/2020-04-14/1062347993890_6fd142c15ffef426eeba_72.png)
ex:
- path: mixins/services/**/*-default
context:
stage: dev
environment: ue1
- path: mixins/services/**/*-dev
context:
stage: dev
environment: ue1
![johncblandii avatar](https://avatars.slack-edge.com/2020-04-14/1062347993890_6fd142c15ffef426eeba_72.png)
if no files have -dev
in them, it will error
![johncblandii avatar](https://avatars.slack-edge.com/2020-04-14/1062347993890_6fd142c15ffef426eeba_72.png)
and, technically, default
is going to be for most envs anyway so there probably won’t be many overrides
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
if no files have -dev
in them
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
then why do you import it? (in any language, if you import/include something that does not exist, it will error)
![johncblandii avatar](https://avatars.slack-edge.com/2020-04-14/1062347993890_6fd142c15ffef426eeba_72.png)
basically, we want to support an override pattern so when we want to override a particular stacks var we can
![johncblandii avatar](https://avatars.slack-edge.com/2020-04-14/1062347993890_6fd142c15ffef426eeba_72.png)
services/all
has imports for all services
plat-ue1-prod
(dev, etc) imports services/all
, but that’s the default.
for instances where we want to override an SSM param’s value or ECS service ENV var, I’m thinking we have a filename-{{ .stage }}.yaml
file pattern for consistency
![johncblandii avatar](https://avatars.slack-edge.com/2020-04-14/1062347993890_6fd142c15ffef426eeba_72.png)
seems a bit hacky, but this works. it would technically import the same default
file twice:
import:
- path: "mixins/services/**/*-default*"
context:
stage: "{{ .stage }}"
environment: "{{ .environment }}"
- path: "mixins/services/**/*-[default][{{ .stage }}]*"
context:
stage: "{{ .stage }}"
environment: "{{ .environment }}"
![johncblandii avatar](https://avatars.slack-edge.com/2020-04-14/1062347993890_6fd142c15ffef426eeba_72.png)
the key is in **/*-[default][{{ .stage }}]*
where it matches “default” (so that will never fail) and .stage
(which would be dev
, etc)
![johncblandii avatar](https://avatars.slack-edge.com/2020-04-14/1062347993890_6fd142c15ffef426eeba_72.png)
but yeah…that does botch the order for sure
![johncblandii avatar](https://avatars.slack-edge.com/2020-04-14/1062347993890_6fd142c15ffef426eeba_72.png)
if a file is created alphabetically below default
, it will put default
after that file
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
looks like a regular expression :)
![johncblandii avatar](https://avatars.slack-edge.com/2020-04-14/1062347993890_6fd142c15ffef426eeba_72.png)
![johncblandii avatar](https://avatars.slack-edge.com/2020-04-14/1062347993890_6fd142c15ffef426eeba_72.png)
i’ve tested a lot of patterns, but none of the fit perfectly.
going to just ensure all envs have at least one file for now.
![johncblandii avatar](https://avatars.slack-edge.com/2020-04-14/1062347993890_6fd142c15ffef426eeba_72.png)
atmos validate stacks
has a problem where context is included in part of the import.
example:
import:
- path: "mixins/services/**/*{{ .stage }}.yaml"
context:
stage: "{{ .stage }}"
environment: "{{ .environment }}"
no matches found for the import ‘mixins/services//{{ .stage }}.yaml’ in the file ‘mixins/services/all.yaml’
Error: failed to find a match for the import ‘/localhost/work/infrastructure/stacks/mixins/services/**/{{ .stage }}.yaml’ (‘/localhost/work/infrastructure/stacks/mixins/services’ + ‘/*{{ .stage }}.yaml’)
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
yes, we know that. The command was not updated after the Go templates were introduced
![johncblandii avatar](https://avatars.slack-edge.com/2020-04-14/1062347993890_6fd142c15ffef426eeba_72.png)
Oh ok. I disabled it on CI since we’re already running affected
anyway.
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
will update the command asap
2023-08-16
![sytten avatar](https://avatars.slack-edge.com/2019-03-01/564418310496_16e796635f1b40fcea36_72.jpg)
Might be a dumb question, but can you reference an output of a top level component to the variable of another component
![Brian avatar](https://secure.gravatar.com/avatar/22efa2a78045ecdab74494db85b41ba7.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0021-72.png)
No. Not directly.
![Brian avatar](https://secure.gravatar.com/avatar/22efa2a78045ecdab74494db85b41ba7.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0021-72.png)
This isn’t a future of atmos
at this time.
![sytten avatar](https://avatars.slack-edge.com/2019-03-01/564418310496_16e796635f1b40fcea36_72.jpg)
Generally you would put the whitelist in my example in the same waf-acl component?
![Brian avatar](https://secure.gravatar.com/avatar/22efa2a78045ecdab74494db85b41ba7.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0021-72.png)
So, the only pure terraform
way to do it is via the remote-state
pattern that is used through the components in cloudposse/aws-terraform-components
.
![Brian avatar](https://secure.gravatar.com/avatar/22efa2a78045ecdab74494db85b41ba7.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0021-72.png)
Generally you would put the whitelist in my example in the same waf-acl component?
I do it the way you have in your example for most IPSets for WAFs. However, I also do it all within one component if the IPSets only is used by a single WAF.
![sytten avatar](https://avatars.slack-edge.com/2019-03-01/564418310496_16e796635f1b40fcea36_72.jpg)
Ok make sense, I like the way I did it I just was not sure how to pass the ARN to create the rule
![Brian avatar](https://secure.gravatar.com/avatar/22efa2a78045ecdab74494db85b41ba7.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0021-72.png)
My waf
component is custom. If I pass a reference to a IPSet or another WAF object type, I do a data lookup within the component to get the arn.
![sytten avatar](https://avatars.slack-edge.com/2019-03-01/564418310496_16e796635f1b40fcea36_72.jpg)
Right, I think we just vendored the cloudposse one. Right now the rules are in yaml
ip_set_reference_statement_rules:
- name: IP-Whitelist
action: allow
priority: 1
statement:
arn:
visibility_config:
cloudwatch_metrics_enabled: true
metric_name: IP-Whitelist
sampled_requests_enabled: true
![Brian avatar](https://secure.gravatar.com/avatar/22efa2a78045ecdab74494db85b41ba7.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0021-72.png)
Yep. You can modify so to check for <waf-object-type>.statement.name
instead. If arn
is not defined but name
is, then get the ARN via a lookup.
![sytten avatar](https://avatars.slack-edge.com/2019-03-01/564418310496_16e796635f1b40fcea36_72.jpg)
right good idea
![sytten avatar](https://avatars.slack-edge.com/2019-03-01/564418310496_16e796635f1b40fcea36_72.jpg)
Basically can I do
components:
terraform:
waf-whitelist:
metadata:
component: aws-waf-ipset
vars:
enabled: true
addresses: ["17.0.0.0/8"]
waf-acl:
metadata:
component: aws-waf-acl
vars:
some_var: { reference waf-whitelist here }
![jose.amengual avatar](https://secure.gravatar.com/avatar/32f267b819eac9e0ea6a8324b53064a0.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0024-72.png)
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Well, it depends.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
We just use YAML, so depending on how you want to reference it, you can use YAML anchors.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Please note, that anchors don’t work across YAML documents. Not a limitation of atmos, but a limitation of YAML.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
When using Bitbucket Pipelines, YAML anchors reduce effort and make updating in bulk, easier.
![sytten avatar](https://avatars.slack-edge.com/2019-03-01/564418310496_16e796635f1b40fcea36_72.jpg)
I need the ARN created for the IPSet
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Oh, but that’s not { reference waf-whitelist here }
that’s { reference some terraform output of waf-whitelist here }
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
This should be implemented by your components.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Terraform can easily reference the ARN (e.g. in SSM or with remote state, data lookups, etc), if your components are written properly, which is why we don’t implement it in atmos.
![this](/assets/images/custom_emojis/this.png)
![sytten avatar](https://avatars.slack-edge.com/2019-03-01/564418310496_16e796635f1b40fcea36_72.jpg)
I think our components might be too generic
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
I consider it generic to write outputs to SSM
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
It’s a consistent, generic convention
![sytten avatar](https://avatars.slack-edge.com/2019-03-01/564418310496_16e796635f1b40fcea36_72.jpg)
Like in case of the WAF, the rules are currently in yaml but they should probably be in terraform?
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Aha I see
![jose.amengual avatar](https://secure.gravatar.com/avatar/32f267b819eac9e0ea6a8324b53064a0.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0024-72.png)
you could create and outputs/inputs module that do that for you to bridge the gap
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Well, No, the rules should be in YAML
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
So, what you want is the corresponding rule ARN that was created for each of the inputs. Perhaps this should be in the outputs of the component.
![sytten avatar](https://avatars.slack-edge.com/2019-03-01/564418310496_16e796635f1b40fcea36_72.jpg)
Would it be worth adding IP support in the cloudposse waf component
![sytten avatar](https://avatars.slack-edge.com/2019-03-01/564418310496_16e796635f1b40fcea36_72.jpg)
instead of just the ARN an ipset
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
I think the easiest thing would be to propose what the change would look like in YAML or outputs
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Then we can advise
![sytten avatar](https://avatars.slack-edge.com/2019-03-01/564418310496_16e796635f1b40fcea36_72.jpg)
Right now it looks like:
ip_set_reference_statement_rules:
- name: IP-Whitelist
action: allow
priority: 1
statement:
arn: <SOME ARN>
visibility_config:
cloudwatch_metrics_enabled: true
metric_name: IP-Whitelist
sampled_requests_enabled: true
![sytten avatar](https://avatars.slack-edge.com/2019-03-01/564418310496_16e796635f1b40fcea36_72.jpg)
We could add:
ip_set_reference_statement_rules:
- name: IP-Whitelist
action: allow
priority: 1
statement:
ips:
addresses: ["17.0.0.0/8"]
visibility_config:
cloudwatch_metrics_enabled: true
metric_name: IP-Whitelist
sampled_requests_enabled: true
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Yes, something like that makes sense to me. @Andriy Knysh (Cloud Posse) who would be best to chime in from our side?
![sytten avatar](https://avatars.slack-edge.com/2019-03-01/564418310496_16e796635f1b40fcea36_72.jpg)
(naming is bad, but its the idea)
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
the resource need the ARN of the IPSet https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/wafv2_web_acl.html#ip_set_reference_statement-block
![sytten avatar](https://avatars.slack-edge.com/2019-03-01/564418310496_16e796635f1b40fcea36_72.jpg)
The idea would be to create it dynamically
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
at least in the current TF odule, we can’t use the IPs https://github.com/cloudposse/terraform-aws-waf/blob/main/rules.tf#L93
resource "aws_wafv2_web_acl" "default" {
![sytten avatar](https://avatars.slack-edge.com/2019-03-01/564418310496_16e796635f1b40fcea36_72.jpg)
Yeah I propose to code the support for it!
![sytten avatar](https://avatars.slack-edge.com/2019-03-01/564418310496_16e796635f1b40fcea36_72.jpg)
If you guys think it makes sense
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
we deal with cases like that all the time. With Atmos, it’s not supported in YAML directly, but using the remote-state
module. In YAML, we provide a component selector which consists of the component name, environment, tenant, stage (if the component is in a diff stack)
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
Yeah I propose to code the support for it!
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
you can open a PR for https://github.com/cloudposse/terraform-aws-waf and add code to get the IPSet ARN using terraform, yes
![sytten avatar](https://avatars.slack-edge.com/2019-03-01/564418310496_16e796635f1b40fcea36_72.jpg)
I dont think we are on the same page, basically in WAF rules are created in yaml. So the idea would be for the module to create the necessary IPSet based on the input as an alternative to the ARN as input.
![sytten avatar](https://avatars.slack-edge.com/2019-03-01/564418310496_16e796635f1b40fcea36_72.jpg)
Basically https://sweetops.slack.com/archives/C031919U8A0/p1692216132927799?thread_ts=1692209301.581579&cid=C031919U8A0 would create an IPSet and use its ARN in the rule
We could add:
ip_set_reference_statement_rules:
- name: IP-Whitelist
action: allow
priority: 1
statement:
ips:
addresses: ["17.0.0.0/8"]
visibility_config:
cloudwatch_metrics_enabled: true
metric_name: IP-Whitelist
sampled_requests_enabled: true
![sytten avatar](https://avatars.slack-edge.com/2019-03-01/564418310496_16e796635f1b40fcea36_72.jpg)
We might not want to do it, we might prefer to support a name import I was just proposing
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
we are on the same page What I’m saying is
• Atmos does not support custom functions in YAML, and the way to do it is to add some Terraform support which can be used in YAML by Atmos
• To add that Terraform support, there are two ways of doing it
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
- Use
remote-state
to get the outputs from thewaf-whitelist
component, and use its ARN in the other component
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
- Update the
waf
module with some TF code. Then, in YAML, you could provide the ARN of the IPSet, OR a list of IP addresses. Then here in the code https://github.com/cloudposse/terraform-aws-waf/blob/main/rules.tf#L423, check if the ARN is provided, use it. If the ARN is not provided, use the outputs from ` TF data source which looks up the IPSet ARN by IP (or other parameters). The IP (or other parameters) will be specified in YAML
arn = ip_set_reference_statement.value.arn
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
once that support is aded to the waf
module, then in YAML you will be able to use either ARN or IP
![sytten avatar](https://avatars.slack-edge.com/2019-03-01/564418310496_16e796635f1b40fcea36_72.jpg)
Agreed so you dont think the waf should create the IPSet resource, just reference it
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
in any case, some TF support/code needs to be added to support the functionality
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
Agreed so you dont think the waf should create the IPSet resource, just reference it
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
it can. It’s yet another way of doing it
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
we can add IPSet resource to the module, and if you provide the inputs for it and it gets created, we can use its ARN. Or the ARN can be provided separately as it is now
![sytten avatar](https://avatars.slack-edge.com/2019-03-01/564418310496_16e796635f1b40fcea36_72.jpg)
Perfect, I can work on that tomorrow and push a PR
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
thank you
![sytten avatar](https://avatars.slack-edge.com/2019-03-01/564418310496_16e796635f1b40fcea36_72.jpg)
I did a prototype https://github.com/cloudposse/terraform-aws-waf/compare/main...masterpointio:terraform-aws-waf:main?expand=1, but I am having a hard time with the dynamic block because the resource arn is not evaluated at runtime, terraform evaluates it a planning time. So it spits out aws_wafv2_ip_set.default is object with no attributes
, I tried a few different ways to try to force it to understand the dependency but it just doesn’t seem to like it. Have you guys seen that before?
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
can you open a PR so we could review it? (there are many ways to deal with the issue, but all of them are different and we deal with them on a case by case basis)
![sytten avatar](https://avatars.slack-edge.com/2019-03-01/564418310496_16e796635f1b40fcea36_72.jpg)
what
• Add ip_set
variable in ip_set_reference_statement_rules
to create IP Set automatically
why
• If the IP set is not shared, there is no need to have separate from the WAF module • It allows a user of atmos to continue defining all their rules in yaml, otherwise they have to use tf code to forward the ARN to this module
![sytten avatar](https://avatars.slack-edge.com/2019-03-01/564418310496_16e796635f1b40fcea36_72.jpg)
I did a few experiment in the commits
![sytten avatar](https://avatars.slack-edge.com/2019-03-01/564418310496_16e796635f1b40fcea36_72.jpg)
![sytten avatar](https://avatars.slack-edge.com/2019-03-01/564418310496_16e796635f1b40fcea36_72.jpg)
Found the bug, PR ready for review https://github.com/cloudposse/terraform-aws-waf/pull/49
what
• Add ip_set
variable in ip_set_reference_statement_rules
to create IP Set automatically
why
• If the IP set is not shared, there is no need to have separate from the WAF module • It allows a user of atmos to continue defining all their rules in yaml, otherwise they have to use tf code to forward the ARN to this module
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
we’ll review it, thanks
![sytten avatar](https://avatars.slack-edge.com/2019-03-01/564418310496_16e796635f1b40fcea36_72.jpg)
I fixed the readme
2023-08-17
![Christopher Pieper avatar](https://secure.gravatar.com/avatar/534a800903dbed158459a2349029435d.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0017-72.png)
So I see atmos is no longer doing a full stack / component output when running terraform commands.. How would I go about configuring atmos so that it does
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
Use the atmos.yaml
configuration file to control the behavior of the atmos
CLI.
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
set the log level Debug or Trace
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
![Zain Zahran avatar](https://avatars.slack-edge.com/2023-08-17/5782637359920_6364b8d5365cdf6522b4_72.png)
Hey all. I had 2 questions
- What is the best way to handle global or shared variables across different Atmos components? How do you combat warnings from Terraform about excess variables when some components only require a subset? In my stack I’m doing something like the following:
vars: environment: env_name foo: example1 bar: example2 components: terraform: component1: vars: {} component2: vars: {}
- When using the terraform s3 backend property, how come things are stored as component/stack and not stack/component in the aws bucket? Is there ways to change that or is it an important design choice?
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
- Globals are not ok in any language. Even if Terraform did not throw the warnings, by using a bunch of globals you would arrive at a big mess very soon. If you have a need to use variables that are common for a few components, you can create an abstract base component (mixin) and inherit from it in your components. By inheriting from the base component, all its variables and other settings will be available in the derived components. Atmos supports many types of component inheritance, see https://atmos.tools/core-concepts/components/inheritance
Component Inheritance is one of the principles of Component-Oriented Programming (COP)
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
- In the TF state S3 bucket, things are stored as component/stack b/c we use the component as the workspace key prefix and the stack as the workspace. So under the key prefix, you see the component name, and inside the folder you see all the workspaces (Atmos stacks) where the component is provisioned
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
@Zain Zahran if you need help with Atmos configurations, let us know, we could suggest some ideas and review your code
![Zain Zahran avatar](https://avatars.slack-edge.com/2023-08-17/5782637359920_6364b8d5365cdf6522b4_72.png)
- Got it. I figured as much. Glad to learn this. I had taken a brief look at the atmos’ abstract mixins before. I think that could prove useful, but also definitely the components and modules should be written in a way that enforces a specific set of vars. Passing in unused things would be bad practice.
- Understood, that makes more sense. @Andriy Knysh (Cloud Posse) sounds good I’ll let y’all know. Thanks for your help!!
![Release notes from atmos avatar](https://a.slack-edge.com/80588/img/services/rss_72.png)
v1.44.0 what Update Spacelift deps Update atmos validate stacks command Update docs Update tests why
Update Spacelift deps - use the new settings.depends_on schema (introduced for the atmos describe dependents command) to specify stack dependencies instead of the deprecated settings.spacelift.depends_on schema (note that the old settings.spacelift.depends_on schema is still supported for backwards compatibility)…
what
Update Spacelift deps Update atmos validate stacks command Update docs Update tests
why
Update Spacelift deps - use the new settings.depends_on schema (introduced for the atmos describe de…
This command produces a list of Atmos components in Atmos stacks that depend on the provided Atmos component.
![Release notes from atmos avatar](https://a.slack-edge.com/80588/img/services/rss_72.png)
v1.44.0 what Update Spacelift deps Update atmos validate stacks command Update docs Update tests why
Update Spacelift deps - use the new settings.depends_on schema (introduced for the atmos describe dependents command) to specify stack dependencies instead of the deprecated settings.spacelift.depends_on schema (note that the old settings.spacelift.depends_on schema is still supported for backwards compatibility)…
![party_parrot](/assets/images/custom_emojis/party_parrot.gif)
![johncblandii avatar](https://avatars.slack-edge.com/2020-04-14/1062347993890_6fd142c15ffef426eeba_72.png)
We have, what looks like, runs triggering based on dependencies (as expected), but the dependencies are not showing up in Spacelift.
Is this expected based on the latest Atmos version, @Andriy Knysh (Cloud Posse)?
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
check that the spacelift
component uses the latest version of yam-stack-config
which should use the latest version of the utils
provider which supports the latest version of Atmos (since it uses Atmos Go modules)
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
there are many things involved here, not only the latest Atmos CLI version (which is just used locally)
![johncblandii avatar](https://avatars.slack-edge.com/2020-04-14/1062347993890_6fd142c15ffef426eeba_72.png)
no, it is 1.0.0 not 1.2.0 of https://github.com/cloudposse/terraform-spacelift-cloud-infrastructure-automation/releases
![johncblandii avatar](https://avatars.slack-edge.com/2020-04-14/1062347993890_6fd142c15ffef426eeba_72.png)
we’re on this version of the refarch: https://github.com/cloudposse/terraform-aws-components/blob/main/modules/spacelift/admin-stack/child-stacks.tf#L47
version = "1.0.0"
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
there were so many diff updates to the Spacelift modules and components by many people to add many changes (some of them breaking), I can’t say from the top of my head which version needs to be used
![johncblandii avatar](https://avatars.slack-edge.com/2020-04-14/1062347993890_6fd142c15ffef426eeba_72.png)
ok
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
I know that the latest version of yaml-stack-config/spacelift
module needs to be used which uses the latest version of the utils
provider
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
![johncblandii avatar](https://avatars.slack-edge.com/2020-04-14/1062347993890_6fd142c15ffef426eeba_72.png)
yeah, we’re stuck in a weird loop of the policy triggering dependencies
![johncblandii avatar](https://avatars.slack-edge.com/2020-04-14/1062347993890_6fd142c15ffef426eeba_72.png)
we need to audit our dependencies for sure
![johncblandii avatar](https://avatars.slack-edge.com/2020-04-14/1062347993890_6fd142c15ffef426eeba_72.png)
just an example
2023-08-29
2023-08-31
![Release notes from atmos avatar](https://a.slack-edge.com/80588/img/services/rss_72.png)
v1.45.0 what Add dependencies on external files and folders to settings.depends_on Fix context inheritance in Atmos imports with Go templates with Multiple inheritance Add/update docs why
Add dependencies on external files and folders to settings.depends_on. Allow specifying that an Atmos component depends on external files or folders. When the files or folders are modified, the command atmos describe affected will detect that and include the dependent component to the affected output list. This is…
what
Add dependencies on external files and folders to settings.depends_on Fix context inheritance in Atmos imports with Go templates with Multiple inheritance Add/update docs
why
Add dependenc…
![Release notes from atmos avatar](https://a.slack-edge.com/80588/img/services/rss_72.png)
v1.45.0 what Add dependencies on external files and folders to settings.depends_on Fix context inheritance in Atmos imports with Go templates with Multiple inheritance Add/update docs why
Add dependencies on external files and folders to settings.depends_on. Allow specifying that an Atmos component depends on external files or folders. When the files or folders are modified, the command atmos describe affected will detect that and include the dependent component to the affected output list. This is…