#refarch (2024-02)
Cloud Posse Reference Architecture
2024-02-12
![Matthew Reggler avatar](https://secure.gravatar.com/avatar/2cdf969f30a522e03ebbd1acf653fa49.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0022-72.png)
The CloudPosse style seems to prefer “single~ton~” components (e.g. lambda
component deploys a single lambda, sns-topic
deploys one topic). The exception to this is ssm-parameters
, I guess because one terraform workspace per parameter would be painful to access via remote-state and slow to deploy.
I am considering creating “plural” component variants to maximise DRYness and reduce workspace sprawl for a few deployments. For example, an sns-topics
component
# format TBC, not convinced the `try` is required if the "default/global" vars have null or empty defaults
module "sns_topic" {
source = "cloudposse/sns-topic/aws"
version = "0.20.1"
for_each = var.sns_topics
subscribers = try(coalesce(each.value.subscribers, var.subscribers), {})
...
kms_master_key_id = try(coalesce(each.value.kms_master_key_id, var.kms_master_key_id), null)
encryption_enabled = try(coalesce(each.value.encryption_enabled, var.encryption_enabled), null)
...
context = module.this.context
}
This *seems* like an OK idea to me, assuming that the resources have similar lifecycles, and in cases where you want to access outputs downstream in remote-state
it would make that YAML cleaner. I notice that CloudPosse don’t seem to ever do this in the core component library, perhaps there is a very good reason why not… Is there a downside to this approach?
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
@Matthew Reggler great to see you poke your head in! It’s been a while.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
We have a MASSIVE docs update coming that introduces our best practices on this.
what
• Update readme to be more consistent with atmos.tools • Fix links • Add features/benefits • Add use-cases • Add glossary
why
• Better explain what atmos does and why
references
• grnet/docusaurus-terminology#28 (cannot use use markdown links in terms) • grnet/docusaurus-terminology#29 (tooltips incompatible with slugs)
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Of course, there’s never a steadfast rule other than the laws of physics. Bend them as you need.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
The “singleton” component, is an interesting way of referring to them. I don’t think of them as singletons, in that sense since they are used anytime (multiple times) you need that particular thing. Calling it a singleton to me sounds more like moving the factory inside the component, which we recognize (after having done this for years now) is bad.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
We still have some public components that do this bad thing I discuss. Components like accounts
should be redesigned to only provision an account, not a factory of accounts.
![Matthew Reggler avatar](https://secure.gravatar.com/avatar/2cdf969f30a522e03ebbd1acf653fa49.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0022-72.png)
Thanks for the reply, and yes, still very much around . Agree with you on my misuse of singleton, not quite the right concept for what is encapsulated within a component really. I see your point about avoiding factories within a component, makes a lot of sense.
![Matthew Reggler avatar](https://secure.gravatar.com/avatar/2cdf969f30a522e03ebbd1acf653fa49.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0022-72.png)
I suppose I’m slightly too attached to the idea hooking groups of connected resources around a loop of some kind, and a terraform for_each
is slightly more digestable that something arcane and terrifying in a Go template
![Matthew Reggler avatar](https://secure.gravatar.com/avatar/2cdf969f30a522e03ebbd1acf653fa49.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0022-72.png)
I can see cases where I might bend the rules (e.g. a common set of sns topics + subscribers deployed per env for chatops/alerting — defined as a (meta-)component instead of a stack to batch the ARNs in one state file), but think otherwise you’ve convinced me there. Thanks!
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
for_each
is slightly more digestable that something arcane and terrifying in a Go template
Agree, if this is the result, we are not advocating using Go templating over doing it in HCL.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
We don’t want to replace one bad thing with an even worse one.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Go templating in stacks is an escape hatch.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
So in refarch v2, we’ll be doubling down on this pattern (not the go templating!!!)
but I want to make sure I understand why you need this loop over subscribers.
subscribers = try(coalesce(each.value.subscribers, var.subscribers), {})
...
kms_master_key_id = try(coalesce(each.value.kms_master_key_id, var.kms_master_key_id), null)
encryption_enabled = try(coalesce(each.value.encryption_enabled, var.encryption_enabled), null)
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
e.g. the pattern we want to ascribe to is each subscriber subscribes as part of getting provisioned.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Put another way. Imagine one component provisions a security group. Then the cluster component adds it’s rules to it. The database component adds it’s rules to it. The VPN component adds it’s rules to it.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Intead of, provision the cluster component, database component, vpn component. Then provision the security group component with a list of other components to permit.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Another example. Provision OU component. The provision an account component for each account, giving it a reference by name to the OU.
![Matthew Reggler avatar](https://secure.gravatar.com/avatar/2cdf969f30a522e03ebbd1acf653fa49.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0022-72.png)
To root this in actual context, I have two examples:
- we have a pre-SweetOps “notifications module” that deploys a series of sns-topics with env-specific Teams channels/PagerDuty endpoints with which they communicate, grouping this up as a single component allows the email/service endpoints for each subscriber to be injectable as a Go template string, and serves to make the component appear consistent (for good or ill) with our previous approach. (A bad reason for using a factory, I’d guess)
- we have a collection of lambdas whose ARNs are all fed into a Step Function to define the source for various steps, having these as one component allows for a single remote state lookup in the Step Function’s component. (A good? reason for using a factory)
![Matthew Reggler avatar](https://secure.gravatar.com/avatar/2cdf969f30a522e03ebbd1acf653fa49.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0022-72.png)
As for the pattern for using try(coalesce(<local>, <global>), <default>)
that is my attempt to be maximally transparent, and produce a maximally configurable component, in alignment with the general ethos of SweetOps. I accept that it is probably unnecessary in this case
2024-02-14
![johncblandii avatar](https://avatars.slack-edge.com/2020-04-14/1062347993890_6fd142c15ffef426eeba_72.png)
DynamoDB: Add support for the import_table block
https://github.com/cloudposse/terraform-aws-dynamodb/pull/120
A preemptive PR for refarch: https://github.com/cloudposse/terraform-aws-components/pull/981
![johncblandii avatar](https://avatars.slack-edge.com/2020-04-14/1062347993890_6fd142c15ffef426eeba_72.png)
@Dan Miller (Cloud Posse) is the terratest using an old TF version?
Keyword “optional” is not a valid type constructor.
https://github.com/cloudposse/actions/actions/runs/7905463589/job/21578153852#step<i class="em em-9"413
![johncblandii avatar](https://avatars.slack-edge.com/2020-04-14/1062347993890_6fd142c15ffef426eeba_72.png)
this failed, but i can run it locally with no prob:
https://github.com/cloudposse/actions/actions/runs/7905463589/job/21578153852
![johncblandii avatar](https://avatars.slack-edge.com/2020-04-14/1062347993890_6fd142c15ffef426eeba_72.png)
oh, yes. https://github.com/cloudposse/terraform-aws-dynamodb/blob/main/versions.tf#L2
what’s the approach to handling TF version updates?
required_version = ">= 0.13.0"
![Dan Miller (Cloud Posse) avatar](https://avatars.slack-edge.com/2021-08-12/2389147782305_5729c9d69c393852d209_72.jpg)
ah yeah it’s the Terraform version. Set versions to required_version = ">= 1.0"
and then add the terraform/1.0
label to the PR
![johncblandii avatar](https://avatars.slack-edge.com/2020-04-14/1062347993890_6fd142c15ffef426eeba_72.png)
![johncblandii avatar](https://avatars.slack-edge.com/2020-04-14/1062347993890_6fd142c15ffef426eeba_72.png)
I already pushed the versions
![johncblandii avatar](https://avatars.slack-edge.com/2020-04-14/1062347993890_6fd142c15ffef426eeba_72.png)
Done
![johncblandii avatar](https://avatars.slack-edge.com/2020-04-14/1062347993890_6fd142c15ffef426eeba_72.png)
…the label, that is
![Dan Miller (Cloud Posse) avatar](https://avatars.slack-edge.com/2021-08-12/2389147782305_5729c9d69c393852d209_72.jpg)
approved and merged
![johncblandii avatar](https://avatars.slack-edge.com/2020-04-14/1062347993890_6fd142c15ffef426eeba_72.png)
sweet
![johncblandii avatar](https://avatars.slack-edge.com/2020-04-14/1062347993890_6fd142c15ffef426eeba_72.png)
https://github.com/cloudposse/terraform-aws-components/pull/981 is ready now, but it is pending a run (unsure how that gets triggered)
what
• Added DynamoDB import_table support
why
• This adds support for using S3 data to import into a new table
references
• Depends on cloudposse/terraform-aws-dynamodb#120
![Dan Miller (Cloud Posse) avatar](https://avatars.slack-edge.com/2021-08-12/2389147782305_5729c9d69c393852d209_72.jpg)
I’m not sure what the deal is with run-pre-commit-checks-and-autocommit-changes
. Looks like it’s a required check we’ve added to the Organization but may be broken. I pinged the team about it
![Dan Miller (Cloud Posse) avatar](https://avatars.slack-edge.com/2021-08-12/2389147782305_5729c9d69c393852d209_72.jpg)
resolved and merged