#terraform (2023-01)

use users, groups and permission sets This generally works well when you have one or two accounts, but more than that and you’re gunna start running into management issues. Operations will have a time keeping that list curated as people come and go. Security would be cumbersome. As a user, you’d have to maintain multiple API keys and passwords. Switching accounts would suck.

It’s easier to manage policies and roles than users, so just deploy those via TF into the accounts.
use users, groups and custom roles across accounts IMO, a much better way to travel. One account that manages only the users. Accessing accounts is much cleaner since you have a single set of credentials but n roles to leverage. This helps curate a least privileged access across the board. You can move around accounts from within the UI using the roles, this means you can log into the account and then have a collection of links to different accounts so that you can seemlessly navigate around. If a users creds get compromised, the threat can be reduced to, it doesn’t really matter unless they know what roles they have access to and what accounts they can access. (I am forgetting if the user needs permission to list roles to assume a role).

@aimbotd thanks for the feedback. is this all related to users and groups with AWS SSO (IAM Identity Center)

You can use a life cycle ignore definition in terraform to ignore those changes

Apply the ignore to the task_definition, you mean?

on the ecs_service I can add ignore task_definition

seems to work for now

thanks

does that 40 USD/apply is directly proportional to the 15,000? doing the math is 600K USD, is that correct?

btw happy new year man! glad to hear from you!

@Giomar Garcia haha hey brother, glad to hear from you as well. Yep, you got it. Literally $40/apply => $600K USD.

Drift detection is important (terraform cloud has this), beyond that it’s a matter of wack-a-mole till a plan has no changes. Sometimes this will be a state rm and import, sometimes a state mv, but hopefully it’s just changing config to align. I haven’t found a better way across hundreds of states. Make a liberal usage of —target on your plans to limit their scope (increase speed).

@Chris Dobbyn Drift detection is important. So for you this isn’t an idealism? By this I mean you have an alerting system setup where your TACO runs and posts to a slack channel or whatever every time state drifts? Or you have drift detection setup to auto-resolve back to what’s in state storage? I’m definitely not a fan of the latter because I have repeatedly found that what happens in that case is people learn to shut off the --auto-resolve feature and it becomes drift detection in name only and then things get out of control.

I really appreciate the state surgery takes above because I’m thinking that’s really my only option!

Drift detection will detect and notify, I do not trust it to resolve.

Hm. Well in spacelift I don’t think it notifies out of the box, I believe you have to hand-jam a webhook, effectively. I think it’s the same for TFC, but I could be wrong there.

So in terraform > 0.15.3 there is terraform plan -refresh-only which will apparently only STDOUT drift. Fyi.

Hi @Jonas Steinberg I’m a noob in this and don’t have any solutions to your question but I’m curious as to how this drift occurs

From a very simplistic view, I’d assume if the state is hosted remotely, a few people given access to the state and TF server and only the TF server has credentials to apply changes, there shouldn’t be any drift

by a few people, I mean the team, and since the state is shared, then each change is applied from a single point/ state

@David There is a difference between theory/ideology and practice. In theory and ideology yes state, as you’ve implied above, should never drift. In practice, “humans gonna human” and will change stuff in the cloud console or wherever for various reasons either in an outage, because they didn’t understand how those cloud objects were created, but still have access to the cloud console via federation or because people were “testing stuff” and forgot to revert.

It is possible to have orgs without drift. One such department at my current shop has no drift and they’re a large group; they have a sibling department that is much smaller and has tremendous drift.

If you work with terraform for any length of time you’re going to encounter drift.

Here are some resources in case you’re interested in learning about what it is, how it happens and how to deal with it:

1. https://developer.hashicorp.com/terraform/tutorials/state/resource-drift
2. https://www.hashicorp.com/resources/the-state-of-infrastructure-drift-what-we-learned-from-100-devops-teams
3. https://developer.hashicorp.com/terraform/cli/commands/plan#planning-modes
4. https://developer.hashicorp.com/terraform/tutorials/state/refresh
5. https://developer.hashicorp.com/terraform/cli/state The most important aspect of recovering from drift is understanding how to safely manipulate state.

Oh, very true, thanks a lot for the clarity

and awesome saying, “humans gonna human”

also, AWS changes/improves their API all the time, terraform changes their API often, some things go obsolete or depreciated, some things that are in TF state change inside AWS (e.g. a resource state changes and TF sees the diff), with many stacks you will always have drift almost every day, and that’s a big issue b/c all of those reasons require diff approaches to reconcile the drift

oh, true, thanks for the info @Andriy Knysh (Cloud Posse)

did some research on generating terraform code from existing infra, seems there are a number of articles on this, but no real solution except for terraformer for GCP

the above might be helpful @Jonas Steinberg

https://github.com/GoogleCloudPlatform/terraformer/blob/master/docs/aws.md

@David Thanks. I’m very familiar with terraformer. FYI GCP has a native --export-to-terraform command that can be used as a bulk export operation so terraform isn’t even needed.

1. I am sure you guys know all of it so here is my bit, if there is any mistake I would happily like to see your comments so I can learn more or correct misconception ; 1. Understand the current state of the resources across all environments: Before attempting to recover from drift, it is essential to understand the current state of the resources in each environment. This can be done by using Terraform’s “terraform state show” command to view the state file for each environment. This will give an overview of the current state of the resources and their configurations in each environment.
2. Identify and document the resources that are drifted: After understanding the current state of the resources, the next step is to identify and document the resources that have drifted. This can be done by comparing the state files for each environment against the configuration files that were used to create the resources in the first place.
3. Create a plan of action: Once the resources that have drifted have been identified, the next step is to create a plan of action to recover from the drift. This plan should include a list of the resources that need to be updated, the desired state for each resource, and the steps that will be taken to bring the resources back to their desired state.
4. Use Terraform import command to recover from drift: The terraform import command can be used to safely manipulate state in order to recover from drift. The import command can be used to update the state file with the desired state of the resources. This can be done by providing the import command with the resource address and the id of the resource. For example: “terraform import <resource_address> <resource_id>”
5. Test the resources in a staging environment: Before implementing the plan in all environments, it’s a good practice to test the changes in a staging environment first. This will allow you to verify that the changes will work as intended and fix any issues that may arise before implementing the plan in the production environment.
6. Implement the plan in all environments: Once the changes have been

terraform state list It will return a list of all the resources that terraform is managing. If you find that a specific resource has drifted, you can use the terraform import command to update the state file to the desired state. For example, to import an S3 bucket with the resource address aws_s3_bucket.example and the resource ID example-bucket, you would use the following command: terraform import aws_s3_bucket.example example-bucket - screeshoot as it didn;t fit the line here;

In this example, it’s creating a plan of action for the S3 bucket resource that has drifted. To create the plan, you might want to take a look at the current state of the resource, so you can check what changes to make to reach the desired state of the resource.

Once you know what changes need to be done to bring the resources back to the desired state, you can test the changes in a staging environment before implementing them in production. For example, you can use the terraform plan command to check what changes will be made to the S3 bucket resource: Then apply terraform apply when confident that its good to go. After implementing the plan, you should continuously monitor and maintain the resource to ensure that it remains in the desired state. You can run the terraform plan command periodically to check for drift and terraform apply if necessary. Also, before importing it is always a good idea to make a backup of your state file in case things went wrong you can roll back to a working state: terraform state pull > tf-state-backup.json

Please for you guys who got more experience is this process correct?

Seems like it’d be nice for managing user on- and off-boarding at a minimum. Especially if you’re using it for auth for external services

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lb_target_group#stickiness

Thank you Warren, if this helps others using this module - great.

target_groups = [
    {
      .
      .
      .
      stickiness = {
        type            = "lb_cookie"
        cookie_duration = 86400
        enabled         = true
      }

      health_check = {
        healthy_threshold   = 2
        unhealthy_threshold = 2
        timeout             = 15
        interval            = 6
        path                = "/healthCheck"
        port                = 80
        matcher             = "200"
      }
    }
  ]

• To configure an Application Load Balancer (ALB) in Terraform to set Target Group attributes such as stickiness, you’ll need to use the terraform-aws-alb module. This module allows you to create an ALB and its associated resources, such as listeners and target groups, in Terraform. • To use the terraform-aws-alb module, you’ll need to:

1. add this module to your terraform configuration file [main.tf](http://main.tf), you can use the following command: • module “alb” { source = “terraform-aws-modules/alb/aws” version = “2.2.0”
2. then configure the attributes of the target group according to your needs • resource “aws_alb_target_group” “example” { name = “example” port = 80 protocol = “HTTP” vpc_id = aws_vpc.main.id stickiness { type = “lb_cookie” enabled = true duration = 1800 } • name: is the name of the target group • port: the port the target group listens on • protocol: the protocol the target group uses • vpc_id: the VPC the target group belongs to ◦ stickiness: the stickiness attributes of the target group,type: determines the type of stickiness that is used. In this case, it’s “lb_cookie”, which uses a cookie to ensure that requests are sent to the same instance. ◦ enabled: enable stickiness for the target group ◦ duration: the time, in seconds, for which requests are sent to the same instance.
3. Add instances to the target group • resource “aws_alb_target_group_attachment” “example” { target_group_arn = aws_alb_target_group.example.arn target_id = aws_instance.example.id port = 80 } • target_group_arn: the ARN of the target group to which the instances are added. • target_id: the ID of the instances that are added to the target group. • port: the port the instances listen on.
4. Define a listener on the ALB that forwards requests to the target group • resource “aws_alb_listener” “example” { load_balancer_arn = aws_alb.example.arn protocol = “HTTP” port = 80 default_action { type = “forward” target_group_arn = aws_alb_target_group.example.arn } } • load_balancer_arn: the ARN of the ALB • protocol: the protocol the listener uses • port: the port the listener listens on ◦ default_action: the action that is taken when a request is receivedtype: the type of action. In this case, it’s “forward”, which forwards requests to the target

Not really a good way to do this natively in terraform. There’s a shell provider that might do the job

https://registry.terraform.io/providers/scottwinkler/shell/latest/docs

The way most people do this is with their workflow automation tool.

You can use atmos for this or #terragrunt or things like Makefiles

thanks @Erik Osterman (Cloud Posse) we have some controls already in Jenkins before triggering plan/apply. But there are some folks using VCS directly to connect to Terraform Cloud. So I was wondering how to bake in controls on VCS runs. sounds like we have to force the VCS folks to use workflow automation tool like jenkins

bash script

I have implemented drift detection in my org. The flow is:

• there is a script(bash) which gets the module name, account name, product (unique) name and send a asynchronous call to lambda.

• Lambda received the payload and runs the terraform plan and stores the plan output in an s3 bucket which is rewritten everyday. (the drift detection framework runs everyday). Our lambda code is written using custom runtime (https://docs.aws.amazon.com/lambda/latest/dg/runtimes-custom.html)

and how do you deal with the drift?

what is the User interaction if any

planning and applying so it will remove any drifts. There are some places where there is a genuine need of change so that is coded in terraform regularly based on reviews

I see ok so it auto-remediated it , cool

We do in spacelift, does that count? It shows up as a proposed terraform run that should get applied. Separately, we have policies to auto confirm changes in non-production environments. So that would then auto remediate.

it counts, any product workflow is welcome in this thread

if you’re using github actions to run terraform, you can extend it to include drift detection on a schedule. I wrote about it here https://www.taccoform.com/posts/tfg_p3/

you also don’t need github actions and can use the same solution in your preferred automation system as a cron-like task

A scheduled plan is how we do it also, using -detailed-exitcode to fail the run if any drift is detected

interesting

we are planning on adding this to Atlantis and I’m gathering ideas to implement it

I used https://driftctl.com/ and pass down multiple state files, across all project stacks, seems promising knowing that it still have a big list of un-supported services, but for the basics its working really well

A scheduled plan is how we do it also, using -detailed-exitcode to fail the run if any drift is detected @loren how are you reporting on it? Are you just relying on status check badges (:x: ) in the commit log against main or did you do some other instrumentation?

nothing so fancy… no “dashboard” where we’d get a visual for such a failed status check… no, just a webhook to a slack channel. job runs in the ~morning, so we see it as we sign in

Makes sense - that’s a simple enough solution

Jenkins job that runs on scheduled and goes through terraform repository looking for version files. The existense of such file means this is a location for a terraform plan. For each folder where it finds that file, it runs terraform plan -detailed-exitcode and sends a message to Slack with summary for for failed or drifiting plans

I know way less about it than everyone here but researching that I found the following below don’t know if it is usefull..;

The issue is that karpenter_instance_family variable is defined as a list of strings, and the provisioner resource yaml_body is using it as list without any manipulation, this is causing an issue in the yaml formation and it will not work. To fix this issue, i would suggest to modify the resource yaml_body, you need to convert this variable karpenter_instance_family list into string and then add it to the yaml_body. Here is an example of how you can fix it.

resource “kubectl_manifest” “default-pa” { yaml_body = <<YAML apiVersion: karpenter.sh/v1alpha5 kind: Provisioner metadata: name: default-pa spec: requirements: - key: karpenter.k8s.aws/instance-family operator: In values: [${join(“,”, var.karpenter_instance_family)}] limits: resources: cpu: 1000 provider: launchTemplate: ‘karpenter’ subnetSelector: kubernetes.io/cluster/karpenter: ‘*’ labels: type: karpenter provisioner: default-pa ttlSecondsAfterEmpty: 30 ttlSecondsUntilExpired: 2592000 YAML }

variable “karpenter_instance_family” { description = “Instance family” type = list(string) default = [] }

In case it is confuse above ; To fix the issue, I added the [${join(“,”, var.karpenter_instance_family)}] statement in the values field. This statement is used to join the list of values in the variable “karpenter_instance_family” and separate them with a comma. This ensures that the values are passed as a list of strings instead of a single string, which is the correct format for this field. It will make sure that the module is going to work with the correct input and match the request format the variable should have been in.

thank you , it worked both ways

Let me know if you have more questions, its good because give me more scenarios that I can learn from as well.

Sorry, not sure on the best place to get eyes on this PR.

approved and merged, thanks @David Lundgren

a good resource https://cloud.google.com/docs/terraform/best-practices-for-terraform

Thank you very much!

this isn’t official support, so we can’t answer questions like “will this issue be addressed?”

I think the cloudposse team is pretty quick to review PRs. Might be worth asking it to be assigned to you and then just submitting a PR with the proposed changes.

lifecycles cannot be related to an input, afaik

i got a working example going with something like:

variable "destroy" {
  default = false
}

resource "aws_security_group" "safe_sg" {
  count = var.destroy ? 1 : 0
  lifecycle {
    prevent_destroy = true
  }
  name        = "allow_tls"
  ...
}

resource "aws_security_group" "unsafe_sg" {
  count = var.destroy ? 0 : 1
  lifecycle {
    prevent_destroy = false
  }

  name        = "allow_tls"
  ...
}

but now trying to take into a s3 is fun

oh I thought you were trying to do something like prevent_destroy = var.some_input

yeah that failed real quick! it was trying to find a workout around since that doesnt work..

yes, if you instantiate a completely new resource with a hard coded prevent_destroy then it should work.

sorry properly poorly worded

would a local like this work?

locals {
  security_group_id = var.destroy ? one(aws_security_group.safe_sg).id : one(aws_security_group.unsafe_sg).id
}

output "security_group_id" {
  value = local.security_group_id
}

Generally, trying to run procedural things like this inside Terraform is an anti-pattern. It’s not designed for this and leads to pain like you’re experiencing.

Instead, I’d suggest you wrap your build process in a script like ./deploy.sh which does:

../build-lambda.sh
terraform apply -var lambda_zipfile=../dist/lambda.zip

Then Terraform can statically know the etag and so on.

We already got a deploy script wrapping the terraform actions. Interesting idea to move the build out.

same here on terraform cloud with aws provider on 4.50.0

https://github.com/hashicorp/terraform-provider-aws/issues/17860

also @Soren Jensen I might need your help as it seems that we write the same code

Have you seen terraform-aws-components repo? Thats how all the root tf directories are structured

https://github.com/cloudposse/terraform-aws-components/tree/master/modules

You may also want to look at https://atmos.tools since its helpful to separate configuration from code

if the module doesn’t output it, you can’t get it

what are all the outputs of the module, could you try using one with an AWS data resource to pull it back in?

you could kind of thought Do a datalookup based on the instance arn or id and take it from there ? ]]

aw, great minds think alike @Warren Parad

Thanks for the suggestions!

Could you create an issue with the appropriate inputs you’re providing it?

I created an issue in the module: https://github.com/cloudposse/terraform-aws-ec2-client-vpn/issues

The client vpn endpoint uses another module to create the ssl cert btw

I wonder if this issue is in a recent aws provider version ?

Yeah.. I inspect the child module and never sets the domain, Probably should be set somewhere here? https://github.com/cloudposse/terraform-aws-ssm-tls-self-signed-cert/blob/master/acm.tf

here’s the example that is run by terratest btw

https://github.com/cloudposse/terraform-aws-ec2-client-vpn/blob/master/examples/complete/main.tf

kicked off here too https://github.com/cloudposse/terraform-aws-ec2-client-vpn/pull/56#issuecomment-1387290569

if this succeeds, then the module is still working as expected https://github.com/cloudposse/actions/actions/runs/3950558796

Once I set these values: ca_common_name , root_common_name , server_common_name as shown in the example it works, however those values have a default in place. Maybe the module should enforce those inputs? I commented back into the issue for visibilty: https://github.com/cloudposse/terraform-aws-ec2-client-vpn/issues/57

Nope, that’s what we all do

Somehow I’ve never had to migrate state until now. Been lucky I guess. Lucky again with it’s simplicity.

It’s considered complicated because mistakes can cause massive problems. Also, I think what’s generally considered complicated is moving resources within state

even thats not actually complicated but more like a royal pain in the **

State migrations are not fun and can be error prone. The import commands in the documentation are also incomplete, so there’s extra fumbling there

I’ve blogged a little bit about these areas https://www.taccoform.com/posts/tfg_p1/

https://www.taccoform.com/posts/tfg_p4/

AWS proton does this

Oh yes, but we use multicloud env hence why we need a self served platform, however the focus is actually on getting terraform variable values from redisdb idea, whether it makes sense or not or if there are other smarter ways to go about it

Fwiw, we do something similar with atmos. All configuration is defined as YAML and then atmos writes the .tfvar.json files which get passed to terraform. We don’t yet have a UI, as we’ve optimized more for a gitops workflow. Is the UI the selling point for this? or automation?

https://atmos.tools/core-concepts/stacks/

Yes, the UI is the selling point, the automation bit has pretty much been figured out, just a way to slap a dashboard to it

EB is terrible, Lambda always and sometimes ECS, you never need anything else

migrate to lambda

Thank you

it looks like your stack is not configured with a backend

jq: error: Could not open file backend.tf.json: No such file or directory

Another win for reading the error message

Maybe better for spacelift channel

@Christopher Pieper are you using atmos with spacelift?

So I noticed the jq error after sharing, and fixed.. Still having issue.. And yes we are using atmos with spacelift..

I was thinking this was a terraform issue, re: workspace state mgmt

@Dan Miller (Cloud Posse) @Andriy Knysh (Cloud Posse)

if in atmos.yaml you have

auto_generate_backend_file: true

then you need to execute a command to generate the backend for the component

in .spacelift/config.yml you need to have something like this:

in spacelift-configure script you need to have this:

hmmm so what I have in place currently is the three scripts found in the terraform-aws-components/modules/spacelift/bin spacelift-configure spacelift-tf-workspace spacelift-write-vars

setup to run in that order.. I will give this a shot and report back with results momentarily

check that you have atmos generate backed command in your script

the number of scripts doesn’t matter

wasn’t suggesting that they did.. just providing context as to what is in place currently.. from the looks of it your script is consolidation of those 3

so that appears to have broken past the issue

thinks appear to be working now

thank you SO MUCH

So I made the suggested modifications to spacelift-configure and it seemed to work, but once again I am running into same issue.. It appears that terraform workspace is somehow screwing with the backend setup.. I am not sure.. Been searching the google for anything similar and nothing is jumping out at me.

not clear why it’s looking for the backend file in the .terraform folder (which is def should not be)

please show what you have in .spacelift/config.yml file

so I don’t seem to have a .spacelift/config.yml file

everything seems to work if I add the “nobackend” label to the stack

I’m not sure that is a correct way of doing things. This is a working example of the file, you can try adding it to the repo

version: "1"

stack_defaults:
  before_init:
    - spacelift-configure

  before_plan:
    - spacelift-configure

  before_apply:
    - spacelift-configure

  environment:
    AWS_SDK_LOAD_CONFIG: true
    AWS_CONFIG_FILE: /etc/aws-config/aws-config-spacelift
    ATMOS_BASE_PATH: /mnt/workspace/source
    TF_CLI_ARGS_plan:  -parallelism=50
    TF_CLI_ARGS_apply:  -parallelism=50
    TF_CLI_ARGS_destroy: -parallelism=50

stacks:
  infra:
    before_init: []
    before_plan: []
    before_apply: []

where do I add it? rootfs?

.spacelift/config.yml in the root of the repo, not in rootfs

ok

https://docs.spacelift.io/concepts/configuration/

and this will need to be deployed by pushing the docker container up to artifacts ?

Spacelift reads the file when it downloads the repo

so the file needs to be in the repo, not in the container

ok just making sure…

so no luck with that chnage

in cases like that (when it’s not easy to understand what’s going on), I usually say please zip up the repo and DM me - I could look if there is something obvious

honestly I would be tremendously grateful.. let me get that for you

maybe use #terragrunt

use alias option in provider file and use it to create resource in another region

We use atmos to reuse root modules and expose the region as an input

Then we use tfvars that are populated from yaml and pass it to the root module in a custom region-account workspace e.g. ue1-dev, ue2-prod, etc

Here is the website https://atmos.tools/ and the repo if you’re interested https://github.com/cloudposse/atmos

Thank you @Andriy Knysh (Cloud Posse)! worked without the OR part:

locals {
  newlist = var.aaa == null ? var.bbb : concat(var.bbb, [var.aaa])
}

otherwise if I include the OR

│ Error: Invalid function argument
│ 
│   on main.tf line 27, in locals:
│   27:   newlist = var.aaa == null || contains(var.bbb, var.aaa) ? concat(var.bbb, [var.aaa]) : var.bbb
│     ├────────────────
│     │ var.aaa is null
│ 
│ Invalid value for "value" parameter: argument must not be null.

this newlist = var.aaa == null ? var.bbb : concat(var.bbb, [var.aaa]) will allow duplicates added to the list

here

on main.tf line 27, in locals:
│   27:   newlist = var.aaa == null || contains(var.bbb, var.aaa) ? concat(var.bbb, [var.aaa]) : var.bbb

looks like you reverted the logic, and that’s why it’s not working

try

newlist = var.aaa == null || contains(var.bbb, var.aaa) ? var.bbb : concat(var.bbb, [var.aaa])

same error, I actually left the logic but removed your dupe check (the || part) which seemed to cause the problem, but, adding it as a list worked:

newlist = var.aaa == null || contains(var.bbb, [var.aaa]) ? var.bbb : concat(var.bbb, [var.aaa])

Your alerts require certs?

Could you explain more @Wil

They’re synthetic checks, one of them has a cert associated yes

see the request_client_certificate key and cert

Here’s where support was added

https://github.com/DataDog/terraform-provider-datadog/pull/711

Here’s this also https://github.com/cloudposse/terraform-datadog-platform/blob/3240c6488c777b96ebebcdc30727fdce045ae85a/modules/synthetics/main.tf#L173

I suppose request_basicauth would have the same issue here

The error says that it’s trying to save to a directory instead of a file

Try changing to a file

if i give directory. its shows same error

Try changing to a file

ex?

it shows this one

i have tried in many ways

but no progress if anyone can help that would be great help to me

It’s hard to say without being able to see the code

Looks like you’re trying to create a file in a read only directory

Try commenting it out and see if this example works

resource "local_file" "foo" {
  content  = "foo!"
  filename = "${path.module}/foo.bar"
}

https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file#example-usage

Spot instances can be interrupted and removed any time (although how often that happens depends on the instance types and many other factors)

you prob don’t want your NAT traffic to be interrupted

Thats…actually a super valid point that I dont know why I didn’t consider.

Thanks for the sanity.

This is worth looking into if you’re trying to cut nat costs

https://github.com/1debit/alternat

Thanks @RB

jsonencode ?

mmm that could work

the idea is to avoid hardcoding of values on the github action task definition and potentially discover those values from another source

one idea I had was to output my ECS module output to parameterStore and then search for parameter store and find the cluster base on a service name and all the values related to that service

for example

like this error :

 no matching Route53Zone found

Correct. Singular data sources will fail if none found. Plural data sources will return an empty list. This is specific to the provider, not general/enforced terraform core semantics

https://hashicorp.github.io/terraform-provider-aws/provider-design/#plural-data-sources

mmm ok, I was hoping this changed but nope