#terraform (2021-05)

How does that differ from dynamic subnets?

How does that differ from dynamic subnets? That’s actually one of the documentation issues https://github.com/cloudposse/terraform-aws-multi-az-subnets/issues/23

nice, didn’t even know about this aws feature

Yeah save yourself re-implementing the wheel.

There is an #atlantis channel if you have any questions.

We use terraform output. Note that it parses the state file… you shouldn’t do it yourself

people also push to key/value stores directly, instead of outputs. for example, there is a consul provider… https://registry.terraform.io/providers/hashicorp/consul/latest/docs/resources/keys

aws parameter store is also popular. or s3, or dynamodb…

+1 for SSM param store

thanks gents.

@loren we’d considering adding to dynamodb (don’t ask) just this way. just seems…laborious? but don’t think there’s an easy way out of it

Haha the dynamodb item syntax is annoying, could use some sugar

We have a keystore module that tries to be a wrapper for this kind of thing… https://github.com/plus3it/terraform-aws-tardigrade-keystore

+1 to using another service to store your Terraform outputs. It lets you decouple the consumer and use another technology.

We use Terraform outputs purely as diagnostic help in our CD logs. Nothing automated reads them, even other Terraform configurations

@Alex Jurkiewicz meaning using a terraform resource to store the results, correct? This has the definite advantage as well of not having to pipe module output up the stack, effectively making one declare it multiple times. thx

sorry, I think terraform outputs are useful to send data from a module to the calling Terraform configuration. I meant “outputs” as in top-level outputs only

i see - thx. just trying to figure out the easiest way to gather and store these outputs. seems it’s either parse outputs and push them to an external service, or use a terraform resource to store them in one..

Hey Petro, I’d open an issue if nobody response and try digging in yourself. If the type is S3 then you should look at the corresponding resource that is failing and look where you need to add a new variable / value to provide the bucket name / ARN.

hey @Matt Gowie, thanks for the input. noob question, but are you referring to modifying the modules that are imported to resolve the issue? or are you mentioning that within terraform-aws-ecs-web-app

the ecs-codepipeline module has a cache_type variable which defaults to ‘S3’. ecs-web-app doesn’t set it explicitly or expose it as a variable. @Petro Gorobchenko you could open a PR to make this change, if you are able.

sounds good. Ill play around with it and see if I can create a PR for it.

added a PR, unsure if I’m missing any steps for the process. https://github.com/cloudposse/terraform-aws-ecs-web-app/pull/147

hello, you say you want to use a list of map and you are using a map of map.

it will be simplier with this

variable "role" {
  type        = list(any)
  description = "The permission block for roles assignment"
  default = [
      {
      scope                = ""
      role_definition_name = "Reader"
      principal_id         = ""
    }
    ]
 }

then you can use a for_each for loop to go through it

@matt

Sure, I’ll look into that

Although I think it won’t be possible right now as discussed in this issue

Oh interesting. @dansimau used to be at uber. He wrote https://github.com/uber/astro. I guess that’s why development stalled on it.

I’m using a queue policy with:

      "Action": [
        "sqs:SendMessage",
        "sqs:ReceiveMessage",
        "sqs:DeleteMessage"
      ],

does it need Send to read SQS messages?

Probably not, but my app uses the same perms for producer and consumers

You may need KMS key access too if you are using a cmk

Huh -replace is interesting. Figure it’s the same as doing a terraform state rm $RESOURCE && terraform apply -target=$RESOURCE. I’m sure I’ve done that… but not too many times.

more like ‘terraform taint <resource>; terraform apply’

doing state rm doesn’t destroy the resource, you’d have a conflict with anything that had the same names/ids on the apply

From the hangops slack…

Ah yeah, that is my goof in regards to rm > taint

Is there a big presence of terraform folks in the hangops slack? I haven’t poked my head in there in a couple years.

Mostly just apparentlymart, with any regulatory, but that’s just enough to be useful

Yeah I could see just him being around being a big benefit.

I muted every other channel there, that slack has a way low signal:noise otherwise

terraform and aws channels are incredibly helpful on that slack though

You can try doing the following:

 query               = <<-EOT
logs("service:service-name platform:platform-name environment:prod status:error env:env-name region:us-east-1").index("*").rollup("count").last("10m") > 50
EOT

I believe that will do it.

Or use the https://github.com/cloudposse/terraform-datadog-monitor module and define your monitors via YAML, which is great.

Yup that worked. I had tried to use the heredoc style, but i got an error because i put everything in the same single line. This works however. Thank you so much!

And the reason why i cannot use cloudposse’s module is because of the yaml configuration. I will have to reference that for my monitor values. And almost all the remaining monitors/infra that we have uses tfvars in its native form.

I usually start by TG_LOG=DEBUG and running plan to observe the logs

@SecOH not totally sure. I’m not sure if the AMQ service requires an outside connection to do updates or similar… but I would guess so. If you want to put up a PR to add an additional _enabled bool flag which disabling adding the egress rule then we’d be happy to review and get merged I’m sure.

@Matt Gowie Happy to get your reply, Thank you. I will make a PR if I get some free time.!

@greg n GH issue or a small PR to help support would be much appreciated!

For stuff like that I usually set a var from a bash script in my pipeline before terraform runs and have it conditionally create based off that

interesting, do you have an example?

you would have a variable say a bool create_lambda and only create if that var is true before terraform runs for ecs you could do something like

aws ecr list-images --repository-name anchore | jq -r '.imageIds[].imageTag' | grep -w -q 1.0.0 && export TF_VAR_create_lambda=true || export TF_VAR_create_lambda=false

so if the tag 1.0.0 exists it sets the var to true otherwise false

i’m using TF Cloud, does that make a difference?

idk ive never used TF cloud

I’ll figure it out. Thanks for the tip!

That will give you an external data source that won’t fail if an AMI isn’t found.

data "external" "imagebuilder_ami" {
  program = ["bash", "${path.module}/files/jq-ext-latest-imagebuilder-arn.sh"]
  query = {
    JQ_AWS_REGION = var.AWS_REGION
    JQ_IMAGE_NAME = "${local.full_name}-ami-builder"
  }
}

amazing, i’m going to look at this over the weekend

and I use it l like this:

image_id  = length(keys(data.external.imagebuilder_ami.result)) > 0 ? data.external.imagebuilder_ami.result.image : data.aws_ami.ubuntu.id

Hey @Paul Robinson — I don’t believe that was discussed. More just a unilateral decision. But your point about gateway load balancers in your final comment seems like a valid one one to me.

@Jeremy G (Cloud Posse) can you please review Paul’s comment and discuss? If the functionality is disabled by default and it’s supporting a valid use-case with newer AWS patterns then I don’t see why we would turn away an eager contributor who is willing to implement.

Thanks both. Yeah it was one of the reasons for choosing this module.

Private subnets without routes to nat gateways are standard with the advent of transit gateway and the gateway load balancer.

Linking again to an aws blog with reference VPCs. https://aws.amazon.com/blogs/networking-and-content-delivery/introducing-aws-gateway-load-balancer-supported-architecture-patterns/

It doesn’t seem like the existing module is incompatible here.

@Paul Robinson Thank you for bringing this up.

First, this pattern of using a Gateway Load Balancer appliance is new to me, and AFAIK, new to Cloud Posse. In general, as the saying goes, we like to “eat our own dog food”, which means we only publish modules and features that we use so we can design the features appropriately and ensure that the module works in practice. This is not a hard-and-fast rule, but rather a reflection of our values. We do often accept contributions to modules that add features we have not used and do not plan on using, and we are grateful for the community support in enhancing the modules to be more useful to more people.

We recently had an internal discussion on the topic of private subnets without gateways due to a different PR and decided not to support them, because it adds a surprising amount of complexity to a module to make them completely optional. This was also in part due to the fact that we had never seen a use case for them and did not contemplate one, so this was not dog food we were ever going to eat. Thank you for educating me on this emerging use case; I will keep it in mind in future PR reviews.

Also, it was not at all clear to me that part of the enhancement you were seeking with your PR was enabling private subnets without NAT. I did not think it was important to you, I thought you were just trying to generalize.

We have 3 modules for creating subnets: • terraform-aws-named-subnets • terraform-aws-dynamic-subnets • terraform-aws-multi-az-subnets As far as I can recall, for the past 2 years we have only used terraform-aws-dynamic-subnets in client engagements, which makes me personally (and this is definitely me and not Erik or Cloud Posse in general) less interested in maintaining or enhancing the other 2 modules, because we have limited resources and are having trouble keeping up with all the PRs across all our modules, so I would rather we not try to keep 3 modules whose differences are difficult to articulate. (See, for comparison, our decision to deprecate and eventually freeze terraform-terraform-label in favor of terraform-null-label ).

Furthermore, when it comes to creating subnets, if you are only creating private subnets without gateways, I personally (and again, not speaking for Cloud Posse) do not see much point in using a Terraform module. It is easy enough to do just using the AWS provider directly and the Terraform built-in function cidrsubnet.

So that is the long story behind the terse
We are not going to support the use case of private subnets without NAT gateways, at least not in this module. I suggest you look at terraform-aws-named-subnets if you want to create private subnets without gateways. If that doesn’t work for you, then we can discuss what you need and how to get it done in the most appropriate way.

@Paul Robinson @Matt Gowie https://github.com/cloudposse/terraform-aws-multi-az-subnets/pull/51

Thanks @Jeremy G (Cloud Posse) for your detailed and insightful response and also your PR.

I can appreciate the pain of maintaining so many modules and the need to deprecate the lesser used/older ones. I also appreciate the time you spend building community and encouraging contributions :raised_hands:

I found a few people asking about the differences between the various subnets modules in this channel. Indeed, there is an open issue on the terraform-aws-multi-az-subnets module. Perhaps I can add my thinking which informed my choice of module.

I did also consider the terraform-aws-dynamic-subnets and terraform-aws-named-subnets modules, but found the former to be simplistic for my needs and the latter to be limited to a single AZ. It does seem at least 2 modules are needed here.

The low barrier to entry terraform-aws-dynamic-subnets module is perfect as a starter/utility/shared services VPC . It splits the VPC CIDR range equally into public and private subnets. This means you are likely to choose a larger CIDR range due to generally needing more ips in the private subnet ranges. Also if you use VPC endpoints, they would probably be located in the public subnets since there are likely spare ips in those subnets.

However I like to use smaller VPC CIDR ranges and make best use of the available CIDR range by having smaller public subnets and relatively larger private subnets. I usually split the private subnets into app/data and a separate one for vpc endpoints. This is a safety design such that, for example, scaling lambda VPC ips cannot exhaust the data subnet range. Also a security design since AWS service traffic and gateway traffic is routed internally to the AWS backbone (which can also lead to a design with no NAT gateway as with this thread). Also a separate VPC endpoints subnet allows NACL rules and/or security groups on the endpoints. For that VPC design, the terraform-aws-multi-az-subnets is perfect.

As to the other benefits of the cloudposse modules, there’s the tagging support and features of the terraform-null-label, the clearly thoughtful implementation and versioning, the CI github actions/PR process, community and terraform registry. Many reasons to contribute rather than roll my own .

I also note the similarities between multi-az-subnets and dynamic-subnets. Your recent switch to using for_each from count in multi-az-subnets is a good example of an important contribution to robustness btw . I can see how some of these PRs would ideally be done on each subnet module (eg/ #49).

I did also consider the named-subnets module but would have had to use for_each at the module level (one for each AZ), and I saw the multi-az-subnets properly calculates the subnet ranges for n AZs for the user, so multi-az-subnets was the winner for our needs.

All considered, it’s difficult to see how to reduce the number of modules to reduce maintenance burden. Perhaps if dynamic-subnets was extended to support multiple categories of private subnets and varying subnet cidr ranges per category, possibly using a sub-module? And make the public subnet optional? And incorporate the named subnets differences if any. This might complicate the interface tbh. I like the simplicity of dynamic-subnets, which I’m sure contributes to its popularity.

Thanks again!

Thank you @Paul Robinson for your review of the 3 modules and the differences between them. I would appreciate it if you would edit it slightly and post it as a comment to https://github.com/cloudposse/terraform-aws-multi-az-subnets/issues/23 to help us get started on the requested documentation.

Thank you also for educating me about uses of subnets without gateways.

great yes I think that I can help there

@Jeremy G (Cloud Posse) @Matt Gowie I’ve added my comment to https://github.com/cloudposse/terraform-aws-multi-az-subnets/issues/23.

Any opinions welcome. Hope it’s useful

@Paul Robinson This is great! Thank you so much. @Erik Osterman (Cloud Posse) I suggest you look at Paul’s documentation because I agree with him on some key points moving forward: • All the modules are missing support for some attributes/features, such as IPv6 support and map_customer_owned_ip_on_launch • dynamic-subnets and multi-az-subnets are practically identical, but adding support for the new features would mean updating both of them. • dynamic and named are both still using count instead of for_each , and one reason for not changing them is that it would cause existing installations to attempt to delete and re-create the subnets • Given the feature overlap, missing features, and outdated nature of some of the code, it probably would be best going forward to either merge these 3 into 1 or create a modern 4th module and deprecate all 3 of the existing ones. I favor the latter, because if we merge them all into, say multi-az (which is the only one using for_each), it may make it harder for existing users of multi-az to transition. Also, a new module could address the multiple outstanding issues with the existing modules without having to worry about backward compatibility.

Glad to help :slightly_smiling_face:

Just a comment on the count change to for_each in the multi-az module in v0.12.0 I used the tf state management tools to upgrade an array index to the availability zone string but it a manual process and great care is required. eg/ module.private_app_subnets.aws_subnet.private[0] becomes module.private_app_subnets.aws_subnet.private["us-east-1a"] and the same for the route tables etc.

You are very clear that modules should be pinned, so these sorts of upgrades are to be expected of course.

I think the missing feature that could force another change is the availability_zone_id which is useful in multi-account peering/transit-gateway scenarios. To minimise costs in a peering scenario you would want the subnets deployed in the same az_id across accounts. In terms of module implementation, this would likely mean adding the az_id to the for_each instance index (probably replacing the availability zone name). That change would result in the same state management headache.

Another one I thought of is the case where you want m subnets placed in n availability zones, where m>n. In that case, the state needs a unique identifier for the for_each instance index, so again there’s a state move. This might be what’s required to add subnets to an existing vpc, for example allocating new subnets in a large VPC CIDR range, or when allocating new subnets after adding a new CIDR range to the VPC.

Migrating from one module to another comes with additional complexity however. It would be nice if there was a tool which could manage the state moves needed from one module to another. Are you aware of anything like that?

No, I know of no tools other than Terraform for managing Terraform state.

I don’t know of any resources that take availability zone IDs as inputs, and I would not expect there to be any, as that sort of defeats the purpose of having a dynamic mapping of AZs to AZ IDs. I think you need to manually manage your AZ selection based on AZ IDs, so I do not think AZ IDs will come into play in modules or for_each keys.

@Paul Robinson there is https://github.com/minamijoyo/tfmigrate but it isn’t full featured and is still a good bit of work.

Cool thanks for the link @Matt Gowie. tfmigrate using with the json form and jq might work for loops.

@Jeremy G (Cloud Posse) it is useful in multi-account scenarios where you are using NLBs to expose a service in the service account as a local ENI in the client account. For that case, the NLB is defined in the service account and needs to know which AZ IDs to enable in the NLB. https://docs.aws.amazon.com/vpc/latest/privatelink/endpoint-service-overview.html#vpce-endpoint-service-availability-zones

Gateway Load Balancer also tries to route within the same AZ iirc.

The underlying subnet resource allows it as an input to handle these scenarios. https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/subnet#availability_zone_id

Also see https://docs.aws.amazon.com/vpc/latest/privatelink/vpce-gateway-load-balancer.html#gwlbe-limitations
When you create a Gateway Load Balancer endpoint, the endpoint is created in the Availability Zone that is mapped to your account and that is independent from other accounts. When the service provider and the consumer are in different accounts, use the Availability Zone ID to uniquely and consistently identify the endpoint Availability Zone. For example, use1-az1 is an Availability Zone ID for the us-east-1 Region and maps to the same location in every AWS account.

To keep traffic within the same Availability Zone, we recommend that you create a Gateway Load Balancer endpoint in each Availability Zone that you will send traffic to.

only Terraformer from google

Ah I see multi-az is more explicit in defining each public and private subnets where dynamic creates both subnets for you.

If starting fresh would there be any benefits to using multi-az?

we usually use dynamic subnets

It looks like you’re creating public subnets in the vpc module, but using private subnets in the eks module, so subnets is indeed an empty list

oh man, good eye! thanks

ttl_atribute -> ttl_attribute_name

Hi David, I thinnk the variable name is ttl_attribute_name

oh you beat me to it

The fastest in the West

verified - thanks for the quick response!

Does anyone face same issue like me ?

@Yoni Leitersdorf (Indeni Cloudrail) if i was going that route, i would use the AWS managed policies for job functions. They are hecka easy to implement in TF and can be tweaked with additional permissions as needed. So depending on what the vendor is asking for (DB, support, billing, etc) you can just assign that role.

One thing to note: the read ony role allows access to the contents of parameters and secrets that might contain sensitive values. if you just want someone to “see” resources but not their configuration or values, the “view only” role is the way to go.

https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_job-functions.html

Yeah, that’s one of the known pitfalls with using AWS managed policies. People don’t fully understand what ReadOnly means…

also, the managed policies change pretty frequently… https://twitter.com/mamip_aws

specifically with this module - “cloudposse/ec2-instance/aws”

an ami we were using has been deleted, now when we run terraform we get the following error

Your query returned no results. Please change your search criteria and try again

i don’t want to destroy the instance and redeploy with a new ami - i just want cloudposse to ignore it

Unfortunately, lifecycle blocks don’t support interpolation, so there’s no nice way to do it. in the past, we’ll define 2 versions of the identical resource and feature flag the lifecycle behavior in one of them. This though adds a lot of overhead for maintenance we do it very seldom.

for this particular module cloudposse/ec2-instance/aws would it be better to not perform a data lookup on the AMI which fails when the AMI no longer exists?

you can’t access the secret access key after creation

it’s only returned as a response to the ‘create access key’ api action

if you are creating the access key in terraform, you can save it somewhere secure and then load it from that location in other Terraform configurations

do you by any chance have a sample of how i can save the returned value

resource "aws_iam_access_key" "test" {
  user = "alex"
}

If you create an access key like ^, you can access the secret key with aws_iam_access.key.test.secret

ahh okey

You can use a pattern like this to write it to SSM: https://github.com/cloudposse/terraform-aws-ssm-tls-ssh-key-pair

okey guys thanks so much for the tips. You ve been really helpful

thanks man, its a facepalm moment as i can’t spell my alias correctly

gotta love troubleshooting for an hour, to discover a single typo… administrator vs adminstrator, so many times

cc: @matt

I think you’re probably missing the nuance in the documentation here…

It only auto-enables accounts that are added to the organization after GuardDuty was enabled

This is why we created the guardduty sub-command on our turf utility

https://github.com/cloudposse/turf#deploy-guardduty-to-aws-organization

@RB I’m not optimistic

https://github.com/hashicorp/terraform/issues/26838#issuecomment-839918748

instead, we might want to add this to cloudposse/terraform-provider-utils

ya i just started reading up on the terraform-provider-utils and i think that might be the best way to go

this pr is older and it looks like pselle is trying to push it into terraform core

https://github.com/hashicorp/terraform/pull/24978#issuecomment-721867351

i think we can recreate the templatestring functionality using a hacky replace() function across a map of variables. not my favorite but would work until hashicorp changes their mind

The nice part about the template functions, is being able to use complex objects and other terraform functions in your template. Would love a templatestring function that supported that

I don’t think so. :thinking_face:

Locals are more like temp values that TF uses during execution. Their values are usually set by some expression or combination of other variables.

Maybe try using a [locals.tf](http://locals.tf) that only contains your locals?

If you are trying to set some local value before execution starts, you probably want a variable instead.

ahh I figured, thanks! Just double checking

I use terragrunt to share locals across different directories but in pure tf there isn’t anyway to do that I think

The purpose is just to add a hostname to a zone (e.g. a cluster’s hostname). I hate the name of the module and would like to rename it.

fun fact, this was probably the first module we released about 5 years

Hi (sorry for the digging up) I was looking for why this name for this module, now I know

Do you know if it would be a difficult job to rename it (without breaking the current uses)?

Use ingress {

Yes

Did you try using more recent versions of those modules?

For example, the first one now supports 0.13 and up: https://github.com/cloudposse/terraform-aws-route53-cluster-hostname/blob/master/versions.tf

i am not specifying them directly - i am only specifying the redis module

module "redis" {
  source = "cloudposse/elasticache-redis/aws"
  # Cloud Posse recommends pinning every module to a specific version
  version = "0.13.0"
  ....
}

they seem to be referenced from the cloudposse module, not from my tf file

Why version 0.13.0 of the module?

There’s 0.38.0 available.

ok - that fixed it - i was going off of this

i thought the version was in reference to terraform

Ah - you’re confusing between the version of Terraform and the version of the module.

Yep

thanks

What do you mean by “messed up”?

Using the terraform state rm/add commands are what you would use here. But I’m not convinced from your info that the state is messed up.

I didn’t provide full info. however, doing a terraform state rm/import ended up doing the trick.

# create a dummy.tfvars file from my TFE workspace variables
export AWS_DEFAULT_REGION=us-west-2
export AWS_ACCESS_KEY=XXXX
export AWS_SECRET_ACCESS_KEY=XXX

terraform state rm  module.rds_instance.aws_db_subnet_group.default
terraform import module.rds_instance.aws_db_subnet_group.default  my-existing-db-subnet-grp-01

I really don’t like having to handle AWS-dependencies with terraform.. things like db option groups and db subnet groups

using module version 0.20.0 or above should work

the latest is 0.17.2

i’ve worked around the issue by passing in arn_format and doing a lookup on the current partition

Hmm I guess the GitHub tags and terraform ones are different

small A and lots of B

in A put sensitive variables, and everything else in B, you cloud even generate B with Terraform it self and mange it

This may help: https://github.com/cloudposse/terraform-aws-iam-assumed-roles/blob/master/main.tf

thanks man thats awesome

found it, https://archive.sweetops.com/office-hours/2020/12/#dca83093-2eb8-4828-a95e-6584485099d5

Sounds like you want to use the error page handler logic of cloudfront. No need to need with a S3 website backend

Thank you @Alex Jurkiewicz. I’m guessing that’s not something that this module’s api exposes? Would you suggest doing it in aws management console?

it does expose it https://github.com/cloudposse/terraform-aws-cloudfront-s3-cdn/#input_custom_error_response

Ah brilliant, thanks

what backend are you using? s3?

You can create the state storage area by hand to bootstrap

I am using s3. Should have mentioned that. The bucket is already created, the key/path is obviously not there yet. I could put an empty file up there, but doesn’t seem much better than -lock=false when working alone. Thanks for the comments.

makes sense to me - we use “env” to align things with datadogs unified service tagging model, just because it was something to follow: https://docs.datadoghq.com/getting_started/tagging/unified_service_tagging

This also makes a ton of sense to me as stage is often repetitive. While i appreciate the ability to be ultimately expressive about what i’m building, i find its often never really used, i like the optional nature of this RFP b/c my/our situation is not everyone’s.

We were confronted with the same situation. What we ended up doing is setting up new CloudTrail settings that send everything to the audit account, shutting down the old CloudTrail settings, and copying the data over to separate buckets.

The idea here is that you don’t have to merge the old data with the new data, as long as you know when the cutover was done and adapt your queries accordingly.

Re your question for using TF - you referring to how to create a central cloudtrail audit account?

Or what specifically?

Like moving it from the master where its sits now to the audit account, essentially what we want to do is alter our base line this will include the AWS CONFIG

Oh we didn’t do that part via TF. We used TF to build the new audit.

Thank you

can you be so kind to send a sample of the TF for the audit?

@Karl Webster you can always bump any PR in #pr-reviews to get some on it. I’ll try to get this one merged for ya.

Merged — Thanks for the bump @Karl Webster

Ah, I did not even realise that was a channel, noted for next time

looks like atlantis, right?

yes.

what’s the difference?

A much more clever business model, definitely.

I did not test it yet, it is open for beta tester

what’s the difference? atlantis does the apply before you merge right? The screenshots look like terrateam does it on/after the merge.

I would do it this way: All features branches should display planning for upcoming changes, and only one branch ( main or master ) can apply changes after PR reviewed and merged. so your CICD should watch for main branch changes and trigger based on changes there.

If each feature branch can apply TF changes, some branches will overwritten the previous one, and could be two branches run at the same time within big teams.

so to summarize my apprach: • create feature branch • display cost using infracost • display security findings using checkov or cloudrail

• display graph of new resouces

• display plan output of changes

• assign senior engineers for review

• approve

• merge Then CICD will be triggered and apply changes and send notification to slack

This can be achieve using multiple ways - so no secret recipe here.

yep, we follow the above model. When you add a commit to a pull request, your CD system runs a speculative plan with your changes.

The key is that you only allow PRs to merge when they are up to date with the base branch. This means your speculative plans are always “fresh”. On the downside, if you have many PRs open, you can get into an annoying cycle of “oh, someone else merged, let’s update my branch, oh, someone else merged”. But it’s not a real issue

this workflow is described in detail in the Spacelift docs. We use Spacelift, but this workflow is generic and can be used with any CD system: https://docs.spacelift.io/integrations/source-control/github#proposed-workflow

you also mention that reviews become “obsolete” in this situation. You’re exactly right, and Github has an option to dismiss review approvals when new commits are pushed to a PR. I don’t know why it’s not enabled by default, but IMO it should be

Thanks for the feedback, really appreciate this:)

Actually we are working like what was mentioned:

• create a feature branch

• open a PR/MR

• assign reviewers

• checkov is going to be implemented soon

• we showed graph, but for some reason stopped doing this

• display plan

• wait for approvals

• merge But additionally we have to:

• check a plan in the mainline after the merge

• apply manually in the mainline if everything is as expected The problem is that we either need to merge the latest code to all PRs manually (usually we don’t have many of them, that’s true) or double check the plan in the mainline. And actually the latter is done in both cases to have this confidence:)
Github has an option to dismiss review approvals when new commits are pushed to a PR. That’s true, but it removes approvals if a new commit is landed into this PR’s branch, not in case of another PR that’s created recently, correct?:)

All in all, I’m more concerned about these two final steps I mentioned. And there is something in the air that things could have been more simple or done differently with even a better confidence. At least, probably there should be some automatic rebase hook that update all PRs if new code appears in the mainline. Hence approvals will be removed with the aforementioned feature enabled.

The most common github integration is atlantis: https://www.runatlantis.io

It operates by applying from your feature branch in combination with PRs locking the state for that project (so only one PR is planning and applying at any given time)

That’s true, but it remove approvals if a new commit is landed into this PR’s branch, not in case of another PR that’s created recently, correct?:) If the base branch gets changed, your pull request cannot be merged because it’s out of date. You update it, and this causes the plan to be regenerated and old approvals to be dismissed.

@Alex Jurkiewicz Depending on the merge strategy and settings, but I see your point:) Huge thanks for your comments!

I’m trying to find anything related to the reasons why we put Atlantis on hold years ago. No luck so far, however occasionally I’ve found a saved thread started by @sheldonh and just want to link it here as well: https://sweetops.slack.com/archives/CB6GHNLG0/p1617922933434300

I downloaded the module by Cloudposse for Atlantis. Was interested in running in ECS fargate but found Azure DevOps didn’t seem to be supported except in a fork/PR version and never reconciled this.

I’ve been running from my machine directly for now as i’m the only one writing terragrunt/terraform at this time. I wrote a Go wrapper for coordinating the runs but would like to get back to a PR comment based workflow in the future if others start contributing.

I’m finding that working directly in a dev team the need for collaborative PR workflow is less than when in the Cloud Operations team as it’s more application deployment focused and less shared.

Thanks for sharing:)

Basically: Add new managed node groups “on the side”…

cordon and drain the old worker ASGs and let the new managed ones scale up..

Then remove the non-managed ones…

I’ll give that a shot, thanks! I was thinking about other ways beyond just running them alongside but it’ll do, I need to dismantle our cluster autoscaler in the pod as well probably…

The cluster-autoscaler can handle any number of ASGs.

But do read some of the notes here https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html

iirc, doesn’t amazon manage autoscaling?

for node groups

It’s worker groups where cluster-autoscaler needs to be installed, right?

The Managed Node Group is an “upper” layer on top of the “regular” resources such as an underlying Autoscaling Group.

Ahhh, gotcha, must’ve been lost in my readings, so autoscalers does need to be installed right?

So to actually automatically scale it - you still need the cluster-autoscaler.

Personally I suggest having a node group (or worker group) which is more or less “static”, for some particular jobs such as the cluster-autoscaler itself.

Gotcha, thanks! Not a problem to install

Usually not much cpu or ram needed… it all depends on how many of those jobs you have.

What do you mean by that last point?

the “static” group for “Cluster tools”, usually doesn’t need to be big and costly …

so I feel most of the time - it is worth it.

Yeah, this is mostly for a small RC environment our devs are using.

sounds cool but i’m tied up this week. i added a slack reminder to give this a shot over the weekend

Thanks so much!

@Sean Turner i took the code for a spin! here’s an update: • I used GitHub as my repo source and created a token but it may be helpful to provide the exact permissions the token needs. People will likely want to scope the token to the exact permissions vs creating an admin token with all access. • I used the code in moot/terraform_examples/complete. I did not add a custom domain so I had to add the following to get the domain that was created. Might be nice to already have that in the example.

output "moot" {
  value = module.moot
}

• I used an admin email address and successfully received the email with the password but when i tried to log in, nothing happened.

• The cloudwatch logs for the API gateway were useful to indicate that there is some sort of integration error but I didn’t go much further than this in terms of debugging:

{
    "httpMethod": "POST",
    "integrationError ": "-",
    "ip": "...",
    "protocol": "HTTP/1.1",
    "requestId": "AHRliheDIAMEMrQ=",
    "requestTime": "29/May/2021:23:08:06 +0000",
    "responseLength": "109",
    "routeKey": "POST /auth/login",
    "status": "400"
}
{
    "httpMethod": "OPTIONS",
    "integrationError ": "-",
    "ip": "...",
    "protocol": "HTTP/1.1",
    "requestId": "AHRlmgJpoAMEMmQ=",
    "requestTime": "29/May/2021:23:08:06 +0000",
    "responseLength": "0",
    "routeKey": "-",
    "status": "204"
}
{
    "httpMethod": "GET",
    "integrationError ": "-",
    "ip": "...",
    "protocol": "HTTP/1.1",
    "requestId": "AHRlngCSoAMEMvw=",
    "requestTime": "29/May/2021:23:08:06 +0000",
    "responseLength": "26",
    "routeKey": "GET /repositories/list",
    "status": "401"
}

• I didn’t try the Slack integration

Thanks! Great feedback Weird that you were getting an integration error. I think I got something like that once and recreating the lambdas fixed it?

Really appreciate the follow up.

no problem. i will give the lambda recreate a try

The good news as that terraform apply worked ;) There’s quite a few dependencies in there

actually, i may be reading the logs wrong. if there is a - next to integrationError, it means there’s no data which likely means there isn’t an integration error.

so i’ll try the lambdas and update

Any lambda cloud watch logs?

ahh let me see

If not then they aren’t being triggered

ok i found the lambda logs. maybe my password is bad?

No older events at this moment. 
Retry
START RequestId: a803588c-7553-4b72-b32a-7ac0e9f6dbe7 Version: $LATEST
{
    "level": "info",
    "msg": "handling request on /auth/login",
    "time": "2021-05-29T23:04:58Z"
}

{
    "level": "error",
    "msg": "InvalidParameterException: Missing required parameter USERNAME",
    "time": "2021-05-29T23:04:59Z"
}

END RequestId: a803588c-7553-4b72-b32a-7ac0e9f6dbe7
REPORT RequestId: a803588c-7553-4b72-b32a-7ac0e9f6dbe7	Duration: 928.26 ms	Billed Duration: 929 ms	Memory Size: 128 MB	Max Memory Used: 45 MB	Init Duration: 115.99 ms	
START RequestId: 74db7a5f-0d1c-4b66-8fc2-df367d977d70 Version: $LATEST
{
    "level": "info",
    "msg": "handling request on /auth/login",
    "time": "2021-05-29T23:08:06Z"
}

{
    "level": "error",
    "msg": "NotAuthorizedException: Incorrect username or password.",
    "time": "2021-05-29T23:08:06Z"
}

END RequestId: 74db7a5f-0d1c-4b66-8fc2-df367d977d70
REPORT RequestId: 74db7a5f-0d1c-4b66-8fc2-df367d977d70	Duration: 210.39 ms	Billed Duration: 211 ms	Memory Size: 128 MB	Max Memory Used: 45 MB	

yep, that would be it ha. my first frontend

Need to add a toast or something if it fails

ok i copied the password with care and got the “new password” prompt. entered new pass and now nothing. will check the log again

no update in the logs

I usually just add a $ to the password from the email and it logs me in

Might be the password updated already. Do a refresh and use the new password?

i refreshed and tried the temp pwd with the $. that worked. if there is password complexity requirement, that might be useful to share. my password was good but still not as complex as the generated one. shorter for sure. one special char. maybe no caps.

Yeah, might be worth parameterising the cognito password requirements

i was able to add a repo in github. i clicked the link to get to the repo and the link is malformed…no .com : https://github/console6500/cherokee

Ah interesting! I’ll need to fix that

ok i added a note and a version, clicked deploy. not sure what’s supposed to happen

Generally you would need to have a change on the HEAD branch. That gets merged into the BASE branch, and then a github release is created from the BASE branch

got it.

My head branch was new, I was using this echo 1 >> some_new_file && ga . && gcmsg "blah" && git push origin new to test easily

ok i’ll give it a shot later.

Thanks again !!

Added the toasts to the login view, added outputs.tf to the examples, and fixed the no .com issue as well .

Actually, this looks like the place where it is failing:

2021/05/26 02:40:08 [TRACE] ExecuteWriteOutput: Saving Create change for output.cicd_roles in changeset
2021/05/26 02:40:08 [TRACE] EvalWriteOutput: Saving value for output.cicd_roles in state
2021/05/26 02:40:08 [TRACE] vertex "output.cicd_roles": visit complete
2021/05/26 02:40:08 [TRACE] dag/walk: upstream of "meta.count-boundary (EachMode fixup)" errored, so skipping
2021-05-26T02:40:08.127Z [DEBUG] plugin: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/aws/3.42.0/linux_amd64/terraform-provider-aws_v3.42.0_x5 pid=19511
2021-05-26T02:40:08.127Z [DEBUG] plugin: plugin exited

It looks like somehow the response object is not getting reset. ‘output.cicd_roles’. seems to come up every time.

@Andriy Knysh (Cloud Posse) would likely be the best to weigh in on this one if he’s got a minute.

must be an invalid YAML config

I’ll double check…

I’d look into these examples https://github.com/cloudposse/terraform-yaml-stack-config/tree/main/examples

and replace the config files with your own to test

ok, will do. I did run all my files through yamllint. No changes. I’ll try your files next.

you can DM your YAML config and code so I can take a look if you still having issues

You were completely right. I am so embarrassed. I was using anchors to define my tags in a way that was passing the lint checks, but was not producing a valid config. Mea culpa, and thanks for the help.

no problem

in any case, we don’t have good enough documentation, so any help on finding issues and improving the modules is greatly appreciated, thanks

Don’t think it would’ve helped here, but we do have some docs on stack files which might be useful: https://docs.cloudposse.com/reference/stacks/

cool! I will check these out. I’ve been making lots of notes as I ramp up – will have some tutorial material fairly soon I think.

you need to create an output in your top level module that prints the value you are looking for:

output "public_ip" {
    value = module.default.module.aws_ec2.aws_instance.debian.public_ip
    description = "The EC2 public IP"
}

then run terraform refresh

No but how about trying env0 self hosted agent? https://docs.env0.com/docs/security-overview#self-hosted-agents is that relevant for you? (Disclaimer, i am founder at env0)

cc: @Sebastian Stadil

I used it yesterday to import accounts into the account component of the CP ref arch aws components.

Posting it as example. I had to pass the region arg too.

Thanks, Brian – this really helps. Let you know how it goes. Cheers –

After a little playing around with turning on/off sections of the main.tf in the account component, I was able to rebuild the entire account component from scratch – Org, OUs, Accounts, Policies – everything. Thanks so much for your pointer on this!

No problem. Anytime.

I think there is a new function unsensitive() or something, to remove the sensitive attribute from the object

There it is, nonsensitive()… https://www.terraform.io/docs/language/functions/nonsensitive.html

awesome, thanks!

it’s pretty cool that the meta of values carry all the way through.

yeah, the system is actually pretty good. Just that there’s no way to know ahead of time what resource attributes are sensitive, so you have to add sensitive and nonsensitive incrementally as you find problems

Relatively new to TF, so I haven’t run into that issue. The alternatives are CDK and CloudFormation. Although I cannot speak to CDK, I would not switch back to CFN over TF. CFN has its problems…

• roughly 50% slower to deploy the same stacks

• import functionality is minimal and clunky

• very slow to add support for new services

• works only with aws resources

• deployments to k8s are more complex due to the need for multiple ioc languages

• error messages are cryptic for the unexperienced

• require much more profound knowledge of aws Experience: I spent the past 4-5 years writing hundreds of CFN templates. I wrote nearly all the automation using those templates to deploy thousands of stacks across multi-region and several dozen accounts.

Ok. Thank you.

I find it quite frustrating to see code break without changes. I haven’t seen anything like this before, I think.

(A bit of a shameless plug as I’m part of the team, sorry about that ) but if you have issues spotting unwanted changes, you might want to try running driftctl (OSS) against your infra to get a clear overview of what has actually changed. It will compare your state against your AWS account and list all resources not managed by TF. Once you’ve done that, you can create a baseline with a .driftignore file (here’s a link to a tutorial), and you’ll see each new change to your infra appear when you run the tool.

make sure you pin your terraform version in your root module, and at least be aware of changes in the provider versions that you are using

and for sure pin all module versions

Hi Loren. That’s a good point. I think quite a few versions are pinned, but I’ll check. Stupid question - is there a good way to follow provider changes (in our case AWS)?

i’d recommend subscribing to the github releases… if you haven’t done that before, go here: https://github.com/hashicorp/terraform-provider-aws. click “Watch”, Custom, check Releases, Apply

you can also use the github integration for slack to post releases to a channel

@Fabian - what do you mean terraform validations broke? Can you go into a bit more detail? Terraform still has not reach an official v1 release. There are typically more headaches associated with a tool or language that moves as fast as Terraform has. If you’re not pinning to a specific version and stay on the latest…you may have migraines instead of a caffeine headache.

For example this alert started (my engineers said that nothing in our code had changed to trigger this)

[TF-WRAPPER][Terraform][Plan] Planning Terraform Code

Error: Unsupported argument

  on vpc.tf line 25, in module "vpc":
  25:   enable_s3_endpoint   = true

An argument named "enable_s3_endpoint" is not expected here.

that’s almost certainly from not pinning, and getting the latest module version, where the module has removed that argument

pin your modules!

ok. I’ll have a look at that! thank you for the direction.

Right. There are also some other best practices that CloudPosse suggests you follow. I’ll link them later on unless someone else has them handy and can share.

Awesome! Thank you!

@Fabian https://docs.cloudposse.com/reference/best-practices/

This is a cool idea! How does it work? Static analysis of the source code?

I imagine with Terraform there will be difficulties around supporting modules and in particular the CloudPosse “null label” convention. But still, I can see a lot of value here!

do you need the rule to be what you said? Or do you just need 50% of servers to go into each zone?

Hi the Foqal bot, told me that I can use the modulo % math

I suggest instead https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/shuffle

haha that’s better then I can support 3 zones but I may have no luck when there is low number of vms ..

as I see no even or odd num func defined https://www.terraform.io/docs/language/functions/index.html so you need to go for remainder operator https://www.terraform.io/docs/language/expressions/operators.html

@Zoltan K thanks that’s what I was looking for

welcome. finally a conditional to setup your zone… https://www.terraform.io/docs/language/expressions/conditionals.html

yes that’s what I am doing right now

and we need to load up terrafrom manual to foqal to make it smarter I guess

well he gave me the math “%” to use :)

so I end up selecting my zone_id for servers named testXXX with a “% 3” as there are 3 zone in Azure Region regex("([1-9][0-9]{0,2})$", srv_name)[0] % 3 + 1 which produce as a result a zone number in 1, 2, 3

This would be great to have to be able to customize the NACLs managed by this module.. Or is there another preferred Cloudposse way to manage NACLs?

I found https://github.com/cloudposse/terraform-aws-named-subnets Is this the alternative to dynamic-subnets?

I think we could have a more robust way of managing NACLs more like our new security group module

https://github.com/cloudposse/terraform-aws-security-group

But we haven’t gotten to that

I have a workaround — testing and will update for a PR soon, thanks!

This is more on debugging the deployed configuration and seeing what is different about your configured cognito backed ES instance vs an AWS blog post cognito backed ES instance. The module should be providing the correct flags or otherwise there would likely be an issue open about this, so I would suggest just digging into what you’re missing by comparing it to an AWS blog post.

Hey @DevOpsGuy 1 - You should use a seperate AWS Account for dev and prod 2 - Here is an example of how I have done this in the past - https://github.com/msharma24/multi-env-aws-terraform

( I just pushed this example for you )

@msharma24 I really thank you so much. It worked.

Mate

hey, yep, see this thread -> https://sweetops.slack.com/archives/CB6GHNLG0/p1614979106313600