#office-hours (2020-11)

https://github.com/cloudposse/terraform-aws-efs-backup

https://github.com/cloudposse/terraform-aws-backup

I want to use AWS backup, but it does EBS snapshots only, not native backups.

If I have a 2TB drive with databases and assuming i enable VSS aware mode… I’m pretty sure I won’t get 15 min RPO from EBS snapshots with AWS Backup.

If I’m wrong and they are fast then let me know! Last time I checked it took a while and also freezes sets of db’s during the snapshots

I want to use AWS backup, but it does EBS snapshots only, not native backups. Our AWS Backup module also does EFS.

what do you mean by “Native backups”? … especially when a database is involved.

SQL Server (MSSQL) has it’s own backup mechanism. They are compressed and per individual database. If I use EBS i need it to be where i can maintain 15 min max data loss which I doubt EBS snapshots will give me.

EFS is promising for where to push my backups but might seems simplier to just backup and sync to s3

Hi Tim, Tim here… You could use ignore_changes: https://www.terraform.io/docs/configuration/resources.html#ignore_changes

To ignore changes in AMI IDs but that might lead to another change causing old AMIs to be deployed if you use a static AMI ID in your Terraform code…

You could retrieve the AMI ID using the aws_ami data source: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/ami based on a name regex or tags.

Maybe combining the 2 ignore_changes and aws_ami would be a reasonable experience.

You could also have the lambda that updates everything store the AMI IDs in something like parameter store and look up the IDs in Terraform… that might add a bit of complexity when you can probably get by with the above option ( combining the 2 ignore_changes and aws_ami ).

I did this for the customer, but didn’t find anything complete, so used to follow items one by one from the list and adjust each service. I am interested to know if you find something better (preferably not very $$$).

I plan on making it from scratch. I have to figure it out by next tuesday

https://d1.awsstatic.com/whitepapers/compliance/AWS_CIS_Foundations_Benchmark.pdf

@Erik Osterman (Cloud Posse) thank you for the reference. I’m using that doc to guide me through the process. I’m going to add a cloudtrail resource, s3 resources, and KMS key resource to encrypt the logs. Hopefully, things go smoothly. Still learning a lot though.

https://github.com/cloudposse/terraform-aws-security-hub

https://github.com/cloudposse/terraform-aws-cloudtrail-cloudwatch-alarms

@Erik Osterman (Cloud Posse) I don’t really understand the point of the security-hub or cloudtrail-cloudwatch-alarms. I think I only need to create a cloudtrail resource, s3 resource, and kms resource. I’m still learning AWS and Terraform. My boss basically handed me an ambiguous project and now I need to figure it out.

cloudtrail.tf

resource "aws_cloudtrail" "nfcisbenchmark" {
  name                          = "nf-cis-benchmark"
  s3_bucket_name                = aws_s3_bucket.nfcisbenchmark.id
  enable_logging                = var.enable_logging
  enable_log_file_validation    = var.enable_log_file_validation
  is_multi_region_trail         = var.is_multi_region_trail
  include_global_service_events = var.include_global_service_events
  is_organization_trail         = var.is_organization_trail

  # CIS Benchmark 3.1 Ensure CloudTrail is enabled in all regions
  # for a multi-regions trail, ensuring that management events configured for all type of
  # Read/Writes ensures recording of management operations that are performed on
  # all resources in an AWS account
  event_selector {
    # Specify if you want your event selector to include management events for your trail.
    include_management_events = true
    # Specify if you want your trail to log read-only events, write-only events, or all. By default, 
    # the value is All. Needed for logging management events.
  }
}

resource aws_s3_bucket "nfcisbenchmark" {
  bucket = "nf-cis-benchmark"
  acl  = "private"
}

variables.tf

# CIS Benchmark 3.2 Ensure CloudTrail log file validation is enabled (Automated)
variable "enable_log_file_validation" {
  type        = bool
  default     = true
  description = "Specifies whether log file integrity validation is enabled. Creates signed digest for validated contents of logs"
}

# CIS Benchmark 3.1 Ensure CloudTrail is enabled in all regions
# ensuring that a multi-regions trail exists will ensure that unexpected activity
# occurring in otherwise unused regions is detected
variable "is_multi_region_trail" {
  type        = bool
  default     = true
  description = "Specifies whether the trail is created in the current region or in all regions"
}

# CIS Benchmark 3.1 Ensure CloudTrail is enabled in all regions
# ensuring that a multi-regions trail exists will ensure that Global Service Logging
# is enabled for a trail by default to capture recording of events generated on AWS
# global services
variable "include_global_service_events" {
  type        = bool
  default     = true
  description = "Specifies whether the trail is publishing events from global services such as IAM to the log files"
}

# CIS Benchmark 3.1 Ensure CloudTrail is enabled in all regions
# Ensure Logging is set to ON
variable "enable_logging" {
  type        = bool
  default     = true
  description = "Enable logging for the trail"
}

variable "is_organization_trail" {
  type        = bool
  default     = false
  description = "The trail is an AWS Organizations trail"
}

Actually here is everything for CIS Benchmark 3 Logging

resource "aws_cloudtrail" "nfcisbenchmark" {
  name                          = "nf-cis-benchmark"
  s3_bucket_name                = aws_s3_bucket.nfcisbenchmark.id
  enable_logging                = var.enable_logging
  enable_log_file_validation    = var.enable_log_file_validation
  is_multi_region_trail         = var.is_multi_region_trail
  include_global_service_events = var.include_global_service_events
  is_organization_trail         = var.is_organization_trail
  kms_key_id                    = aws_kms_key.nfcisbenchmark.arn

  # CIS Benchmark 3.1 Ensure CloudTrail is enabled in all regions
  # for a multi-regions trail, ensuring that management events configured for all type of
  # Read/Writes ensures recording of management operations that are performed on
  # all resources in an AWS account
  event_selector {
    # Specify if you want your event selector to include management events for your trail.
    include_management_events = true
    # Specify if you want your trail to log read-only events, write-only events, or all. By default, 
    # the value is All. Needed for logging management events.
  }
}

resource aws_s3_bucket "nfcisbenchmark" {
  bucket = "nf-cis-benchmark"
  acl  = "private"
  #  ensure the CloudTrail S3 bucket has access logging is enabled.
  logging {
    target_bucket = aws_s3_bucket.log_bucket.id
    target_prefix = "log/"
  } 
}

resource aws_s3_bucket "log_bucket" {
  bucket = "nfcisbenchmark-log-bucket"
  acl    = "log-delivery-write"
}

# 3.7 Ensure CloudTrail logs are encrypted at rest using KMS CMKs (Automated)
# Defaults to ENCRYPT_DECRYPT
resource aws_kms_key "nfcisbenchmark" {
  description             = "nf-cis-benchmark"
}

and here are the vars

variable workspace_iam_roles {
  type = map
  default = {
    nf-sandbox         = "arn:aws:iam::721086286010:role/ops-mgmt-admin"
    nf-integration-int = "arn:aws:iam::472879144981:role/ops-mgmt-admin"
    nf-staging-int     = "arn:aws:iam::560269805515:role/ops-mgmt-admin"
    nf-staging-cust    = "arn:aws:iam::337684097865:role/ops-mgmt-admin"
    nf-devops-tools    = "arn:aws:iam::447513199460:role/ops-mgmt-admin"
    nf-prod-int        = "arn:aws:iam::745435643501:role/ops-mgmt-admin"
  }
}

# CIS Benchmark 3.2 Ensure CloudTrail log file validation is enabled (Automated)
variable "enable_log_file_validation" {
  type        = bool
  default     = true
  description = "Specifies whether log file integrity validation is enabled. Creates signed digest for validated contents of logs"
}

# CIS Benchmark 3.1 Ensure CloudTrail is enabled in all regions
# ensuring that a multi-regions trail exists will ensure that unexpected activity
# occurring in otherwise unused regions is detected
variable "is_multi_region_trail" {
  type        = bool
  default     = true
  description = "Specifies whether the trail is created in the current region or in all regions"
}

# CIS Benchmark 3.1 Ensure CloudTrail is enabled in all regions
# ensuring that a multi-regions trail exists will ensure that Global Service Logging
# is enabled for a trail by default to capture recording of events generated on AWS
# global services
variable "include_global_service_events" {
  type        = bool
  default     = true
  description = "Specifies whether the trail is publishing events from global services such as IAM to the log files"
}

# CIS Benchmark 3.1 Ensure CloudTrail is enabled in all regions
# Ensure Logging is set to ON
variable "enable_logging" {
  type        = bool
  default     = true
  description = "Enable logging for the trail"
}

variable "is_organization_trail" {
  type        = bool
  default     = false
  description = "The trail is an AWS Organizations trail"
}

I stole some of this from your repos

I don’t really understand a lot of the dynamic blocks and what not. I’m just getting started in terraform and I would really like some help with this project if anyone has time.

I figured it all out btw.

@Erik Osterman (Cloud Posse) wow actually I just looked at SecurityHub and I can see that it’s an auditing tool. I made the terraform for cloudtrail, kms, and s3, for CIS. Would you recommend using Security Hub to audit the CIS benchmark foundations or create custom config rules in AWS config?

@matt has maybe more insight on this

@Erik Osterman (Cloud Posse) @matt I’d love to get some more insight on this. I’ve been having issues with the aws_config_delivery_channel most of the code out there is quite old. One repo was using format for parsing an arn. I think I’m going to need to make it all from scratch. My plan is to set up an AWS Config resource with a delivery channel, s3 bucket, and sns messaging queue. I’ll use the same s3 bucket that I’m using for cloudtrail, but use two different prefixes i.e. cloudtrail and config. Then I will add an aws_securityhub_standards_subscription. I’m not sure how cost effective this solution is though. Also, I’m learning terraform and AWS at the same time. Probably going to need to make it all manually and then figure out the terraform. Is this cost effective? I’m trying to stay lean, but I gotta meet CIS Foundations.

@Erik Osterman (Cloud Posse) still running this down. I’ve knocked out 1, 2, and 3. I am running into a bump on 4.1 Ensure no security groups allow ingress from 0.0.0.0/0 to port 22. Did you guys find a way to solve this?

Btw, we released this yesterday: https://github.com/cloudposse/terraform-aws-config/tree/master/modules/cis-1-2-rules

Which leverages the full catalog of prepared config rules by AWS: https://github.com/cloudposse/terraform-aws-config/tree/master/catalog

@matt can maybe talk to the specific question you have though

Yes, the AWS Config rule resstricted-ssh can detect that this is compliant and let you know of any SGs that are violating it…

https://docs.aws.amazon.com/config/latest/developerguide/restricted-ssh.html

We are in the process of putting together a series of modules and configurations that allow our customers to achieve CIS compliance. So hopefully we’ll have a good “end-to-end” story on this to share shortly.

@Erik Osterman (Cloud Posse) does it make sense to use the newer version of CIS?

Not really, because SecurityHub doesn’t yet support it.

Okay, thank you. I am struggling to find an optimal way to knock out the security groups no ingress 0.0.0.0/0 to port 22. Seems like the solution is to go through and individually remove the non-compliant sgs, add a cloudwatch alert, trigger an alert when an sg with port 22 open is created, then use auto-remediation to delete it. I feel like this is going to be painful.

Actually better idea. Get a list of running instances, run a port scan for each ip, if 22 is open and not a bastion then remove the ingress rule

Also, has anyone figured out why vpc_flow_log resources always need to be recreated? It looks like the AWS iam arn isn’t getting attached.

@Erik Osterman (Cloud Posse) are you enabling AWS accessanalyzer for via

aws_accessanalyzer_analyzer 

resource? https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/accessanalyzer_analyzer I’m very close to being done with the entire automation project for CIS foundations.

Oh also I’m handing the no ingress on port 22 or port 3389 by creating config rules that check each condition. Then if the compliance status on either of those rules change I’ve set up a cloudwatch rule that will send an alert to a central email account. It’s better than the nmap script that I was previously using.

Haven’t looked into that one yet. Looks nice - I think we should build support for that.

Using the https://github.com/cloudposse/terraform-aws-ec2-autoscale-group. I am following the example. I am able to plan it but getting the following error.

module.autoscale_group.aws_autoscaling_group.default[0]: Creating... Error: One of `id` or `name` must be set for `launch_template

`

See you in 30 mins

What would be the best way to modify the terraform on this package: https://github.com/terraform-aws-modules/terraform-aws-ec2-instance so that we could optionally have it create a ENI and bind it, as an alternative to just providing the ENI

From my post in #general: I have searched through the archives and google is not helping either… so hopefully this isn’t described somewhere I have just missed. I am working on setting up a new infra utilizing cloudposse OSS but have bumped into a few issued. I got all the accounts setup and that is working great, also got EKS and other application backing services up (eg RDS) all using terraform… but I am having troubles finding out how to make the next steps through IaC.

In particular I am trying to get Codefresh setup for GitOps but it is not clear how to do this as IaC. I got my Codefresh account setup (using Github to auth) but to add the K8s EKS integration for example I used the helm chart (cloudposse-incubator/codefresh-service-account ). But I did this by hand which seem wrong. Then to setup the project and pipelines I can only find how to do this by hand.

I am working through the cloudposse/codefresh repo but this seems to be make based, not terraform. So it is not clear how this would be applied using gitops. So I come into the same problem as with the helm chart above.

Does anyone have information about how to do these programmatically? Is just doesn’t seem correct to have these be done by hand.

https://github.com/cloudposse/terraform-aws-eks-workers/blob/master/main.tf

I have just gone through this for PCI compliance. You do not want to use the CIS images. Although they are hardened they are not complete with the docker and Kubernetes components. Installing those is not trivial and will not result in a process you can easily repeat. Instead use the recommended Amazon Linux 2 Optimized AMi ( ami-0c62450bce8f4f57f). It works perfectly with EKS. If you need to show hardening use kube-bench tool found here (https://aws.amazon.com/blogs/containers/introducing-cis-amazon-eks-benchmark and https://www.eksworkshop.com/intermediate/300_cis_eks_benchmark/).

https://gravitational.com/gravity/

https://www.arhea.net/posts/2020-04-29-hardened-images-amazon-eks.html

Aaaaaand I was wrong.

In the call I said that Fargate finally supports config files from S3. That’s false. Fargate now supports env-files from S3 Sorry

We’re working to have all the SaaS services on an office hours showdown

@marcinw @Sebastian Stadil @Jake Lundberg (HashiCorp) so the vendors can showcase their differentiators.

Multiple ways to skin the cat, but one thing bridgecrew does is automatically open PRs to Fix problems (optional)

@barak

@Erik Osterman (Cloud Posse) thanks for the tagging! @David Lozano sorry for the late reply. bridgecrew is the company behind checkov, and it is part of the platform. Some features that are only in the platform:

1. Remediation - the platform can open “fix” pull requests where possible to a terraform/cloudformation misconfig
2. Compliance reporting
3. Runtime analysis of misconfig of you Cloud (AWS, GCP, AZURE) and K8 projects
4. AWS IAM rightsizing - analysing AWS IAM policies and recommending editions by the principle of least privilege
5. Dependency graph analysis ( if a resource, depends on a variable in another file - the platform will know that)
6. Organization dashbaord
7. Notifications (jira, slack, splunk etc..) and many more to come

I’m here if you want to drill into any of those in DM

@RB agreed! we started doing this. you can find notes in the youtube video descriptions with timestamps to jump to those, check out last week’s video for example: https://www.youtube.com/watch?v=XR9pIWAMNlE&list=PLhRztDM6Uvne8MUuwXrv2truMl6gVZ0D8&index=1&t=160s

we sometimes also create short segments for interesting topics you can find in our playlists “Cloud Posse Explains” and “Terraform News”

notes are also uploaded below the videos on our blog: https://cloudposse.com/blog/

would a question like this be acceptable for office hours? https://sweetops.slack.com/archives/CB6GHNLG0/p1605655766298600

maybe too specific

I’ll be in today! I’m curious if anyone has an opinionated pattern for running a personal cluster (independent of employer) that is portable enough to generally run on a laptop and occasionally push to a cloud. https://sweetops.slack.com/archives/CHDR1EWNA/p1604519888152500

@Tomek, I think the #aws channel is good for the role TTL question. IMHO, office hours is good for questions that may have multiple solutions or that might elicit the wisdom borne from painful experience.

hi Tarlan! we uploaded this segment a few weeks ago on Waypoint, hope it helps https://www.youtube.com/watch?v=DoE6OteOUSo&feature=youtu.be

Awesome, thanks mate

Hrmmm This looks like helm and not helmfile notation

@Erik Osterman (Cloud Posse) what do you mean by notation?

{{ include "common.name" . }} looks like what you would write in a helm chart and not in a helmfile

yeah, i was just giving an example of how my monochart helm chart doesn’t make any assumptions currently except things like those very high level labels

You can use bases in helmfiles to layer options.

go here and scroll UP to Advanced Configuration: Layering above  bases in code block

please post here

I wanted to avoid a VPN if possible. Plus AWS VPN is terribly expensive from what I remember.

and TF does not support all the confid for aws vpn

@Matt Gowie I haven’t tried it myself yet, but maybe this one? https://github.com/abutaha/aws-es-proxy/

I don’t know if this is intended to work with signed requests to Kibana… or ES only

Hey @Matt Gowie I just had to solve this for a client. We were using AWS Client VPN but it wasn’t ideal.

The app @Vugar linked to publishes a Docker image so I used it with cloudposse/terraform-aws-ecs-alb-service-task.

Then used the alb (for TLS and easy target registration with ECS) and alb-ingress modules to set up authentication to Google OIDC.

The modules are customised for the client, but let me know if I can send you some snippets.

Thank you @Joe Niland! Sounds great! I So I take it ‘aws-es-proxy’ works both for signing requests to Kibana and ES… , am I correct?

Yes it seems to @Vugar

Thanks for the suggestion @Joe Niland. I’ve added it to the list.

I’ve been looking into multiple solutions for this… I think I’m moving towards something like https://tailscale.com/. It’s a SaaS product, but damn is it impressive!

wierd!

i see what you mean

Yeah, as @jose.amengual suggested, when you subscribe to the Zoom invitation, it’s got a link to add the events to a few different calendars, including Google’s.

yea, if you register here: cloudposse.com/office-hours you’ll get a calendar invite

@Andy Miguel (Cloud Posse) we should create a public calendar too for people to subscribe to