#aws (2020-11)

aws Discussion related to Amazon Web Services (AWS)

aws Discussion related to Amazon Web Services (AWS) Archive: https://archive.sweetops.com/aws/

2020-11-30

Steve Wade avatar
Steve Wade

is anyone able to help me understand VPC peering, I have created a terraform module and applied it but am struggling to understand one thing ….

I want to peer the private subnets from account X with the database subnets from account Y

Where do I need to enable dns resolution to allow the instances in the private subnet to use internal hosted zones in account Y ?

Santiago Campuzano avatar
Santiago Campuzano

@ you need to enable DNS resolution at the VPC peering config

Santiago Campuzano avatar
Santiago Campuzano

Let me show you

Steve Wade avatar
Steve Wade

thanks man your help is much appreciated

Santiago Campuzano avatar
Santiago Campuzano
02:53:39 PM
Santiago Campuzano avatar
Santiago Campuzano

We enabled it for some of our VPC peerings.

Steve Wade avatar
Steve Wade

so i need to enable in the VPC connection in account X, right?

Santiago Campuzano avatar
Santiago Campuzano

Yep//// you’re right !

Steve Wade avatar
Steve Wade

let me try that

Steve Wade avatar
Steve Wade

it doesn’t seem to be working with nslookup

Santiago Campuzano avatar
Santiago Campuzano

Wait… I think you should do it the other way….

Steve Wade avatar
Steve Wade

ok let me try that

Santiago Campuzano avatar
Santiago Campuzano

I read your config again

Steve Wade avatar
Steve Wade

trying …

Santiago Campuzano avatar
Santiago Campuzano

Cool… let me know how that goes

Steve Wade avatar
Steve Wade

will do

Steve Wade avatar
Steve Wade

nope

Steve Wade avatar
Steve Wade

could it be security group related?

Santiago Campuzano avatar
Santiago Campuzano

Wait…weird…. are you having issues with DNS resolution or with TCP connectivity ?

Steve Wade avatar
Steve Wade
[email protected]:/# nslookup elastic.logging.de.qa.underwriteme.internal
Server:         172.20.0.10
Address:        172.20.0.10#53

** server can't find elastic.logging.de.qa.underwriteme.internal: NXDOMAIN
Santiago Campuzano avatar
Santiago Campuzano

Is that K8S by any chance ?

Steve Wade avatar
Steve Wade

100% its EKS

Santiago Campuzano avatar
Santiago Campuzano

Are you doing that from a POD ?

Steve Wade avatar
Steve Wade

yes i am

Santiago Campuzano avatar
Santiago Campuzano

I’m wondering if that could be an issue… would you be able to do that from a regular EC2 instance ?

Steve Wade avatar
Steve Wade

we are using bottlerocket so going onto the node isn’t the easiest for me unfortunately

Santiago Campuzano avatar
Santiago Campuzano

Hmmm gotcha

Steve Wade avatar
Steve Wade

i might just try dns resolution from both sides

Steve Wade avatar
Steve Wade

oh shoot is it because i am peering with the database subnets from the upstream VPC module which deny inbound, don’t they?

Santiago Campuzano avatar
Santiago Campuzano

What do you mean. When you configure VPC peering, you peer the entire VPC

Santiago Campuzano avatar
Santiago Campuzano

And then you configure SGs and Routing Tables to allow certain traffic

Santiago Campuzano avatar
Santiago Campuzano

Have you configured both ?

Steve Wade avatar
Steve Wade

let me create a gist it might be easier

Steve Wade avatar
Steve Wade

i tried traceroute as well to no avail

Steve Wade avatar
Steve Wade

i think my routing logic is correct

Steve Wade avatar
Steve Wade

@Santiago Campuzano i have an update, i can nslookup from account X to an instance in the database subnet in account Y without a problem, the issue is just DNS resolution is not working

Steve Wade avatar
Steve Wade
[email protected]:/# nslookup redacted.cssoccfscup8.eu-west-1.rds.amazonaws.com
Server:         172.20.0.10
Address:        172.20.0.10#53

Non-authoritative answer:
Name:   redacted.cssoccfscup8.eu-west-1.rds.amazonaws.com
Address: 10.60.26.86
Santiago Campuzano avatar
Santiago Campuzano

@ What do you mean by you can nslookup but DNS resolution is not working. ?

Santiago Campuzano avatar
Santiago Campuzano

That is like contradictory

Santiago Campuzano avatar
Santiago Campuzano

I mean… nslookup is pretty much DNS resolution

Steve Wade avatar
Steve Wade

i can nslookup to an AWS known DNS record (see above)

Santiago Campuzano avatar
Santiago Campuzano

Ok… and you’re able to get the private IP address from that RDS instance

Steve Wade avatar
Steve Wade

i can’t resolve custom private DNS addresses

Steve Wade avatar
Steve Wade

e.g. [yellowfin.rds.de.qa](http://yellowfin.rds.de.qa).underwriteme.internal which is a CNAME to an RDS endpoint

Santiago Campuzano avatar
Santiago Campuzano

Hmmm got it… you’re resolving the public IP, not the private one

Steve Wade avatar
Steve Wade

currently i have dns resolution setup on both sides

Steve Wade avatar
Steve Wade

maybe i need to turn off account X like you said before?

Santiago Campuzano avatar
Santiago Campuzano

I would give it a try….

Steve Wade avatar
Steve Wade

made no difference

Steve Wade avatar
Steve Wade

i wonder if this has something to do with coredns configuration?

Steve Wade avatar
Steve Wade

or maybe i need to create a route53 resolver?

Santiago Campuzano avatar
Santiago Campuzano

I was checking my config, and I don’t have the R53 resolver configured

Steve Wade avatar
Steve Wade

but are you peering the whole VPCs together?

Steve Wade avatar
Steve Wade

or subnets in account X to others in account Y ?

Santiago Campuzano avatar
Santiago Campuzano

The whole VPCs together

Steve Wade avatar
Steve Wade

yeh i think that might be the difference

Santiago Campuzano avatar
Santiago Campuzano

Well…. wait.. is it possible to peer certain subnets ?

Santiago Campuzano avatar
Santiago Campuzano

AFAIR you peer the entire VPC

Santiago Campuzano avatar
Santiago Campuzano

not certain subnets/CIDRs

Steve Wade avatar
Steve Wade

well the route tables are subnets

Santiago Campuzano avatar
Santiago Campuzano

Ok… The route tables indicate what traffic to send through the VPC peering…

Steve Wade avatar
Steve Wade

yeh mine looks right

Steve Wade avatar
Steve Wade

the issue is just the private dns resolution is not working

Santiago Campuzano avatar
Santiago Campuzano

I would discard. the EKS component of this equation

Santiago Campuzano avatar
Santiago Campuzano

I’d create a simple Ec2 instance and try from there

Santiago Campuzano avatar
Santiago Campuzano

The DNS resolution

Santiago Campuzano avatar
Santiago Campuzano

EKS/K8S DNS resolution is weird sometimes

Steve Wade avatar
Steve Wade
Enabling DNS resolution for Amazon EKS cluster endpoints | Amazon Web Services attachment image

Update – December 2019 Amazon EKS now supports automatic DNS resolution for private cluster endpoints. This feature works automatically for all EKS clusters. You can still implement the solution described below, but this is not required for the majority of use cases. Learn more in the What’s New post or Amazon EKS documentation. This post […]

Santiago Campuzano avatar
Santiago Campuzano

Morning @ !

Santiago Campuzano avatar
Santiago Campuzano

How this ended up for you ?

Santiago Campuzano avatar
Santiago Campuzano

The DNS resolution issue ?

Steve Wade avatar
Steve Wade

after talking to AWS you can do this via the command line

bazbremner avatar
bazbremner

Anyone using ACM private CA (PCA) with root CA and subordinate CAs in separate accounts (as recommended by the best practice section of the PCA docs)? I’m bashing my head against the wall trying to work out how to have the root CA sign the subordinate’s certificate and import the certificate to get it into an active state.

bazbremner avatar
bazbremner

If I try a RAM share of the subordinate into the root CA’s account (which uses the RAM-managed permissions) trying to sign the subordinate from the root account fails with 1 validation error detected: Value at 'csr' failed to satisfy constraint: Member must satisfy regular expression pattern: -----BEGIN CERTIFICATE REQUEST-----... which makes me suspect that the root CA/account can’t read the CSR for the subordinate, but of course I can’t change the permissions since RAM manages them. Can’t see what the alternative workflow would look like - I don’t see anything in the docs.

bazbremner avatar
bazbremner

Oh and yes, trying to view the CSR of the subordinate as shared by RAM in the root CA account just beachballs, which is another tint on the permissions side of things.

bazbremner avatar
bazbremner

Going the other way, and sharing the root CA into the account with the subordinate doesn’t work with the “Import CA” -> “ACM private CA” flow in the console, as the root isn’t presented as an option.

bazbremner avatar
bazbremner

…And Terraform doesn’t support most of the PCA operations.

Rob Williams avatar
Rob Williams

Hey folks Whats the recommended way to partition environments and related infra? want to keep prod env as much isolated as possible, but still want to ensure pace of development. We are a relatively new startup ~6 eng, so am just setting things and process up.

roth.andy avatar
roth.andy

either separate AWS accounts or separate VPCs is what we do (depending on scale)

roth.andy avatar
roth.andy

@Erik Osterman (Cloud Posse) did a good overview in office hours a few weeks ago

Rob Williams avatar
Rob Williams

ah thanks, do we have recording of those? Till now I have decided to go with separate vpc, not sure where shared services like logs/metrics etc should go. I have read that they should generally go into a different “shared” environment, but not sure of how their config etc would be tested. Like where should staging version of such shared services go?

roth.andy avatar
roth.andy
Cloud Posse

We’re a DevOps accelerator. That means we help companies own their infrastructure in record time by building it with you and then showing you the ropes. If t…

roth.andy avatar
roth.andy

I couldn’t find the exact video (there’s a lot there). A “Cloud Posse Explains: AWS Account/Environment Laydown” video would be helpful

roth.andy avatar
roth.andy
05:16:31 PM

Who the hell is that ugly guy @Erik Osterman (Cloud Posse)?? You really should keep out the rifraf

1
Rob Williams avatar
Rob Williams

ah thanks for the help and the direction.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We’re actively releasing stuff every week as part of our reference architecture. It’s too early to point you to something concrete. But by end of december it will be much closer. Documentation will begin in earnest Q1 2021.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

All of our top-level AWS components are here: https://github.com/cloudposse/terraform-aws-components

cloudposse/terraform-aws-components

Catalog of reusable Terraform components and blueprints for provisioning reference architectures - cloudposse/terraform-aws-components

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

this is the future home of our reference-architectures https://github.com/cloudposse/reference-architectures but its documentation is stale and examples incomplete.

cloudposse/reference-architectures

[WIP] Get up and running quickly with one of our reference architecture using our fully automated cold-start process. - cloudposse/reference-architectures

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we’ll definitely be recording lots of videos, demos, walkthrus in addition to more step by step documentation.

Vlad Ionescu (he/him) avatar
Vlad Ionescu (he/him)
jose.amengual avatar
jose.amengual

what, they do not have enough seats?????????

Alex Jurkiewicz avatar
Alex Jurkiewicz

scalpers probably going to sell seats on ebay

jose.amengual avatar
jose.amengual

HAHAHA lol

2020-11-29

RB avatar

Is there a simple explanation of minimum security group configuration for an internal load balancer, target groups, and ec2 instances ? I’m a bit confused

RB avatar
Security groups for your Application Load Balancer - Elastic Load Balancing

Learn how to update the security groups for your Application Load Balancer.

Alex Jurkiewicz avatar
Alex Jurkiewicz

The security group on the EC2 instances should allow ingress and egress to the load balancer security group, and vice versa.

I would suggest adding allow all out egress rule to both groups as well though.

1
RB avatar

Anyone know if the a) asg instance refresh or b) manual termination invokes the lifecycle hooks?

RB avatar

There are only 2 references to lifecycle hooks here https://docs.aws.amazon.com/autoscaling/ec2/userguide/asg-instance-refresh.html

Replacing Auto Scaling instances based on an instance refresh - Amazon EC2 Auto Scaling

Describes the scenario of replacing the Amazon EC2 instances in your Auto Scaling group, based on an instance refresh.

Zach avatar

It does yes

Zach avatar

Well, I can say for sure it certainly invokes the create hook, we don’t make use of the termination one

RB avatar

You’re saying only the insurance refresh will invoke the lifecycle hook?

Zach avatar

No I’m just confirming that instance-refresh definitely does. I assume it triggers in both cases that you’re asking though

1
Chris Fowles avatar
Chris Fowles

i can confirm it triggers the termination hooks

1
Chris Fowles avatar
Chris Fowles

we make use of it regularly

RB avatar

from my tests, it looks like manual termination of the ec2 instance does not invoke the lifecycle hook

RB avatar

ah and it looks like an instance refresh does invoke the lifecycle hook. thank you.

Zach avatar

interesting. I guess they take the stance that if you manually asked for a termination that you want it done right now

RB avatar

weird. i spoke with support and they said that manual terminating should invoke the hook

RB avatar

perhaps the https://github.com/getsocial-rnd/ecs-drain-lambda just doesn’t work for the manual termination

getsocial-rnd/ecs-drain-lambda

Automation of Draining ECS instances with Lambda, based on Autoscaling Group Lifecycle hooks or Spot Instance Interruption Notices - getsocial-rnd/ecs-drain-lambda

RB avatar

ah ok it was a miscommunication. Confirmed with support that manual termination from the UI does not run the lifecycle hook but it does if the aws autoscaling terminate-instance-in-auto-scaling-group cli command is run

RB avatar

oh and what about asg max instance lifetime settings ? does it invoke the lifecycle hook too ?

Replacing Auto Scaling instances based on maximum instance lifetime - Amazon EC2 Auto Scaling

Describes the scenario of replacing the Amazon EC2 instances in your Auto Scaling group, based on a maximum instance lifetime limit.

2020-11-28

Vlad Ionescu (he/him) avatar
Vlad Ionescu (he/him)

AWS Kinesis incident details: https://aws.amazon.com/message/11201/

1
antonbabenko avatar
antonbabenko

Good week-ends reeding In short, use just EC2 instances and don’t use new and fancy services because everything is interconnected internally.

Vlad Ionescu (he/him) avatar
Vlad Ionescu (he/him)

I deeply disagree with that. The best example against it is the experience iRobot had: https://twitter.com/ben11kehoe/status/1332391783829913600

Yes, they had a bad incident, but it was: 1) rare, and by managing EC2 themselves they would’ve definitely had more incidents + higher cost + huge opportunity cost 2) relaxed, as in they had to wait for AWS to recover. No stress, no fighting to figure out the root cause, no marketing hit of “Roombas are down” but “A bunch of IoT things are down

I want to talk a bit about what this was like. tl;dr: it was long and inconvenient timing but, as an operations team, not particularly stressful. Questions of “when”, not how or if systems would come back. A lot of waiting and watching—and that’s desirable. https://twitter.com/ben11kehoe/status/1332028868740354048

I’m still working with my team to mitigate the after effects of the day-long AWS outage yesterday, including dealing with follow-on AWS issues. I’ve gotten three hours of sleep and it’s ruining my Thanksgiving day. Hot take: I am thankful we have built serverless on AWS.

antonbabenko avatar
antonbabenko

… it was trolling from my side. I understand that such incidents happen all the time. Smaller or larger, and it is our job to prepare and minimize impact as much as we can.

Vlad Ionescu (he/him) avatar
Vlad Ionescu (he/him)

welp, I did not catch that. Sorry!

antonbabenko avatar
antonbabenko

Some people posted on linkedin during an incident that their EC2+RDS services were 100% up during that period, so they are lucky that they don’t use fancy services… and then it started a “discussion” ec2 VS serverless :)))

Yoni Leitersdorf (Indeni Cloudrail) avatar
Yoni Leitersdorf (Indeni Cloudrail)

The discussion we’re having is: “should we replace AZ-level redundancy (within us-east-1), with the more resource intensive region-level redundancy”.

2020-11-27

Joan Porta avatar
Joan Porta

Hi guys! QQ we are in AWS, and we are finding some global cron scheduler to run tasks. I know Rundeck but something more SaaS? Something that dev teams do not depend on IT operations to create things in terraform like lambdas triggered by cloud events, or ECS tasks that run based on events…. need something easy and fast for dev’s. Thx!!!!!

loren avatar
loren

Cloudwatch events has cron support for scheduled triggers, codebuild works as a target and can run arbitrary commands

loren avatar
loren

Or cloudwatch events to lambda works fine

loren avatar
loren

I think ssm automation is also an option

Joan Porta avatar
Joan Porta
blinkist/terraform-aws-airship-ecs-service

Terraform module which creates an ECS Service, IAM roles, Scaling, ALB listener rules.. Fargate & AWSVPC compatible - blinkist/terraform-aws-airship-ecs-service

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We even have an #airship channel :-)

Joan Porta avatar
Joan Porta

cool!

Joe Niland avatar
Joe Niland

Sounds like you want something managed? But how about Serverless Framework? It satisfies your Lambda requirement easily. Defining events is really easy in the serverless.yml file.

Not sure how well it’ll fit with ECS but found this: https://www.serverless.com/plugins/serverless-fargate-tasks

Part of the challenge here is coming up with sufficiently restrictive policies to allow developers to deploy without being able to change things they shouldn’t.

Also managing everything for cost and tracking purposes will require some thought. Serverless.com paid offering might help with that but I’ve not used it yet.

Serverless Fargate Tasks

A plugin to run fargate tasks as part of your Serverless project

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

also, since you mentioned something easy one thing we use a lot of is Zapier and Integromat. Both of them have cron like functionality.

Joe Niland avatar
Joe Niland

Oh yeah, ofc! @Erik Osterman (Cloud Posse) just curious - would you choose one over the other? I was looking at doing something with Xero the other day, and for some reason the Integromat connector was more full-featured.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yea, we started with Zapier so we have ~300 zaps there. However, if we were starting from scratch would 100% use Integromat. Easy things are a little bit harder to do in integromat (compared to Zapier), but it more than makes up for it with all the advanced features. I feel like Integromat is more geared towards advanced users / programmers. It has iterators, routers, variables, etc. If their integration doesn’t provide the API endpoint you want, they usually provide a raw method to use the integration. Integromat supports handling errors, while Zapier doesn’t.

Joe Niland avatar
Joe Niland

Ah thank you for the insight! Very useful.

Joe Niland avatar
Joe Niland

Integromat’s API looks pretty good too

Joan Porta avatar
Joan Porta

very valuable info, thx all!

2020-11-25

maarten avatar
maarten
1
1
Matt Gowie avatar
Matt Gowie

Yeah, not a fun morning.

jose.amengual avatar
jose.amengual

and this is why I tell people DO NOT DEPLOY in us-east-1

1
Matt Gowie avatar
Matt Gowie

I know better and yet one of my larger clients is running in us-east-1

rei avatar

Do you mean us-east-1 is the aws staging environment?

Maciek Strömich avatar
Maciek Strömich

tbh for me it’s the 2nd larger outage that I’m affected by in roughly 7 years of running stuff in us-east-1. first one being s3 outage (happily we lived through it without downtime) few years back.

Maciek Strömich avatar
Maciek Strömich

and currently i see longer response times from firehose endpoint (800-1000ms instead of normal 50-80ms) and cognito endpoint is completely down

Matt Gowie avatar
Matt Gowie

We’re having troubles with ECS auto scaling and scaling out. Desired tasks threshold is not triggering new task creation and some tasks are failing to be placed due to “limited capacity”. I think it’d all tied to CloudWatch problems.

Maciek Strömich avatar
Maciek Strömich

yeah and CW is failing because of kinesis streams which IMO started the domino to fall. Happily for me I’ve quite good understanding of CPU needs and cosidering those few k rpm i’m receiving on a constant basis if no instance will be replaced i should be fine

Issif avatar
Issif

@ I always thought that

Issif avatar
Issif

when AWS has issues, a lot have too

Issif avatar
Issif
07:34:46 PM
jose.amengual avatar
jose.amengual

still down…..shit tomorrow 3000 people from amazon fired

jose.amengual avatar
jose.amengual

so now that shit hit the fan people will start migration to us-east-2 and then it will go down

voidSurfr avatar
voidSurfr

hey guys, just dipping my toe in with https://registry.terraform.io/modules/cloudposse/tfstate-backend/aws/latest and I can’t get it to create the bucket, lol

Initializing the backend...

Successfully configured the backend "s3"!
...
Error: Failed to get existing workspaces: S3 bucket does not exist.             <--

The referenced S3 bucket must have been previously created. If the S3 bucket    <-- uhhh, what?
was created within the last minute, please wait for a minute or two and try
again.

Error: NoSuchBucket: The specified bucket does not exist
	status code: 404, request id: 08C447AD410DF430, host id: a6btOtHixZYbFcNJ8E+gpLoFP9vw4MIvFfibWxHrtQwB+tf2HSDJ9bbMvmGRBDt9BmqW/XoZUzY=
Matt Gowie avatar
Matt Gowie

You need to first create the resources with a local backend and then transition over.

Matt Gowie avatar
Matt Gowie

Usually you should do this through having a separate root module for your tfstate-backend which bootstraps your backend

1
Matt Gowie avatar
Matt Gowie

@voidSurfr did you see these? I think the above is your immediate problem.

voidSurfr avatar
voidSurfr

sorry, I was attending office hours. I’ll double-check your notes against my issue.

jose.amengual avatar
jose.amengual

you do something like : terraform apply -var-file='production.tfvars' -target module.terraform_state_backend

1
jose.amengual avatar
jose.amengual

with local state defined

jose.amengual avatar
jose.amengual

once the s3 bucket is created , hen you can define the backend-config on a file or main.tf and then

jose.amengual avatar
jose.amengual

terraform init -var-file='production.tfvars' -backend-config=production-backend-us-east-2.tfvars

jose.amengual avatar
jose.amengual

and it will ask you to move the local state over

jose.amengual avatar
jose.amengual

this is all on the README

voidSurfr avatar
voidSurfr

can someone help me to understand this?

2020-11-24

Stephen Bennett avatar
Stephen Bennett

using: https://github.com/cloudposse/terraform-aws-efs is it possible to turn off the creation of a security group and pass one to it instead?

cloudposse/terraform-aws-efs

Terraform Module to define an EFS Filesystem (aka NFS) - cloudposse/terraform-aws-efs

Alex Jurkiewicz avatar
Alex Jurkiewicz
cloudposse/terraform-aws-efs

Terraform Module to define an EFS Filesystem (aka NFS) - cloudposse/terraform-aws-efs

Stephen Bennett avatar
Stephen Bennett

thanks

Shreyank Sharma avatar
Shreyank Sharma

Hi All, We are having an Application in Lambda, is there any way we can backup that for Disaster Recovery. or how Amazon does this for us. Thank you.

ms16 avatar

Hey @ Im not sure what do u mean by backing up a lambda function for DR - I would just keep the infra configuration and the lambda application + cicd code in a git repo ?

Zach avatar

The backup is simply your code in your repo, the ‘lambda’ in AWS is just the runtime configuration

uselessuseofcat avatar
uselessuseofcat

Help, is there any way to list all services that I’m using on my AWS account?

Santiago Campuzano avatar
Santiago Campuzano

Yes.. the simplest way is check your AWS bill

2
Santiago Campuzano avatar
Santiago Campuzano

You can use CostExplorer to get the list of AWS resources/services that you are being charged for

uselessuseofcat avatar
uselessuseofcat

Thanks @Santiago Campuzano

1
RB avatar

my attempt at lifecycle hook niche awesome list https://github.com/nitrocode/awesome-aws-lifecycle-hooks

nitrocode/awesome-aws-lifecycle-hooks

Awesome aws autoscaling lifecycle hooks for ECS, EKS - nitrocode/awesome-aws-lifecycle-hooks

1
RB avatar

i couldn’t find a resource that aggregated all of it. it was a bit disparate. i tried to put it all together.

nitrocode/awesome-aws-lifecycle-hooks

Awesome aws autoscaling lifecycle hooks for ECS, EKS - nitrocode/awesome-aws-lifecycle-hooks

RB avatar

if anyone knows of additional lifecycle hooks for other services or whatever, please let me know

RB avatar

im still trying ot figure out the difference of using only cloudwatch (ecs-drain-lambda), using sqs (claranet method), and using sns

2020-11-23

uselessuseofcat avatar
uselessuseofcat
Hi, I have a domain, for example [test.com> and I want to delegate [delegate.test.com> to another AWS account. I was able to do that but I can only create A records for a.delegate.test.com and <http://b.delegate.test.com b.delegate.test.com](http://delegate.test.com) for example but I need to create A record for <http://delegate.test.com delegate.test.com](http://test.com). Can I do that from the account to which I delegated the zone to? Thanks!
Maciek Strömich avatar
Maciek Strömich

you have two hosted zones now: test.com delegated.test.com

[test.com> has an entry <http://delegate.test.com|delegate.test.com](http://test.com) IN NS …

This means that all records for the domain [delegate.test.com> (including the A record for the top level domain) will be handled by the NS servers configured in the test.com IN NS record and those records are handled by <http://delegate.test.com delegate.test.com](http://test.com) hostedzone (in whatever aws account this hosted zone is deployed)
uselessuseofcat avatar
uselessuseofcat

Thanks Maciek!

2020-11-21

Riak avatar

Hello All, How are you today? ‘Im looking for a cloud formation stack to deploy traefik reverse proxy on ecs

2020-11-20

uselessuseofcat avatar
uselessuseofcat

Has anyone restricted outgoing traffic from ec2 instances via security groups with success? I’ve enabled a lot of VPC endpoints, but I still see a lot of outbound traffic towards AWS subnets for which I cannot identify the service that they belong to.

Darren Cunningham avatar
Darren Cunningham

do you have non-VPC Lambdas? those are usually the culprit for me

Darren Cunningham avatar
Darren Cunningham

wait, you mean just from the ec2? Sorry I thought you meant account wide

curious deviant avatar
curious deviant

Hello

My company has a domain purchased say [mydomain.com> with GoDaddy and it is today mapped to servers running on prem. We are migrating to AWS and have our Route 53 DNS and public hosted zone setup as [ourcoolcloud.com> (say). We want to setup the DNS routing such that when our clients hit [mydomain.com> it actually gets proxied to [ourcoolcloud.com>. We do not want the <http://ourcoolcloud.com ourcoolcloud.com](http://ourcoolcloud.com) to appear in the client browser and we may drop <http://mydomain.com mydomain.com](http://mydomain.com) at some point and purchase a cooler name. We do not want to keep changing our AWS HZs. Is there some DNS voodoo we can do to make this routing happening from <http://mydomain.com mydomain.com](http://ourcoolcloud.com) - > <http://ourcoolcloud.com ourcoolcloud.com](http://mydomain.com) without the client seeing this in the browser ?
roth.andy avatar
roth.andy
Can you change the A record for [mydomain.com> to be the same value as the one for <http://ourcoolcloud.com ourcoolcloud.com](http://mydomain.com)? If you are using an alias to an AWS ALB/ELB then probably not, but if you are just pointing at an IP address then you should be able to just point both A records to the same IP address
curious deviant avatar
curious deviant

I am thinking not since ourcoolcloud.com would typically point to a CloudFront distribution and would have other record sets say ALBs etc.

curious deviant avatar
curious deviant

Would a CNAME record suffice ?

roth.andy avatar
roth.andy

I’m not an expert at DNS. I know just enough to be dangerous. If I was in your shoes I’d be finding someone who is an expert, and if there aren’t any that I can get, hiring a consultant.

DNS can be quite complicated, and in most cases it is a single point of failure. If you screw something up your whole site will go down. Most of the downtime for the big players like CloudFlare, AWS, etc, have been because someone screwed up a DNS setting

1
curious deviant avatar
curious deviant

thank you so much for taking the time to understand the problem. I’ll try and see if I can do a spike or something to test how this would work using a test DNS maybe. I’ll share the results here just for future reference. Thank you again

roth.andy avatar
roth.andy

Alex Jurkiewicz avatar
Alex Jurkiewicz

https://github.com/awsdocs/aws-cloudformation-user-guide/pull/438

A >1 year old trivial doc change finally getting attention, but they require YOU to rebase it. Contributing to AWS docs in a nutshell.

Move docs about accessing nested stack outputs by alexjurkiewicz · Pull Request #438 · awsdocs/aws-cloudformation-user-guide

Issue #, if available: Description of changes: By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

1

2020-11-18

Tomek avatar
Tomek

hey all, was going to cross post here as it sounds like this is a better channel for these types of questions https://sweetops.slack.com/archives/CB6GHNLG0/p1605655766298600

:wave: Is there a way to define the session expiration time for the role an ECS task assumes in terraform? The AWS docs state that the default is 6 hours. max_session_duration for aws_iam_role only sets the allowed max session but it looks like when changing that to 12 hours, the ECS task’s role still uses the default 6 hour session duration

Darren Cunningham avatar
Darren Cunningham

I’m annoyed by the AWS Support Feedback experience. I selected Good rather than Excellent and I got an email from the support engineers reporting manager asking why I gave the engineer a “low” grade. I can’t stop thinking about how awful that is and now every time I go to fill out the AWS Support Feedback form now a little part of me dies. I felt that it was necessary to try and do something about it. http://chng.it/gmtG8xqF

Darren Cunningham avatar
Darren Cunningham

and yes I realize that “that’s how all feedback systems work” but I feel like this is something that we collectively can ask for better options

Darren Cunningham avatar
Darren Cunningham

I’m totally open to feedback too

Alex Jurkiewicz avatar
Alex Jurkiewicz

good old “5 stars is normal service” rating as popularised by the service economy

Alex Jurkiewicz avatar
Alex Jurkiewicz

“help keep our employees precariously employed by giving us an excuse to fire them if they get close to a benefit date”

Alex Jurkiewicz avatar
Alex Jurkiewicz

send this to Corey Quinn

Darren Cunningham avatar
Darren Cunningham

forgive me I’m socially dysfunctional …who’s that?

Darren Cunningham avatar
Darren Cunningham

I can Google, but I just feel weird sending somebody I don’t know a DM…again bad at the social thing. I don’t really take part in the social medias. Slack is the only thing I stick with because it feels like less of a blackhole of dialog..and people are generally more rational.

Zach avatar

he runs the Last Week In AWS blog

Zach avatar

and does AWS consulting

Zach avatar

very funny guy

Darren Cunningham avatar
Darren Cunningham

if others agree with my position and want to share it out I’m all for it. I want to try to do something to change the stats quo because (1) it hurts our fellow engineers (2) it doesn’t help us at all

Darren Cunningham avatar
Darren Cunningham

buuuut, my amount of trying probably equates to about 1hr of effort

Zach avatar

you could just email that to jeff bezos and ask him “wtf”

Darren Cunningham avatar
Darren Cunningham

I could, or I could just drop a dumbbell on my foot

Zach avatar

? I mean that Bezos pretty famously deals with customer feedback like this in a very serious way

Darren Cunningham avatar
Darren Cunningham

oh sorry, I thought you were being facetious

Zach avatar

heh, no - his email at amazon is public and people will on occasion send him product complaints about amazon or aws

Darren Cunningham avatar
Darren Cunningham

I can give that a shot too but just figured that the only way real change was going to happen if Amazon saw that it’s something we collectively cared about as it’s customers…you k now how typical large enterprises operate

Darren Cunningham avatar
Darren Cunningham

thanks @ I sent it off – who knows if it will go anywhere but makes me feel like I’m doing my part to try to help our fellow engineers working the support desk at AWS

1

2020-11-17

jfraley avatar
jfraley

so I have a distribution working with 2x origins, but having trouble getting the /bat/*, /biz/** routes to flow to bucketB in example above ^

jfraley avatar
jfraley
404 Not Found
Code: NoSuchKey
Message: The specified key does not exist.
Key: bat/index.html
jfraley avatar
jfraley

so all content needs to be in the /bat/ directory on the target origin ^

maarten avatar
maarten

Instead of using a bucket as origin, you can also use a bucket as “Custom origin”, this way it should be possible to change the default path.

Alex Jurkiewicz avatar
Alex Jurkiewicz

yep

2020-11-14

jfraley avatar
jfraley

planning out a future website structure and wondering if I can have a single cloudfront distro with two S3 bucket origins

cloudfront distro: sample.cloudfront.net CNAME/Alias that points foo.com to sample.cloudfront.net

S3 bucketA contains: foo.com [foo.com/foo/](http://foo.com/foo/) [foo.com/bar/](http://foo.com/bar/)

S3 bucketB contains: [foo.com/bat/](http://foo.com/bat/) [foo.com/biz/](http://foo.com/biz/)

seems like I cannot do this at the cloudfront origin level even with using a custom behavior and a pattern match

can I: put redirects in bucket A to redirect [foo.com/bat/> and <http://foo.com/biz|foo.com/biz](http://foo.com/bat/) to bucket B ? if I do that ^ should I put another cloudfront distro in front of bucket B (bucketB.cloudfront.net) and point the redirects at that ?

so that:

[foo.com/](http://foo.com/)-> alias for sample.cloudfront.net -> bucketA [foo.com/foo/](http://foo.com/foo/) -> alias for sample.cloudfront.net -> bucketA [foo.com/bar/](http://foo.com/bar/) -> alias for sample.cloudfront.net -> bucketA

[foo.com/bat/*> -> alias for sample.cloudfront.net -> bucketA -> bucketA redirect to <http://bucketB.cloudfront.net bucketB.cloudfront.net](http://sample.cloudfront.net) -> bucketB
[foo.com/biz/*> -> alias for sample.cloudfront.net -> bucketA -> bucketA redirect to <http://bucketB.cloudfront.net bucketB.cloudfront.net](http://sample.cloudfront.net) -> bucketB

any other options ?

some top level router thing that points paths to cloudfront distros ?

Alex Jurkiewicz avatar
Alex Jurkiewicz

You can do that. It would be represented as two origins, four url behaviours, and one default url behaviour

Alex Jurkiewicz avatar
Alex Jurkiewicz

There’s a rather low limit on url behaviours by default though (I think 20) so you can’t scale it up too far if that’s a concern

jfraley avatar
jfraley

@ thanks! testing it out today.

jose.amengual avatar
jose.amengual

I don’t know if cloudfront supports multiple origins but I have done this with cloudfront and a nginx proxy and many buckets with different routes

2020-11-13

Francesco Ciocchetti avatar
Francesco Ciocchetti

Hi Everyone. I just stared experimenting with IRSA on EKS and it is working great.

I just have one quesiton about the projected volume with the JWT token from AWS

  volumes:
  - name: aws-iam-token
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          audience: sts.amazonaws.com
          expirationSeconds: 86400
          path: token

What happens when the expirationSeconds expire ? I know from the k8s documentation ( so one where audinece is not external ) that it will renew at 80% … is it the same in this case ?

maarten avatar
maarten
Meet the newest AWS Heroes including the first DevTools Heroes! | Amazon Web Services attachment image

The AWS Heroes program recognizes individuals from around the world who have extensive AWS knowledge and go above and beyond to share their expertise with others. The program continues to grow, to better recognize the most influential community leaders across a variety of technical disciplines. Introducing AWS DevTools Heroes Today we are introducing AWS DevTools […]

Shreyank Sharma avatar
Shreyank Sharma

Am trying cluster migration in AWS, Both  k8s clusters are in same region. Cluster 1 : Deployed 2 Application with PV reclaim policy one as Delete and another as Retain, and annotated so it will take Restic backup. Cluster 2: Restored those 2 applications, worked fine.

again Cluster 1: Deployed same 2 application with Reclaim policy as Delete and Retain but not annotated so it took snapshot when i backup. Cluster 2: Restore did not work as PV volume is failed to attach with the following Warning FailedAttachVolume pod/<pod-name> AttachVolume.Attach failed for volume "pvc-<id>" : Error attaching EBS volume "vol-<id>" to instance "i-<instance-id>": "UnauthorizedOperation: You are not authorized to perform this operation.

So, Snapshot restore feature will work in the same AWS region or am only getting this error????

2020-11-12

Stephen Bennett avatar
Stephen Bennett

hi, for the cloudposse/elasticache-redis/aws module, there is no mention of tags. anyone able to tel me how to?

RB avatar

So what kind of problems are you folks trying to or have successfully solved recently?

rei avatar

I am trying to get IaaC right in oder to deploy the same EKS cluster with service roles and helmfiles for alb, ingress, cert-manager, external-dns, etc. And for several environments (dev, prod, stg, …) And keeping the differences between them low. At the end Gitlab CI will deploy each branch to an own namespace “deployment” to be accessed using an unique URL. VPN access too. And you?

RB avatar

that sounds like a beautiful endeavor! I tried to do something similar. EKS has been lowered in priority for us so we won’t get to conquer it until Q1 next year.

I’m currently trying to solve giving devs iam permissions using a permission boundary. I’m hoping I can get this out the door this week and unblock a lot of devs. Meanwhile, I’ve been exploring arm a lot in our ECS clusters so it always seems to be a balancing act.

loren avatar
loren

@RB here’s how we manage the requirement for the permissions boundary…

RB avatar
Thanks Loren. I’m using basically this [same policy> from the <https://aws.amazon.com/blogs/security/delegate-permission-management-to-developers-using-iam-permissions-boundaries/ aws blog](https://aws.amazon.com/premiumsupport/knowledge-center/iam-permission-boundaries/) on perm boundary
Delegate permission management to developers by using IAM permissions boundaries | Amazon Web Services attachment image

Today, AWS released a new IAM feature that makes it easier for you to delegate permissions management to trusted employees. As your organization grows, you might want to allow trusted employees to configure and manage IAM permissions to help your organization scale permission management and move workloads to AWS faster. For example, you might want […]

loren avatar
loren

yeah, that one flips the logic, using Deny with StringNotEquals instead of Allow with StringEquals

1
RB avatar

may i ask what you use for the permission boundary policy ? we’re still trying to figure out what’s the best maximum permissions to give without giving too much. we were thinking of removing all deletion, for instance.

loren avatar
loren

we actually use the same policy we assign to the role. it’s primarily to prevent privilege-escalation, for our use case

loren avatar
loren

that doesn’t work of course if you apply multiple policies to the role

loren avatar
loren

but say you have a DEV role, with only the DEV policy attached, you can require the DEV policy as the permissions boundary for any role/user they create to prevent privilege escalation

loren avatar
loren

pain points have been places where the console will auto-create roles when the user first attempts to use the service. such as service-linked roles, and the rds-monitoring role, etc… so we pre-create those in the account as we identify them

RB avatar

ah so whatever access the DEV role has, it can also create roles with max permissions as the DEV role since the single DEV policy is used as the boundary

RB avatar

our issue is that we have multiple iam policies associated to our DEV role

RB avatar

so i was going to create a separate maximum iam policy to use as a perm boundary policy

loren avatar
loren

yeah, so that doesn’t work so well then, and you have to come up with an actual policy for the perm boundary separately

RB avatar


pain points have been places where the console will auto-create roles when the user first attempts to use the service. such as service-linked roles, and the rds-monitoring role, etc… so we pre-create those in the account as we identify them
yes, i can see this being an issue. we have most of these created in our accounts so hopefully this wont be an issue. we’re worried about existing iam roles exceeding the perm boundary so retroactively attaching may be an issue

RB avatar


yeah, so that doesn’t work so well then, and you have to come up with an actual policy for the perm boundary separately
exactly. we’ll have to be crafty here. not much online about this except for the aws blog side.

loren avatar
loren

yes, we also rolled through all accounts and slapped the perm boundary on pre-existing roles/users when we shipped the perm boundary requirement

RB avatar

awesome! how did you verify that each role did not exceed the permission boundary tho ?

loren avatar
loren

it’s a fairly permissive policy for this environment, so we didn’t expect issues and encountered none

RB avatar

ah I see. We have a more strict environment. Perhaps that’s something we can work on.. making things looser.

loren avatar
loren

yeah, the stricter the perm boundary policy, the more likely you’ll have problems

loren avatar
loren

or, you may need multiple perm boundary policies for different use cases

1
loren avatar
loren

we stick with basic stuff around security and auditing that we manage in the account. if we create it, you can’t touch it. that kind of thing

kskewes avatar
kskewes

Thanks everyone this looks like will help us out with IAM permissions for devs in their sandpit account. Currently IAM:* is basically off but we need to enable so they can do lambda etc. Will do some reading.

kskewes avatar
kskewes

We’re currently investigating rabbitmq service. Want to kick our k8s one out of cluster and get from 115 to 118 etc.

maarten avatar
maarten

Has anyone played with https://github.com/aws-samples/aws-secure-environment-accelerator/ before ? It’s massive.

aws-samples/aws-secure-environment-accelerator

The AWS Secure Environment Accelerator is a tool designed to help deploy and operate secure multi-account AWS environments on an ongoing basis. The power of the solution is the configuration file w…

RB avatar

i have not. i dont think i would touch it because it is massive. i’d prefer if it was broken up into modules.

we dont use code commit either.

aws-samples/aws-secure-environment-accelerator

The AWS Secure Environment Accelerator is a tool designed to help deploy and operate secure multi-account AWS environments on an ongoing basis. The power of the solution is the configuration file w…

1
uselessuseofcat avatar
uselessuseofcat

Hi, one noob question - what about ALB outbound traffic? When client make request on port 443, for example, to some application the request will go through ALB to that EC2 instance and then response will go through ALB again (on high number unprivileged ports), so I was wondering, if I restrict egress traffic on my EC2 instances’ SG, should I also do the same for ALB? Is there a reason to restrict egress traffic on ALB? Thakns!

EvanG avatar
EvanG

Is there a reason that this can’t be handled directly on the ALB itself? I’m a beginner too.

uselessuseofcat avatar
uselessuseofcat

ALB has it’s own security group.

EvanG avatar
EvanG

How did you teach yourself about SGs?

uselessuseofcat avatar
uselessuseofcat

The hard way. You should go and buy Stephane Mareek course on Udemy!

1
EvanG avatar
EvanG

not enough I need to just read some simple code and keep moving

EvanG avatar
EvanG

Does anyone here use a framework for testing CIS Benchmark foundations in AWS? I’ve seen things like: https://github.com/mberger/aws-cis-security-benchmark#description, but I’m wondering if they’re worth the risk.

mberger/aws-cis-security-benchmark

Tool based on AWS-CLI commands for AWS account hardening, following guidelines of the CIS Amazon Web Services Foundations Benchmark (https://d0.awsstatic.com/whitepapers/compliance/AWS_CIS_Foundati

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

and by testing, do you mean ensuring your safeguards for CIS compliance are working, or putting the safeguards in place?

mberger/aws-cis-security-benchmark

Tool based on AWS-CLI commands for AWS account hardening, following guidelines of the CIS Amazon Web Services Foundations Benchmark (https://d0.awsstatic.com/whitepapers/compliance/AWS_CIS_Foundati

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Because if you mean the latter, that’s what you get with SecurityHub standards.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We’ll have support for security hub in the next day or so. @matt is working out the last details fixing our tests for https://github.com/cloudposse/terraform-aws-security-hub/pull/3

EvanG avatar
EvanG

I really don’t know what that code is doing. I learned what a resource and datasource was last week. Time to give it the old college try.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

are you using terraform at your company?

EvanG avatar
EvanG

Yes

EvanG avatar
EvanG

I’m trying to develop my skills as a devops engineer.

1
EvanG avatar
EvanG

I keep on getting

Error updating CloudTrail: InsufficientEncryptionPolicyException: Insufficient permissions to access S3 bucket nf-cis-benchmark or KMS key arn:aws:kms:us-east-1:721086286010:key/af37e59b-e5fa-446c-b43c-180b425c6222.

when I try to apply my terraform with a kms_key

resource aws_kms_key "cisbenchmark" {
  description             = "cis-benchmark"
  enable_key_rotation     = true
}

I’m trying to setup my cloudtrail with a key like so

resource "aws_cloudtrail" "cisbenchmark" {
  name                          = "cis-benchmark"
  s3_bucket_name                = aws_s3_bucket.cisbenchmark.id
  enable_logging                = var.enable_logging
  enable_log_file_validation    = var.enable_log_file_validation
  is_multi_region_trail         = var.is_multi_region_trail
  include_global_service_events = var.include_global_service_events
  is_organization_trail         = var.is_organization_trail
  kms_key_id                    = aws_kms_key.cisbenchmark.arn

  # CIS Benchmark 3.1 Ensure CloudTrail is enabled in all regions
  # for a multi-regions trail, ensuring that management events configured for all type of
  # Read/Writes ensures recording of management operations that are performed on
  # all resources in an AWS account
  event_selector {
    # Specify if you want your event selector to include management events for your trail.
    include_management_events = true
    # Specify if you want your trail to log read-only events, write-only events, or all. By default, 
    # the value is All. Needed for logging management events.
  }
}

I stole some of this from your guys code.

matt avatar
matt
08:25:49 PM

@matt has joined the channel

aaratn avatar
aaratn
Using AWS Lambda extensions to send logs to custom destinations | Amazon Web Services attachment image

You can now send logs from AWS Lambda functions directly to a destination of your choice using AWS Lambda Extensions. Lambda Extensions are a new way for monitoring, observability, security, and governance tools to easily integrate with AWS Lambda. For more information, see “Introducing AWS Lambda Extensions – In preview”. To help you troubleshoot failures […]

1

2020-11-11

kalyan M avatar
kalyan M

is there any software that can restrict the users to just view the code. instead of modifying or downloading the code. even copy/Paste?

Jonathan Marcus avatar
Jonathan Marcus

There are solutions for this but they tend to be very intrusive and enterprise-y. Not really an #aws issue. Some examples:

• Citrix makes a remote-access virtualization product. You open up your app through it, and then users log in via a special Citrix agent that prohibits copy/paste.

• On iOS and Android there are enterprise management tools (i.e., rootkits but from your employer) that can create special sandboxes that prohibit copy/paste, screenshots, etc.

• A simpler one is Windows Remote Desktop. IIRC you can set it so that copy+paste works within the RDP session but is disallowed between remote and local.

RB avatar

if you can view the code, couldn’t you also copy and paste the code ?

Issif avatar
Issif

screenshot and paste the image troll

Alex Jurkiewicz avatar
Alex Jurkiewicz

The most suitable software is a contract making them pay you lots of money if they copy the code, I think.

rms1000watt avatar
rms1000watt

Does anyone have any experience around Aurora Failures in Prod?

(We’re going through planning on migrating to Aurora, but just curious of pitfalls to be aware of.. like.. “too many writes will knock Aurora over” or something based on past experience, instead of hypothetical: ‘it should be great!’)

Alex Jurkiewicz avatar
Alex Jurkiewicz

We migrated to aurora (MySQL) and very happy.

The replication is disk block based, so during failovers you lose almost no data, and they are very fast.

The biggest caveat is that Aurora is slower for writes than a traditional MySQL master. Because it waits for writes to be acked across multiple AZs. Not that much slower, but AWS don’t really mention this anywhere as it’s the only meaningful regression

1
kskewes avatar
kskewes

Can’t update from 5.6 to 5.7 in place. Cloudposse module wants to recreate instances on minor engine version change. You can avoid by forking module and add lifecycle version ignores…

1
kskewes avatar
kskewes

Otherwise been great for us too. Fail over quick. Odd MySQL error resulted in crash and fail over but fixed on later versions.

jose.amengual avatar
jose.amengual

very heavy concurrent writes will force a failover of the cluster instances due to the things mention about the storage so as long as you can slow down your writes then you will be ok

jose.amengual avatar
jose.amengual

I’m talking about millions of rows here

jose.amengual avatar
jose.amengual

we have tables with more than 1B rows

Alex Jurkiewicz avatar
Alex Jurkiewicz

I haven’t seen failover caused by high write load. We’ve hit >30k WIOPS sustained without issues

1
jose.amengual avatar
jose.amengual

we can literally crash it at will on an aurora 5.6 but again that is usually not a limit you will get to and there is still ways to avoid it

jose.amengual avatar
jose.amengual

we use aurora for many other things and no issues

jose.amengual avatar
jose.amengual

One of the things I like the most is how fast the clones are since is basically “sharing” the same storage it is pretty fast

rms1000watt avatar
rms1000watt

@jose.amengual We have like ~3TB of data we’re going to migrate and I do think one or more of our tables have > 1B rows

We’re also going to migrate to Aurora Postgres rather than MySQL. @jose.amengual curious if you could clue me into how many writes would cause you to fall over?

jose.amengual avatar
jose.amengual

mmmm I do not know exactly how many but I can tell you how I did it

rms1000watt avatar
rms1000watt

yeah, that’d be good to know, if possible

Ronak avatar
Ronak

I work with @rms1000watt and we just did a sync of data using pglogical from rds pg -> aurora pg (11.8) and it went up to 100k WIOPS and stayed there without any issue. This is not live traffic just syncing data over. Really pleased with the performance as of now.

2
jose.amengual avatar
jose.amengual
I had 15x10GB files with two columns Name MD5 hash and I had 15 tables, each table named after the file like file_1, file_2 etc and I was importing those files into each table, it was able to load 3 in parallel ( although writes are sequential) when I aded to more then it will just failover
jose.amengual avatar
jose.amengual

but you need to keep in mind that the size of the instances matter

jose.amengual avatar
jose.amengual

so it is a combination of the two, I was using db.r5.4xl

Ronak avatar
Ronak

yes we are using r5.24xlarge

jose.amengual avatar
jose.amengual

lol

Ronak avatar
Ronak

I do want to scale it down but not during this transition

jose.amengual avatar
jose.amengual

and keep in mind too, no tuning nothing, straight aurora

jose.amengual avatar
jose.amengual

we use mostly the reader for the app so we can basically write all the time

jose.amengual avatar
jose.amengual

you can do multimaster too that can double the WIOPS

jose.amengual avatar
jose.amengual

and does some sort of sharding capabilities

Ronak avatar
Ronak

I am really interested in multimaster!! Do you have any doc or article around this?

Ronak avatar
Ronak
Working with Aurora multi-master clusters - Amazon Aurora

Use multi-master clusters for continuous availability. All DB instances in such clusters have read/write capability.

Ronak avatar
Ronak

I will check it out.

Ronak avatar
Ronak

This seems to not support PG

Currently, multi-master clusters require Aurora MySQL version 1, which is compatible with MySQL 5.6. When specifying the DB engine version in the AWS Management Console, AWS CLI, or RDS API, choose 5.6.10a.
jose.amengual avatar
jose.amengual

oohhhh yes PG is always behind

Alex Jurkiewicz avatar
Alex Jurkiewicz

You can tune things. For example we disable per commit sync for MySQL Aurora through parameter group tuning. I’m sure there are similar knobs for postgres

jose.amengual avatar
jose.amengual

yes they are, but you can’t disable fsync in postgres aurora, you need to play with workmem, temp tables and such

2020-11-10

RB avatar

Requesting upvotes https://github.com/aws/containers-roadmap/issues/256 (for ecs automatic asg draining)

[ECS] [RFC]: Automatic management of instance draining in an ASG · Issue #256 · aws/containers-roadmap

Request for comment: please share your thoughts on this proposed improvement to ECS! With this improvement, ECS will automate instance and task draining. Customers can opt-in to automated instance …

1
aaratn avatar
aaratn
Introducing AWS Gateway Load Balancer – Easy Deployment, Scalability, and High Availability for Partner Appliances | Amazon Web Services attachment image

Last year, we launched Virtual Private Cloud (VPC) Ingress Routing to allow routing of all incoming and outgoing traffic to/from an Internet Gateway (IGW) or Virtual Private Gateway (VGW) to the Elastic Network Interface of a specific Amazon Elastic Compute Cloud (EC2) instance. With VPC Ingress Routing, you can now configure your VPC to send all […]

2
paultath81 avatar
paultath81

Can anyone tell me if it’s possible to decrease an FSX once it has been increased?

Alex Jurkiewicz avatar
Alex Jurkiewicz

The FAQ https://aws.amazon.com/fsx/windows/faqs/ says:
Q: Can I change my file system’s storage capacity and throughput capacity?
A: Yes, you can increase the storage capacity, and increase or decrease the throughput capacity of your file system – while continuing to use it – at any time by clicking “Update storage” or “Update throughput” in the Amazon FSx Console, or by calling “update-file-system” in the AWS CLI/API and specifying the desired level.

paultath81 avatar
paultath81

Thx @ I also saw that as well. I’m guessing the only option is increasing but you cannot decrease as aws doc doesn’t mention this.

Alex Jurkiewicz avatar
Alex Jurkiewicz

I’m sure they would mention if it’s possible. AWS prefer to say nothing than to say “you can’t do X”

paultath81 avatar
paultath81

jmccollum avatar
jmccollum

Can’t decrease the storage space of a FSX. Only thing that can be decreased after creation is the throughput.

paultath81 avatar
paultath81

Thx

2020-11-09

Andreas P avatar
Andreas P

Hey all is there a way to create multiple databases in an RDS instance as part of terraform provisioning ??

Zach avatar

use the postgresql provisioner

Zach avatar

and I’m sure there’s a mysql one

Andreas P avatar
Andreas P

You mean using postgresql provider in the terraform specification?

Zach avatar

yah

Andreas P avatar
Andreas P

Interesting didn’t think of that thanks!

Andreas P avatar
Andreas P

That one right?

Matt Gowie avatar
Matt Gowie

That’s it. I do this exact thing. The only rough part about this is that you typically want a separate root module to do this because :

  1. You typically need a SSH tunnel into your VPC so you can access your private RDS instances for the postgresql provider to do its work.
  2. It’s not done often, so keeping it isolated and not having to share that tunnel across multiple root modules is the right way to go.
Zach avatar

I’ve also found that the postgres provider doesn’t seem to integrate very well with the terraform dependency graph. It tries to ‘jump the gun’ and execute before things are ready

Matt Gowie avatar
Matt Gowie

Anyone know any tooling to keep AWS Config files (+ Kube config files as well) up-to-date across an org? I’m considering writing a script around a gomplate template for this but before I do I figured I should check that this isn’t already a thing.

tim.j.birkett avatar
tim.j.birkett

You mean like users AWS configs, configs for the “AWS Extend Switch Roles” plugin etc…? I created a simple Jenkins job at a previous employer so that users could self-serve getting configs for their user (email). Currently, I have some janky Ruby / rake tasks that do some of those things. I haven’t found anything out in the wild that does it.

1
Matt Gowie avatar
Matt Gowie

Yeah, that’s what I was referring to.

Matt Gowie avatar
Matt Gowie

What is AWS Extend Switch Roles? I should check this out…

Matt Gowie avatar
Matt Gowie

Oh a browser extension. Interesting.

tim.j.birkett avatar
tim.j.birkett

It’s great if you work across many accounts without SSO or some sort of app portal like you get with Azure AD.

2020-11-08

2020-11-07

btai avatar

I have a cloudwatch alarm for read iops where I want to set the threshold for alarming at a certain number (i.e 5000) but every night at 2am we run some sync jobs that are read intensive that spike up to higher than that number for a short period of time (i.e 7000) is there a way to configure the alarm threshold to be higher for that short period of time

btai avatar

i don’t necessarily want to set it at the higher threshold all the time as prolonged iops will lower our burst balance but the 2am spike is when we are at our lowest traffic and the sync jobs don’t last too long

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

And what about increasing the duration of elevated iops?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Also, this is something you should be able to control with your escalation platform

Alex Jurkiewicz avatar
Alex Jurkiewicz

Yeah, if you feed these alerts into pager duty, you can have scheduled maintenance to silence alarms for a certain time period. But that brush might be a little broad.

There’s no way to do what you want natively in AWS, except the hammer of a lambda scheduled action.

I wonder how useful this alert is though. Why do you want to be notified about high write iops? Can you instead be notified about degradations in application performance directly? Or if you must alert on the database, maybe there are other metrics which are more useful. Like number of connections or CPU.

btai avatar

Hmm, I don’t necessarily want to increase the duration as I think it could lead to slower reaction time to a system that might be impaired soon. Also I’m not sure if we can get that granular in our escalation platform (victorops) to be in maintenance mode for just a specific alert type? I’m reading that people have set up lambdas to disable/enable alarms. So I could have an alarm with a higher threshold enabled between 2am PST and 3am PST while disabling the general alarm during that period.

@ we alert on many things including cpu, connections, and iops. for us, we are very read heavy and increased read iops past our specific threshold for a longer period of time can drop our burst balance (which once depleted will cause read latency and service degradation) I like to alert on read iops because it generally gives us the most time to verify if we’re likely to be in trouble soon. Alerts on low burst balance, queue depth, read latency generally would come after as they are lagging symptoms of the problem. I feel like alerting on degradation in application performance to me is reactionary and you definitely want proactive alerting like prolonged read iops spikes as it gives you time to debug and hopefully prevent an issue that hasn’t happened yet.

Alex Jurkiewicz avatar
Alex Jurkiewicz

Thanks for the explanation. That makes a lot of sense, alerts are always application/system-specific. I agree with your point that if something is going wrong, it’s better to know before than after. But I think this situation is a great example of how difficult it is to build an alert that predicts problems in a reliable manner.

I’m reminded of a tweet from one of the original YouTube SREs, who said something like “in the early days, YouTube monitoring alerted on two metrics only: stream starts and number of uploads”. If you are a large company with a big complement of on-call staff, you can get tricky and predictive and build automated remediation based on this alert! But if not, you will end up with overly sensitive alarms and a culture of alert fatigue.

1
btai avatar

agree on alert fatigue!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I still see this as not an alert problem but an escalation problem. That alert escalation should be associated with a schedule. Very easy to do in something like opsgenie, but not sure about victorops

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
11:56:51 PM
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

opsgenie

btai avatar

got it @Erik Osterman (Cloud Posse)

2020-11-06

2020-11-05

uselessuseofcat avatar
uselessuseofcat

Hi! Is there any way to increase Security groups per network interface other than through service quotas. Maximum number is 16, is there any way to have it set on, let’s say 30 or 50? Should I contact AWS support? Thanks!

Henry Carter avatar
Henry Carter

Be aware that if you increase the number of sg-per-eni then the number of rules-per-sg will decrease.

uselessuseofcat avatar
uselessuseofcat

Thanks! I use only a few rules-per-sg. The thing is, I’ve created separate sg for every ECS cluster, and I have a lot of ECS clusters…

uselessuseofcat avatar
uselessuseofcat

I’ve created everything trough Terraform. So other option is to redesign everything

Alex Jurkiewicz avatar
Alex Jurkiewicz

You can redesign things by creating a common security group, and having each ECS cluster add rules to the common group.

If you have this many clusters you might want to consider some other form of security. You could have a single rule based on CIDR and use dedicated subnets

uselessuseofcat avatar
uselessuseofcat

Thanks @ - at the end there was no problem at all to begin with since I’ve misunderstood how VPC endpoints work :)

Alex Jurkiewicz avatar
Alex Jurkiewicz

Even better

1

2020-11-03

Shankar Kumar Chaudhary avatar
Shankar Kumar Chaudhary

anyone have successfully updated eks from 1.14 using terraform terragrunt? using terraform-root-modules

kalyan M avatar
kalyan M

How can we restrict aws IAM users not to generate their own access or secret keys by themselves.

roth.andy avatar
roth.andy

Deny iam:CreateAccessKey

roth.andy avatar
roth.andy

Though, is that really the right way to go? If you do that, you have to manage them, including rotating them.

Yoni Leitersdorf (Indeni Cloudrail) avatar
Yoni Leitersdorf (Indeni Cloudrail)

You can allow them to create keys for themselves

roth.andy avatar
roth.andy

right

roth.andy avatar
roth.andy

If you create my access keys for me and give them to me, that means you know them. As the user I don’t like that. I want to be the only person on the planet that knows those keys

roth.andy avatar
roth.andy

Do you allow users to set their own passwords (within the confines of a secure password policy)?

Yoni Leitersdorf (Indeni Cloudrail) avatar
Yoni Leitersdorf (Indeni Cloudrail)

@ what are you trying to achieve? Or what is the risk you’re trying to mitigate?

kalyan M avatar
kalyan M

previously one of the developer machine keys got malwared and there are c9 large clusters spinned in the Ireland region, other regions, which we are not aware. the ec2 spin up is done from china region. we got a a billing of $10K on that month. Looking for a way to remove their keys after work is done.

roth.andy avatar
roth.andy

Encouraging use of secure development practices like pre-commit hooks that check for AWS keys before a commit, or use of tools like aws-vault which always creates a temporary key using AWS STS (and better training on protecting keys and other security practices) would be a better strategy. What you mentioned is a serious problem and one that really needs to be solved, but it should be solved in a way that increases trust in the dev culture, not locking things away

roth.andy avatar
roth.andy
99designs/aws-vault

A vault for securely storing and accessing AWS credentials in development environments - 99designs/aws-vault

1
roth.andy avatar
roth.andy
Trust is the foundation of DevOps | SAP Blogs

[tl;dr] Organizations going through a DevOps transformation must review their culture, to evaluate whether it is meeting the requirements of a true DevOps organization An absence of trust is the root cause

roth.andy avatar
roth.andy

Okay here’s an actual strategy you can use instead of what you are proposing: You can configure IAM policies such that MFA must be used, even when running console commands with an access key. aws-vault supports it beautifully.

roth.andy avatar
roth.andy

You can also lock out regions that you don’t use via IAM

roth.andy avatar
roth.andy

Or use something like CloudCustodian to monitor AWS activity

Yoni Leitersdorf (Indeni Cloudrail) avatar
Yoni Leitersdorf (Indeni Cloudrail)

Agree with Andrew 100% here. Making things harder for devs won’t solve your problem here. Also, dev’s are very resourceful and find ways around the blocks you set up. You need to make sure you build something that’s easy for them to adopt.

We use aws-vault and AWS SSO (with G Suite IDP). The keys are never saved to the HD. Malware can try and make aws-vault calls, so it’s not fool-proof, but you’re adding more hurdles for the malware to try to overcome.

1
    keyboard_arrow_up