#aws (2019-10)

Anyone have any recommendations for a tool for disaster recovery on AWS? especially for Aurora, DynamoDB and EBS.

a tool ? as like something that could do what ?

centralized cross regions backup for services or at least for RDS. I look for https://aws.amazon.com/backup but it’s region dependency.

I will just use global aurora cluster and backup the main region

or you can have read replica cluster in other regions/accounts or same account

so instead of having snapshots you have replicas that can promote in very little time

Does anyone know the default duration of the session when using aws-vault?

1h I think

but that is not on aws-vault side

is aws side

and you can change that with a policy

https://github.com/99designs/aws-vault/blob/master/USAGE.md#environment-variables ASSUME ROLE TTL

I had been facing a problem when I tried to provision a cloudfront (estimate time 18 min) and using aws-vault to work properly I’ve needed to set this flag --assume-role-ttl=1h, like:

aws-vault exec <profile-name> --assume-role-ttl=1h

Hey folks – IAM Policy questions: What’s the standard operating procedure for dev teams on AWS and requiring MFA? I’ve created a policy to require MFA for all actions so users need to assign an MFA on first login and then on subsequent logins they need to provide MFA before they can do anything in the console, which is what I want. My problem with this is that I can’t distinguish between requiring MFA for console usage vs CLI usage. I’d like to empower devs to push to ECR or use certain CLI functionality without having them put their MFA in every morning.

I have a way to add IAM actions the user is allowed to do via the following policy statement:

{
            "Sid": "DenyAllExceptListedIfNoMFA",
            "Effect": "Deny",
            "NotAction": [
                // Bunch of management actions the user is allowed to do.
            ],
            "Resource": "*",
            "Condition": {
                "BoolIfExists": {
                    "aws:MultiFactorAuthPresent": "false"
                }
            }
        }

Should I just push all my CLI allowed actions into that NotAction and manage it that way? Or is there a better way?

I recommend having the ability to change password and manage own MFA available by default, and everything else locked behind having MFA present. Providing all access through assumed roles, that means you only have to lock down role assumption, and the only thing an IAM user is allowed to do is manage their MFA and login

That said, the second half of your statement, (allowing certain actions via MFA) could easily just be added as allows, since everything is an implicit deny. To my brain “allowing if” is simpler than “denying unless…”

I don’t think there is a way in IAM to distinguish between API and console access, so you’d have to be okay with it being available in both places without MFA. I mean maybe you could do something with aws:UserAgent but those are spoofable

Also aws-vault in server mode can help reduce the frequency of entering MFA code

Is every 12 hours really such a bad thing? :-)

Got it — Thanks gents. Think I need to do some reading on role assumptions + aws-vault, but overall I think I’ll move forward with supplying explicit allows for things I don’t want to hinder this dev team with and I’ll try to just keep that list short.

Is every 12 hours really such a bad thing? :-) Haha I personally don’t think so… but since I’m consulting for a dev agency who is more cavalier about security I just don’t want to rub them the wrong way.

Hi guys, Is there anyone familiar with IAM role and Instance Profile ?> I have a case like this: I would like to create an Instance Profile with suitable policy to allow access to ECR repo ( include download image from ECR as well). Then I attach that Instance Profile for a Launch Configuration to spin up an instance. The reason why I mentioned Policy for ECR is that I would like to set aws-credential- helper on the instance to use with Docker (Cred-helper). when it launch, so that when that instance want to pull image from ECR, it wont need AWS credential on the host itself at first. All of that module, I would like to put in Terraform format as well. Any help would be appreciated so much.

@Phuc i sort of do this with Elasticbeanstalk. I use the module https://github.com/cloudposse/terraform-aws-ecr.git and pass the elasticbeanstalk roles that get created.

Hi, If I use transit gw with multiple vpcs attached. Each VPC use its own private DNS zone in Route53. Traffic is working between VPC’s, but is there way to somehow delegate DNS resolving between VPCs?

Route 53 resolver I believe

Thanks @oscar. I was able to configure this using Route53 Resolver

Nice one

definitely awesome stuff. It will allow me to connect on-prem solutions with AWS later on, and to use on-prem private dns server

https://aws.amazon.com/blogs/aws/migration-complete-amazons-consumer-business-just-turned-off-its-final-oracle-database/

https://aws.amazon.com/blogs/security/aws-achieves-fedramp-jab-high-moderate-provisional-authorization-18-services/

Guys I need a bit of help with AWS networking Issue - we have RDS instances running in private subnets in VPC. From our office network we want to be able to always ssh into these instances (without client VPN). How should we do that? I guess we need a site-to-site VPN connection from our local network to the VPC. However how to enable traffic only to the rds instances. I do not want all the Internet network to go via the VPC/VPN So the local network should still have internet connection as is, only that there should be a direct connection possible to the RDS instances in VPC only

what’s the easiest way to clean up an aws account prior to account deletion?

AWS nuke

https://github.com/gruntwork-io/cloud-nuke https://github.com/rebuy-de/aws-nuke

last time i’ve checked nuke wasn’t supporting all of the services

https://github.com/1Strategy/automated-aws-multi-account-cleanup

but maybe i need to recheck

Not personally used it, just no of it. Can’t comment

sure. i will try it on one of the test accounts which will be deleted

thanks for a tip

https://aws.amazon.com/blogs/aws/now-available-amazon-relational-database-service-rds-on-vmware/

I see AWS recommend VPC’s of /16 or smaller. Given a /16 is split into further subnets (at least by AZ but potentially further, eg: different app ASG’s, k8s, etc), I’m curious where the /16 recommendation comes from. Any ideas? Hard follow or ignore? https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Subnets.html#vpc-sizing-ipv4 EDIT: The allowed block size is between a /28 netmask and /16 netmask.

Hello everyone, how do you manage ecs container logs? Example, now when a container dies I have to connect at EC2 that hosts ECS service and execute a docker logs xxxx. Which stack or strategy do you use to handle that?

You can send the logs to Cloudwatch on ecs/ec2 & fargate. On ec2, there are more logging options

Usually they go to CloudWatch and then either to 1) external log system, 2) Lambda function that triggers some action if something happened, or 3) nowhere cause they are fully ignored( observability in the application, tracing with exhaustive context sent to other systems so a bit similar to option 1)

Depends on the app and the company and the usecase

you can also stream you logs to elasticsearch in AWS

https://www.lastweekinaws.com/blog/aws-cross-az-data-transfer-costs-more-than-aws-says/

There’s a limit of 10 policies per role. Am i doing something wrong if I hit this limit?

I suppose I can concatenate policy statements and build one large JSON instead of defining one policy per statement, each with its own json…

did you considered using groups?

with 10 groups per iam user/role and 10 policies attached to a group you can increase it to 100 policies per user

or you can create inline policies as long as you don’t exceed 10,240 chars in total aggregate policy size

so to answer your question. it depends

aws will never say that you’re doing something wrong. it may be suboptimal from their perspective but it may be perfectly fine from yours

What would be the go to way for allowing k8s ExternalDNS to change records on a different account to that of the cluster?

cross account iam access to route53 or k8s zone delegation to the cluster account?

hi all, im getting this error, using the ECR module, aws_ecr_repository_policy.default: InvalidParameterException: Invalid parameter at 'PolicyText' failed to satisfy constraint: 'Invalid repository policy provided' is there a fix? do i need to provide the principals_full_access field ?

https://aws.amazon.com/chatbot/

Yeah I checked that out a month or so ago when they first released it.

It’s promising but doesn’t do what I need for now

Hey everyone, am working through our IP Addressing on a new AWS migration and would be great to hear some thoughts on how to structure public/private subnets please. Curious how others manage. I’m pretty fresh to AWS on a large scale.

Our setup:

1. We have a mix of Kubernetes, VM and Managed Service (DB/Queuing/etc) workloads. There is communication between them.
2. We are multi-AZ and are planning to minimise inter AZ traffic. ie: apps target other apps in same AZ via DNS.
3. We are in multiple regions (identical setup, just different VPC’s with non-clashing addressing, not peered)

AWS subnetting option 1 (my proposal):

• VPC /16
• AZ /18 (one per AZ plus one /18 spare)
• Split an AZ in /20 per workload (assume even distribution, but not important for this chat). eg: /20 for public k8s/loadbalancers, /20 private k8s (nodes/pods), /20 VM’s, /20 managed services. Reality is different sizing but it’s still plenty addresses.
• Pros: AZ boundary very clear because /18 defines it - for troubleshooting/etc. Suspect we will have separate (k8s and vm) node pools per AZ. We have had odd hacky IP whitelist in current cloud due to lack of auth between apps.
• Cons: Overhead managing?

AWS subnetting option 2 (Partner proposed):

• VPC /16
• Public /17
• Private /17
• Pros: Don’t think about IP’s (micro manage). Based on EC2 metadata target az# service, use some selector mechanism in private loadbalancer/security group (We would do this above anyway).
• Cons: Troubleshooting traffic path will be non-obvious by endpoint name and require AZ membership lookup (potentially retrospectively) ? K8s node pool rebalancing (I’ve heard this is a thing)? Does this apply to ASG’s?

In both cases the Cloudposse named-subnets module looks like a great fit and we would pass in the subnets id in as required.

@kskewes will get back to you tomorrow

Anyone using AWS MSK resource (Apache Kafka), with data (topics, kafka and zookeeper state) stored in bucket ?

@kskewes FWIW I’m doing /17 non-clashing per environment per region with 8 (4 public and 4 private) /19 subnets per region which allows for 4 AZs per region.

Not sure what your capacity planning looks like but that allows for 8,192 address per subnet or 32,768 per VPC

https://www.theregister.co.uk/2019/10/28/amazon_ddos_attack

Este jueves 31/10 tendrá lugar el 8vo capítulo en la serie de webinars de DinoCloud, “Desplegando entornos altamente disponibles en la nube”! En esta edición Nicolás Sánchez (COO) y Juan Diaz (Cloud Architect) comentarán sobre las mejores prácticas a la hora de planear un disaster recovery. Además sortearemos 2000 U$D en vivo!

Registrate gratis aquí: https://www.eventbrite.com.ar/e/desplegando-entornos-altamente-disponibles-en-la-nube-tickets-78054520171

GM everyone! Has anyone used Beats Central Management feature with ElasticCloud (Official Elastic hosted on cloud)? I am getting add_cloud_metadata/add_cloud_metadata.go:87 add_cloud_metadata: hosting provider type not detected. error message.

https://aws.amazon.com/about-aws/whats-new/2019/10/announcing-image-scanning-for-amazon-ecr/

Interesting, thanks for sharing

https://www.robustperception.io/getting-cloudwatch-data-into-prometheus

Hey folks, Any suggestions for a alternative to beanstalk environment variables for managing environment variables? I still want my application to consume environment variables for config, but using terraform to manage those variables seems like it’s going to be painful.

Is AWS Parameter store solid? Is there an alternative that folks would suggest?

Chamber+SSM parameter Store

and you can expose them directly in the Taskdef

cloudposse have modules for chamber and SSM with examples

also, there are a few ways of doing that using elastic beanstalk:

1. Store all the ENV vars in SSM (using chamber for example as @jose.amengual pointed out), then during terraform apply read them from the SSM and put them into https://github.com/cloudposse/terraform-aws-elastic-beanstalk-environment/blob/master/main.tf#L778

In this case, EB will provide those ENV vars to your app automatically

1. Store the ENV vars in SSM and let the app read them from SSM using an AWS SDK (if you want to go this route)

Huh, I’ll look into those. I’ve seen Chamber around before, but haven’t dug in too deeply. I like the idea of reading in at deploy time and then passing to EBS from TF. Good stuff. Thanks gents.

also, in TF, when you create some resources like RDS, you can write all required variables (e.g. host, user, password) to SSM directly after you create the resources, e.g. https://github.com/cloudposse/terraform-root-modules/blob/master/aws/grafana-backing-services/aurora-mysql.tf#L194

in this case, you will have all secrets automatically saved in SSM, and then in EB you read them from SSM and populate ENV vars

no user involved, no moving those secrets around

@Andriy Knysh (Cloud Posse) Do you know of an example of reading in values from parameter store? Realize I can use the “aws_ssm_parameter” data source, but I’m wondering how I can read in all SSM parameters for a particular path /$PROJECT/$ENV/* without having to pass all the names of the parameters I’m storing in as a var.

Maybe that’s not possible though.

Wouldn’t be too painful if it wasn’t really…