#aws (2018-12)

aws Discussion related to Amazon Web Services (AWS)

aws Discussion related to Amazon Web Services (AWS) Archive: https://archive.sweetops.com/aws/

2018-12-28

2018-12-27

imiltchman avatar
imiltchman

Novice question. Why do the ephemeral ports on an inbound ACL rule for a private subnet need to have a source of 0.0.0.0/0, when the traffic is coming from a NAT Gateway that should be running within the same VPC on a public subnet? Setting the source to the VPC CIDR doesn’t seem to work.

maarten avatar
maarten

@imiltchman is your question related to a specific module or in general ?

imiltchman avatar
imiltchman

general question

maarten avatar
maarten

Ah ok, are you talking NACL or Security Group now btw. For both I don’t have an answer tbh, would love to replicate your issue however.

I do know that a NLB will most likely send the traffic through as if it would originate from the source. So for NLB I see the point.

But I was under the impression that one can change the rules the way they want them, without limitations, so I’d like to know what’s going on .

imiltchman avatar
imiltchman

I’m talking about NACLs. I was stuck on an issue with my config, and setting the source to 0.0.0.0/0 on private subnet NACL fixed the issue for me. I referred to https://docs.aws.amazon.com/vpc/latest/userguide/vpc-recommended-nacl-rules.html and sure enough, it recommends this configuration as well

Recommended Network ACL Rules for Your VPC - Amazon Virtual Private Cloud

Use the network ACL rules we recommend to provide an additional layer of security for your subnets.

maarten avatar
maarten

Hardening with NACLs is tricky, and I personally would never start working on them unless someone is really requesting me to do so. Even when I had AWS consultants over auditing multiple times, NACLS were never part of the audit.

:100:1
maarten avatar
maarten

What is the reason you want to use them ?

imiltchman avatar
imiltchman

Security, but I am also trying to make sense of how the AWS NAT Gateway works.

imiltchman avatar
imiltchman

I’m pretty sure things are locked down regardless of ACL rules given that there is no route to the IGW from the private subnet

maarten avatar
maarten

is the nat gw working for you now ?

imiltchman avatar
imiltchman

Yup, got it to work by changing the source of the NACL rule for ephemeral ports.

maarten avatar
maarten

I don’t think NAT is being translated both ways normally.

maarten avatar
maarten

so you would still receive from an outside IP and you would need to allow traffic from it in your private subnet

imiltchman avatar
imiltchman

Makes sense

2018-12-26

maarten avatar
maarten

Would someone be interested to do a project together on the ASG Launch Configuration update. Thinking of a module with a Lambda / Step functions. A full ASG recreate is just too heavy sometimes.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

That would be pretty neat @maarten .

maarten avatar
maarten

Cool, I was thinking there are proably similarities updating EKS workers and ECS nodes. Maybe there is a way to catch both of them.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so if they use the same core module

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

… that would help

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but for kubernetes, technically we’d want to cordon the node first, then drain it - for a smooth rolling update

maarten avatar
maarten

maybe push amount of running services per instance to cloudwatch, and use that to make decisions with for further rolling

Nikola Velkovski avatar
Nikola Velkovski

What about autoscaling lifecycleHooks that should also do the trick imho.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Have you seen our latest terraform external artifact module? Makes it easy to distribute and deploy complex lambdas without baking zips into git

maarten avatar
maarten

I did, although I never disliked very small zips inside a repo and I find external urls confusing for people who’re not used to the Cloudposse eco-system, but we can talk about it .

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

The problem is fundmentally that we cannot rely on a “local” toolchain required for packaging (E.g. npm dependencies)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

the artifact module supports local files (e.g. file://) as well as remote.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I guess if the CI system built the zip and committed back to the repo, that could work too

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but I don’t want users committing zips to repos b/c it puts the onus on us to verify every zip

maarten avatar
maarten

Or utilize Codebuild, but then it gets massive

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

hrm

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yes, that’s true - could use codebuild approach

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

albeit heavy handed - but at least self-contained

maarten avatar
maarten

A type of Dockerfile for Lambda’s would have been helpful.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, seriously

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

one day…

maarten avatar
maarten

When singularity is reached.

maarten avatar
maarten

Very small node scripts which just work with AWS don’t need packages, not sure what you think of that.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, when that’s the case - no need to use the external artifacts

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but requiring single nodejs scripts with no dependencies, is an undue limitation

maarten avatar
maarten

yes

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I wish it was possible to attach artifacts to a commit in github the way it’s possible to attach them to a release

2018-12-24

imiltchman avatar
imiltchman

When updating the Launch Configuration on an AutoScaling group, it seems like the only way to complete the deployment is to change the min instances to double the size, wait for the new ones to spin up, and then change the min back to what’s expected. Is there a better way to accomplish this?

imiltchman avatar
imiltchman

Looks like CloudFormation has support for that

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yes, I don’t believe there’s a way to use launch configurations to trigger a rebuild of an ASG.

antonbabenko avatar
antonbabenko

Change ASG name to be recreated every time new LC is created… see var.recreate_asg_when_lc_changes - https://github.com/terraform-aws-modules/terraform-aws-autoscaling/blob/master/main.tf#L34

terraform-aws-modules/terraform-aws-autoscaling

Terraform module which creates Auto Scaling resources on AWS - terraform-aws-modules/terraform-aws-autoscaling

:--1:4
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Clever fix for this

terraform-aws-modules/terraform-aws-autoscaling

Terraform module which creates Auto Scaling resources on AWS - terraform-aws-modules/terraform-aws-autoscaling

imiltchman avatar
imiltchman

This should be used with wait_for_elb_capacity & create_before_destroy to avoid leaving the target group with no instances, right?

antonbabenko avatar
antonbabenko

If I remember correctly now wait_for_elb_capacity is optional, while create_before_destroy is set to true already in the module.

imiltchman avatar
imiltchman

I’ll do a test and get back to you, but I suspect without wait_for_elb_capacity, there won’t be any healthy instances on the target group during the cut-over (which may or may not be okay depending on the environment).

2018-12-20

antonbabenko avatar
antonbabenko

They promised to deliver it before holidays during this re:invent. Looks promising! $108 per single user full-day connection at least. Not sure how comparable it is to existing OpenVPN setup which has to be maintained.

maarten avatar
maarten

lol, that’s scary expensive

maarten avatar
maarten

A good Openvpn module is missing, but they have an okish commercial byol and it would suck for them to have something free competing.

pericdaniel avatar
pericdaniel

we use the pfsense ami in aws that uses the openvpn client

pericdaniel avatar
pericdaniel

its not a bad setup

pericdaniel avatar
pericdaniel

and theres an aws vpn integration piece

antonbabenko avatar
antonbabenko

If I understand correctly you pay per time you actually use VPN per user, so it should be cheaper on week-ends, for example. Still expensive?

antonbabenko avatar
antonbabenko

I have not used VPN on AWS for the last 3 years Last time I used it in Russia where linkedin was blocked

:--1:1
pericdaniel avatar
pericdaniel

you pay for the instance and then pay for the software/hr depending on instance size

maarten avatar
maarten

If you use it together with a few alarms it’s probably totally ok. What you don’t want is that johnny and his team forget about the vpn connections and just add the office manager’s salary to the monthly bill

antonbabenko avatar
antonbabenko

pfsense’s one cost $288 in total per month, which is $0,34/hour

pericdaniel avatar
pericdaniel

the caveat to this, is that im sure you pay for the internet traffic hitting the vpn etc. but thats where you would want to use a split tunnel

pericdaniel avatar
pericdaniel

t2.medium $0.08 $0.046 $0.126

Nikola Velkovski avatar
Nikola Velkovski

But wait,, this means tha tif the instance is down then the vpn is down as well ?

pericdaniel avatar
pericdaniel

yea so y ou want to try to setup HA with it

antonbabenko avatar
antonbabenko

exactly, as typical service you have to manage…

pericdaniel avatar
pericdaniel

truue

Nikola Velkovski avatar
Nikola Velkovski

I’ll be waiting for lambda vpn

pericdaniel avatar
pericdaniel

pfsense also comes with other stuff liek a FW and whatnot

Nikola Velkovski avatar
Nikola Velkovski

not just manage but, it can become a bottleneck due to network limit per instance type.

antonbabenko avatar
antonbabenko

I wishlisted AWS Lambda with GPU support, didn’t happen…

Nikola Velkovski avatar
Nikola Velkovski

so It doens’t really sound like a cool thing to have to route all the traffic through one instance.

pericdaniel avatar
pericdaniel

yea i guess it depends on your budget cause im sure the aws service isnt cheap

pericdaniel avatar
pericdaniel

where the link for the lamda vpn thing

Nikola Velkovski avatar
Nikola Velkovski

it was a joke

antonbabenko avatar
antonbabenko

that is what we are trying to compare now $288 for manage it yourself for unlimited users vs $108 per single user, no management requried.

pericdaniel avatar
pericdaniel

i guess you could also look at https://www.netgate.com/support/

Netgate Global Support

From configuration assistance to ensuring minimal downtime for mission-critical circuits, we have the expertise to assist you for any size network.

pericdaniel avatar
pericdaniel

probably talk them down in price

Nikola Velkovski avatar
Nikola Velkovski

hmmm then , taking into consideration the management costs and possible incidents should be taken into account.

Nikola Velkovski avatar
Nikola Velkovski

and that is hard to convert into money usually

antonbabenko avatar
antonbabenko

yes, true, for such critical piece as VPN tend to be

2018-12-19

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
What Is AWS Client VPN? - AWS Client VPN

Enable access to your VPC and on-premises network from anywhere, on any device.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Finally

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

apparently it’s openvpn behind the scenes

2018-12-13

Daren avatar
Daren

@Erik Osterman (Cloud Posse) have you setup an EC2 health check feed into slack before?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yes - quite easily done

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

or wait, i thought you mean the status page for EC2

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(rss)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

… if that’s the case, just use the /feed command to add the RSS feed (after adding the app)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we do something like this for #releases

Daren avatar
Daren

No, instance health/status

Daren avatar
Daren

Looking for any tips before we dive in

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so that would be SNS notifications, no?

Daren avatar
Daren

Yes, and presumably a lambda to post

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

cloudwatch -> sns -> lambda

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ya

Daren avatar
Daren

I was hoping you would say: absolutely, heres our a proven TF module

Daren avatar
Daren

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/terraform-aws-sns-lambda-notify-slack

Terraform module to provision a lambda function that subscribes to SNS and notifies to Slack. - cloudposse/terraform-aws-sns-lambda-notify-slack

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so we’ve used this module to send SNS alarms to slack

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so now, you just gotta whip up the cloudwatch part

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we have some examples for that. sec.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@sarkis wrote them

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/terraform-aws-alb-target-group-cloudwatch-sns-alarms

Terraform module to create CloudWatch Alarms on ALB Target level metrics. - cloudposse/terraform-aws-alb-target-group-cloudwatch-sns-alarms

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)


EC2 health check

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

hrm…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yes, one sec

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/terraform-aws-ec2-instance

Terraform Module for providing a general EC2 instance provisioned by Ansible - cloudposse/terraform-aws-ec2-instance

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

this implements one for StatusCheckFailed_Instance

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

now need to glue the parts together. maybe someone else has written a module though… haven’t looked

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(check the registry)

Daren avatar
Daren

ya, thats my next step

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

do let me know what you find

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

it would be sweet to get those notices into slack

2018-12-12

2018-12-11

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
davidvasandani avatar
davidvasandani

Recreantly migrated our Postgres database to RDS Postgres using DMS and while there were some gotchas it was an excellent experience!

davidvasandani avatar
davidvasandani

@daveyu I’m happy to answer any questions if you have them.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

You didn’t by chance happen to migrate from Heroku, did you?

davidvasandani avatar
davidvasandani

Nope. Self managed EC2 instance but I can’t imagine it would be much different (though I haven’t used Heroku Postgres) as DMS just uses the postgres connection.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Migration from Heroku PostgreSQL to AWS RDS using AWS Data Migration Service (DMS) not working

I’m trying to use AWS DMS to copy and replicate data from a Heroku PostgreSQL database to an AWS RDS PostgreSQL instance but it isn’t working (more infos below). In DMS Log I can see the following

:-1:1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Needs super user

daveyu avatar
daveyu

one of the primary reasons for moving off heroku becomes an obstacle

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

hah, yea

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think this works in Heroku’s favor: vendor lock-in

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@daveyu

daveyu avatar
daveyu
02:00:31 AM

@daveyu has joined the channel

sarkis avatar
sarkis

TIL don’t try to use the s3 bucket url (even though AWS only auto completes this) in a Cloudfront distribution pointing to a s3 website… be sure to use the s3 website endpoint url!

sarkis avatar
sarkis

It’s easy to know you are doing it wrong… you will be greeted with an Access Denied XML page on your cloudfront distribution url if you did the former

:--1:3
1

2018-12-05

joshmyers avatar
joshmyers

@Gabe not really asset mgmt, but CVE complaince checking - have used https://github.com/future-architect/vuls before which was pretty nice

future-architect/vuls

Vulnerability scanner for Linux/FreeBSD, agentless, written in Go - future-architect/vuls

joshmyers avatar
joshmyers

and re compliance for AWS account stuff - https://github.com/alphagov/pay-aws-compliance

alphagov/pay-aws-compliance

Contribute to alphagov/pay-aws-compliance development by creating an account on GitHub.

joshmyers avatar
joshmyers

Vuls runs on every node for CVE aggregation and dumps reports to S3. pay-aws-compliance runs as a scheduled lambda function against an AWS account and pulling these reports and emails operators if something is not in compliance.

Gabe avatar

awesome thanks @joshmyers i’ll check them out

joshmyers avatar
joshmyers

It doesn’t look as pretty as pacbot

Gabe avatar

haha but i’m sure it works better

Gabe avatar

i’ve tried once a week every week to install pacbot and every time i get an error

2018-12-04

Gabe avatar

anyone have any good alternatives to https://github.com/tmobile/pacbot ? I’ve tried getting pacbot setup numerous times and no luck.

tmobile/pacbot

PacBot (Policy as Code Bot). Contribute to tmobile/pacbot development by creating an account on GitHub.

    keyboard_arrow_up