#aws (2018-12)
Discussion related to Amazon Web Services (AWS)
Discussion related to Amazon Web Services (AWS)
Archive: https://archive.sweetops.com/aws/
2018-12-04
anyone have any good alternatives to https://github.com/tmobile/pacbot ? I’ve tried getting pacbot setup numerous times and no luck.
2018-12-05
@Gabe not really asset mgmt, but CVE complaince checking - have used https://github.com/future-architect/vuls before which was pretty nice
and re compliance for AWS account stuff - https://github.com/alphagov/pay-aws-compliance
Vuls runs on every node for CVE aggregation and dumps reports to S3. pay-aws-compliance runs as a scheduled lambda function against an AWS account and pulling these reports and emails operators if something is not in compliance.
awesome thanks @joshmyers i’ll check them out
It doesn’t look as pretty as pacbot
haha but i’m sure it works better
i’ve tried once a week every week to install pacbot and every time i get an error
2018-12-06
2018-12-11
Recreantly migrated our Postgres database to RDS Postgres using DMS and while there were some gotchas it was an excellent experience!
@daveyu I’m happy to answer any questions if you have them.
You didn’t by chance happen to migrate from Heroku, did you?
Nope. Self managed EC2 instance but I can’t imagine it would be much different (though I haven’t used Heroku Postgres) as DMS just uses the postgres connection.
Found this. Damn. https://stackoverflow.com/questions/46939176/migration-from-heroku-postgresql-to-aws-rds-using-aws-data-migration-service-dm
Needs super user
one of the primary reasons for moving off heroku becomes an obstacle
hah, yea
I think this works in Heroku’s favor: vendor lock-in
@daveyu
@daveyu has joined the channel
TIL don’t try to use the s3 bucket url (even though AWS only auto completes this) in a Cloudfront distribution pointing to a s3 website… be sure to use the s3 website endpoint url!
It’s easy to know you are doing it wrong… you will be greeted with an Access Denied XML page on your cloudfront distribution url if you did the former
2018-12-12
2018-12-13
@Erik Osterman (Cloud Posse) have you setup an EC2 health check feed into slack before?
yes - quite easily done
or wait, i thought you mean the status page for EC2
(rss)
… if that’s the case, just use the /feed command to add the RSS feed (after adding the app)
we do something like this for #releases
No, instance health/status
Looking for any tips before we dive in
so that would be SNS notifications, no?
Yes, and presumably a lambda to post
cloudwatch -> sns -> lambda
ya
I was hoping you would say: absolutely, heres our a proven TF module
so we’ve used this module to send SNS alarms to slack
so now, you just gotta whip up the cloudwatch part
we have some examples for that. sec.
@sarkis wrote them
EC2 health check
hrm…
yes, one sec
this implements one for StatusCheckFailed_Instance
now need to glue the parts together. maybe someone else has written a module though… haven’t looked
(check the registry)
ya, thats my next step
do let me know what you find
it would be sweet to get those notices into slack
2018-12-19
Finally
apparently it’s openvpn behind the scenes
2018-12-20
They promised to deliver it before holidays during this re:invent. Looks promising! $108 per single user full-day connection at least. Not sure how comparable it is to existing OpenVPN setup which has to be maintained.
lol, that’s scary expensive
A good Openvpn module is missing, but they have an okish commercial byol and it would suck for them to have something free competing.
we use the pfsense ami in aws that uses the openvpn client
its not a bad setup
and theres an aws vpn integration piece
If I understand correctly you pay per time you actually use VPN per user, so it should be cheaper on week-ends, for example. Still expensive?
I have not used VPN on AWS for the last 3 years Last time I used it in Russia where linkedin was blocked
you pay for the instance and then pay for the software/hr depending on instance size
If you use it together with a few alarms it’s probably totally ok. What you don’t want is that johnny and his team forget about the vpn connections and just add the office manager’s salary to the monthly bill
pfsense’s one cost $288 in total per month, which is $0,34/hour
the caveat to this, is that im sure you pay for the internet traffic hitting the vpn etc. but thats where you would want to use a split tunnel
t2.medium $0.08 $0.046 $0.126
But wait,, this means tha tif the instance is down then the vpn is down as well ?
yea so y ou want to try to setup HA with it
exactly, as typical service you have to manage…
truue
I’ll be waiting for lambda vpn
pfsense also comes with other stuff liek a FW and whatnot
not just manage but, it can become a bottleneck due to network limit per instance type.
I wishlisted AWS Lambda with GPU support, didn’t happen…
so It doens’t really sound like a cool thing to have to route all the traffic through one instance.
yea i guess it depends on your budget cause im sure the aws service isnt cheap
where the link for the lamda vpn thing
it was a joke
that is what we are trying to compare now $288 for manage it yourself for unlimited users vs $108 per single user, no management requried.
i guess you could also look at https://www.netgate.com/support/
probably talk them down in price
hmmm then , taking into consideration the management costs and possible incidents should be taken into account.
and that is hard to convert into money usually
yes, true, for such critical piece as VPN tend to be
2018-12-24
When updating the Launch Configuration on an AutoScaling group, it seems like the only way to complete the deployment is to change the min instances to double the size, wait for the new ones to spin up, and then change the min back to what’s expected. Is there a better way to accomplish this?
Looks like CloudFormation has support for that
Yes, I don’t believe there’s a way to use launch configurations to trigger a rebuild of an ASG.
Change ASG name to be recreated every time new LC is created… see var.recreate_asg_when_lc_changes - https://github.com/terraform-aws-modules/terraform-aws-autoscaling/blob/master/main.tf#L34
Clever fix for this
This should be used with wait_for_elb_capacity & create_before_destroy to avoid leaving the target group with no instances, right?
If I remember correctly now wait_for_elb_capacity is optional, while create_before_destroy is set to true already in the module.
I’ll do a test and get back to you, but I suspect without wait_for_elb_capacity, there won’t be any healthy instances on the target group during the cut-over (which may or may not be okay depending on the environment).
2018-12-26
Would someone be interested to do a project together on the ASG Launch Configuration update. Thinking of a module with a Lambda / Step functions. A full ASG recreate is just too heavy sometimes.
That would be pretty neat @maarten .
Cool, I was thinking there are proably similarities updating EKS workers and ECS nodes. Maybe there is a way to catch both of them.
so if they use the same core module
… that would help
but for kubernetes, technically we’d want to cordon the node first, then drain it - for a smooth rolling update
maybe push amount of running services per instance to cloudwatch, and use that to make decisions with for further rolling
What about autoscaling lifecycleHooks that should also do the trick imho.
Have you seen our latest terraform external artifact module? Makes it easy to distribute and deploy complex lambdas without baking zips into git
I did, although I never disliked very small zips inside a repo and I find external urls confusing for people who’re not used to the Cloudposse eco-system, but we can talk about it .
The problem is fundmentally that we cannot rely on a “local” toolchain required for packaging (E.g. npm dependencies)
the artifact module supports local files (e.g. file://) as well as remote.
I guess if the CI system built the zip and committed back to the repo, that could work too
but I don’t want users committing zips to repos b/c it puts the onus on us to verify every zip
Or utilize Codebuild, but then it gets massive
hrm
yes, that’s true - could use codebuild approach
albeit heavy handed - but at least self-contained
A type of Dockerfile for Lambda’s would have been helpful.
yea, seriously
one day…
When singularity is reached.
Very small node scripts which just work with AWS don’t need packages, not sure what you think of that.
yea, when that’s the case - no need to use the external artifacts
but requiring single nodejs scripts with no dependencies, is an undue limitation
yes
I wish it was possible to attach artifacts to a commit in github the way it’s possible to attach them to a release
2018-12-27
Novice question. Why do the ephemeral ports on an inbound ACL rule for a private subnet need to have a source of 0.0.0.0/0, when the traffic is coming from a NAT Gateway that should be running within the same VPC on a public subnet? Setting the source to the VPC CIDR doesn’t seem to work.
@Igor is your question related to a specific module or in general ?
general question
Ah ok, are you talking NACL or Security Group now btw. For both I don’t have an answer tbh, would love to replicate your issue however.
I do know that a NLB will most likely send the traffic through as if it would originate from the source. So for NLB I see the point.
But I was under the impression that one can change the rules the way they want them, without limitations, so I’d like to know what’s going on .
I’m talking about NACLs. I was stuck on an issue with my config, and setting the source to 0.0.0.0/0 on private subnet NACL fixed the issue for me. I referred to https://docs.aws.amazon.com/vpc/latest/userguide/vpc-recommended-nacl-rules.html and sure enough, it recommends this configuration as well
Hardening with NACLs is tricky, and I personally would never start working on them unless someone is really requesting me to do so. Even when I had AWS consultants over auditing multiple times, NACLS were never part of the audit.
What is the reason you want to use them ?
Security, but I am also trying to make sense of how the AWS NAT Gateway works.
I’m pretty sure things are locked down regardless of ACL rules given that there is no route to the IGW from the private subnet
is the nat gw working for you now ?
Yup, got it to work by changing the source of the NACL rule for ephemeral ports.
I don’t think NAT is being translated both ways normally.
so you would still receive from an outside IP and you would need to allow traffic from it in your private subnet
Makes sense