#aws (2019-03)

Fricking safety gloves … On a command line Like people will pop it open and accidentally blow windows up

Are you really doubting that?

People bypass any warnings you put to them, even when it says “this will delete your system”, and once its deleted, they write on reddit/twitter how bad Windows is, as its so easily deleted

They can type sudo rm -Rf /

you can’t cure stupid

you can’t cure stupid No but you can safety-net it

The darwinian effect of letting the stupid rid the world of themselves is fine with me http://www.weirduniverse.net/blog/comments/tullock_spike

Realistically though if you broke your WSL install, you can just remove + reinstall it. It’s supposed to be an app

I’ve figured it out… watch this space https://github.com/osulli/aws-multi-account-setup

Ok.. published! Would appreciate someone suggesting a good way around the limitation listed

Does anyone know how to use aws-vault login x with SSO / Federation? There’s clearly some sort of support in https://github.com/99designs/aws-vault/blob/master/cli/login.go but I can’t work out what config I’m missing in ~/.aws/config… I think it’s missing session token from the SSO portal?

aws-vault does not support SSO

for that, you need a purpose built tool

e.g. aws-okta for okta (by segmentio)

ther are others for gsuite, etc

I’m only using AWS SSO

I’m going to create a new root account that my company isn’t currently using and try reference architecture. Feel like all my problems stem from avoiding it!

If I am only using the master node for Redis in my application, is there any advantage to having more than 1 replica in ElastiCache cluster?

Anyone who used ACM’s “private certificate authority (CA) ” for having a CA infra out of the box, for use with Kafka for example,

I didn’t use it but I saved the terraform for setting up an ACM

do you want??

Ah that’s cool, yes please.

It’s been about a month, but I remember thinking “I should save that if I’m not going to use it”.. so I don’t think it’s just the default example lol

Yeah it’s quite expensive

It had nothing to do with cost for me.. we just manage domains weirdly at my place and have yet to move CA and domain control to AWS

what causes random spikes in read/write ipos on rds databases? where can i look to debug

I think RDS snapshots will influence that

hmm no snapshot at the time of the spike

has anyone here migrated hosted zones between aws accounts before?

I followed this guide to the tee, and it seems to have worked (running nslookup/dig shows the new nameservers) https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hosted-zones-migrating.html

ive left the old hosted zone up while the switchover occurs (its supposed to take up to two days because of dns resolver caching) and its been 7 days now.

but it seems like i am getting err_name_not_resolved browser errors. this is happening extremely (emphasis on extremely) rarely but i was wondering is the hosted zone cutover not a completely clean and error prone process?

Could always open an aws ticket. They take about a week

It took us about a week as well

If you have business support it can be expedited

So in testing reference-architectures stuff, I made a few accounts I don’t want anymore. I went in and closed the accounts but they are still in my org as “suspended” I’ve worked on hundreds of AWS accounts, but I realized today I’ve never closed one. Any clue on if those will eventually go away in my AWS Org?

https://aws.amazon.com/blogs/aws/new-open-distro-for-elasticsearch/

I hope this leads to some nice alternative tooling for handling auth etal from x-pack for those who just want a bloody search engine

hope

the current stuff aws have thrown up is a bit meh though

bad the perf tool

i dooooo like that ascii graphing

Any thoughts on why after having setup an openvpn instance (that does change my IP, confirmed), I still can’t use private IPs from my local to ssh into other machines?

Can you reach the machines in any way, like pinging them?

security groups?

aknysh [4:28 PM] security groups?

Bingo

yep that’s got to be it

or not

ingress 22 for my public IP

If you VPN you should have a private IP that needs access.

open the SG for all traffic and test if you can access it

ooo what the heck

that worked @Andriy Knysh (Cloud Posse)

Usually OpenVPN will put you in a subnet and you can give the all subnet IPs SSH access for your machines.

I did ALL UDP and ALL TCP from anywhere

~ah~

yea, then @mmuehlberger is correct, VPN uses you private local IP, open the SG for it

Why would it use my local private IP to ssh

surely that will change all the time and I can’t possibly open a SG rule for it?

It uses your private IP in the VPC, that you get after connecting via VPN.

Let me try putting the machine in a public VPC

Ah it already is

Oh wait not what you’re saying

Ah so If I allow the CIDR for the private IP of the subnet which I’m tunnelling into…..

does your VPN have its own SG?

Yes

then add it to the other SG

I’m unsure that would work across accounts

across accounts you have just a few choices I guess: in the bastion SG, open a hardcoded IP or CIDR from the VPC (not good), or do VPC peering; maybe there are other solutions?

I have got VPC peering active weirdly

I have:

MGMT: VPC 1 containing VPN

Sandbox: VPC 2 peered t oVPC 1 containing anything

oh, then adding the VPC SG to ingress for the bastion SG should work?

Oh right, didn’t realise I could reference a SG from another account - hadn’t tried

or, you know the CIDR of VPC 1, add it to ingress of VPC 2

Hmmm easier for me to do the CIDR I suppose

because I manually set them with terraform

so I know what they’ll be

(to avoid overlapping when peering)

(don’t remember if SGs accross accounts work with VPC peering)

(and make it easier to tell things based on IP CIDR)

ok cool so sounds like CIDR is aactually the best route

if you know them and have peering, then yes

ooooh yeh nice.

Removed the wildcard ingress rules and allowed the CIDR of my VPC

Nice

And cross account works!

amazing

Great!

anyone know of any magic ways to get Ubuntu 18lts to fuck off caching DNS

or more concisely stopping it from seemingly caching things with short ttls for all of existence

nvm only on one box… at least its 5pm

is it running nscd?

Is there a way to log complete request body at loadbalancer level or VPC (AWS ELB/VPC) ?

@rohit Why do you need to log it at the loadbalancer, and not at the server/lambda? Do you suspect the LB modifies the request somehow?

nope, i don’t see the request body when it reaches nginx

so was wondering if there is a way to log the complete request body at loadbalancer

I don’t think so. I am surprised you don’t have that capability within nginx

I mean there is a way to do it in nginx but i want to log the entire request at the loadbalancer before it reaches my app

I guess you could put something in front of the LB like a WAF or CloudFlare workers

I am sure there is a way to do it, but I don’t think AWS ELB has this logging ability

Or use an NLB

then you’ll see the full unadulterated request body at your app

thanks @Erik Osterman (Cloud Posse) Will check if i can use NLB

https://aws.amazon.com/about-aws/whats-new/2019/03/amazon-eks-introduces-kubernetes-api-server-endpoint-access-cont/

Has anyone experienced a delay between when you have an issued certificate in ACM (passed DNS validation and in us-east-1), and when it becomes available for use within CloudFront via the console?

I’m creating a new CF distro, and have 2 available, issued certs in ACM in us-east-1. In CloudFront, the option to choose “Custom SSL Certificate” is not available

is that CF distro in us-east-1 as well?

CF Distro is global, not associated to a specific region

ooh nvm – i see them now. Looks like there’s a bit of a lag between when the certificate is validated/issued within ACM and when its available for use with other AWS resources (at least CloudFront distros)

it used to be at least that ACM certs were required to exist in us-east-1

though I think they recently lightened that restriction

(for CF distros)

yeah i made sure the ACM certs were in that region… i didn’t change anything, just reopened the new Distro console and finally the Custom SSL Certificate radio button became available. but there was at least a 20m lag between when ACM showed it issused and when i could associate to CF Distro

anyway, thanks for the attention haha

cool, thanks for the update

that lag seems unusual - but might’ve been a temporary/isolated thing

Guys, question about EC2 ENI limits which is related to ECS clusters. I have 2 EC2 instances, where each has 2 ENIs on it, which means I can launch just 2 containers there (right?). My tasks are rather lightweight, so I have a lot of unused resources but need to scale-out EC2 instances in the cluster because I need more tasks/containers running. I want to be able to run 10 small tasks on a single EC2 instance t3.large (for eg).

Are there better ways to utilise resources and have more ENIs available? I have been evaluating bigger instances also, but there are not so many ENIs comparing to amount of resources.

Can Fargate be a better option in term of price to utilise just what I need and get ENIs allocated as requested?

/cc @maarten

ENI limit is only applicable when you are using ECS tasks in awsvpc mode, normally you would use bridged mode with dynamic port allocation.

ok, let me read more about that one. Thanks!

https://github.com/aws/containers-roadmap/issues/7

If you create a module with a script in it how do you load the module script in the module? template = "${file("./scripts/userscript.sh")}" Loads based on the working path rather than the module.

${path.module}

one of those rtfm moments (aka rtfm find m useless go to github find other peoples shit)

Is there a way to use a custom certificate with ElastiCache Redis for in-transit encryption? I can’t seem to find a way.

Hrmmm good question

is there no way to specify the specific ACM cert to use?

(haven’t looked)

Doesn’t seem to be. “You don’t have to manage the lifecycle of your certificates because ElastiCache for Redis automatically manages the issuance, renewal, and expiration of your certificates.” Sounds like I’m being ungrateful.

Lol

Unfortunately, this means I can’t create a standardized hostname for the Redis cluster

Crap, you’re right. That sucks!

Guess we have been using the canonical cname

I tried using a CNAME, but it doesn’t seem to work with Tls enabled

Which makes sense given that the certificate subject doesn’t match

It’s gotta be the hostname returned by redis

Nothing you generate

Yeah, exactly

Anyone attend AWSome day today? Ric Harvey was really good

What is this AWSome you speak of?

RKE (ranchers version of KOPS) added a cluster.rkestate file output after its ran… yeeeeey more state to move around.

It doesn’t support remote state?

Kops has a state bucket

no; had to jimmy-rig s3 pull/push in

which is shit

but functionaly

lol I keep getting told off by the anti-swearing bot

Wow surprised that would be the case. Not very team friendly.

Checkout goofys

Also we have support for that in geodesic

Mount s3 as a filesystem

currently trying to decide if its best to packer build an nginx ami, then packer the configs in using that as a base, or sync the configs in on userscript certainly know which would be quicker to update

goofys looks pretty neat; its always reassuring when somethings written in GO as I dont have to spend 20 minutes looking for the “wont work on x system” crap

Yea it’s a big qualifier for me

Never blindly apply CIS benchmark changes Nothing like spending 2 hours wondering why you’re K8 deployment has broken only to discover it disabled ipv4 forwarding … docker kinda needs that

its always reassuring when somethings written in GO Why? Because you know GO or because of a special trait of GO?

Because the code compiles into a single binary per OS / tends to be agnostic

whereas try faffing with Python + Matlab on windows Or anything node on windows

its nice to be able to drop a binary and run

without having to install a crap load of dependencies

When C# can do the same I may change my mind due to familiarity alone

loyalties are fleeting

https://www.youtube.com/watch?v=P6iY6ovhbfc

fargate made easy

@oscarsullivan_old you might dig this

Thanks! I couldn’t sleep last night because all I could think about was K8s vs fargate Vs not using either and instead orchestrating with ansible/ ecs/ lambda

Ansible, do you idempotence ?

here’s a pic from AWS just the other day

i saw in the sweetops docs somewhere that if you create an account when adding to an org (versus creating it independtly then importing it to the org) you can’t ever spin out that account if necessary

is that still the case?

cause the new reference architecture impl seems to auto provision the sub-accounts rather

you can spin it out of the org, but you’ll usually have to do some extra config first

Yeh it’s a real pain

I created a few too many org accounts

And I CBA to log in and configure them to be independent so I can detatch

When you create an account in an organization using the AWS Organizations console, API, or AWS CLI commands, all the information that is required of standalone accounts is not automatically collected. For each account that you want to make standalone, you must accept the AWS Customer Agreement, choose a support plan, provide and verify the required contact information, and provide a current payment method. https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_accounts_remove.html

yea i’ve been trying to decide how finely to split stacks among accounts

we have multiple deployments of our main stack for different customers, and each deployment needs its own dev/staging/prod

https://aws.amazon.com/blogs/aws/new-advanced-request-routing-for-aws-application-load-balancers/

https://venturebeat.com/2019/03/27/amazons-aws-deep-learning-containers-simplify-ai-app-development/

I don’t understand the concept behind why we need to enable backups in order to use read replicas for aws rds?

can anyone help me with this ?

Turn off backups, on the replicas? I think that depends on the engine version you’re using. Not supported on MySQL 5.5 IIRC, but is on MySQL 5.6, for instance. Is that what you meant?

@Tim Malone I meant to ask - why do i have to enable backups in order to use read replicas ?

Is there any advantage in using S3 transfer acceleration if i am already using cloudfront to serve s3 files ?