#aws (2019-02)

aws Discussion related to Amazon Web Services (AWS)

aws Discussion related to Amazon Web Services (AWS)

Archive: https://archive.sweetops.com/aws/

2019-02-01

i5okie avatar

in ECS, with autodiscovery ALB.. is it possible to set the subdomain name to be different from the service name?

2019-02-04

i5okie avatar

I’ve been trying to figure out how to ‘architect’ the stack for our apps. we have 4 different services. This is a Ruby on Rails stack.

    1. the main app, public dns,
    1. auth, public dns.
  • 3 background scheduler - private dns needs to be accessible from main app.
    1. another app with private dns to be accessible by main app.

Fargate, RDS, elasticache.

I guess i need two ALBs. one private, and one public.

— questions — Is there any advantage/benefit to throwing Traefik in there somewhere?

  • if so, what would the stack look like (a secondary ecs cluster just for traefik? or stick traefik in the same cluster as the apps?)

Is there a tutorial / guide on how to use SSM Parameter Store to get ENVVARS into each task/container on deploy?

joshmyers avatar
joshmyers

@i5okie this sounds like a pretty contained app, I’m not sure what Traefik would add to it being more of an edge proxy

i5okie avatar

so just the dual ALB public and private?

joshmyers avatar
joshmyers

Is auth an internal app in the above?

joshmyers avatar
joshmyers
Specifying Sensitive Data - Amazon Elastic Container Service

Amazon ECS enables you to inject sensitive data into your containers by storing your sensitive data in either AWS Secrets Manager secrets or AWS Systems Manager Parameter Store parameters and then referencing them in your container definition.

i5okie avatar

yeah just a microservice type thing

joshmyers avatar
joshmyers

yeah make that an internal alb

i5okie avatar

thank you

2019-02-05

chrism avatar

Anyone using 99designs/aws-vault with packer? set the variables to use the aws env vars

    "aws_access_key": "{{env `AWS_ACCESS_KEY_ID`}}",
    "aws_secret_key": "{{env `AWS_SECRET_ACCESS_KEY`}}",

but aws just spits it back out Build 'amazon-ebs' errored: error validating regions: AuthFailure: AWS was not able to validate the provided access credentials 99 aws-vault

Works fine with terraform though

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

You also need to set 2 more variables

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I forget exactly, but they are AWS*SESSION_*

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

AWS_SECURITY_TOKEN=

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

AWS_SESSION_TOKEN=

chrism avatar

AHH I did wonder if i needed the session. Terraform just kinda works (outside of geodesic)

chrism avatar

Thanks; naming it aws-vault is a killer as all the results are aws (hashicorp) vault related

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i know!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

bad name

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

couldn’t they just have called it aws-assume-role

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

chrism avatar

lol almost as bad as another tech we use called EventStore. Sure… name your product after the actual bloody pattern

chrism avatar

one day terraform will produce errors that don’t fill me with rage

chrism avatar

the token = AWS_SESSION_TOKEN bit sorted packer out. Thanks for that Terraform automagically picks up env vars. Damn the inconsistency

chrism avatar

more confused after poking through other peoples shit in github as they’re using the same thing.

dave.udit avatar
dave.udit

We are looking for implementing transit vpc. Most third party solutions (Cisco, Aviatrix etc.) are expensive. Has anyone used AWS transit gateway? If yes, how was the experience?

Steven avatar

We decided not to use because of the increased cost over using VPC peering

dave.udit avatar
dave.udit

I see

dave.udit avatar
dave.udit

In our case there are multiple vpc and giving vpn access to Transit VPC makes more sense

dave.udit avatar
dave.udit

So evaluating this option

Steven avatar

It simplifies things but at additional cost for all traffic. So depends use case, If only for user VPN, it could make sense. But we needed that as well as apps going cross accounts for shared resources

dave.udit avatar
dave.udit
VPC sharing: A new approach to multiple accounts and VPC management | Amazon Web Servicesattachment image

My first interaction with AWS was immediately after the launch of the Asia Pacific (Sydney) AWS Region, just a bit over 6 years ago. Back then, the AWS Management Console had fewer services, and I quickly found the Amazon Virtual Private Cloud (VPC). In under 10 minutes, I could define a new VPC, with subnets, […]

2019-02-06

Steven avatar

Thanks

Pablo Costa avatar
Pablo Costa

Hello, I’m looking to vpc sharing as a way to reduce VPN costs, but I already implemented a vpc transit solution using FRR with BGP and Strongswan (IPsec) https://frrouting.org/

Igor avatar

How does VPC Sharing fit in with the practice of having separate AWS accounts for each environment? As an example, would it be a bad idea to share a VPC between Staging and Production?

dave.udit avatar
dave.udit

@Igor VPC sharing is not meant to be cross environment. The idea is to a) centralize management/billing from one account 2) make account boundaries transparent per environment. So for example if dev environment has components spread across multiple AWS accounts then all those components can be part of a single VPC (which earlier required separate VPC)

dave.udit avatar
dave.udit

@Pablo Costa I am curious how are you creating/updating transit vpc. Does Strongswan give any tools to do it or you are manually (from aws console or terraform) creating all required routing?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
eeg3/workspaces-portal

Amazon WorkSpaces Self-Service Portal. Contribute to eeg3/workspaces-portal development by creating an account on GitHub.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

This looks pretty cool

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
06:27:20 AM

2019-02-13

Maciek Strömich avatar
Maciek Strömich

Hey, did anyone upgrade dc1 to dc2 redshift clusters using Cloudformation? I know there are 2 paths using snapshot-restore and elastic resizing but I don’t want to cause a drift from CF state doing it manually and I wonder if CF does handle the upgrade in a data-loss-less way

Maciek Strömich avatar
Maciek Strömich

FYI if anyone is interested upon dc1 to dc2 redshift upgrade via cloudformation data is being migrated. the only thing to remember is that during the migration cluster is available only in read only mode

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
AWS Limit Monitor – AWS Answers

Automatically monitor your AWS service usage and receive notifications as you approach limits.

Nikola Velkovski avatar
Nikola Velkovski

Morning People, anyone can help me with how ECS decides to place the tasks when multiple ordered_placement_strategy blocks are used? Given the following example: I’ve 4 tasks ( dockers ) and 2 machines in 2 different AZs

ordered_placement_strategy {
    type  = "spread"
    field = "attribute:ecs.availability-zone"
  }

  ordered_placement_strategy {
    field = "memory"
    type  = "binpack"
  }

Will ECS stop bringing tasks up because it won’t be able to do so because the first placement block says different AZ or something entirely different will happen? Anyone got any ideas ?

pecigonzalo avatar
pecigonzalo

Are they 4 tasks of 1 service? could you explain a bit more

Nikola Velkovski avatar
Nikola Velkovski

SO

  • 1 service
  • 4 tasks
  • 2 instances
pecigonzalo avatar
pecigonzalo

And it places 2 and stops?

pecigonzalo avatar
pecigonzalo

Or you are just asking?

pecigonzalo avatar
pecigonzalo

If asking, then it should put 2 Tasks per Instance

Nikola Velkovski avatar
Nikola Velkovski

damn this slack doesn’t notify me when I get replies from a thread…

Nikola Velkovski avatar
Nikola Velkovski

Ok so the situation is even weirder because we are using awsvpc mode and it has restrictions on ENIS…

Nikola Velkovski avatar
Nikola Velkovski

meaning that the spread based on AZ is not needed

Nikola Velkovski avatar
Nikola Velkovski

because t2.small can have only 2 tasks per machine ..

Nikola Velkovski avatar
Nikola Velkovski

it supports 3 ENIs 1 for the machine 2 for tasks

Nikola Velkovski avatar
Nikola Velkovski

so sticking with memory only just in case the app has a memory leak.

Nikola Velkovski avatar
Nikola Velkovski

and with 2 machines and 4 tasks we get an even spread.

Nikola Velkovski avatar
Nikola Velkovski

tbh I do not like awsvpc mode, it kills the docker vibe

pecigonzalo avatar
pecigonzalo

Yeah, no idea why using that

pecigonzalo avatar
pecigonzalo

2019-02-14

Abel Luck avatar
Abel Luck

The cloudfront docs on which types of certs are valid is very confusing It seems you can use an ECC cert between cloudfront <—> origin but only an RSA cert between viewer <—> cloudfront but i can’t find this explicitly stated anywhere

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

hrm…. not something we’ve run in to

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(that said, we have some modules for cloudfront and acm)

2019-02-18

Maciek Strömich avatar
Maciek Strömich

do you observe RDS hostname resolution issues in us-east-1?

2019-02-19

chrism avatar

When you waste time writing code around aws config … then look at the pricing.

joshmyers avatar
joshmyers

@chrism Wrote https://github.com/alphagov/pay-aws-compliance a while back to help with auditing of AWS resources (and some others) - run as a scheduled Lambda - cheap!

alphagov/pay-aws-compliance

Contribute to alphagov/pay-aws-compliance development by creating an account on GitHub.

1
chrism avatar
toniblyx/prowler

AWS Security Best Practices Assessment, Auditing, Hardening and Forensics Readiness Tool. It follows guidelines of the CIS Amazon Web Services Foundations Benchmark and DOZENS of additional checks …

3
chrism avatar

which is pretty nice

chrism avatar

I’ll add the alpha gov one to my list

joshmyers avatar
joshmyers

Prowler looks pretty full featured, but bash!

chrism avatar

Well we run it via teamcity within docker and output the report as an artifact

1
chrism avatar
chrism
12:21:32 PM

plops it in a report tab; though I need to look at the background colour of the html at some point.

joshmyers avatar
joshmyers

Nice

joshmyers avatar
joshmyers

Can hook up Lambdas to SNS to send email reports etc too

chrism avatar

cuts down on the drudgery

chrism avatar

It’s a nice tool

joshmyers avatar
joshmyers

Looks good

chrism avatar

Drowning in things analysing aws; though Im yet to get anything of value from GuardDuty beyond 3rd parties appreciating it as a tick box

joshmyers avatar
joshmyers

Never looked at GuardDuty

joshmyers avatar
joshmyers

It’s impossible to keep up with all the AWS services hah

chrism avatar

they certainly dont miss a trick; just wish they’d have a chat with other areas of aws before building them

chrism avatar

annoying levels of disconnect between products; you set your multi region cloudtrail… and they bring out guard duty that has to be enabled in each regions. Oh and you can hook it up to another account so you can read from your root account the guard duty events of prod/testing etc…. but you have to do it per region, per account via invites

chrism avatar

… one of those moments you terraform the creation and realise you’re going to have to go hand-ball all the invite acceptance

chrism avatar

They almost got it right, but then didnt

chrism avatar

Pretty cheap though compared to most aws things so it has that going for it

chrism avatar
chrism
12:29:42 PM
chrism avatar

meh think i’ll just configure the hell out of existing scanners. AWS configs one of those things that looks like it’ll drive me up the f’ing wall rather than save me time

chrism avatar

I was looking at https://github.com/cloudposse/terraform-aws-cloudwatch-flow-logs but it doesnt seem to fit with the ref-architecture way of splitting audit off / storing to s3. its just flowlog>kinesis>CW

cloudposse/terraform-aws-cloudwatch-flow-logs

Terraform module for enabling flow logs for vpc and subnets. - cloudposse/terraform-aws-cloudwatch-flow-logs

chrism avatar

its kinda either or; I may be overthinking this

joshmyers avatar
joshmyers

Not sure on that one tbh, but have you looked at flowlogs or thought about what you wanna do with them?

joshmyers avatar
joshmyers

Have used mainly as a checkbox exercise before

joshmyers avatar
joshmyers

Creating/managing alerts/dashboards with them is quite expensive in terms of mgmt cost

chrism avatar

Yeah tbh the idea seems utterly pointless for the most point

chrism avatar

tickboxes for everyone

1
chrism avatar

Lets put it this way, if you’re having to dig into tb’s of flow logs to find evidence of something; you’ve missed a bigger step elsewhere

joshmyers avatar
joshmyers

How else are you going to find out how much data was exfiltrated from your environment for realz?!

chrism avatar
chrism
12:51:04 PM
chrism avatar

nothing like a terraform-github safari to find how other people dealt with stuff, and realising you’ve been here before

joshmyers avatar
joshmyers

lol

chrism avatar

reaches for the star already starred

joshmyers avatar
joshmyers

seriously though, storing flowlogs in something like $LOGGING_SAAAS_PROVIDER generates so much data, is not cheap to do anything proactive with the data

joshmyers avatar
joshmyers

cloudwatch would be cheaper but limited

chrism avatar

More worrying is when our SIEM tool hooks on, and reprocesses the same data

chrism avatar

You know your siem monitoring is worth the cash when you terraform up 30 machines in vsphere; tear them all down and they alert you a day later

1
chrism avatar

Well worth the email notification at 7am on a saturday

chrism avatar

GuardDuty will also look for compromised EC2 instances talking to malicious entities or services, data exfiltration attempts, and instances that are mining cryptocurrency.

chrism avatar

well at least guard duty covers the ex-filtration side; though id be interested to know how that works if its someones shitty app code being abused to haul data out

chrism avatar

my feeling is it wouldnt notice

1
joshmyers avatar
joshmyers

Bet it won’t pick up exfiltrating to an attacker owned S3 bucket

chrism avatar

“its in aws, everything is awesome”

chrism avatar

best place to pull data to; aws would take an age to lock you out and the transfers that fast you’ve less chance of the owner locking you out before you get it

joshmyers avatar
joshmyers

Need to watch out for that too if using S3 endpoints and aren’t locking down your buckets it bypasses any egress proxies you may have in place

joshmyers avatar
joshmyers

Found that out during a pentest :D

chrism avatar

we only really have public stuff in buckets; outside of the whole snowplow usage and thats all anonomised noise

chrism avatar

keeping data in SAAS, aka flick switch for encryption, flick switch for firewall, flick switch for logs

1
chrism avatar

after years of terraform its nice at times to be able to keep shit pretty secure and not have to spend days tweaking bollocks to get there

chrism avatar

pretty secure, as nothing is totally secure

chrism avatar

I need to look at s3 bucket security again as they love to change things

chrism avatar

The UI change was nice, more explicit options to not allow buckets to become public by accident

chrism avatar

Don’t wholly understand why they’re terrified of making it default-on

chrism avatar

the IAM stuff around S3 is pretty powerful; ridiculously so compared to Azures

joshmyers avatar
joshmyers

yeah but too many folks left things open which leads to breaches

chrism avatar

I mean why they don’t default it to most secure and make people open it up. They’ve made the UI better but it wont stop people doing dumb shit

chrism avatar

I mean I know it wont I’m still sat skimming tls updates for bucket names

joshmyers avatar
joshmyers

ah, hadn’t spent too much time poking in the UI recently

chrism avatar

our original account is old as f’; so where no terraform lives, one has to spelunk in aws ui.

chrism avatar

Compared to azure portal though everythings a dream

joshmyers avatar
joshmyers

hah, so I’ve heard

chrism avatar

I imagine its where the Windows 8 UI creator ended up as punishment

1
chrism avatar

terraform and azure sounded like a dream come true then I used it in anger for a month lots of anger lots and lots

chrism avatar

3 machine cluster 22minutes… still not complete TIMES THE HELL OUT

chrism avatar

wasn’t very impressive

2019-02-21

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
AWS Firecracker: 10 things every tech pro should knowattachment image

AWS Firecracker is tiny, efficient, fast, and might redefine the virtual machine. Here’s what you need to know about this AWS product.

Igor avatar

I have been getting a lot of We currently do not have sufficient {instance-type} capacity in the Availability Zone you requested {zone}. messages in EC2. Is this common? Does AWS address these capacity issues?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

this is where RIs help

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

…as they provide capacity reservations

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Igor can’t say we’ve been seeing it lately, but it highly depends on (a) the region you’re operating in (b) the type of instance

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

for example, we’ve seen this in the past when AWS has had a regional zone failure in a region and everyone auto scales out to the other zones

Igor avatar

So I need to purchase RIs for the specific AZ?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yes, the “capacity” reservations are tied to the AZ, but the cost savings span all AZs

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

though AWS has been revamping this, so maybe it’s easier now?

Igor avatar

At least that’s something

Igor avatar

@Erik Osterman (Cloud Posse) Do you know if there is a way to attach the RIs to an ASG? I purchased a couple, but they seem to immediately be used up on existing running instances.. which makes sense in retrospect

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

RIs are not a directly addressable resource from the EC2 perspective

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

it’s a billing instrument

Igor avatar

Yeah, but when it comes to capacity

Igor avatar

I want PROD instances to take priority over non-PROD, if that makes sense

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but they are not running in the same account, right? so that won’t happen

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i am not sure how to prioritize RI capacity reservations within an account

Igor avatar

It’s a legacy pre-SweetOps account

1
Igor avatar

I am locked for some customers due to not being able to move around NAT elastic IPs, and whitelisting changes are a pain

Igor avatar

#HarshRealityOps ?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, there are in the end always some things outside of our control

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

the NAT IP thing comes up regularly

2019-02-27

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

has anyone seen something to enforce arbitrary limits on the number of specific kinds of resources which match a given tag?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

basically a policy enforcement engine for aws which would let us say untagged instances are automatically destroyed

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

if the count of instances with tag Developer=foobar, is greater than X, then alert

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

if the count of instances with tag Developer=baz is greater than Y, then kill until less than Z

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
RiotGames/cloud-inquisitor

Enforce ownership and data security within AWS. Contribute to RiotGames/cloud-inquisitor development by creating an account on GitHub.

2019-02-28

oscarsullivan_old avatar
oscarsullivan_old

AWS Organizations and AWS SSO setup guide here: https://github.com/osulli/aws-sso-setup

osulli/aws-sso-setup

A guide on how to setup AWS Organizations and AWS SSO and an example permissions matrix. - osulli/aws-sso-setup

pecigonzalo avatar
pecigonzalo

Do you know if this works with things like aws-vault?

osulli/aws-sso-setup

A guide on how to setup AWS Organizations and AWS SSO and an example permissions matrix. - osulli/aws-sso-setup

oscarsullivan_old avatar
oscarsullivan_old

@pecigonzalo I don’t really see why not. The only trouble is if you create a resource on one account you have to create policies to share it. So for instance, I can share my S3 bucket storing my state files to all my accounts reasonably easily… but my dynamodb that contains the lock hash… not proving so easy to allow access from other accounts!

pecigonzalo avatar
pecigonzalo

Well im not sure, since I could not find any docs to get SSO to work with aws cli even, with assume roles or something

chrism avatar

If you setup WSL via the store (as they say you should now) geodesic fails to map the path :grimacing: :gun: It assumes its still in the local/lxss when in reality its in AppData\Local\Packages\CanonicalGroupLimited.Ubuntu18.04onWindows_79rhkp1fndgsc

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
WSL Bugs · Issue #379 · cloudposse/geodesic

what Not working on some WSL environments Here is a list of the changes I had to make to get root.cloud.posse script to run on WSL: In root.*.com: DOCKER_NAME was using $NAME environment variable. …

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Does this help?

chrism avatar

Guess this needs to be smarter

chrism avatar
cloudposse/geodesic

Geodesic is a cloud automation shell. It&#39;s the fastest way to get up and running with a rock solid, production grade cloud platform built on top of strictly Open Source tools. ★ this repo! h…

chrism avatar

joys of changing

chrism avatar

infact

chrism avatar

simpler

chrism avatar

the if/else just needs reversing; check for canonical … but i imagine that might have unexpected consiquences

chrism avatar

or msft need to stop being random af

chrism avatar

I’m taking the lazy way out and just blowing the folder away seeing ms’ command for removing lxss full didn’t actually do what anyone of reasonably sound mind would call “FULLY remove”

chrism avatar

kinda stuff that makes you think ubuntu desktop maybe the future; then you remember how much that blows

chrism avatar

AWS v2 providers out

chrism avatar
terraform-providers/terraform-provider-aws

Terraform AWS provider. Contribute to terraform-providers/terraform-provider-aws development by creating an account on GitHub.

chrism avatar

aka as good a time as any to add that version = “1.60.0” to your provider

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@chrism any chance we could get your help improving that path extrapolation? I’d be happy to jump on a call

chrism avatar

Think it might be worth a ticket on WSL to add some sort of inbuilt var of winpath

chrism avatar

He asks, and the lord microsoft giveth

chrism avatar

already exists

chrism avatar

wslpath -wa . in WSL returns the windows path to the current folder

chrism avatar
MicrosoftDocs/WSL

Source code behind the Windows Subsystem for Linux documentation. - MicrosoftDocs/WSL

chrism avatar

Been around a while too

chrism avatar

I shall give that a stab

chrism avatar
wslpath command does not return the windows path for home directory · Issue #3146 · Microsoft/WSL

$ wslpath -w ~~~ wslpath: /path/to/home: Result not representable Is this intended behavior? I expect following result: $ wslpath -w ~~~:\Users\mkt\AppData\Local\Packages\CanonicalGroupLimited.Ubuntu…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Hah! Damn… looked like a good option

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Just windows being windows. Getting in the way.

    keyboard_arrow_up