#aws (2020-09)

Not required https://d1.awsstatic.com/whitepapers/compliance/AWS_CIS_Foundations_Benchmark.pdf

any breaking changes?

it depends on whatever is using it if it can handle the rotation

in our case it will be somewhat catastrofic

no way, really ?

i pinged aws support and they said that it wouldn’t change the arn or alias the rotation is completely done in the backend. it doesn’t even require re-encrypting data.

id like to know why it would be catastrophic for your setup in case it also affects mine

the arn does not change?

well I thought the arn will change if you rotate it

if that was the case then it will ber catastrofic

the arn does not change

well then it will not be catastrofic for us

I though it did

i’ll let you know what happens after we enable it

lol

Back end key rotation harms nothing. It’s abstracted completely.

it’s the key material that changes, nothing else about the key changes. Yes, you should rotate.

we have way too many AMIs and it’s due to combination of installing agents

id like to switch this so we can use a single AMI, install all the agents, turn them all on by default, and turn them off using tags

Bake a script or config mgmt agent into the ami, set it to “run once” at startup, read the tags and apply the desired state?

Or maybe use ssm documents/automation?

ah interesting so maybe i can configure something like this.

https://serverfault.com/a/148355/320458

can an ssm document run on all new instances only once they are brought up ?

cloud-init also has a “run once per instance” feature… pretty sure you can drop a script in the directory and it will run it. linux-only, of course

+1 to cloud-init and tags

i’m pretty sure ssm could do it somehow, but i find the service to be highly confusing, so it’s usually my last choice

ah ok, ill stay clear of ssm for now then regarding the enablement of agents

i’ll try the cloud-init and tags method. thank you everyone.

im surprised there isn’t a blog post on this but guess we cannot rely on blog posts for everything

someone has to be first! go for it!

AMI tags can’t get read cross account fyi, if that changes your approach

yes that’s true. zach, you can configure a lambda that copies the tags across accounts

i have not configured it, but it seems like a good method to keep tagging consistent.

i also did not mean tags on the AMIs itself but on the EC2 itself

so if you spin up EC2 with tag datadog_agent=false , the cloud-init script can run, check the ec2s tag, if it is true, then install the agent, if it is false, then do not install

Oh sure if you are copying the AMIs that works. We keep the AMIs in a single account and pull from it. It caused some problems when we tried to do something clever, only to find it was not so clever once we went to the other account

Yes I was talking about ec2 tags retrieved via metadata

gotcha

ssm association for sure based on my quick scan

SSM isn’t intuitive. i should do a blog post on this as i found it super confusing and have written a bunch of internal things on it.

Basically ssm association targed to tag can be just like cron job, or can be a one time bootstrap. I use it to bootstrap 200+ instances so anything newly created gets agents installed, and i can also reapply on a whim.

the key is that you ensure your docs are idempotent so you feel safe running at anytime.

You can do an automation dog for adhoc runs and execute against a tag with a few clicks or automatically too.

Just be warned it’s not super intuitive on the naming, that’s what makes it tough.

I have basically zero dependency on user data or any init now. This means I can update my scripts independently of the infra to patch/fix issues too.

My packer pipeline also runs the same scripts pretty much so my effort for better tooling also helps me try to move stuff to better golden images (i’ve given up on immutable at this time )

That’s fantastic sheldonh

I’d love to read a blog post from you on this subject

Do you use any terraform modules when creating your ssm docs?

I like to post blog posts on issues people are asking about so I’ll do my best to

I use a mixture to create my SSM docs.

What are some tips with ssm and creating a run once doc?

If I was creating something in the future I might explore the template file

Right now I use Powershell to generate the docks from native Powershell scripts so I can let them for format and so on.

And the build script output is a yaml doc. Terraform manages it after that. Probably better way to do… But it worked for generating docs as code at the time. Can even use vaporshell and validate syntax.

Probably would first look at template file in future

What you need to understand about the docks is there is no concept of a run once doc.

They are just docs. It’s up to you to execute it or to link it up to an association so that it can run automatically

ssm association with no schedule applied will automatically apply on a new resource or existing resources that haven’t yet had it run and that’s it. Unless you re deploy the doc

Once you add a cron then it’s running on a schedule. Otherwise it is already a 1x run.

Will catch up tomorrow in case you have more questions

Think of SSM as a perhaps less featured puppet, chef etc. Some bugs and all but overall I’ve had a reasonable experience. It’s not super intuitive and like any aws service gotta figure it out

Thanks a lot Sheldon! I’ll try this out. Ssm seems like a good strategy

I’ve used Firefox Send many times. Very happy with it

“It was launched on March 12, 2019 and taken offline on July 7, 2020”

I’m talking about enterprise clients with a sensitive database that miught be 100GB. They need to drop it over to us and S3 is preferred destination for us. I want to do the equivalent of an upload a file only to a specific place with an access token only and minimize the need to have them run scripts if possible.

If no online service with s3 to do this, then maybe a golang cli app that takes a single run token, askes for a file path, and does an upload to a desiginated s3 bucket only might be a cool way to do it, Just mulling over ideas

I’ve not done it myself, but you could run your own firefox send instance https://github.com/mozilla/send

Not sure what this is getting me over some s3 solution? I still have to get it to s3 right?

firefox send supports using s3 as the file store

interesting. thanks for this i will review further then

looks promising as it simplifies the process

aws sso has its fans, but i still can’t get past the limited ability to define granular policies with per-account references to resources/ids. closest i can get to is a permission set that only allows assume role to a more constrained role

We’ve been using https://github.com/godaddy/aws-okta-processor and really liking how it works. It’s a nicer workflow than aws-okta was.

there are so many methods to do this. it would be ncie to have a comparison of all of them

ideally we’d want granular permissions like iam role perms as we do now.

and be able to click a button in okta to auto login to a specific account in AWS

aws sso has its fans, but i still can’t get past the limited ability to define granular policies with per-account references to resources/ids. closest i can get to is a permission set that only allows assume role to a more constrained role could you explain more on this limitation ?

Gimme creds by Nike works great

Gimme creds by Nike works great how do you open aws console using nike’s tool ?

plus nike’s tool came out before the sso integration. wouldn’t the new solution be better than nikes ?

aws sso has its fans, but i still can’t get past the limited ability to define granular policies with per-account references to resources/ids. closest i can get to is a permission set that only allows assume role to a more constrained role two links to forum responses i posted, that might get at the underlying issue

• https://forums.aws.amazon.com/thread.jspa?threadID=282793&tstart=0

• https://forums.aws.amazon.com/thread.jspa?threadID=312303&tstart=0

wow. aws sso users cannot assume a specific role !?

it’s more making it automatic… and that’s a workaround to a limitation of permissions sets. i either have to create a permission set for every combination of account/role in order to reference the account id (think arn) or a resource id for policy resource-level restrictions, or create a permission set per role that only allows assumerole to the “real” role in the account

ah i see the issue. so the workflow is…

1. idp (like okta) click button to login
2. aws console opens up but is not using a specific role
3. manually assume role xyz through console this requires the role xyz ‘s policy to have every sso user enabled to assume it

i don’t think the last bit is an issue, far as allowing people access to accounts they shouldn’t have. more just that the workflow is burdensome on the user. a workaround for the limitation of permissions sets would be for aws sso to do step 3 itself and perform the assume-role

idp (like okta) click button to login
aws console opens up but is not using a specific role That’s how our okta/aws saml federation works anyways but thats so that you can choose particular roles for things. You get a little selection menu of which account/role you want to use, and only the ones that your okta groups map to are visible.

That’s the okta->iam workflow, which is great. okta->aws-sso is different and subject to the limitations of aws-sso

Ah

unfortunate because my teams really dislike having to use a tool at the CLI to authenticate before using the AWS CLI

I just told them to suck it up because security is hard

and at least they aren’t working in a SCIF like I used to

aws-okta-processor integrates authentication into the aws-cli usage, so the login is inline

if I had an existing service in AWS

with an alb

and the dns of the service is https://pepe.null.com and this service runs on ECS and is a simple webapp with a mysql backend

if I wanted to add an envoy proxy to my task def ( with or without app mess)

does envoy requires to know what is the hostname if the request that this webapp supposed to respond to?

or just by adding the proxy I could just getting it going with minimal config like a listener ?

I want to know what is the minimum configuration required to have it running

and then plug in to other services

I’ve been working on it just yesterday , Im not there yet but but this is a great resource https://youtu.be/D0cuv1AEftE

I saw that video, made me much more confused about everything

there is a lot of assumptions about microservices and service discovery

but moving from monolithic to this is what I want

Is this what you’re looking for? https://github.com/aws/aws-cli/issues/819

hmm not quite what im looking for, but turns out it’s just this: aws iam get-login-profile

interesting idea… i don’t see anything on the aws side that reports on that… could you write those exceptions to Cloudwatch Logs, and use a filter or subscription to collect the events?

you could tweet the idea at #awswishlist

This is a good question. I would start with cw monitoring for “RequestLimitExceeded” on the AWS/EC2/API Namespace and then also make sure CloudTrail was on.

Hopefully when API throttling kicked in, I could go to CloudTrail 10 minutes later and see what was hitting often.

Haven’t tried turning on “RequestLimitExceeded” monitoring yet though - in the docs, it says you need to contact AWS support to turn it on:

This is an opt-in feature. To enable this feature for your AWS account, contact AWS Support.

i did see that, wasn’t sure if it was specific to just calls to the ec2 api…?

i think it is specific for ec2, but lots of services ride on ec2 - it still might give a good picture.

i’m quite certain that dynamodb has different api throttling limits than ec2, so i guess it depends on what you’re after.

for me, when i see “RateExceeded” in the AWS Console, I have a hunch it’s the EC2 API. too many describes* within a short period of time,

Thanks for the tips. I am going to tweet that, and unfortunately it’s Cognito not EC2. I’ll check if I can enable CloudTrail logging for Cognito requests

https://twitter.com/conductotech/status/1301901319746314240

Cloudtrail can give you some insights into what’s being called a lot (if it’s something that is logged by cloudtrail)

and Instance profile = a profile attached to a instance

I think using instance profiles is not possible to allow the instance to modify it’s own security group since is very unsafe from a security prospective

Yeah I get that , the use case was there are some services exposed to the internet , the Only way around it is to give the guys using the machine add/remove rules on the fly and delete the 0.0.0.0 rule , since the users know what they’re doing and security group changes are very frequent so that was it

I figured we can only update them using the console or pragmatically

Thanks for the help

np

Troubleshooting IAM policies is a nightmare , use IAM policy simulator if that helps , in cross account access both sides have to agree and there shouldn’t be any SCPs denying the action ( in case you’re using scps )

no its working fine with kiam but facing access denied when assuming iam role via service-account.

I did not know you could assume rule in terraform cloud

Well, I thought it is part of the provider SDK

I thought f TF cloud account was tight to each aws account only

it’s just one of the arguments you provide for assume role, so I “assumed” that that’s it.

No. Terraform Cloud agents (not self-hosted) are hosted by them. It has no knowledge of your AWS infra without what you provide. There is no “system level aws connection”

Basically I use a data source lookup right now for one workspace that manages service accounts and get data.terraform_[cloud.credentials.my](http://cloud.credentials.my)_user.access_key type of thing. I want to stop using access keys for each and instead use assumed roles since it seems to be a better practice

I have a primary/root account in our AWS Organization. I set up TF Cloud with keys to a user in the master account that has the ability to assume roles in the other accounts. We use provider defs like this, using a single set of AWS keys:

provider "aws" {
  region  = "us-east-2"
  profile = "infra"
  assume_role {
    role_arn     = "arn:aws:iam::${var.account_number}:role/OrganizationAccountAccessRole"
    session_name = "Terraform"
  }
}

This way, the account_number is passed into TF as a variable and used to assume this role in the subaccount.

Very nice. We aren’t using orgs right now. See any issues with this without org?

This doc has some info on using multiple browsers with aws-vault login https://github.com/FernandoMiguel/aws-vault-quick-guide

Ty, good tip

B - I like the consistency. The difference between the old/new icon styles in A makes me wonder if the new ones are third-party tools or something at first glance.

Nonono, all the icons in A are part of the latest and greatest pack from AWS.

Some just have backgrounds, while others don’t. It looks like generic service icons have backgrounds, but specifics don’t. Like ElastiCache has a background, but ElastiCache for Redis does not

This must be what happens when design decisions are made by the same people who name aws services.

Colour is good, consistency in diagrams is overrated. Infra diagrams become outdated instantly anyway

I’d say A cause I prefer the color / official icons as well.

colors are good, but i would recommend making them lighter colors so the black is easier to read.

Compliance is expensive. Can you avoid needing to be compliant?

Hahaha savage , exactly what we were thinking

Datadog?

nah. it’s a security agent

we have a few of this policies attached to difference instance roles

a nice way to solve this could be using SCPs in AWS Organizations

it will be easier to manage

that is brilliant

i completely forgot about SCPs

Any reason not to go for Fargate? Takes a bit of the headache off of you.

I just want to try it out with ec2, but I’m not sure what is the best practice for it.

Ah, I don’t have much experience with EC2 mode.

mmm, so how does it work with Fargate?

With Fargate, you just care about the task definitions and the service. All the management of the VM is done by ECS for you.

So, start with the task definition.

Then, define the service, and use the task definition there.

And that’s pretty much it.

(depending on what the task is)

Oh, that’s cool. do you use any external tool for your deployment?

Terraform

Here’s one example (it’s not the best, because you should have all the resources defined in terraform and not use ARNs, but it’ll give you an idea):

resource "aws_ecs_task_definition" "streamdelayer" {
  family = "streamdelayer"
  container_definitions = file("task-definitions/service.json")

  task_role_arn = "arn:aws:iam::123456789012:role/ecsTaskExecutionRole"
  execution_role_arn = "arn:aws:iam::123456789012:role/ecsTaskExecutionRole"
  network_mode = "awsvpc" 

  cpu = "256"
  memory = "512"

  requires_compatibilities = ["FARGATE"]

  volume {
    name = "chunks_efs"

    efs_volume_configuration {
      file_system_id = aws_efs_file_system.chunks.id
      root_directory = "/"
      /*transit_encryption = "ENABLED"
      authorization_config {
        access_point_id = aws_efs_access_point.aws_efs_mount_target.id
        iam             = "ENABLED"
      }*/
    }
  }
}

resource "aws_ecs_cluster" "cluster" {
  name = "streamdelayer-cluster"

  setting {
    name = "containerInsights"
    value = "enabled"
  }
}

resource "aws_ecs_service" "service" {
  name             = "streamdelayer-service"
  cluster          = aws_ecs_cluster.cluster.id
  task_definition  = aws_ecs_task_definition.streamdelayer.arn
  desired_count    = 1
  platform_version = "1.4.0"
  launch_type      = "FARGATE"

  network_configuration {
    subnets          = [aws_subnet.external.id]
    security_groups  = [aws_security_group.allow_ecr.id]
    assign_public_ip = "true"
  }

  load_balancer {
    target_group_arn = aws_lb_target_group.main.arn
    container_name   = "server"
    container_port   = 8000
  }
  depends_on = [aws_lb_target_group.main]

}

so every time you hard code the new version for the image in the task definition?

nope, I use “:latest”

Oh, nice that’s good way, but if you want to rollback you will have to build and push the old image to ecr?

Yes. I could always peg a specific version in the task definition and update that.

Cool cool, thanks man I really appreciate your help.

NP

Highly recommend Fargate Spot if you’re going to try it out, the normal on-demand Fargate pricing is pretty steep

this tool is nice https://github.com/fabfuel/ecs-deploy

we initially create the task definition in terraform and then use ecs-deploy tool to deploy

we use both Fargate and ECS+EC2, do I like any? No I HATE ECS but we use different deployment models

we use jenking to trigger a deploy in some cases with the task def managed in TF since it does not change much

and we created a Go tool to deploy in multiple regions since we have multi region deployments

ecs-deploy tool is nice an can do a lot for you

What do you hate about ECS?

it is 100% Vendor Locking crap

Welcome to PaaS

You could run kube instead

it is hard to setup, you need to understand a lot concepts the make it run etc

and that is why we will be moving to K8s

I do not see any benefit in ECS that K8 could not do and K8s makes the deployments way easier

to be honest ElasticBeanstack is way easier than pure ECS and it does what most people will need

I think it’s a matter of use cases, as always. ECS can make things really easy if you don’t want to deal with the complexities of K8s (workers, kubectl, etc. etc.). However, once you grow beyond certain levels of complexity in your application, k8s probably better. Also it’s portable.

how is ECS vendor lock? You can take your containers and walk away anytime you want

thats the illusion

Yes, but you build a whole operation around it. Think of monitoring for example - you end up using CloudWatch to monitor it (because other solutions aren’t great). You’re entire way of running it becomes tethered to ECS.

Not necessarily a bad thing, just something to be realized.

can you go easily from one paas (like ecs) to another ?

By that definition your just as vendor locked onto K8s when “the next big thing” comes out

it’s kind of comparing apples to oranges.

ecs is paas and is closed source which means i cannot deploy it to GCP

in comparison, eks or kubernetes, is at least open source, and can be deployed to any cloud, no ?

I’m just looking at it from an application standpoint more than ‘the thing running it’

K8s have been out for a while and we are a Open Source Company so we are more tan willing to contribute

which you can’t on ECS

I never saw or though of ECS as the next big thing, for me it felt like learning Chef after using ansible

and I felt the same way with K8 in the early stages but now there is so many resources that does not hold true anymore

this may help: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-optimize-cpu.html
In most cases, there is an Amazon EC2 instance type that has a combination of memory and number of vCPUs to suit your workloads. However, you can specify the following CPU options to optimize your instance for specific workloads or business needs:
Number of CPU cores: You can customize the number of CPU cores for the instance. You might do this to potentially optimize the licensing costs of your software with an instance that has sufficient amounts of RAM for memory-intensive workloads but fewer CPU cores.
Threads per core: You can disable multithreading by specifying a single thread per CPU core. You might do this for certain workloads, such as high performance computing (HPC) workloads.

got it thanks, so you still pay for whatever instance type you pick but you can customize if you need to for some special reason

Why not use terraform?

Playing with AWS APIs directly may be painful. Unless there’s something TF does that you don’t like (it can sometimes destroy resources without need).

I’ve never had success using Terraform as a deployment tool

I use it with ECS/Fargate + Task Definitions etc without issues

I’m using Terraform to just get the ecs service and all supporting resources like vpc, lb, etc bootstrapped

But maybe my use cases are different to yours

yea i don’t know, it just feels a bit clunky to me… do you update the image tag you are deploying manually and terraform apply?

Well, I use “latest” in the taskdef. I upload to ECR via an Orb in CircleCI. The I use “aws cli” to restart the task.

I preferred to use aws-cli over APIs directly. Less headache that way.

ah okay - so you roll forward in case of issue? what happens if your latest tag has a bug and need to go back to last tag?

Correct, I roll forward. I could change the task def to go back, but I noticed roll forward works a lot smoother.

I could rollback the code changes and re-run the build, which will upload a new latest image which is the same as the previous version

yep - valid strategy … thanks for sharing!

https://github.com/fabfuel/ecs-deploy

https://github.com/silinternational/ecs-deploy

This one is 1.6k , never tried it myself though

ecs is complex enough to deserve its own deployment tool IMO.

always used https://github.com/silinternational/ecs-deploy

I liked fabfuel/ecs-deploy when evaluating the two because it wasn’t built in bash. Bash is a necessary tool, but it isn’t a great tool to actually write a library in.

After name change and 3 complete destroy and restart with TF the instances are not registering with ECS service

the error on ecs looks like

"status": "REGISTRATION_FAILED",
            "statusReason": "Unexpected EC2 error while attempting to Create Network Interface in subnet 'subnet-0f1c1aff459ba2755': InvalidParameterValue",
            "agentConnected": true,

and I do not have trunking enabled

this is running in host mode not awsvpc

I did this yesterday in staging

I do it in prod today and it

id create an urgent aws support ticket in chat mode

its been 2 hrs since your last reply in this thread. did you figure it out ?

we do not have a support contract

I did not figure it out

I used the same TF project, changed the task def to add CPU and wen to Fargate

started first try

ah dang, at least it’s working again!

but now I need to figure out why

this is one of the most useless command ever

aws ecs list-account-settings --effective-settings

set trunking to disable and still shows as enabled

sooo it looks like this was the setting that was not enabled:

aws ecs put-account-setting-default --name containerInstanceLongArnFormat --value enabled

although the useless cli shows are everything enabled

now the instance is registering

no that was not it it was this :

aws ecs put-account-setting-default --name awsvpcTrunking --value disabled --region us-east-2

``

so we are using Shared Vpcs

and Trunking was enabled but is not compatible with shared vpcs

ahhhh wow, i would never have guessed

are these account level settings configurable via terraform ? i wonder if there is a terraform module for sane defaults that include the setting that caused that issue

they are not

you need to run those stupid commands

wow youre right, it’s not: https://github.com/terraform-providers/terraform-provider-aws/issues/10168

I enabled trunking again and it failed

so now I need to know who changed this setting in my account recently since this was working before

isnt there a way to disable toggling it via iam ?

cloudtrail should show who changed it if cloudtrail is enabled

cloudtrail is enabled but this are account settings that I think can be done from organizations or at least that is my theory since we have been moving things recently

but there is one guy that is working on TGWs and networking stuff and I wonder if he did something to all accounts

it’s on a different cadence, and I can’t figure out why on earth it does this if I don’t have my health check pointing to /

I should mention the target group does have ECS Fargate instances that roll in and out - now I wonder if this is some default target group behavior i haven’t overriden

Any chance you have another health check configured?

The default one is indeed on “/”

arg, so task def

hmm my task definitions have healthCheck: null

What about the target group?

Try describing target groups: https://docs.aws.amazon.com/cli/latest/reference/elbv2/describe-target-groups.html

only 1 target group:

            "HealthCheckProtocol": "HTTP",
            "HealthCheckPort": "traffic-port",
            "HealthCheckEnabled": true,
            "HealthCheckIntervalSeconds": 30,
            "HealthCheckTimeoutSeconds": 5,
            "HealthyThresholdCount": 3,
            "UnhealthyThresholdCount": 3,
            "HealthCheckPath": "/health",
            "Matcher": {
                "HttpCode": "200"
            },

do you have datadog or any other test from outside ? OR a container level check?

if you are using ecs

no container level check that i can find at least… no external test or service from outside.. hmm actually let me confirm the latter

confirmed nothing from external sources

Got a source IP showing in your logs?

i have the logs from the app itself and it shows the ALB IP, I need to go look at the ALB access logs i think

thanks for talking this through all! I’ll continue to dig into the access logs - if I do find anything I’ll come back and close the loop here

we’re thinking of creating a new cluster, migrating services to it, destroying the existing, and migrating clusters back

you do not have to

is is ECS+EC2 it will not affect running clusters

but the easiest way is to create another cluster

otherwise you need to recreate it

I did that yesterday like 15 times

we have around 200 services in our cluster tho

I have enabled long format etc on running services and we did not have an issue

i dont believe it works on a previously created cluster

maybe im having trouble conceptualizing migrating services from 1 cluster to another

hmm ya looks like it will be a manual process with the new cluster…

https://aws.amazon.com/blogs/compute/migrating-your-amazon-ecs-deployment-to-the-new-arn-and-resource-id-format-2/

in our very old dev account we had like 50 services

I just went an enable the new format for the whole account

all services keep working

but new services will adopt the new id format

i think the cluster has to be recreated tho

i also enabled the new format

oh i see so new services on an old cluster will be able to use the task tags ?

no, only new cluster I think

we do not use it so we don’t care

but if you want to use it then you are screw you will have to recreate or migrate

We used SigSci at my last gig. It was pretty simple to set up and pretty transparent. I can’t speak to the functionality. I didn’t suck…mainly, because my exposure was limited.

Currently – in fact 5 mins ago – we started our first PoC for the Cloudflare WAF. This is the second attempt at insinuating CF between our browser apps and our load balancer. So far, their docs are a little all over the place (the dark side of documentation), and left us hanging at a number of places. We worked with their tech support, via the on-page chat, and even though the engineer that was hellping us was cool and helplful, it feels very commoditized. Obviously, i’m not new to curl, but at the level we play at, a company providing curl commands to hit its api, rather than wrapping that up in code in a command-line tool or web page, they give you curl commands in which you need to figure out by trial-and-error what values are to be replaced.

All that said, we are starting our Cloudflare WAF evaluation today.

Thank you this was very helpful, and sounds about right.

Cool. Glad to help. One of the things we like about Cloudflare is that it will allow us to manage WAV implementations across multiple accounts/client installations, and we need to be able to manage resources globally like that. Let me know how things go with your search. We are definnitely not sold on CF. In fact, we’re leaing toward the AWS WAF offering, in no small part, becuase it’s integrated wtih everything else we have in AWS. If this doesn’t go smoothly, we will probably opt for a vendor with more of a stake in our sucess, whcih means AWS where EVERYTHING lives over CF, where we are planning to use only the least amount of of one of their non-core offerings (WAF)

Yea the problem with the aws waf is it has no capability for learning, and adding exceptions is a pain and can be too permissive. Also version to of waf with terraform the readability of the code is basically non existent compared to version 1

However having said that being able to have lambdas auto attach lbs to different web acls and such is pretty nice

Using assumed roles much better than actual users in my opinion

Yeah I was thinking assumed roles with SSO okta integration. Then having short-term credential support for api/cli

thats what we use

much easier to manage 3 or 4 roles than 30 iam users in 10 accounts

Looks amazing!

it will not replace everything of my old code : https://github.com/claranet/aws-inventory-graph

I never heard a feature like that, what you can do else, it to set up 2 ALB, with 2 target groups, and you use a route53 entry set to the 2 ALB and with an healthcheck

https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/dns-failover-configuring.html

gotcha @Issif

Yes, this sounds similar to canary deployments using multiple ALB + route53 weighted records

unfortunately thats not the problem im trying to solve. We have a WAF (sigsci) deployed in our k8s cluster as a reverse proxy and it’s a part of the request path. We want to allow ourselves to upgrade these WAF agents w/o causing downtime so we were thinking of a solution where the ALB will generally route traffic to the WAF agents which routes to our ingress controller agents unless the WAF agents return an unhealthy healthcheck, otherwise route straight to the ingress controller

Like issif said, two ALBs should let you do this

If you have a lot of load on the ALB, it might not be at the correct prewarmed size if you use a failover design with a hot/cold ALB. ALB scale fast, but if it’s a huge rp/s cutover, I’d expect lots of timeouts in the beginning of the switch.

in k8s two waf deployments, every deployment with an alb and on top of that route53

s3 bucket with a vpc endpoint and a bucket policy that restricts access to the vpc endpoint?

@loren beat me to it

Nice. But if the folks needing to access it are not in the vpc and access is controlled through security groups would that work. Would the vpc endpoint have this option? I’m new to vpc endpoints. Don’t know if easy to plug one in and do that

i suppose i was going off “internal”… depends on really what you mean by that

but you can have a bucket policy with a statement that allows the vpc endpoint, and another statement that allows specific IPs

https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-sourceip

https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-sourcevpc

I think i get it. Not sure that solves what I’m looking for. I want anyone at my company to be able to access, but I don’t want to host internally. I want to host in AWS. This means I’d need to use security groups to avoid it being a pain. If I could gate keep this behind SSO with microsoft that would be fine too, but didn’t want to complicate this or deal with any security other than security groups if possible. I’ve done lots of reading in the past on this and determined that s3 buckets just can easily be used like that

well now with SSO you’re talking about an authenticated site. that’s a whole different thing again

with security groups, that’s not authenticated. or at least, it is delegating the authentication to something else.

i feel like now we have a classic X/Y problem, where the question doesn’t really address the problem

Simplest static site hosting in aws that I can use security groups with to keep internal? I don’t think I’m off from what I posted I’m up for other things if I’m missing something basic I could do, but mostly I’m talking about having an security group protected instance that serves up my static content… or I have to move to sharepoint

alright, if you’re that focused on security groups being the answer, then you’re right. but that’s the part that is the X/Y issue, to me. they are a solution looking for a problem

you could try lightsail. it’s more than a static site needs, but has a specialized cost model that might work, while being easier or more intuitive to manage than fargate. have you looked at this page yet? https://aws.amazon.com/websites/

does it need to be a security group, or can it be CIDR based ACL?

https://aws.amazon.com/premiumsupport/knowledge-center/s3-aws-ip-addresses-access/

I could probably use the security groups as a data source and write the cidr blocks this way. That might work.

this is not really what security groups are for and it sounds like you’re painting yourself into a corner by trying to make it work with security groups.

Internal only static sites are annoying. Love static but the placement is annoying.

It could be that they already have a list of acceptable CIDRs associated with the security group, so this would keep it “dry” by looking up those permitted ranges and then adding that to the bucket policy.

Yes. That’s what I was wanting to use

if I query aws ssm describe-instance-information it only returns ec2 instances that have the ssm agent running

hacky but the only way i can figure out how to do this is to use the above command to dump out all the ec2 instances that have ssm and then filter them out from aws ec2 describe-instances to get the instances w/o ssm

is there an easier way to do this ?

that way is the way I know, find the ones that have, find all, do diff

so this doesnt really answer your question, but I have a lambda that tags all instances with ssm so I can then do a query to find all instances that doesnt have that tag

Aws config has a rule for this

ah I didn’t know aws config supported this!

if an aws config rule can auto tag ec2 instances with/without ssm

then could an aws config rule also trigger the installation of ssm on non-ssm compliant resources ?

at that point, we wouldn’t even need the ssm tag.

so i enabled the ec2-instance-managed-by-systems-manager aws config rule

next is the lambda to install ssm

@pjaudiomv how does your lambda determine if an ec2 has ssm or not ?

my ec2 instances are tagged with EX. SSM-Detected: true

so if that tag isnt there or isnt true

how do you set the SSM-Detected tag ?

does each EC2 have a script on it that checks if ssm agent is running and then add the tag to itself ?

Im acutally using a bash script on a pipeline schedule

basically something like this

#!/bin/bash

INSTANCE_IDS=$(aws ssm describe-instance-information --query 'InstanceInformationList[?starts_with(InstanceId, `i-`) == `true` && starts_with(PingStatus, `Online`) == `true`].InstanceId' --output json)

echo "$INSTANCE_IDS" | jq -r '.[]' | while read INSTANCE; do
    aws ec2 create-tags --resources $INSTANCE --tags "Key=SSM-Detected,Value=true"
done

ah it checks PingStatus

so PingStatus can only be checked on ec2 instances that are already managed by ssm

yea

and thats the problem, i need to know which ones have ssm agent installed on them which means they don’t even show up as being managed by ssm

ahhh

in fact the aws config rule ec2-instance-managed-by-systems-manager that @sheldonh was alluding too, also checks pingstatus unfortunately

so it looks like i’m back to square 1 and im going to use the solution @jose.amengual agreed with which is hacky but it should work

yea seems best way I know of

wow. cloud custodian to the freaking rescue.

      - name: ec2-ssm-check
        resource: ec2
        filters:
          - type: ssm
            key: PingStatus
            value: Online
        actions:
          - type: tag
            key: ssm
            value: true

nice

I want to deploy cloud custodian. I was thinking fargate task

we use cloud custodian periodic mode which creates lambdas for us

pretty low cost for us

Wait I can use it to create lambdas for it’s functionality ? So cool! Even better

Reading docs. Very cool! Going to try and deploy at least one today

ya, ive implemented it at 3 companies and it’s a surprisingly underused tool

it’s basically creating lambdas as a free and open source service

you can toss out so many custom scripts that lay there in dying jenkins instances

Very cool! I bet it’s easier to create my own rules than aws config sdk. That is definitely not quick to learn.

you can create a cloudwatch event that triggers off of cloudtrail

then when the cloudwatch event triggers, you can then make it invoke a lambda if you like

here’s a nice example https://aws.amazon.com/blogs/mt/monitor-and-notify-on-aws-account-root-user-activity/

nice, ok. Hm, there’s no way to skip the lambda and go straight from cloudwatch logs into SNS?

Like how cloudwatch alarms go straight to SNS?

i believe it would be a cloudwatch event that would trigger on cloudtrail event

you dont have to invoke the lambda

i use cloudtrail events to trigger a cloudwatch alarm that invokes a lambda that alerts me in slack/email using a cloud custodian policy

example: https://cloudcustodian.io/docs/aws/examples/accountrootlogin.html

we have a base ami and teams that use packer to build off the base ami

now we wonder what the original amazon amis were that the team amis are based off of

we technically tag base amis with source ami but it would be easier if there is a file already on the instance itself that contains this info

if one doesnt exist, we’re thinking of figurign out a way to bundle the metadata into our base amis to child amis will also be aware of their parent (or grand parent)

ah wow. there is one.

$ cat /etc/image-id

curl <http://169.254.169.254/latest/meta-data/ami-id>

is that what you mean

nah thats the current ami

the parent ami

gotcha

process

1. amazon outputs ami
2. we create a base ami
3. team creates ami off of base ami now what source ami is the team using? and amazingly it’s in the /etc/image-id file :D

nooice

Use assumed roles and then they only ever have temp credentials

I am using assumed roles for accessing other accounts, but team members still need a user account and a single access ID / secret for assuming the roles from the CLI. I don’t want those access ID / secrets to live too long.

Or maybe I’m misunderstanding your suggestion?

Can do cloud custodian I think and it can help setup notification and aws config rule.

@sheldonh I’ve seen Cloud Custodian before — didn’t have a need back then, but it would likely be beneficial for this company. Thanks for the suggestion!

Are you talking of an static site served from s3?

yessir

there is few ways, you can do it with a fargate task, nginx proxy to s3

with an alb or not

or cloudfront+s3

that is about it

you could abiously use any other proxy like envoy, haproxy etc

but it means an ALB before your proxy, you can’t extract ACM certs

You can do alb+nginx proxy+S3 using acm, it works just fine

exactly my point, or directly traefik with let’s encrypt

ah sounds like a pain to setup. i was hoping to do it with only aws stuff. i guess cloudfront+s3+acm is the only way (besides lightsail).

it’s the easiest way (and cheapest)

I used the clousposse module with cloudfront and s3 and It was VERY easy

those would have been private ones, but not doable since the api they are going to balance against:

1. is reachable by vpn only
2. has public address that combo rules out nlb/alb to front the api

I went with a small lambda function as reverse proxy instead

you can use a “child” account, I do this across 8 acccounts and the one i aggregate to is not the org “primary”

Did you use some other automation to deploy the role across every account?

yes so i used terraform but theres no way to accep[t an invitation right now so you can only enable it and submit a request with terraform

aws_securityhub_account and aws_securityhub_member

you could probably use a null_resouce and cli or python script but i decided i could make better use of my time for a one time operation like that

gotcha, guess I’ll go the same route

your module isn’t public is it?

no but its super basic i can post it

aws_securityhub_account just enables it for that account and takes no arguments

right on thanks

Basically that’s a management network. Sort of the equivalent of out-of-band of data centers.

You set up a mgmt and then peer with each of the other VPCs. You allow traffic between the mgmt and the peered VPCs, but not between the peered VPCs.

Of course, it means you have one point that, if compromised, will be very bad for you.

It needs to be very protected.

I would encourage you to rethink the “bastion” need, and see if there are other ways to access the data you need, while implementing strong IAM enforcement.

Thanks @Yoni Leitersdorf (Indeni Cloudrail) for the response

Do you have a preferred way to access RDS instances in a private subnet across different accounts with strong IAM/MFA enforcement?

This is the exact issue I’ve been trying to solve the last 2 days, the only solution I’ve come to is the management VPC with MFA at the VPN level.

Can you access the data not using the DB protocol directly? Through an API or something of the sort?

I don’t know what data you are looking to access, but maybe there’s a way to auto-collect it with Lambda and place it elsewhere.

I think Gravitational Teleport goes some way to help tidy up bastion ops but haven’t looked at RDS and other managed service access. Curious to see what other ideas come out of thread.

Yes there is a risk because you are only using a single instance type. If there is no available spot capacity for that instance type you will loose your spot requests until sufficient spot capacity is available.

I’ve seen this happen when there has been a service outage in a specific availability zone which has caused resource contention. Both on-demand and spot instances were not able to start as there wasn’t any availability of that instance type.

If you are going to go 100% ideally you want to diversify across instance types and families to ensure your cluster can survive at least a loss of one instance type.

makes sense. I will add more types of instances but I don’t want to make it too complicated. What about spot.io platform is it worth to try them or are asg with spot instances enough?

Just having multiple instance types in your Launch Template and running 100%-spot might well be good enough for you. It depends on your workload, if it’s critical production I’d always keep at least 1 on-demand instance in an asg to ensure that spot market fluctuation don’t impact you. But if your workload can handle an outage without service or data loss you might not need this.

relevant blog: https://aws.amazon.com/blogs/aws/new-ec2-auto-scaling-groups-with-multiple-instance-types-purchase-options/

Use as many AZ as possible. 1 of our services is 100% fargate spot and scales to 2000 tasks across 6 AZ. Been this way for about 6 months with no issues.

Yes indeed use all the AZ’s you can. Second nature to use them all I forgot to even mention it!

perfect, we use all az’s already. The production cluster actually has one base on demand instance. I’m not sure what would happen if no spot instances were available would it launch on demand or not?

My old team at GumGum really loved Spot.io.

If believe if your Launch Template doesn’t specify any on-demand in Optional On-Demand Base or On-Demand Percentage Above Base your instances won’t start and you’ll get instance-terminated-no-capacity status

https://aws.amazon.com/premiumsupport/knowledge-center/ec2-spot-instance-termination-reasons/

i use this https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/organizations_account

oh interesting, i’ve been dumbly doing it from the command line.

aws organizations create-account --email <newAccEmail> --account-name "<newAccName>" --role-name <roleName>

thanks for sharing!

welcome

is it still 5000+ lines of cloudformation and a couple of step functions?

Yes it is still CF. May be some improvements as it was turned into a product. But basics are the same

I’ve been dealing with almost two days of environmental issues on my build agent box. I’d like to transition to more docker related jobs and stop managing more ec2 instances

if I deploy an ECS fargate task … It still requires ECS cluster. Is it basically just metadata at that point since I don’t have any Yeah. There are nodes under the hood, but since it’s fargate, they’re abstracted from you and you don’t deal with them at all. You just think of tasks.

Windows — I have no clue unfortunately. I wouldn’t be surprised if Fargate did not support windows. It’s still young in my mind.

Does the new docker and ECS integration hold a lot of promise for me to quickly deploy something I tested locally? I feel it’s better than doing things through EC2 for sure, but there is a good bit of domain model cruft in any container orchestrator.

I would suggest looking at CLI tools that help manage ECS easily for this type of things:

1. AWS’ ecs-cli — https://github.com/aws/amazon-ecs-cli / https://github.com/aws/copilot-cli — Haven’t used, but they’re putting a lot of effort into this toolset so I would imagine it’s got some good stuff going for it.
2. https://github.com/fabfuel/ecs-deploy — I’ve used this lightly, but would suggest this over doing the bash scripting that everyone does to manage ECS deployments / scaling.
3. https://github.com/masterpointio/ecsrun — This is my own project for maintaining YAML configurable ad-hoc jobs like you’re talking about.

Windows containers are only supported for tasks that use the EC2 launch type. The Fargate launch type is not currently supported for Windows containers.

i don’t see why not. it’s just a zone in route53, so it should resolve fine as long as your dns resolver uses the vpc dns server

Ok, yes I can retrieve the zone id. I am going to try configuring the resolver. Thanks @loren!