#aws (2020-09)

aws Discussion related to Amazon Web Services (AWS)

aws Discussion related to Amazon Web Services (AWS) Archive: https://archive.sweetops.com/aws/

2020-09-23

chrism avatar
chrism

Has anyone migrated their orgs into control tower? looks like pain in the butt

sheldonh avatar
sheldonh

Simplest static site hosting in aws that I can use security groups with to keep internal?

Thinking a fargate task that cicd builds with static site and hosts with something like “ran” and done. S3 buckets don’t seem to have anything with groups and ec2 while ok wouldn’t allow me to set target tasks at 1 for it to autoheal itself.

Any better way?

loren avatar
loren

s3 bucket with a vpc endpoint and a bucket policy that restricts access to the vpc endpoint?

:--1:2
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@loren beat me to it

2
sheldonh avatar
sheldonh

Nice. But if the folks needing to access it are not in the vpc and access is controlled through security groups would that work. Would the vpc endpoint have this option? I’m new to vpc endpoints. Don’t know if easy to plug one in and do that

loren avatar
loren

i suppose i was going off “internal”… depends on really what you mean by that

loren avatar
loren

but you can have a bucket policy with a statement that allows the vpc endpoint, and another statement that allows specific IPs

sheldonh avatar
sheldonh

I think i get it. Not sure that solves what I’m looking for. I want anyone at my company to be able to access, but I don’t want to host internally. I want to host in AWS. This means I’d need to use security groups to avoid it being a pain. If I could gate keep this behind SSO with microsoft that would be fine too, but didn’t want to complicate this or deal with any security other than security groups if possible. I’ve done lots of reading in the past on this and determined that s3 buckets just can easily be used like that

loren avatar
loren

well now with SSO you’re talking about an authenticated site. that’s a whole different thing again

loren avatar
loren

with security groups, that’s not authenticated. or at least, it is delegating the authentication to something else.

loren avatar
loren

i feel like now we have a classic X/Y problem, where the question doesn’t really address the problem

sheldonh avatar
sheldonh


Simplest static site hosting in aws that I can use security groups with to keep internal?
I don’t think I’m off from what I posted I’m up for other things if I’m missing something basic I could do, but mostly I’m talking about having an security group protected instance that serves up my static content… or I have to move to sharepoint

loren avatar
loren

alright, if you’re that focused on security groups being the answer, then you’re right. but that’s the part that is the X/Y issue, to me. they are a solution looking for a problem

loren avatar
loren

you could try lightsail. it’s more than a static site needs, but has a specialized cost model that might work, while being easier or more intuitive to manage than fargate. have you looked at this page yet? https://aws.amazon.com/websites/

Web Hosting - Amazon Web Services (AWS)

AWS offers a cloud web hosting solution that provides businesses & organizations with a flexible, scalable, low-cost way to deliver websites & applications.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

does it need to be a security group, or can it be CIDR based ACL?

sheldonh avatar
sheldonh

I could probably use the security groups as a data source and write the cidr blocks this way. That might work.

1
Chris Fowles avatar
Chris Fowles

this is not really what security groups are for and it sounds like you’re painting yourself into a corner by trying to make it work with security groups.

1
sheldonh avatar
sheldonh

Internal only static sites are annoying. Love static but the placement is annoying.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

It could be that they already have a list of acceptable CIDRs associated with the security group, so this would keep it “dry” by looking up those permitted ranges and then adding that to the bucket policy.

sheldonh avatar
sheldonh

Yes. That’s what I was wanting to use

RB avatar

anyone know any fancy cli magic to find ec2 instances that are NOT in SSM ?

RB avatar

if I query aws ssm describe-instance-information it only returns ec2 instances that have the ssm agent running

RB avatar

hacky but the only way i can figure out how to do this is to use the above command to dump out all the ec2 instances that have ssm and then filter them out from aws ec2 describe-instances to get the instances w/o ssm

RB avatar

is there an easier way to do this ?

PePe avatar

that way is the way I know, find the ones that have, find all, do diff

:--1:1
pjaudiomv avatar
pjaudiomv

so this doesnt really answer your question, but I have a lambda that tags all instances with ssm so I can then do a query to find all instances that doesnt have that tag

sheldonh avatar
sheldonh

Aws config has a rule for this

RB avatar

ah I didn’t know aws config supported this!

RB avatar

if an aws config rule can auto tag ec2 instances with/without ssm

RB avatar

then could an aws config rule also trigger the installation of ssm on non-ssm compliant resources ?

RB avatar

at that point, we wouldn’t even need the ssm tag.

RB avatar

so i enabled the ec2-instance-managed-by-systems-manager aws config rule

RB avatar

next is the lambda to install ssm

RB avatar

@ how does your lambda determine if an ec2 has ssm or not ?

pjaudiomv avatar
pjaudiomv

my ec2 instances are tagged with EX. SSM-Detected: true

pjaudiomv avatar
pjaudiomv

so if that tag isnt there or isnt true

RB avatar

how do you set the SSM-Detected tag ?

RB avatar

does each EC2 have a script on it that checks if ssm agent is running and then add the tag to itself ?

pjaudiomv avatar
pjaudiomv

Im acutally using a bash script on a pipeline schedule

pjaudiomv avatar
pjaudiomv

basically something like this


\#!/bin/bash

INSTANCE_IDS=$(aws ssm describe-instance-information --query 'InstanceInformationList[?starts_with(InstanceId, `i-`) == `true` && starts_with(PingStatus, `Online`) == `true`].InstanceId' --output json)

echo "$INSTANCE_IDS" | jq -r '.[]' | while read INSTANCE; do
    aws ec2 create-tags --resources $INSTANCE --tags "Key=SSM-Detected,Value=true"
done
RB avatar

ah it checks PingStatus

RB avatar

so PingStatus can only be checked on ec2 instances that are already managed by ssm

pjaudiomv avatar
pjaudiomv

yea

RB avatar

and thats the problem, i need to know which ones have ssm agent installed on them which means they don’t even show up as being managed by ssm

pjaudiomv avatar
pjaudiomv

ahhh

RB avatar

in fact the aws config rule ec2-instance-managed-by-systems-manager that @sheldonh was alluding too, also checks pingstatus unfortunately

RB avatar

so it looks like i’m back to square 1 and im going to use the solution @PePe agreed with which is hacky but it should work

:--1:1
pjaudiomv avatar
pjaudiomv

yea seems best way I know of

RB avatar

wow. cloud custodian to the freaking rescue.

      - name: ec2-ssm-check
        resource: ec2
        filters:
          - type: ssm
            key: PingStatus
            value: Online
        actions:
          - type: tag
            key: ssm
            value: true
:--1:1
pjaudiomv avatar
pjaudiomv

nice

sheldonh avatar
sheldonh

I want to deploy cloud custodian. I was thinking fargate task

RB avatar

we use cloud custodian periodic mode which creates lambdas for us

RB avatar

pretty low cost for us

sheldonh avatar
sheldonh

Wait I can use it to create lambdas for it’s functionality ? So cool! Even better

sheldonh avatar
sheldonh

Reading docs. Very cool! Going to try and deploy at least one today

RB avatar

ya, ive implemented it at 3 companies and it’s a surprisingly underused tool

RB avatar

it’s basically creating lambdas as a free and open source service

RB avatar

you can toss out so many custom scripts that lay there in dying jenkins instances

sheldonh avatar
sheldonh

Very cool! I bet it’s easier to create my own rules than aws config sdk. That is definitely not quick to learn.

2020-09-22

Issif avatar
Issif
awslabs/aws-perspective

AWS Perspective is a solution to visualize AWS Cloud workloads. Using Perspective you can build, customize, and share detailed architecture diagrams of your workloads based on live data from AWS. P…

:--1:1
Marcin Brański avatar
Marcin Brański

Looks amazing!

awslabs/aws-perspective

AWS Perspective is a solution to visualize AWS Cloud workloads. Using Perspective you can build, customize, and share detailed architecture diagrams of your workloads based on live data from AWS. P…

Issif avatar
Issif

it will not replace everything of my old code : https://github.com/claranet/aws-inventory-graph

claranet/aws-inventory-graph

Explore your AWS platform with, Dgraph, a graph database. - claranet/aws-inventory-graph

btai avatar

is there a way to route your AWS ALB to a different target group if the first target group fails a healthcheck?

Issif avatar
Issif

I never heard a feature like that, what you can do else, it to set up 2 ALB, with 2 target groups, and you use a route53 entry set to the 2 ALB and with an healthcheck

Issif avatar
Issif
Configuring DNS failover - Amazon Route 53
When you have more than one resource performing the same function—for example, more than one HTTP server or mail server—you can configure Amazon Route 53 to check the health of your resources and respond to DNS queries using only the healthy resources. For example, suppose your website, [example.com>, is hosted on six servers, two each in three data centers around the world. You can configure Route 53 to check the health of those servers and to respond to DNS queries for <http://example.com example.com](http://example.com) using only the servers that are currently healthy.
btai avatar

gotcha @Issif

Alex Jurkiewicz avatar
Alex Jurkiewicz

Yes, this sounds similar to canary deployments using multiple ALB + route53 weighted records

btai avatar

unfortunately thats not the problem im trying to solve. We have a WAF (sigsci) deployed in our k8s cluster as a reverse proxy and it’s a part of the request path. We want to allow ourselves to upgrade these WAF agents w/o causing downtime so we were thinking of a solution where the ALB will generally route traffic to the WAF agents which routes to our ingress controller agents unless the WAF agents return an unhealthy healthcheck, otherwise route straight to the ingress controller

Alex Jurkiewicz avatar
Alex Jurkiewicz

Like issif said, two ALBs should let you do this

Jonathan Le avatar
Jonathan Le

If you have a lot of load on the ALB, it might not be at the correct prewarmed size if you use a failover design with a hot/cold ALB. ALB scale fast, but if it’s a huge rp/s cutover, I’d expect lots of timeouts in the beginning of the switch.

2020-09-21

Daniel Pilch avatar
Daniel Pilch

Can anybody help me with advice on what is the best practice for creating IAM users for multiple aws accounts? Ideally I’d like to create users in a “root” type account and then those users are granted access to perform actions in other actions dependent on what type of account it is e.g. Dev/prod

If there’s a way to do this with an SSO type product like Okta that would be even better.

Thank you

Zach avatar

Using assumed roles much better than actual users in my opinion

Daniel Pilch avatar
Daniel Pilch

Yeah I was thinking assumed roles with SSO okta integration. Then having short-term credential support for api/cli

Zach avatar

thats what we use

Zach avatar

much easier to manage 3 or 4 roles than 30 iam users in 10 accounts

2020-09-18

Maciek Strömich avatar
Maciek Strömich

Hey folks, anyone using pro version of localstack to be able to use cognito locally? Does it work as expected and is the $15/per month/per developer worth the price?

2020-09-17

pjaudiomv avatar
pjaudiomv

anybody have any experience with or recommendations for AWS WAF alternatives like signal science or anything.

Eric Berg avatar
Eric Berg

We used SigSci at my last gig. It was pretty simple to set up and pretty transparent. I can’t speak to the functionality. I didn’t suck…mainly, because my exposure was limited.

Currently – in fact 5 mins ago – we started our first PoC for the Cloudflare WAF. This is the second attempt at insinuating CF between our browser apps and our load balancer. So far, their docs are a little all over the place (the dark side of documentation), and left us hanging at a number of places. We worked with their tech support, via the on-page chat, and even though the engineer that was hellping us was cool and helplful, it feels very commoditized. Obviously, i’m not new to curl, but at the level we play at, a company providing curl commands to hit its api, rather than wrapping that up in code in a command-line tool or web page, they give you curl commands in which you need to figure out by trial-and-error what values are to be replaced.

All that said, we are starting our Cloudflare WAF evaluation today.

pjaudiomv avatar
pjaudiomv

Thank you this was very helpful, and sounds about right.

Eric Berg avatar
Eric Berg

Cool. Glad to help. One of the things we like about Cloudflare is that it will allow us to manage WAV implementations across multiple accounts/client installations, and we need to be able to manage resources globally like that. Let me know how things go with your search. We are definnitely not sold on CF. In fact, we’re leaing toward the AWS WAF offering, in no small part, becuase it’s integrated wtih everything else we have in AWS. If this doesn’t go smoothly, we will probably opt for a vendor with more of a stake in our sucess, whcih means AWS where EVERYTHING lives over CF, where we are planning to use only the least amount of of one of their non-core offerings (WAF)

pjaudiomv avatar
pjaudiomv

Yea the problem with the aws waf is it has no capability for learning, and adding exceptions is a pain and can be too permissive. Also version to of waf with terraform the readability of the code is basically non existent compared to version 1

pjaudiomv avatar
pjaudiomv

However having said that being able to have lambdas auto attach lbs to different web acls and such is pretty nice

2020-09-16

sarkis avatar
sarkis

This is really strange, so I have an ALB and a Target Group with a path based health check, for some reason the ALB is hitting my application at / periodically (along with the normal healthcheck that hits the /health-check/path

sarkis avatar
sarkis

it’s on a different cadence, and I can’t figure out why on earth it does this if I don’t have my health check pointing to /

sarkis avatar
sarkis

I should mention the target group does have ECS Fargate instances that roll in and out - now I wonder if this is some default target group behavior i haven’t overriden

Yoni Leitersdorf avatar
Yoni Leitersdorf

Any chance you have another health check configured?

Yoni Leitersdorf avatar
Yoni Leitersdorf

The default one is indeed on “/”

sarkis avatar
sarkis

arg, so task def

sarkis avatar
sarkis

hmm my task definitions have healthCheck: null

Yoni Leitersdorf avatar
Yoni Leitersdorf

What about the target group?

sarkis avatar
sarkis

only 1 target group:

            "HealthCheckProtocol": "HTTP",
            "HealthCheckPort": "traffic-port",
            "HealthCheckEnabled": true,
            "HealthCheckIntervalSeconds": 30,
            "HealthCheckTimeoutSeconds": 5,
            "HealthyThresholdCount": 3,
            "UnhealthyThresholdCount": 3,
            "HealthCheckPath": "/health",
            "Matcher": {
                "HttpCode": "200"
            },
PePe avatar

do you have datadog or any other test from outside ? OR a container level check?

PePe avatar

if you are using ecs

sarkis avatar
sarkis

no container level check that i can find at least… no external test or service from outside.. hmm actually let me confirm the latter

sarkis avatar
sarkis

confirmed nothing from external sources

Yoni Leitersdorf avatar
Yoni Leitersdorf

Got a source IP showing in your logs?

sarkis avatar
sarkis

i have the logs from the app itself and it shows the ALB IP, I need to go look at the ALB access logs i think

sarkis avatar
sarkis

thanks for talking this through all! I’ll continue to dig into the access logs - if I do find anything I’ll come back and close the loop here

:--1:1
RB avatar

anyone know a safe and fast way of rebuilding an ecs cluster ? we recently opted into long arn format but have to recreate the cluster to take advantage

RB avatar

we’re thinking of creating a new cluster, migrating services to it, destroying the existing, and migrating clusters back

PePe avatar

you do not have to

PePe avatar

is is ECS+EC2 it will not affect running clusters

PePe avatar

but the easiest way is to create another cluster

PePe avatar

otherwise you need to recreate it

PePe avatar

I did that yesterday like 15 times

RB avatar

we have around 200 services in our cluster tho

PePe avatar

I have enabled long format etc on running services and we did not have an issue

RB avatar

i dont believe it works on a previously created cluster

RB avatar

maybe im having trouble conceptualizing migrating services from 1 cluster to another

RB avatar
Migrating your Amazon ECS deployment to the new ARN and resource ID format | Amazon Web Services attachment image

Update – August 21, 2020 – Added a section with the latest timeline. Starting today you can opt in to a new Amazon Resource Name (ARN) and resource ID format for Amazon ECS tasks, container instances, and services. The new format enables the enhanced ability to tag resources in your cluster, as well as tracking […]

PePe avatar

in our very old dev account we had like 50 services

PePe avatar

I just went an enable the new format for the whole account

PePe avatar

all services keep working

PePe avatar

but new services will adopt the new id format

RB avatar

i think the cluster has to be recreated tho

RB avatar

i also enabled the new format

RB avatar

oh i see so new services on an old cluster will be able to use the task tags ?

PePe avatar

no, only new cluster I think

PePe avatar

we do not use it so we don’t care

PePe avatar

but if you want to use it then you are screw you will have to recreate or migrate

1
btai avatar

anyone having IAM issues right now?

loren avatar
loren

yes, it’s down hard

loren avatar
loren


3:17 PM PDT We are investigating increased authentication error rates and latencies affecting IAM. IAM related requests to other AWS services may also be impacted.
https://status.aws.amazon.com/

loren avatar
loren
AWS live status. Problems and outages for Amazon Web Services attachment image

Real-time AWS (Amazon Web Services) status. Is AWS down or suffering an outages? Here you see what is going on.

Mikael Fridh avatar
Mikael Fridh

Probably impacting other things since I just got some alerts about ASG activity that shouldn’t happen…

2020-09-15

sarkis avatar
sarkis

What do you all recommend for deploying to ECS, specifically need to just change a task definition json to modify the image tags and then deploy it up to a service. I have this automated via github actions using the aws-actions, but I need a “break glass” in case that isn’t working right or need to revert to an older image tag. I’m leaning just interacting with the AWS api myself and using jq to do the image tag substitution - probably best…

Yoni Leitersdorf avatar
Yoni Leitersdorf

Why not use terraform?

Yoni Leitersdorf avatar
Yoni Leitersdorf

Playing with AWS APIs directly may be painful. Unless there’s something TF does that you don’t like (it can sometimes destroy resources without need).

sarkis avatar
sarkis

I’ve never had success using Terraform as a deployment tool

Yoni Leitersdorf avatar
Yoni Leitersdorf

I use it with ECS/Fargate + Task Definitions etc without issues

sarkis avatar
sarkis

I’m using Terraform to just get the ecs service and all supporting resources like vpc, lb, etc bootstrapped

Yoni Leitersdorf avatar
Yoni Leitersdorf

But maybe my use cases are different to yours

sarkis avatar
sarkis

yea i don’t know, it just feels a bit clunky to me… do you update the image tag you are deploying manually and terraform apply?

Yoni Leitersdorf avatar
Yoni Leitersdorf

Well, I use “latest” in the taskdef. I upload to ECR via an Orb in CircleCI. The I use “aws cli” to restart the task.

Yoni Leitersdorf avatar
Yoni Leitersdorf
      - aws-ecr/build-and-push-image:
          name: build-grabber
          path: grabber
          repo: python-stream-grabber
          tag: 'latest'
          requires:
      - aws-ecr/build-and-push-image:
          name: build-server
          path: server
          repo: python-stream-server
          tag: 'latest'
          requires:
      - restart_task:
          requires:
            - build-grabber
            - build-server

...

  restart_task:
    executor: aws-cli/default
    steps:
      - aws-cli/install
      - run:
          name: Restart task
          command: |
            TASK_ID=`aws ecs list-tasks --desired-status "RUNNING" --cluster "streamdelayer-cluster" | grep arn | sed 's/[ "]//g'`
            AWS_PAGER="" aws ecs stop-task --cluster "streamdelayer-cluster" --task $TASK_ID --reason "Restarting task due to new ECR image deployment"
Yoni Leitersdorf avatar
Yoni Leitersdorf

I preferred to use aws-cli over APIs directly. Less headache that way.

sarkis avatar
sarkis

ah okay - so you roll forward in case of issue? what happens if your latest tag has a bug and need to go back to last tag?

Yoni Leitersdorf avatar
Yoni Leitersdorf

Correct, I roll forward. I could change the task def to go back, but I noticed roll forward works a lot smoother.

:--1:1
Yoni Leitersdorf avatar
Yoni Leitersdorf

I could rollback the code changes and re-run the build, which will upload a new latest image which is the same as the previous version

sarkis avatar
sarkis

yep - valid strategy … thanks for sharing!

Gowiem avatar
Gowiem
fabfuel/ecs-deploy

Powerful CLI tool to simplify Amazon ECS deployments, rollbacks & scaling - fabfuel/ecs-deploy

Iouns avatar
Iouns
silinternational/ecs-deploy

Simple shell script for initiating blue-green deployments on Amazon EC2 Container Service (ECS) - silinternational/ecs-deploy

Iouns avatar
Iouns

This one is 1.6k , never tried it myself though

Iouns avatar
Iouns

ecs is complex enough to deserve its own deployment tool IMO.

maarten avatar
maarten
silinternational/ecs-deploy

Simple shell script for initiating blue-green deployments on Amazon EC2 Container Service (ECS) - silinternational/ecs-deploy

Gowiem avatar
Gowiem

I liked fabfuel/ecs-deploy when evaluating the two because it wasn’t built in bash. Bash is a necessary tool, but it isn’t a great tool to actually write a library in.

PePe avatar

Anyone have seen this error before ?

rror: lost websocket connection with ECS Telemetry service (TCS): websocket: close 1008 (policy violation): InvalidContainerInstance: Missing container instance arn" module=handler.go
PePe avatar

After name change and 3 complete destroy and restart with TF the instances are not registering with ECS service

PePe avatar

the error on ecs looks like

"status": "REGISTRATION_FAILED",
            "statusReason": "Unexpected EC2 error while attempting to Create Network Interface in subnet 'subnet-0f1c1aff459ba2755': InvalidParameterValue",
            "agentConnected": true,
PePe avatar

and I do not have trunking enabled

PePe avatar

this is running in host mode not awsvpc

PePe avatar

I did this yesterday in staging

PePe avatar

I do it in prod today and it

RB avatar

id create an urgent aws support ticket in chat mode

RB avatar

its been 2 hrs since your last reply in this thread. did you figure it out ?

PePe avatar

we do not have a support contract

PePe avatar

I did not figure it out

PePe avatar

I used the same TF project, changed the task def to add CPU and wen to Fargate

PePe avatar

started first try

RB avatar

ah dang, at least it’s working again!

PePe avatar

but now I need to figure out why

PePe avatar

this is one of the most useless command ever

PePe avatar

aws ecs list-account-settings --effective-settings

PePe avatar

set trunking to disable and still shows as enabled

PePe avatar

sooo it looks like this was the setting that was not enabled:

aws ecs put-account-setting-default --name containerInstanceLongArnFormat --value enabled
PePe avatar

although the useless cli shows are everything enabled

PePe avatar

now the instance is registering

PePe avatar

no that was not it it was this :

PePe avatar
aws ecs put-account-setting-default --name awsvpcTrunking --value disabled --region us-east-2

``

PePe avatar

so we are using Shared Vpcs

PePe avatar

and Trunking was enabled but is not compatible with shared vpcs

RB avatar

ahhhh wow, i would never have guessed

RB avatar

are these account level settings configurable via terraform ? i wonder if there is a terraform module for sane defaults that include the setting that caused that issue

PePe avatar

they are not

PePe avatar

you need to run those stupid commands

RB avatar
Support for account settings flags · Issue #10168 · terraform-providers/terraform-provider-aws

Community Note Please vote on this issue by adding a :–1: reaction to the original issue to help the community and maintainers prioritize this request Please do not leave &quot;+1&quot; or &quot;me to…

PePe avatar

I enabled trunking again and it failed

PePe avatar

so now I need to know who changed this setting in my account recently since this was working before

RB avatar

isnt there a way to disable toggling it via iam ?

RB avatar

cloudtrail should show who changed it if cloudtrail is enabled

PePe avatar

cloudtrail is enabled but this are account settings that I think can be done from organizations or at least that is my theory since we have been moving things recently

PePe avatar

but there is one guy that is working on TGWs and networking stuff and I wonder if he did something to all accounts

2020-09-14

zidan avatar
zidan

Hi guys, I’ve a question. In QA environment how do you provision and deploy the containers? For example if I have a lot of QAs and all of them want to run in the same time, images tag are following the branch name, and we are using terraform, and userdate to run the docker-compose file and run the containers, and after we finish the testing we destroy the resources using terraform destroy, but I’m feeling this is a bad way, so any recommendations? or how you guys do it?

walicolc avatar
walicolc
03:26:33 PM

hello peoples, anyone know why the commit hash doesn’t appear when the buildproject is automatically triggered.? I’m expecting refs/head/master^{commit_hash}  but i get refs/head/master when i don’t trigger it manually

2020-09-11

Darren Cunningham avatar
Darren Cunningham

Any recommendations on a dashboard for business users to review upcoming scheduled events (CloudWatch Rules)?

roth.andy avatar
roth.andy
Roach avatar
Roach

this may help: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-optimize-cpu.html
In most cases, there is an Amazon EC2 instance type that has a combination of memory and number of vCPUs to suit your workloads. However, you can specify the following CPU options to optimize your instance for specific workloads or business needs:
Number of CPU cores: You can customize the number of CPU cores for the instance. You might do this to potentially optimize the licensing costs of your software with an instance that has sufficient amounts of RAM for memory-intensive workloads but fewer CPU cores.
Threads per core: You can disable multithreading by specifying a single thread per CPU core. You might do this for certain workloads, such as high performance computing (HPC) workloads.

Optimizing CPU options - Amazon Elastic Compute Cloud

Specify CPU options for your instance by specifying the number of CPU cores and disabling multithreading.

roth.andy avatar
roth.andy

got it thanks, so you still pay for whatever instance type you pick but you can customize if you need to for some special reason

2020-09-10

RB avatar

We use a 3rd party agent that requires the ec2 describetags perms. We were thinking of creating a managed policy and attaching this policy to all of our roles since this agent will be installed on everything.

Is there a better way to solve this problem?

PePe avatar

Datadog?

RB avatar

nah. it’s a security agent

PePe avatar

we have a few of this policies attached to difference instance roles

PePe avatar

a nice way to solve this could be using SCPs in AWS Organizations

PePe avatar

it will be easier to manage

RB avatar

that is brilliant

RB avatar

i completely forgot about SCPs

PePe avatar

:–1:

zidan avatar
zidan

Hello Guys, how do you manage the deployment for containers in ECS? I have one service and one task, how to utilize the ec2 instances because right now I have two ec2 instances but only one has my containers and the other empty. and If I make it one instance only will not able to deploy the new version.

Yoni Leitersdorf avatar
Yoni Leitersdorf

Any reason not to go for Fargate? Takes a bit of the headache off of you.

zidan avatar
zidan

I just want to try it out with ec2, but I’m not sure what is the best practice for it.

Yoni Leitersdorf avatar
Yoni Leitersdorf

Ah, I don’t have much experience with EC2 mode.

zidan avatar
zidan

mmm, so how does it work with Fargate?

Yoni Leitersdorf avatar
Yoni Leitersdorf

With Fargate, you just care about the task definitions and the service. All the management of the VM is done by ECS for you.

Yoni Leitersdorf avatar
Yoni Leitersdorf

So, start with the task definition.

Yoni Leitersdorf avatar
Yoni Leitersdorf

Then, define the service, and use the task definition there.

Yoni Leitersdorf avatar
Yoni Leitersdorf

And that’s pretty much it.

Yoni Leitersdorf avatar
Yoni Leitersdorf

(depending on what the task is)

zidan avatar
zidan

Oh, that’s cool. do you use any external tool for your deployment?

Yoni Leitersdorf avatar
Yoni Leitersdorf

Terraform

Yoni Leitersdorf avatar
Yoni Leitersdorf

Here’s one example (it’s not the best, because you should have all the resources defined in terraform and not use ARNs, but it’ll give you an idea):

resource "aws_ecs_task_definition" "streamdelayer" {
  family = "streamdelayer"
  container_definitions = file("task-definitions/service.json")

  task_role_arn = "arn:aws:iam::123456789012:role/ecsTaskExecutionRole"
  execution_role_arn = "arn:aws:iam::123456789012:role/ecsTaskExecutionRole"
  network_mode = "awsvpc" 

  cpu = "256"
  memory = "512"

  requires_compatibilities = ["FARGATE"]

  volume {
    name = "chunks_efs"

    efs_volume_configuration {
      file_system_id = aws_efs_file_system.chunks.id
      root_directory = "/"
      /*transit_encryption = "ENABLED"
      authorization_config {
        access_point_id = aws_efs_access_point.aws_efs_mount_target.id
        iam             = "ENABLED"
      }*/
    }
  }
}

resource "aws_ecs_cluster" "cluster" {
  name = "streamdelayer-cluster"

  setting {
    name = "containerInsights"
    value = "enabled"
  }
}

resource "aws_ecs_service" "service" {
  name             = "streamdelayer-service"
  cluster          = aws_ecs_cluster.cluster.id
  task_definition  = aws_ecs_task_definition.streamdelayer.arn
  desired_count    = 1
  platform_version = "1.4.0"
  launch_type      = "FARGATE"

  network_configuration {
    subnets          = [aws_subnet.external.id]
    security_groups  = [aws_security_group.allow_ecr.id]
    assign_public_ip = "true"
  }

  load_balancer {
    target_group_arn = aws_lb_target_group.main.arn
    container_name   = "server"
    container_port   = 8000
  }
  depends_on = [aws_lb_target_group.main]

}
zidan avatar
zidan

so every time you hard code the new version for the image in the task definition?

Yoni Leitersdorf avatar
Yoni Leitersdorf

nope, I use “:latest”

zidan avatar
zidan

Oh, nice that’s good way, but if you want to rollback you will have to build and push the old image to ecr?

Yoni Leitersdorf avatar
Yoni Leitersdorf

Yes. I could always peg a specific version in the task definition and update that.

zidan avatar
zidan

Cool cool, thanks man I really appreciate your help.

Yoni Leitersdorf avatar
Yoni Leitersdorf

NP

Zach avatar

Highly recommend Fargate Spot if you’re going to try it out, the normal on-demand Fargate pricing is pretty steep

RB avatar
fabfuel/ecs-deploy

Powerful CLI tool to simplify Amazon ECS deployments, rollbacks & scaling - fabfuel/ecs-deploy

RB avatar

we initially create the task definition in terraform and then use ecs-deploy tool to deploy

PePe avatar

we use both Fargate and ECS+EC2, do I like any? No I HATE ECS but we use different deployment models

PePe avatar

we use jenking to trigger a deploy in some cases with the task def managed in TF since it does not change much

PePe avatar

and we created a Go tool to deploy in multiple regions since we have multi region deployments

PePe avatar

ecs-deploy tool is nice an can do a lot for you

Yoni Leitersdorf avatar
Yoni Leitersdorf

What do you hate about ECS?

PePe avatar

it is 100% Vendor Locking crap

:--1:1
Yoni Leitersdorf avatar
Yoni Leitersdorf

Welcome to PaaS

Yoni Leitersdorf avatar
Yoni Leitersdorf

You could run kube instead

PePe avatar

it is hard to setup, you need to understand a lot concepts the make it run etc

PePe avatar

and that is why we will be moving to K8s

PePe avatar

I do not see any benefit in ECS that K8 could not do and K8s makes the deployments way easier

PePe avatar

to be honest ElasticBeanstack is way easier than pure ECS and it does what most people will need

Yoni Leitersdorf avatar
Yoni Leitersdorf

I think it’s a matter of use cases, as always. ECS can make things really easy if you don’t want to deal with the complexities of K8s (workers, kubectl, etc. etc.). However, once you grow beyond certain levels of complexity in your application, k8s probably better. Also it’s portable.

Zach avatar

how is ECS vendor lock? You can take your containers and walk away anytime you want

RB avatar

thats the illusion

Yoni Leitersdorf avatar
Yoni Leitersdorf

Yes, but you build a whole operation around it. Think of monitoring for example - you end up using CloudWatch to monitor it (because other solutions aren’t great). You’re entire way of running it becomes tethered to ECS.

Not necessarily a bad thing, just something to be realized.

RB avatar

can you go easily from one paas (like ecs) to another ?

Zach avatar

By that definition your just as vendor locked onto K8s when “the next big thing” comes out

RB avatar

it’s kind of comparing apples to oranges.

ecs is paas and is closed source which means i cannot deploy it to GCP

in comparison, eks or kubernetes, is at least open source, and can be deployed to any cloud, no ?

Zach avatar

I’m just looking at it from an application standpoint more than ‘the thing running it’

PePe avatar

K8s have been out for a while and we are a Open Source Company so we are more tan willing to contribute

PePe avatar

which you can’t on ECS

PePe avatar

I never saw or though of ECS as the next big thing, for me it felt like learning Chef after using ansible

PePe avatar

and I felt the same way with K8 in the early stages but now there is so many resources that does not hold true anymore

2020-09-09

sheldonh avatar
sheldonh

AWS IAM Service Account assuming role in various other accounts in Terraform Cloud

Background: I have 8 AWS Accounts. I have 8 terraform cloud plans for deploying an IAM Service Account to each. Each get’s its own key.

I want to have

• create each of the service account users in a “home account”

• deploy the role for the service account to the remaining 7 (probably with group/role)

• in terraform cloud stop using the access keys (using data source lookup from terraform cloud state for the service account). Instead, I want to use assume role for all the other accounts. My main questions

  1. Anyone have a module laying this basic structure out?
  2. Is there a problem with using assume role with terraform cloud agents?
  3. I believe there are limitations when assume role is used, such as can’t update IAM role or something (haven’t gone to double check exact limits). Does this become problematic with using Terraform Cloud because things will start failing due to inherent limitations on an assumed role?

I’m ready to begin refactoring this from the access key approach to better design, but hoping someone else has laid this out in a blog post or willing to talk through it here or on call.

PePe avatar

I did not know you could assume rule in terraform cloud

sheldonh avatar
sheldonh

Well, I thought it is part of the provider SDK

PePe avatar

I thought f TF cloud account was tight to each aws account only

sheldonh avatar
sheldonh

it’s just one of the arguments you provide for assume role, so I “assumed” that that’s it.

sheldonh avatar
sheldonh

No. Terraform Cloud agents (not self-hosted) are hosted by them. It has no knowledge of your AWS infra without what you provide. There is no “system level aws connection”

sheldonh avatar
sheldonh

Basically I use a data source lookup right now for one workspace that manages service accounts and get data.terraform_[cloud.credentials.my](http://cloud\.credentials\.my)_user.access_key type of thing. I want to stop using access keys for each and instead use assumed roles since it seems to be a better practice

Eric Berg avatar
Eric Berg

I have a primary/root account in our AWS Organization. I set up TF Cloud with keys to a user in the master account that has the ability to assume roles in the other accounts. We use provider defs like this, using a single set of AWS keys:

provider "aws" {
  region  = "us-east-2"
  profile = "infra"
  assume_role {
    role_arn     = "arn:aws:iam::${var.account_number}:role/OrganizationAccountAccessRole"
    session_name = "Terraform"
  }
}

This way, the account_number is passed into TF as a variable and used to assume this role in the subaccount.

sheldonh avatar
sheldonh

Very nice. We aren’t using orgs right now. See any issues with this without org?

imiltchman avatar
imiltchman

For someone that switches between accounts a lot, is there a better workflow with aws-vault than to remember to log out first and then use the cli to log in?

1
Joe Niland avatar
Joe Niland

This doc has some info on using multiple browsers with aws-vault login https://github.com/FernandoMiguel/aws-vault-quick-guide

FernandoMiguel/aws-vault-quick-guide

Contribute to FernandoMiguel/aws-vault-quick-guide development by creating an account on GitHub.

imiltchman avatar
imiltchman

Ty, good tip

Vlad Ionescu avatar
Vlad Ionescu

This question’s a bit different, but I need some opinions: https://twitter.com/iamvlaaaaaaad/status/1303727463072239616

attachment image

Which one do you like more, A or B?

A: official AWS icons B: custom icons with no colorful backgrounds https://pbs.twimg.com/media/EhfFMgCXkAMwl8r.jpg

bradym avatar
bradym

B - I like the consistency. The difference between the old/new icon styles in A makes me wonder if the new ones are third-party tools or something at first glance.

attachment image

Which one do you like more, A or B?

A: official AWS icons B: custom icons with no colorful backgrounds https://pbs.twimg.com/media/EhfFMgCXkAMwl8r.jpg

Vlad Ionescu avatar
Vlad Ionescu

Nonono, all the icons in A are part of the latest and greatest pack from AWS.

Some just have backgrounds, while others don’t. It looks like generic service icons have backgrounds, but specifics don’t. Like ElastiCache has a background, but ElastiCache for Redis does not

bradym avatar
bradym

This must be what happens when design decisions are made by the same people who name aws services.

Alex Jurkiewicz avatar
Alex Jurkiewicz

Colour is good, consistency in diagrams is overrated. Infra diagrams become outdated instantly anyway

Gowiem avatar
Gowiem

I’d say A cause I prefer the color / official icons as well.

Chris Picht avatar
Chris Picht

colors are good, but i would recommend making them lighter colors so the black is easier to read.

Vlad Ionescu avatar
Vlad Ionescu

^^^ you can reply here, I just posted the Tweet link so I don’t have to copy/paste the photos

6
4
nileshsharma.0311 avatar
nileshsharma.0311

I know it’s a broad question , but how would you approach compliance in aws Let’s say ISO27017 or any other as a startup granted you’ve limited resources , may be share some of your stories :)

Alex Jurkiewicz avatar
Alex Jurkiewicz

Compliance is expensive. Can you avoid needing to be compliant?

2
nileshsharma.0311 avatar
nileshsharma.0311

Hahaha savage , exactly what we were thinking

2020-09-08

2020-09-06

nileshsharma.0311 avatar
nileshsharma.0311

I’ve been troubleshooting an IAM policy , Objective - Give EC2 permission to add/remove security group rules , Inbound and Outbound , This is the policy I’m using , Funny thing is it works if I attach it to an IAM user and Use awscli ( I checked , the IAM user has no other permissions than what the policy grants ) , but when I attach an IAM role with same policy to ec2 , it doesn’t , does iam users and roles interact with aws differently , Thanks in advance :)

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ec2:AuthorizeSecurityGroupIngress", "ec2:RevokeSecurityGroupIngress", "ec2:AuthorizeSecurityGroupEgress", "ec2:RevokeSecurityGroupEgress", "ec2:UpdateSecurityGroupRuleDescriptionsIngress", "ec2:UpdateSecurityGroupRuleDescriptionsEgress" ], "Resource": [ "arn:aws:ec2:us-east-1:acnumber:security-group/*" ], "Condition": { "StringEquals": { "ec2:ResourceTag/Name": "$NameOfTheSecurityGroup" } } } ] }

PePe avatar

and Instance profile = a profile attached to a instance

PePe avatar

I think using instance profiles is not possible to allow the instance to modify it’s own security group since is very unsafe from a security prospective

nileshsharma.0311 avatar
nileshsharma.0311

Yeah I get that , the use case was there are some services exposed to the internet , the Only way around it is to give the guys using the machine add/remove rules on the fly and delete the 0.0.0.0 rule , since the users know what they’re doing and security group changes are very frequent so that was it

nileshsharma.0311 avatar
nileshsharma.0311

I figured we can only update them using the console or pragmatically

nileshsharma.0311 avatar
nileshsharma.0311

Thanks for the help

PePe avatar

np

Soumya avatar
Soumya

hi has anyone knowledge around the iam role association to pods through service-account , Im able to do assume the role (assume-role-with-web-identity) of the same account , but now I need to assume a role present in a different account, I already tried out with attaching assume-role policy to my role for that (role present in 2nd account ) and even editing its trust relationships. Thanks in advance

nileshsharma.0311 avatar
nileshsharma.0311

Troubleshooting IAM policies is a nightmare , use IAM policy simulator if that helps , in cross account access both sides have to agree and there shouldn’t be any SCPs denying the action ( in case you’re using scps )

Soumya avatar
Soumya

no its working fine with kiam but facing access denied when assuming iam role via service-account.

2020-09-05

2020-09-04

MrAtheist avatar
MrAtheist

anyone knows of a way to identify if a user already has the “login-profile” enabled?

Peter Huynh avatar
Peter Huynh

Is this what you’re looking for? https://github.com/aws/aws-cli/issues/819

Check if profile exists command · Issue #819 · aws/aws-cli

We use several different profiles for uploading our static website to different environments, each environment has it&#39;s own profile. It would be nice to have a way to check if a profile is conf…

MrAtheist avatar
MrAtheist

hmm not quite what im looking for, but turns out it’s just this: aws iam get-login-profile

Jonathan Marcus avatar
Jonathan Marcus

What’s a good way to profile my AWS API usage? We’re hitting some RateLimitExceeded exceptions, and it would be great to start with data on what the calls are (you know, profiling instead of guessing). I’ve googled without success so far.

loren avatar
loren

interesting idea… i don’t see anything on the aws side that reports on that… could you write those exceptions to Cloudwatch Logs, and use a filter or subscription to collect the events?

loren avatar
loren

you could tweet the idea at #awswishlist

1
Jonathan Le avatar
Jonathan Le

This is a good question. I would start with cw monitoring for “RequestLimitExceeded” on the AWS/EC2/API Namespace and then also make sure CloudTrail was on.

Hopefully when API throttling kicked in, I could go to CloudTrail 10 minutes later and see what was hitting often.

Haven’t tried turning on “RequestLimitExceeded” monitoring yet though - in the docs, it says you need to contact AWS support to turn it on:

This is an opt-in feature. To enable this feature for your AWS account, contact AWS Support.
loren avatar
loren

i did see that, wasn’t sure if it was specific to just calls to the ec2 api…?

Jonathan Le avatar
Jonathan Le

i think it is specific for ec2, but lots of services ride on ec2 - it still might give a good picture.

i’m quite certain that dynamodb has different api throttling limits than ec2, so i guess it depends on what you’re after.

for me, when i see “RateExceeded” in the AWS Console, I have a hunch it’s the EC2 API. too many describes* within a short period of time,

Jonathan Marcus avatar
Jonathan Marcus

Thanks for the tips. I am going to tweet that, and unfortunately it’s Cognito not EC2. I’ll check if I can enable CloudTrail logging for Cognito requests

Jonathan Marcus avatar
Jonathan Marcus

Anyone know a way to profile AWS API calls? When you get a TooManyRequestsException it would be great to be able to identify every call to that service, instead of just guessing. #AWSWishlist

:--1:1
Chris Fowles avatar
Chris Fowles

Cloudtrail can give you some insights into what’s being called a lot (if it’s something that is logged by cloudtrail)

2020-09-03

RB avatar

anyone use this for okta as a replacement for gimme-aws-creds from nike?

https://www.okta.com/blog/2020/05/how-okta-aws-sso-simplifies-admin-and-adds-cli-support/

RB avatar

looks promising as it simplifies the process

loren avatar
loren

aws sso has its fans, but i still can’t get past the limited ability to define granular policies with per-account references to resources/ids. closest i can get to is a permission set that only allows assume role to a more constrained role

pib avatar

We’ve been using https://github.com/godaddy/aws-okta-processor and really liking how it works. It’s a nicer workflow than aws-okta was.

godaddy/aws-okta-processor

Okta credential processor for AWS CLI. Contribute to godaddy/aws-okta-processor development by creating an account on GitHub.

:--1:1
RB avatar

there are so many methods to do this. it would be ncie to have a comparison of all of them

RB avatar

ideally we’d want granular permissions like iam role perms as we do now.

RB avatar

and be able to click a button in okta to auto login to a specific account in AWS

RB avatar


aws sso has its fans, but i still can’t get past the limited ability to define granular policies with per-account references to resources/ids. closest i can get to is a permission set that only allows assume role to a more constrained role
could you explain more on this limitation ?

ms16 avatar

Gimme creds by Nike works great

RB avatar


Gimme creds by Nike works great
how do you open aws console using nike’s tool ?

plus nike’s tool came out before the sso integration. wouldn’t the new solution be better than nikes ?

loren avatar
loren


aws sso has its fans, but i still can’t get past the limited ability to define granular policies with per-account references to resources/ids. closest i can get to is a permission set that only allows assume role to a more constrained role
two links to forum responses i posted, that might get at the underlying issue

https://forums.aws.amazon.com/thread.jspa?threadID=282793&tstart=0

https://forums.aws.amazon.com/thread.jspa?threadID=312303&tstart=0

RB avatar

wow. aws sso users cannot assume a specific role !?

loren avatar
loren

it’s more making it automatic… and that’s a workaround to a limitation of permissions sets. i either have to create a permission set for every combination of account/role in order to reference the account id (think arn) or a resource id for policy resource-level restrictions, or create a permission set per role that only allows assumerole to the “real” role in the account

RB avatar

ah i see the issue. so the workflow is…

  1. idp (like okta) click button to login
  2. aws console opens up but is not using a specific role
  3. manually assume role xyz through console this requires the role xyz ‘s policy to have every sso user enabled to assume it
loren avatar
loren

i don’t think the last bit is an issue, far as allowing people access to accounts they shouldn’t have. more just that the workflow is burdensome on the user. a workaround for the limitation of permissions sets would be for aws sso to do step 3 itself and perform the assume-role

Zach avatar


idp (like okta) click button to login
aws console opens up but is not using a specific role
That’s how our okta/aws saml federation works anyways but thats so that you can choose particular roles for things. You get a little selection menu of which account/role you want to use, and only the ones that your okta groups map to are visible.

:--1:1
loren avatar
loren

That’s the okta->iam workflow, which is great. okta->aws-sso is different and subject to the limitations of aws-sso

Zach avatar

Ah

Zach avatar

unfortunate because my teams really dislike having to use a tool at the CLI to authenticate before using the AWS CLI

Zach avatar

I just told them to suck it up because security is hard

Zach avatar

and at least they aren’t working in a SCIF like I used to

:--1:1
loren avatar
loren

aws-okta-processor integrates authentication into the aws-cli usage, so the login is inline

PePe avatar

anyone here knowledgeable about Envoy proxy?

PePe avatar

if I had an existing service in AWS

PePe avatar

with an alb

PePe avatar

and the dns of the service is https://pepe.null.com and this service runs on ECS and is a simple webapp with a mysql backend

PePe avatar

if I wanted to add an envoy proxy to my task def ( with or without app mess)

PePe avatar

does envoy requires to know what is the hostname if the request that this webapp supposed to respond to?

PePe avatar

or just by adding the proxy I could just getting it going with minimal config like a listener ?

PePe avatar

I want to know what is the minimum configuration required to have it running

PePe avatar

and then plug in to other services

nileshsharma.0311 avatar
nileshsharma.0311

I’ve been working on it just yesterday , Im not there yet but but this is a great resource https://youtu.be/D0cuv1AEftE

PePe avatar

I saw that video, made me much more confused about everything

PePe avatar

there is a lot of assumptions about microservices and service discovery

PePe avatar

but moving from monolithic to this is what I want

2020-09-02

RB avatar

what’s a good strategy for turning on and off agents on a golden ami using tags ? cannot find any blog post on it

RB avatar

we have way too many AMIs and it’s due to combination of installing agents

RB avatar

id like to switch this so we can use a single AMI, install all the agents, turn them all on by default, and turn them off using tags

loren avatar
loren

Bake a script or config mgmt agent into the ami, set it to “run once” at startup, read the tags and apply the desired state?

loren avatar
loren

Or maybe use ssm documents/automation?

RB avatar

ah interesting so maybe i can configure something like this.

https://serverfault.com/a/148355/320458

Linux: Schedule command to run once after reboot (RunOnce equivalent)

I’d like to schedule a command to run after reboot on a Linux box. I know how to do this so the command consistently runs after every reboot with a @reboot crontab entry, however I only want the co…

RB avatar

can an ssm document run on all new instances only once they are brought up ?

loren avatar
loren

cloud-init also has a “run once per instance” feature… pretty sure you can drop a script in the directory and it will run it. linux-only, of course

vFondevilla avatar
vFondevilla

+1 to cloud-init and tags

loren avatar
loren

i’m pretty sure ssm could do it somehow, but i find the service to be highly confusing, so it’s usually my last choice

RB avatar

ah ok, ill stay clear of ssm for now then regarding the enablement of agents

RB avatar

i’ll try the cloud-init and tags method. thank you everyone.

RB avatar

im surprised there isn’t a blog post on this but guess we cannot rely on blog posts for everything

loren avatar
loren

someone has to be first! go for it!

Zach avatar

AMI tags can’t get read cross account fyi, if that changes your approach

RB avatar

yes that’s true. zach, you can configure a lambda that copies the tags across accounts

RB avatar

i have not configured it, but it seems like a good method to keep tagging consistent.

RB avatar

i also did not mean tags on the AMIs itself but on the EC2 itself

RB avatar

so if you spin up EC2 with tag datadog_agent=false , the cloud-init script can run, check the ec2s tag, if it is true, then install the agent, if it is false, then do not install

:--1:1
Zach avatar

Oh sure if you are copying the AMIs that works. We keep the AMIs in a single account and pull from it. It caused some problems when we tried to do something clever, only to find it was not so clever once we went to the other account

vFondevilla avatar
vFondevilla

Yes I was talking about ec2 tags retrieved via metadata

Zach avatar

gotcha

sheldonh avatar
sheldonh

ssm association for sure based on my quick scan

sheldonh avatar
sheldonh

SSM isn’t intuitive. i should do a blog post on this as i found it super confusing and have written a bunch of internal things on it.

Basically ssm association targed to tag can be just like cron job, or can be a one time bootstrap. I use it to bootstrap 200+ instances so anything newly created gets agents installed, and i can also reapply on a whim.

sheldonh avatar
sheldonh

the key is that you ensure your docs are idempotent so you feel safe running at anytime.

You can do an automation dog for adhoc runs and execute against a tag with a few clicks or automatically too.

Just be warned it’s not super intuitive on the naming, that’s what makes it tough.

sheldonh avatar
sheldonh

I have basically zero dependency on user data or any init now. This means I can update my scripts independently of the infra to patch/fix issues too.

My packer pipeline also runs the same scripts pretty much so my effort for better tooling also helps me try to move stuff to better golden images (i’ve given up on immutable at this time )

RB avatar

That’s fantastic sheldonh

RB avatar

I’d love to read a blog post from you on this subject

:--1:1
RB avatar

Do you use any terraform modules when creating your ssm docs?

sheldonh avatar
sheldonh

I like to post blog posts on issues people are asking about so I’ll do my best to

:--1:1
sheldonh avatar
sheldonh

I use a mixture to create my SSM docs.

RB avatar

What are some tips with ssm and creating a run once doc?

sheldonh avatar
sheldonh

If I was creating something in the future I might explore the template file

sheldonh avatar
sheldonh

Right now I use Powershell to generate the docks from native Powershell scripts so I can let them for format and so on.

sheldonh avatar
sheldonh

And the build script output is a yaml doc. Terraform manages it after that. Probably better way to do… But it worked for generating docs as code at the time. Can even use vaporshell and validate syntax.

Probably would first look at template file in future

sheldonh avatar
sheldonh

What you need to understand about the docks is there is no concept of a run once doc.

sheldonh avatar
sheldonh

They are just docs. It’s up to you to execute it or to link it up to an association so that it can run automatically

sheldonh avatar
sheldonh

ssm association with no schedule applied will automatically apply on a new resource or existing resources that haven’t yet had it run and that’s it. Unless you re deploy the doc

sheldonh avatar
sheldonh

Once you add a cron then it’s running on a schedule. Otherwise it is already a 1x run.

:--1:1
sheldonh avatar
sheldonh

Will catch up tomorrow in case you have more questions

sheldonh avatar
sheldonh

Think of SSM as a perhaps less featured puppet, chef etc. Some bugs and all but overall I’ve had a reasonable experience. It’s not super intuitive and like any aws service gotta figure it out

RB avatar

Thanks a lot Sheldon! I’ll try this out. Ssm seems like a good strategy

sheldonh avatar
sheldonh

Does anyone have a cool way to generate the equivalent of a temporary s3 “dropbox” for a client to securely upload with a token, and tear down? I was thinking of just doing a AWS Automation doc that would create a randomized s3 bucket, create scoped credentials for just that bucket and upon finish tear it down. This AWS SSM Automation doc would be a self-service dropbox option for larger database transfers and all.

is there a better way to do this?

roth.andy avatar
roth.andy

I’ve used Firefox Send many times. Very happy with it

sheldonh avatar
sheldonh

“It was launched on March 12, 2019 and taken offline on July 7, 2020”

sheldonh avatar
sheldonh

I’m talking about enterprise clients with a sensitive database that miught be 100GB. They need to drop it over to us and S3 is preferred destination for us. I want to do the equivalent of an upload a file only to a specific place with an access token only and minimize the need to have them run scripts if possible.

sheldonh avatar
sheldonh

If no online service with s3 to do this, then maybe a golang cli app that takes a single run token, askes for a file path, and does an upload to a desiginated s3 bucket only might be a cool way to do it, Just mulling over ideas

bradym avatar
bradym

I’ve not done it myself, but you could run your own firefox send instance https://github.com/mozilla/send

mozilla/send

Simple, private file sharing from the makers of Firefox - mozilla/send

sheldonh avatar
sheldonh

Not sure what this is getting me over some s3 solution? I still have to get it to s3 right?

bradym avatar
bradym

firefox send supports using s3 as the file store

sheldonh avatar
sheldonh

interesting. thanks for this i will review further then

loren avatar
loren

aws-vault just released v6, looks like some good stuff in there… https://github.com/99designs/aws-vault/releases/tag/v6.0.0

Release v6.0.0 · 99designs/aws-vault

Added Support for AWS SSO #549 docs Support for Yubikey TOTP #558 docs A shell script for adding a Yubikey to IAM #559 aws-vault exec –ecs-server starts an ECS credential server offering many adv…

5
1

2020-09-01

omerfsen avatar
omerfsen

Hi, Is GuarDuty required for CIS Benchmarks for AWS ?

chrism avatar
chrism

https://aws.amazon.com/bottlerocket/ BottleRocket has gone GA

Csaba avatar
Csaba

Anyone tried S3 upload from browser directly using POST policy - to a bucket with AWS Signature Version 4? I have the code which generates policy and signature - it’s working with old buckets, but not anymore with new ones, as AWS made required sig-v4 for new buckets after June2020. My form contains: <input type=“hidden” name=“x-amz-algorithm” value=“AWS4-HMAC-SHA256” /> still getting “The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.” Any idea?

RB avatar

Is enabling key rotation on an existing kms key safe?

RB avatar

any breaking changes?

PePe avatar

it depends on whatever is using it if it can handle the rotation

PePe avatar

in our case it will be somewhat catastrofic

RB avatar

no way, really ?

RB avatar

i pinged aws support and they said that it wouldn’t change the arn or alias the rotation is completely done in the backend. it doesn’t even require re-encrypting data.

RB avatar

id like to know why it would be catastrophic for your setup in case it also affects mine

PePe avatar

the arn does not change?

PePe avatar

well I thought the arn will change if you rotate it

PePe avatar

if that was the case then it will ber catastrofic

RB avatar

the arn does not change

PePe avatar

well then it will not be catastrofic for us

party_parrot1
PePe avatar

I though it did

RB avatar

i’ll let you know what happens after we enable it

PePe avatar

lol

Chris Wahl avatar
Chris Wahl

Back end key rotation harms nothing. It’s abstracted completely.

:--1:1
corcoran avatar
corcoran

it’s the key material that changes, nothing else about the key changes. Yes, you should rotate.

1
    keyboard_arrow_up