#aws (2020-07)

aws Discussion related to Amazon Web Services (AWS)

aws Discussion related to Amazon Web Services (AWS) Archive: https://archive.sweetops.com/aws/

2020-07-31

2020-07-30

Biswajit Das avatar
Biswajit Das

Hello All, I have been working on a CW Alarm solution for a Project, where we are moving all the logs from text files to CW Logsgroups logs. I have to filter a particular line from AWS CW log-groups log and send SNS . I am able to do so via custom metrics.

However I am looking for customisation where I would look to get the complete Message of CW Log Group if there is a match in string? Any help would be appreciated. Thanks Biswajit

Csaba avatar
Csaba

Anyone knows AWS SSM Documents can be shared between regions? I can see sharing option with other AWS account only I have tried it, and I can see shared document only in the same region. Should I replicate all my documents in every used regions? Thanks

rajeshb avatar
rajeshb

its like any other AWS resource , you have to create a cross account role and Document policy has to be updated accordingly. Hope this helps.

:--1:1
Balazs Varga avatar
Balazs Varga

Aws elb throttling. Any idea how can I avoid it?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Use NLB? :-)

2020-07-29

Vlad Ionescu avatar
Vlad Ionescu

AWS re:Invent 2020 is officially going virtual and free. re:Invent will be presented across 3 weeks from Nov. 30 - Dec. 18, 2020 https://reinvent.awsevents.com/

AWS re:Invent attachment image

Join the AWS Cloud Community

:--1:4

2020-07-28

sheldonh avatar
sheldonh

I setup a terraform service account creator for multiple accounts. It uses Cloudposse iam user service user module. That means I have 7-8 plans in Terraform Cloud that any update automatically runs and deploys.

Then I get the benefit of all my jobs running a data source = this plan = get aws access key.

However, I’d prefer to avoid iam keys and use roles when possible. Is this the correct thought process?

  1. Create home user account with the same approach.
  2. Create roles in each account for the terraform-service account
  3. Add trust relationship and allow the iam home account principle user to assume role
  4. Ensure MFA not set on this particular user
  5. Done? I’ve have such a time of it in the past with roles I had given up due to time. I’m ready to revisit to setup better, and think this is the correct way to go about it, otherwise I have iam creds for each account/user, which isn’t really preferred.
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

So these are accounts for external automated systems?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

e.g. something like CircleCI

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

anywhere your service is running inside of AWS, you should avoid the iam user module and instead use assumed service account roles in ECS or EKS, or instance profiles with EC2.

sheldonh avatar
sheldonh

Yes, they are external

sheldonh avatar
sheldonh

Like Terraform Cloud, Azure DevOps pipelines for example. I can setup an agent in one account but honestly trying to avoid where possible as I prefer to use the hosted agents rather than having another server to manage

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think what you describe is the gist of it. It definitely does suck having those keys laying around in SSM, but hard to avoid storing them somewhere.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

There’s also a terraform-1password-provider for storing secrets that should be shared with teams for integrations

sheldonh avatar
sheldonh

So i can use 1 iam user with assume role setup across accounts instead of using 1 unique IAM user per account? That makes sense to you right?

Eric D. Berg avatar
Eric D. Berg

My TF authenticates to multiple accounts, from a main account, using role assumption. I also have roles in each subaccount with reduced permissions, to which I grant groups of users the ability to assume in subaccounts.

sheldonh avatar
sheldonh

@ external terraform account right?

Eric D. Berg avatar
Eric D. Berg

Not sure what you’re asking. I have a master account, which has DNS and things loike that in there and for each client, we spin up a new aws account and use TF to create our entire stack in that account. We access these subaccounts using role assumption.

RB avatar

anyone come up with a terraform method of switching between launch_type=EC2 and launch_type=FARGATE with zero downtime ? looking for a terraform-y way to do this.

im wondering if the code deploy method would work

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/codedeploy_deployment_group#blue-green-deployments-with-ecs

PePe avatar

I think you will need to do that on the Target group level with Code deploy maybe

David avatar
David

Target group level seems right. I would make a new target group, add the new Fargate service pointed at that new target group, and then update your ALB Listener to point to the new target group. Then clean up the old ECS launch_type service and target group.

If you wanted to test out that your Fargate service actually works, you could use a Listener Rule where it would use the new service if you pass some special header, otherwise it would forward to the old target group by default

PePe avatar

I try this with codedeploy and it worked just fine, I do have the code still and we could make a module out of it

RB avatar

yeah i think this would be super helpful. because we’ve been doing cost comparisons between ecs and fargate, ive been trying to explore a way to switch between those easily with zero downtime

RB avatar

if you have a code already or know of a tf module that already supports this, then id love to use it

PePe avatar

my code I think it was for tf 11 but can be easily converted

PePe avatar

we can work together on this if you want

RB avatar

yeah that would be awesome

2020-07-27

Prasad avatar
Prasad

Hello all, Might be a basic query. is the Resource block Mandatory in Resource Policies? if i’m attaching the resource policy to a bucket i was hoping the only required fields are effect allow and principal as a minimum requirement

Issif avatar
Issif

it’s mandatory for sure

Joe Niland avatar
Joe Niland

it is but you can give it a non-existent resource if you need a neutral default

Prasad avatar
Prasad

thanks @Issif and @Joe Niland

Eric D. Berg avatar
Eric D. Berg

Has anybody seen this error message?

Error: Error setting SSLNegotiationPolicy: InvalidConfigurationRequest: external-tls can be associated only with a listener with one of HTTPS, SSL as frontend protocol
        status code: 409

I haven’t found anything helpful. I opened a case with AWS, but wondering if the genius pool here might come up with something. Thanks!

Alan Kis avatar
Alan Kis

Yes and no. A little bit context here would be helpful, but on the first it looks like there is a misconfiguration with LB’s security policy (ciphers).

Eric D. Berg avatar
Eric D. Berg

I had been told that some old config for nginx-ingress was no longer needed. I put it back and it started working . it was this config:

                rbac:
                  create: true
                controller:
                  config:
                    server-tokens: "false"
                  service:
                    enableHttp: false
                    targetPorts:
                      https: 80
                    annotations:
                      [service.beta.kubernetes.io/aws-load-balancer-ssl-cert](http://service\.beta\.kubernetes\.io/aws\-load\-balancer\-ssl\-cert): "arn:aws:acm:us-east-2:xxxxxxx:certificate/77a312cb-149e-47c3-b243-19e841a253c3"
                      [service.beta.kubernetes.io/aws-load-balancer-backend-protocol](http://service\.beta\.kubernetes\.io/aws\-load\-balancer\-backend\-protocol): "http"
                      [service.beta.kubernetes.io/aws-load-balancer-ssl-ports](http://service\.beta\.kubernetes\.io/aws\-load\-balancer\-ssl\-ports): "https"
                      [service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout](http://service\.beta\.kubernetes\.io/aws\-load\-balancer\-connection\-idle\-timeout): "3600"
Alan Kis avatar
Alan Kis

Now it makes more sense

Eric D. Berg avatar
Eric D. Berg

I’m glad i remembered that, because i found pretty much zero on my first few searches. thanks for looking, Alan.

1
RB avatar

How does everyone here create golden amis with toggles ? such as if you want instance X to use AMI1 with datadog and instance Y to use AMI1 without datadog, you wouldn’t build a whole new AMI, you’d have some kind of flag or feature toggle, right?

Would love to here thoughts on this. I’m wondering if we can do something with SSM or tagging on instances to use as toggles.

David avatar
David

Does Cognito integrate at all into Cloud Map / Service Discovery? I’m moving some ECS fargate service that are behind an ALB (with Cognito auth) to use Service Discovery, but am unsure how to force beyondcorp auth

2020-07-26

Chris Fowles avatar
Chris Fowles

i’ve always found the limit of 10 attached policies to a role to be a real limiting factor in well designed granular iam policies - does anyone have any patterns or techniques that they use to get around this?

i really want to be able to have policies that are “Allow-This-Specific-Thing” rather than “Allow-These-20-Things-That-This-Specific-Service-Needs”, as I think it’s a much easier pattern to reason with

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/terraform-aws-iam-policy-document-aggregator

Terraform module to aggregate multiple IAM policy documents into single policy document. - cloudposse/terraform-aws-iam-policy-document-aggregator

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

you’ve seen this?

Chris Fowles avatar
Chris Fowles

have now

Chris Fowles avatar
Chris Fowles

that’s not a bad workaround

Zach avatar

You can attach a lot more (unlimited?) inline policies if that helps

Jonathan Le avatar
Jonathan Le

That aggregator + the IAM Principal Tag have been really really great for me in reducing IAM pain.

"Condition": {
                "StringEquals": {
                    "ec2:ResourceTag/CostCenter": "${aws:PrincipalTag/CostCenter}"
                }
            }
Jonathan Le avatar
Jonathan Le

Def check out the blog post on aws:PrincipalTag is you haven’t gotten into it yet; TLDR: let say you have an IAM role or User with a tag on it with the k,v being “ec2:ResourceTag/Team_ID” where Team_ID = “Frontend”

in the IAM condition above attached to AWS Resources, the IAM role with the tag “Frontend” can only access that resource if it also has the Tag “Frontend”.

We use it sorta like this in our nonProd Envs: if a user with the Tag for Team_ID == “Data Engineering”, they can access lots of stuff they need to where the AWS Resource has also been tagged with “Data Engineering”.

^^ I’m 2 years late to the game on aws:PrincipalTag - wish I knew about it sooner.

2020-07-24

Abel Luck avatar
Abel Luck

Also.. is is strange that there exists no Cloudwatch metrics for creating Alarms for AWS Config and AWS State Manager failed associations?

rajeshb avatar
rajeshb

I have used State Manager RUN Command and notifications i have used are RUN Command -> SNS -> grafana CW source Actual Document RUN -> logs to loki

2020-07-23

maarten avatar
maarten

I know UI/UX adoption is a thing, but the new Route53 interface… Feels like Plesk.

1
Issif avatar
Issif

I agree 100%

Issif avatar
Issif

all new AWS interfaces are dumb, slow, you can’t sort columns

:--1:1
Issif avatar
Issif

giant icons (like for tablet interface ????)

Issif avatar
Issif

@maarten Have you tried to change a record? I wanted to change a A (Alias) for a CNAME? I needed 2min to figure out I had to delete the value to have the option to change the record type

maarten avatar
maarten

No I went straight here to nag about it, it would be better if they drop the whole UI completely now.

Issif avatar
Issif

vFondevilla avatar
vFondevilla

I agree. I hate that interface. Almost 3 minutes for creating a new record.

Zach avatar

My favorite part is that when you drop into a Hosted Zone, they for some reason don’t set the left-right scroll bar for the records all the way to the left, so you can’t actually see any of the record information until you scroll it over. It only renders correctly if you’re in full screen mode

loren avatar
loren

oh yeah, and i thought the old interface was bad and unintuitive! the new one is waaaay worse! so much clicking!

Zach avatar

there’s a new route53 UI?

github140 avatar
github140

We are asked to only use API.

sheldonh avatar
sheldonh

AWS is great at many things. UI experience is not one of them. AWS ssm is so slow to improve. You can filter by the presence of a tag but not the value last I checked. Seriously.

Abel Luck avatar
Abel Luck

Anyone have a solution for sending AWS Config noncompliant alerts to prometheus?

2020-07-22

rajeshb avatar
rajeshb

Does anyone know if i can CloudWatch event pattern for SNS in same account. i want to come up with a rule, SNS event activity to trigger RUN Command. Thanks.

2020-07-21

Henry Carter avatar
Henry Carter

Does anyone know of a tool to export a report (csv,json, etc) on IAM Users? Something with fields such as Name/Username/Group/Console Signin/Last Creds Use/MFA Status etc…

Issif avatar
Issif

you can do that with aws-cli + jq or awk

Henry Carter avatar
Henry Carter

get-account-authorization-details gets most of it but I’d still have to write something to get the rest. I can’t be the first person to do this so hoped there might be an existing tool

Santiago Campuzano avatar
Santiago Campuzano

@Henry Carter There is an easier way:

Santiago Campuzano avatar
Santiago Campuzano

IAM Credential Report

Issif avatar
Issif

nice tip, thanks

Henry Carter avatar
Henry Carter

Nice, I’d just need to add each users groups to that :–1:

2020-07-20

RB avatar

recently stuck cloudtrail logs into s3 and made them searchable using athena. I’ve found some good queries online by googling them, seeing aws docs, cloudonaut, etc. Any additional queries you folks have found ? We’re doing a hack week at my company and I’d love to add more juicy queries

loren avatar
loren

one of our teammates has been gathering queries here: https://github.com/easttimor/aws-incident-response

easttimor/aws-incident-response

Contribute to easttimor/aws-incident-response development by creating an account on GitHub.

party_parrot2
2
PePe avatar

we used too, it is awesome

sheldonh avatar
sheldonh

I built a poshbot interaction I was proud of. Poshbot (chatops Powershell bot) could run Athena query that was setup via terraform and return formatted tables or excel files

cool-doge1
RB avatar

what are your athena cli tools of choice? im experimenting with this at the moment : https://github.com/dbcli/athenacli

dbcli/athenacli

AthenaCLI is a CLI tool for AWS Athena service that can do auto-completion and syntax highlighting. - dbcli/athenacli

2020-07-19

sheldonh avatar
sheldonh

Is there any framework that would help me add a front end, basic approval , and lambda functions to make a simple “self service portal”?

I’ve moved away from chatops for Microsoft Teams and instead was thinking of simple utilities like

  • lookup a row in database based on fuzzy match
  • turn a test environment off
  • turn a test environment on
  • request a backup of a database, but since it’s prod wait for the manual approval from specific set of users

I’m currently doing some of this in Azure Pipelines but it’s not really a “runbook” tool. If I can’t find a better solution I’ll stick with azure Pipelines but wanted to explore a nice “self-service” option if there was one that I could keep entirely private in my VPC. I just don’t want to reinvent and build all the pieces, esp to hopefully leveraging IAM auth for certain actions.

One last option I was considering, was maybe building a simple go CLI interface. Been wanting to do that anyway. Have a couple options in there, and that would solve using Iam auth. I could just wrap up call in the lambda functions. Any approval workflow would be in step functions.

I guess the last option as much as I’m not a big fan of them could be to use AWS ssm automation docs more. Many of the devs don’t have access to the web console so that would be another thing to figure out I guess. SSM automation docs are a bit annoying to troubleshoot too.

Any ideas?

Zach avatar

Heh, well I know we’ve discussed it already but you’re basically describing Rundeck (for all its warts)

loren avatar
loren

Sounds like AWS amplify to me

sheldonh avatar
sheldonh

Ha!

sheldonh avatar
sheldonh

I got ya. I surrender.

sheldonh avatar
sheldonh

Kinda was hoping for a serverless/step function interface but nothing out there like that from what I see.

sheldonh avatar
sheldonh

Better for now to stick with azure Pipelines I guess. Takes 50 secs to run a task instead of 5-10 secs but worrying about that is probably premature optimization for all the benefit of starting to add some basic automation anyway.

I wish azure Pipelines had an output view that would immediately come up instead of navigating into sub pages and it wouldn’t be so confusing

sheldonh avatar
sheldonh

I even tried adding Microsoft forms tab to channel …but flow required premium subscription and it didn’t support properly sending runtime parameters

It’s all a plot against me

Zach avatar

I hear you, I’ve been trying to find alternatives to rundeck for quite awhile

sheldonh avatar
sheldonh

If i was sticking with slack it would be much easier. Teams is much harder for integrations.

bradym avatar
bradym

coughrundeckcough

Zach avatar

Jeebuz yes. My company recently forced us off slack and onto Teams because thats what the ‘corporate side’ was using. It’s been awful, do not recommend.

2
sheldonh avatar
sheldonh

I’m kinda biased against rundeck because my first impression was lots of whitespace, overlappy busy UI and not quickly intuitive to run Powershell core container workflows. Not saying it can’t but I’m outta time for anything complicated and was hoping for some quick wins. Don’t think I’m going to get that with rundeck right now and going to limp along with azure pipeline yaml based jobs and use that for now. Will help me raise more awareness of it as well for simple automation.

I just need to figure out templates better and I think DRY will be adequate too.

Zach avatar

I was just doing a POC of Azure DevOps over the weekend and I thought the templating was nice, but it also had a lot of verbosity just to use it

sheldonh avatar
sheldonh

Compared to some options it feels super succinct. Don’t forget each level of deploy>job>steps can be ignored to start simple. Can literally run a few lines with just a pool + steps of bash/pwsh and done. It’s only more verbose when you need to add more strategies on it

bradym avatar
bradym

What’s the ux for someone new to it running a pipeline? Is that something you’re worried about?

Zach avatar

The Azure UI was quite nice I thought It feels weird because we run in AWS but are now considering Azure DevOps for build/deploy

sheldonh avatar
sheldonh

The ux for running a pipeline is fine. I meant the UX overall in rundeck made it feel overall busy and complicated vs me just building a yaml pipeline in azure pipelines.

Zach avatar

Yah rundeck UI is garbage

sheldonh avatar
sheldonh

I’m pretty picky on UI experience, i know it’s not the most important thing, but I like any tools I use to provide a nice polish when possible

bradym avatar
bradym

I only ask because while rundeck’s UI isn’t great, it’s really easy for someone who doesn’t know anything about rundeck to login and run a job. Since you mentioned there are a variety of people who will be using this thing (and I have zero experience with azure) I thought it worth asking.

loren avatar
loren

i’ve never before heard anyone say azure devops had a simple interface lol, i can never find anything without at least 20 minutes signing out and back in (trying to find which account owns the resource) and poking through at least a dozen screens

loren avatar
loren

i mean, is this configuration hidden in a page, a tab, or behind an obscure triple-dot context menu? who knows! it’s a mystery!

loren avatar
loren

i literally dread trying to troubleshoot the single azure pipeline we have. we only keep it for the free windows ci. just biding time to cut it over to github actions, or travis-ci, when/if their windows support comes far enough along

Zach avatar

Hmm. I had very little issue standing up three Go projects on it on Sunday afternoon. Only thing tricky was its secrets mapping for our private modules

sheldonh avatar
sheldonh

Sorry to hear that Loren. I’ll tell you having used both GitHub actions and azure Pipelines I think that azure Pipelines if you give it a shot is more intuitive, robust, and feature filled . If you get stuck on your build be more than happy to see what’s up with it.

It can be confusing at first but it’s a best in class in my opinion if you can get past some quirks

2020-07-18

Châu Anh Tuấn avatar
Châu Anh Tuấn

Hi Everyone.

1

2020-07-17

Maciek Strömich avatar
Maciek Strömich
Introducing the Cloud Development Kit for Terraform (Preview) | Amazon Web Services attachment image

Infrastructure as Code (IaC) is a fundamental component of modern DevOps practices because it enables you to deploy any version of your application infrastructure at will, and facilitates the full lifecycle management of all the resources required to run and monitor your application. Organizations who have adopted DevOps practices often deploy hundreds or even thousands […]

2020-07-16

sheldonh avatar
sheldonh

Any happy users of rundeck right now? Thinking of trying it out after waffling between using SSM Docs, Azure Pipelines, and other tools. If I could run in fargate I might be able to demo the benefits and get some buy it, but only want to explore further if it’s really good to help promote more “self-service” activity from teams/devs.

Say:

• runbook — > add me to allow light for specfic group

• Run powershell command to using dbatools and restore a sql server database

• Backup a copy of a database - pause for approval - continue and output the results to teams Etc. Seems like it’s the only common platform for this of it’s type. Is it something I’ll regret or has it made your life better? Not a big fan of ssm docs as they are clunky, hard to stream output and use overall, so not really self-service.

Zach avatar


Is it something I’ll regret or has it made your life better?
yes.

Zach avatar

We use it, it works. Its not my favorite tool, its a little quirky and it doesn’t see a lot of improvement. They only 1 price tier which is enterprise, and SSO is hidden behind that.

Zach avatar

Jobs produce ‘logs’ and not really “output” so its a little annoying if you need to do something like provision something and return some value to the user, they have to dig into the logs to find it.

Zach avatar

You can only ACL down to the job itself, nothing within it. ie you can’t restrict certain options from being used by a particular user/group.

The ‘dynamic options’ are poorly documented but are actually pretty powerful once we figured them out. You can point it at a url that returns a specific json structure and rundeck will extract option values from it. This lets you run a lambda behind an API gateway for example and have it lookup complex things for the job. And the dynamic options can ‘stack’ on each other, so that as you select option 1, the values of option 2 can be dynamically computed.

PePe avatar

we use it and we are deprecating it

Zach avatar

Ooohhh in favor of what

PePe avatar

moving to containers

PePe avatar

ansible does not make much sense for us anymore

PePe avatar

nor any other orchestration tool

Zach avatar

Ah I see. We’re using it for not just ‘deployment’ type stuff although that’s the primary use

PePe avatar

we do not use AMIs anymore, or a few but they are fading away

bradym avatar
bradym

We use it, and while I acknowledge there are issues I like it. I do wish there was some competition to push the rundeck team forward a bit.

I get the sense that it may work better for small teams (like mine) than it does for larger groups.

sheldonh avatar
sheldonh

dang.

So what’s an option for a containered (so I could ideally put into fargate) runbook tool so I could plugin in some powershell scripts , python etc… and basically try to get folks away from tickets being done manually?

I want to create a library of simple tasks that folks keep doing manually. I can use azure pipelines, but it’s not really meant for that, so feels like a hack rather than a nice “runbook” to go and use.

bradym avatar
bradym

I’ve not used it myself, but https://stackstorm.com/ is on my list of things to explore if I ever get a chance.

StackStorm - StackStorm

StackStorm connects all your apps, services, and workflows. Why StackStorm? Get Started Open source and trusted by the enterprise Robust Automation Engine From simple if/then rules to complicated workflows, StackStorm lets you automate DevOps your way. See More Features… Integrates with your Existing Infrastructure No need to change your existing processes or workflows, StackStorm connects…

Zach avatar

I’ve looked at StackStorm. It is a hell of a lot more complicated stack than rundeck, and I found it VERY difficult to get the same sort of “self service push button” jobs.

bradym avatar
bradym

Oof, that’s a shame.

Zach avatar

It’s aimed at being an ‘If this then that’ of operations

sheldonh avatar
sheldonh

I think relay.sh is going to be freaking epic. It’s just too early of access I think for me, though maybe it’s what i should focus on

sheldonh avatar
sheldonh

Anything else? I need a free tier, ideally self-hosted so I don’t have to buy anything and “click button” to run something. I need the target to be devs, qa, and some business folks eventually. RIght now just a couple devops team folks

Zach avatar

I think rundeck is the closest you get

Zach avatar

free, on-prem is why we’re using it too

PePe avatar

Spinaker ?

bradym avatar
bradym

One thing to mention is that while their official SSO is enterprise only, it is possible to use oauth2 with rundeck if you also deploy https://github.com/oauth2-proxy/oauth2-proxy - we’re doing that with gitlab as our auth source since everyone who needs access will also have gitlab access.

oauth2-proxy/oauth2-proxy

A reverse proxy that provides authentication with Google, Github or other providers. - oauth2-proxy/oauth2-proxy

PePe avatar
Spinnaker

Global Continuous Delivery

Zach avatar

I have our rundeck hooked into Okta via the Okta LDAP Connector

bradym avatar
bradym

nice

Zach avatar

I saw the oauth proxy but was hoping to just not have to run 1 more thing

bradym avatar
bradym

Believe me, I hear that

sheldonh avatar
sheldonh

So rundeck still for running various powershell/python aws things, or should I just stick with Azure Pipelines (which isn’t quite what I was hoping for)

Zach avatar

@sheldonh yes it can do those

sheldonh avatar
sheldonh

Tried running docker version and having issues with default credentials to test it out locally. Hoping i could use prebuilt images with ecs task

Zach avatar

our rundeck basically runs a lot of shell/python stuff

bradym avatar
bradym

^ that’s what we use it for too. mostly shell and a handful of python.

sheldonh avatar
sheldonh

Better than azure pipelines/gitlab etc?

The other last option I can see is spin up a self-hosted gitlab (which might meet with some challenges), and use their interactive runbook feature?

Not exactly a self-service thing but possibly a compromise for at least the devs to run stuff

Zach avatar

haven’t ever used Azure so no opinion there

Zach avatar

Hmm no price details on Relay.sh?

bradym avatar
bradym

I haven’t used azure either, but I’m in the middle of moving from github + circleci to gitlab. I would not recommend gitlabci as a general “run this thing” tool.

It could be used that way, but you’d need to have a deep understanding of the pipeline definitions in order to know what variables should be passed in to run the jobs you want to run.

sheldonh avatar
sheldonh

relay.sh is in early access. They told me they will have a free tier regardless. They are working out pricing after they see how things go with users /pipelines etc.

sheldonh avatar
sheldonh

I’m familar with yaml pipelines in azure devops and could pick that up in gitlab pretty quickly I feel.

it’s more about what is the right “interface” experience for making it easy. Azure Pipelines is easy, just feels like a clunky way to use it. Relay is promising but very beta, have to give it access keys and not going to be a quick sell for me at this time

sheldonh avatar
sheldonh

I just got docker rundeck running. Maybe I’ll look at that first

bradym avatar
bradym

Oh, I have no doubt you could pick up ci gitlab stuff. But if you’re looking for something where others can run jobs themselves, those people would also need a deep understanding of the gitlab pipelines.

Running a pipleline is simple, you just choose a branch, enter some variables (if you need) and hit run.

But knowing what variables are available, what jobs need what variables, etc… that’s where you need to understand the pipelines.

sheldonh avatar
sheldonh

got you

sheldonh avatar
sheldonh

So rundeck or azure pipelines i guess is my main option

sheldonh avatar
sheldonh

the UI for rundeck has so much wasted space

Zach avatar

wait until you run a long job and want to view the logs >_<

sheldonh avatar
sheldonh

that’s kinda why i was hoping to use this vs azure pipelines. i think i’m back to just using pipelines for a quick win cause at least i have it and it runs powershell without issues. Even the yaml is preferrable to this workflow builder. Ugggh

bradym avatar
bradym

I mean if you really want yaml… you can declare a job in yaml and upload the definition to rundeck. You can also set it up to sync with git repos

bradym avatar
bradym

If you’re feeling masochistic, you can even use xml.

Zach avatar

Oh yah the rundeck interface for building jobs is AWFUL

Zach avatar

you have to hit save in every section, and then at the end. annoying as heck

Zach avatar

We found that it has a terraform provider and have been defining jobs that way

bradym avatar
bradym

woah, that’s a cool idea

sheldonh avatar
sheldonh

nice. I think for now azure pipelines will be the hacky way to provide this, as much as I prefer something designed for it, seems like nothing fits the bill, so might as well stick with something that quick to get running

Zach avatar

it also let us get around some issues we had with how it would mangle shell scripts in its own interface

sheldonh avatar
sheldonh

Octopus has free tier with 10 targets included. I’m running mostly just container stuff, so their runbook stuff seems interesting too. Anyone used that?

sheldonh avatar
sheldonh
05:31:28 PM

On the subject of yaml… Not sure this gives me immediate confidence in using that

sheldonh avatar
sheldonh

Can I host rundock in ECS and have it just run docker commands for each task?

Not sure if it’s easy to setup like that as Azure Pipelines has their “hosted” pool and I can pick any ubuntu, mac, windows and go.

I might explore rundeck again in the future if I can easily have it run docker based actions.

Zach avatar

Pretty sure you have to pay $ for Enterprise to get that

Zach avatar

At least I think, their docs aren’t really clear on that part

bradym avatar
bradym

There are docker and k8s plugins for rundeck, so yes you can run docker commands. Of course you can also just run docker commands without the plugins as well.

Also, a quick google turns up a third-party ecs plugin: https://github.com/oyama/rundeck-awsecs-plugin

oyama/rundeck-awsecs-plugin

Rundeck Amazon EC2 Container Service Task execution plugin - oyama/rundeck-awsecs-plugin

Zach avatar

I thought the problem was that it relies on Rundeck Clustering which Enterprise only

bradym avatar
bradym

Which part relies on rundeck clustering?

Zach avatar

External nodes like that ecs plugin I thought

bradym avatar
bradym

Nope, I’ve not used those plugins but I’ve definitely used external nodes with rundeck oss.

Zach avatar
Welp ok. I’ll chalk that up their documentation not being very clear then =
bradym avatar
bradym

Totally understandable. Clustering is only needed if you want to have multiple rundeck instances, but you can have any number of nodes where jobs are executed without clustering.

sheldonh avatar
sheldonh

will take a look, thanks!

sheldonh avatar
sheldonh

Got my interest when I read:
Clutch ships as a single binary that contains the frontend and backend, making it trivial to deploy. Many changes can be achieved via configuration rather than recompiling a new binary.

sheldonh avatar
sheldonh

Even more when I saw
Clutch is powered by a Go backend and React frontend.

sheldonh avatar
sheldonh


This component-and-workflow architecture allows a developer with limited frontend experience to replace clunky tooling or command-line scripts with a clear and easy-to-use step-by-step UI in under an hour of development time>.</span

Zach avatar

I question that 1 hour based on their “amiibo” example walkthrough

sheldonh avatar
sheldonh
 docker run --rm -p 8080:8080 -it lyft/clutch:latest
sheldonh avatar
sheldonh

So far, that’s the first tool like this that run the first time without any issue.

Zach avatar

well I’d be curious to see what you do with it, their examples are pretty bare bones

sheldonh avatar
sheldonh

Probably not much right now. No time. Not enough plug and play. But still really cool for when I someday over the rainbow get the time

Luis avatar

Hi everyone!

Issif avatar
Issif

Hi

2020-07-15

2020-07-14

Daniel Pilch avatar
Daniel Pilch

Hi all, I’m looking for a solution to root MFA storage for an enterprise environment where the teams that manage each account are regionally split. Due to the pandemic everyone is now remote first and we would like to have a solution whereby certain members have access to MFA remotely and then a DR emergency backup physical universal 2nd factor device. Does anyone have any experience with a solution like this?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We’ve settled on 1Password for business or enterprise. It supports geofencing and audit trails. It’s one of the few solutions for shared and team managed TOTP.

sweetops1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Other solutions of have seen are private slack channels and totp bots.

Daniel Pilch avatar
Daniel Pilch

Great thanks Erik

sheldonh avatar
sheldonh

I’m looking at using serverless framework to deploy lambda functions with api-gateway and integrating into Microsoft Teams with messaging extensions to try and provide some chatops type solution.

I’m newer to API Gateways and the looking for a 101 on what makes it secure to call from Teams with authentication since I’ll have it do some specific admin functions.

Anyone wrote up something on this or willing to dive into more in a thread or even jump on a zoom call to chat for a few on this? I’m done with using a standard chatbot in Teams as it’s all message queue based, with no activity indicators or anything. I figure Lambda functions with API Gateways will be a better way to go if I can get my head around this.

loren avatar
loren

aws sam, or amplify, or serverless.com make the api gateway and lambda integration really pretty easy

loren avatar
loren

just worked a week-long prototype using aws amplify and came away super impressed… some rough edges, but made it very easy to build out and manage the whole solution… ‘course, if you don’t have a frontend website, it’s not a perfect fit, but it is usable

sheldonh avatar
sheldonh

i’m looking for just api calls so I can have Teams extensions interact with my AWS stuff for chatops “2.0” style interactions

sheldonh avatar
sheldonh

Serverless framework seems great, so I’m starting up on that, but the api gateway stuff is newer to me so hoping someone can give me a quick 101 or dummy version of how it’s secure for this. And yes, I can google it, just been spending too much effort on all the pieces so a quick start on how it’s secure with a token or whatever would be useful to make sure I’m not going down the wrong road

Maycon Santos avatar
Maycon Santos

Hi all, is anybody running ECS with autoscaling and capacity providers?

Maycon Santos avatar
Maycon Santos

I am testing it now, that there is some cloudformation support, but they way it scale in seems to be very unstable as I always end up with at least one idle server in the cluster

Jonathan Marcus avatar
Jonathan Marcus

There’s CF support? I’ve tried and found it very lacking. I had to augment our CF runs with some manual calls to the CLI after the fact, and it was really gross.

Maycon Santos avatar
Maycon Santos

still missing support to autoscaling group name instead of Arn and if you use Managed termination, ASG lacks support for protecting instance from scale in

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Introducing AWS Copilot | Amazon Web Services attachment image

The first official command line tool for Amazon Elastic Container Service (Amazon ECS) was created in 2015. In December 2019, we shared the preview release of a new command line experience, redesigned from the ground up to make it easier for you to deploy your applications on Amazon ECS. Today, we are sharing some of […]

Gowiem avatar
Gowiem

Yeah — Pretty cool to bootstrap a small project! We were chatting about this when they first announced over here: https://sweetops.slack.com/archives/CCT1E7JJY/p1594310815461900

quick reminder that the AWS Cloud Container Conference (C3) is going on now from 9am to 6pm PST

https://www.twitch.tv/aws

https://aws.amazon.com/blogs/containers/the-inaugural-aws-cloud-containers-conference/

:--1:1
maarten avatar
maarten

Cool.. But.. why don’t they come up with a Heroku-like Paas instead I wonder? A type of Elastic Beanstalk done well.

1

2020-07-13

PePe avatar

Anyone familiar with API gateway? is it possible to have public API gateway endpoint that only allow certain aws accounts/roles to access and deny everyone else ?

PePe avatar

this suggest is possible buy it does not work

Steven avatar
Steven

If it is a private API gateway it should work. But for a public one, you’ll need to deal with IP whitelisting

Steven avatar
Steven

I haven’t looked at this specifically for API gateway (may need to soon), but it is part of AWS design

PePe avatar

ohhh so it is a public endpoint you can’t do account/role filtering?

Steven avatar
Steven

Not that I’m aware of. I’ve always done IP whitelisting with Security Groups or WAF. Since it is public there is no AWS login information

Steven avatar
Steven

Some services like S3 are an exception to this

Steven avatar
Steven

But that is because it is an AWS public service. By default all your private traffic goes to the internet to access S3.

loren avatar
loren

if you lean towards private api gateway, read this first: https://st-g.de/2019/07/be-careful-with-aws-private-api-gateway-endpoints

Be careful with AWS Private API Gateway Endpoints attachment image

Don’t kill your production systems!

loren avatar
loren

your other option would be some kind of custom authorizer on the public api gateway… https://aws.amazon.com/blogs/compute/introducing-custom-authorizers-in-amazon-api-gateway/

Introducing custom authorizers in Amazon API Gateway | Amazon Web Services attachment image

Today Amazon API Gateway is launching custom request authorizers. With custom request authorizers, developers can authorize their APIs using bearer token authorization strategies, such as OAuth using an AWS Lambda function. For each incoming request, API Gateway verifies whether a custom authorizer is configured, and if so, API Gateway calls the Lambda function with the […]

PePe avatar

so is a bad idea to use the private api gateway?

PePe avatar

this is an internal api endpoint, it should never been accessible from other accounts

loren avatar
loren

that’s not the primary consideration. it’s whether you have other api gateway endpoints that you want to reach from any account where you create a private api gateway endpoint
But in this case, the setting Enable Private DNS Name, which I would certainly want to activate, will prevent all access to any regional/public API Gateway in the whole AWS region!

loren avatar
loren

there are workarounds mentioned in the article also, if you don’t see any way forward but to use a private api gateway

PePe avatar

well so we tried today and I call this api gateway policies a bunch of lies

PePe avatar

no matter what, you need connectivity to hit the endpoint so you can’t use policies base on account ids on a public endpoint

PePe avatar

and the whole idea of all this was so we could avoid doing vpc peering and filter by account id

loren avatar
loren

i’ve not tried the resource policy approach with api gateway. if it seems to not be working, i’d recommend opening a support ticket

2020-07-10

Henry Carter avatar
Henry Carter

I’ve been looking for a nice way to separate environment/account config from cfn templates, so we can ensure stacks are identically configured between accounts. I’ve used terraform previously, but my new team are cfn based. Does anyone have any experience using Sceptre for deploying their stacks as it seems to do what I want (and also have the bonus of handling assume-role with mfa prompt)

Alan Kis avatar
Alan Kis

What do you mean by environment/account? Sceptre seems much like an overkill.

How are you deploying resource across account/regions using CFN? If using CodePipeline, the pipeline can easily do cross-account/cross-region deployments.

https://docs.aws.amazon.com/codepipeline/latest/userguide/action-reference-CloudFormation.html (TemplateConfiguration).

AWS CloudFormation - AWS CodePipeline

Executes an operation on an AWS CloudFormation stack. A stack is a collection of AWS resources that you can manage as a single unit. The resources in a stack are defined by the stack’s AWS CloudFormation template. A change set creates a comparison that can be viewed without altering the original stack. For information about the types of AWS CloudFormation actions that can be performed on stacks and change sets, see the

btai avatar

anyone here use cloudflare + global accelerator? do they play nicely together?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

global accelerator provides you with static IPs on one side, and connects to ALB, NLB, EIP or EC2 instance on the other side. It should play nicely with any DNS providers, if you are talking about creating A record in CloudFlare and pointing it to the static IP

btai avatar

will our TCP traffic be routed immediately to global accelerator edge server or through cloudflare edge servers (possibly less ideal?) then hit a global accelerator node way later down the line

btai avatar

curious if someones had the same question and if its even a valid concern , if not i can report back once i get it set up

2020-07-09

RB avatar

quick reminder that the AWS Cloud Container Conference (C3) is going on now from 9am to 6pm PST

https://www.twitch.tv/aws

https://aws.amazon.com/blogs/containers/the-inaugural-aws-cloud-containers-conference/

Save the date: AWS Cloud Containers Conference | Amazon Web Services attachment image

The AWS Containers team is hosting the inaugural AWS Cloud Containers Conference (C3), a fully virtual event, solely covering Containers at AWS. The initial event will take place on July 9th from 9:00am – 6:00pm PDT and will be a full day of technical deep dives, demos, and workshops. Attendees can look forward to learning […]

PePe avatar

is that a new tech ? “Cintainers”

Save the date: AWS Cloud Containers Conference | Amazon Web Services attachment image

The AWS Containers team is hosting the inaugural AWS Cloud Containers Conference (C3), a fully virtual event, solely covering Containers at AWS. The initial event will take place on July 9th from 9:00am – 6:00pm PDT and will be a full day of technical deep dives, demos, and workshops. Attendees can look forward to learning […]

PePe avatar

please do not tell me I need to learn something new now….

RB avatar

lol typo my bad

PePe avatar

lol

RB avatar

there is such a thing as a Sintainer. it’s a container that’s been bad

RB avatar

like a container that runs as root user

PePe avatar

hahahahahahahahahahah took me a few second to translate that one

PePe avatar

but isn’r that a problem of the Sinadmin that set the Sintainer in the first place?

RB avatar

You’re thinking of a Badmin

RB avatar

when he’s hacked he becomes a Sadmin

RB avatar

anyone following this thread will want to kill me lol

PePe avatar

I think we are the only too people enjoying the bad humour LOL

PePe avatar

we are going to get kicked out from the @contributors channel

1
imiltchman avatar
imiltchman

anything interesting? any new announcements?

RB avatar

currently going over fargate 1.4 version

Gowiem avatar
Gowiem

@RB well done.

1
RB avatar

containerd is replacing docker. wow.

RB avatar

im sure there will be more performance gains using docker’s core containerd, so thats nice. hoping for cheaper prices soon.

Joe Niland avatar
Joe Niland
Introducing AWS Copilot | Amazon Web Services attachment image

The first official command line tool for Amazon Elastic Container Service (Amazon ECS) was created in 2015. In December 2019, we shared the preview release of a new command line experience, redesigned from the ground up to make it easier for you to deploy your applications on Amazon ECS. Today, we are sharing some of […]

Joe Niland avatar
Joe Niland

They’ve kind of made something like serverless-cli for ECS

Gowiem avatar
Gowiem

@Joe Niland Huh ecs-cli v2… Don’t know why they gave it a new name. But sadly, I personally am already not a fan —
But later when it’s time to create this architecture, there are a lot more cloud components to think about: VPC subnets, load balancers, deployment pipelines, and durable storage for your application’s stateful data. Copilot handles all these underlying details for you.
This is what bugged me about ecs-cli and is why I stayed away from it… I don’t want AWS to abstract all that stuff away from me because it makes it harder when I need further access to those pieces.

Joe Niland avatar
Joe Niland

Yeah know what you mean. I’m thinking about who this cli targeted at exactly. Also without plugins / hooks it’s only going to work with pretty common scenarios.

Gowiem avatar
Gowiem

It does look like it does a lot of heavy lifting after looking at the full post. I would possibly use this for small side projects that I couldn’t just host on Amplify but beyond that I think I’d steer clear.

Joe Niland avatar
Joe Niland

Yes I think it’s pretty cool for Dev/test. But then you need to build out the Terraform modules for production.

Joe Niland avatar
Joe Niland

I’ve got an app to move off elastic beanstalk. I think I’ll give this a try!

Gowiem avatar
Gowiem

Let us know how it goes!

Joe Niland avatar
Joe Niland

Sure. So this thing is super similar to Serverless framework. It’s generating CloudFormation and you can extend it by adding your own CloudFormation.

Gowiem avatar
Gowiem

Ah so you’re allowed to extend the CloudFormation. That’s something at least.

:--1:1
Gowiem avatar
Gowiem

I figured it was using CF under the hood, but figured they wouldn’t expose that.

Gowiem avatar
Gowiem

Seems AWS + Docker worked together on updating compose with the same ecs-cli capability: https://www.docker.com/blog/from-docker-straight-to-aws/

From Docker Straight to AWS - Docker Blog

Just about six years ago to the day Docker hit the first milestone for Docker Compose, a simple way to layout your containers and their connections. A talks to B, B talks to C, and C is a database. Fast forward six years and the container ecosystem has become complex. New managed container services have arrived bringing their own runtime environments, CLIs, and configuration languages. This complexity serves the needs of the operations teams who require fine grained control, but carries a high price for developers.

Joe Niland avatar
Joe Niland

the plot thickens!

Gowiem avatar
Gowiem

Haha yeah… and I find that to be a bigger deal. There is work to make the compose file specification a broader specification so that you can use it to deploy services. And it seems like Docker and ECS have already started on that, which is cool…. If this was a pluggable architecture it would be awesome to build a terraform plugin so you could deploy a compose file with generated terraform.

Gowiem avatar
Gowiem

This is what I’m referring to with the compose spec stuff. Super interesting as this could vastly simplify various cloud setups similar to the way that compose simplified local development setups.

https://www.docker.com/blog/announcing-the-compose-specification/

Announcing the Compose Specification - Docker Blog attachment image

Docker is pleased to announce that we have created a new open community to develop the Compose Specification.

1
imiltchman avatar
imiltchman

Does anyone know how reliable Route53 geolocation is? Is it any worse than [email protected]?

jafow avatar
jafow

we use it and it is mostly reliable — we have < 5% requests that end up routing to the incorrect region

:--1:1
jafow avatar
jafow

I do not know how it compares or relates to edge lambda, i have only used that for simple slicing on the request URI

jafow avatar
jafow


we have < 5% requests that end up routing to the incorrect region
so depending on your tolerance for this and the regions you’re deploying in it’s probably good. It’s actually not good enough for us and we’ve written our own routing in order to pin the region — FWIW I notice it’s much flakier in me-east-1 (bahrain) and Asia. very solid in the US regions

vFondevilla avatar
vFondevilla

We’re using it in the africa region (with our resources deployed in eu-west-1) and it’s working awesomely.

maarten avatar
maarten

@imiltchman what’s your issue with [email protected]

imiltchman avatar
imiltchman

No issue with [email protected], just asking which one is better

2020-07-08

RB avatar

probably a dumb question but what are the cons of running a fargate container as root user instead of a non root user?

RB avatar

thread

Santiago Campuzano avatar
Santiago Campuzano

The cons are mostly security cons… if the Docker image/container has some sort of vulnerability, it could escalate privileges in the EC2 instance running the container

Santiago Campuzano avatar
Santiago Campuzano

It could become a Rogue container

Santiago Campuzano avatar
Santiago Campuzano

@RB

RB avatar

there is no ec2 instance as it is fargate

RB avatar

i mean, sure there is an ec2 but its managed by amazon. im sure their firecracker encapsulation of my fargate docker container will prevent issues if i run inside my docker container as root, no?

RB avatar

@Santiago Campuzano what do you think?

Santiago Campuzano avatar
Santiago Campuzano

Right…. it’s Fargate… so Im pretty sure AWS is gonna make sure that your Container won’t do any harm outside

Santiago Campuzano avatar
Santiago Campuzano

So I’d say go ahead …. no worries

party_parrot1
Maciek Strömich avatar
Maciek Strömich

Except giving potential attackers easy means to install whatever shit they need to become part of a botnet, send email spam, used as a jump box to carry other attacks or whatever. If you’re not replacing your containers (e.g when running a legacy system that requires few to no upgrades a year) this potentially means persistent presence in your network especially if you’re not monitoring what processes are running on the systems you’re managing.

1
RB avatar

That’s a very good point.

Marcin Brański (Cloud Posse) avatar
Marcin Brański (Cloud Posse)

I think short answer to your question would be that it would violate “least privilege” principle/best practice.

RB avatar

ya, but i wanted the long answer. i wanted to see what could possibly go wrong if my fargate container was using a root user to run its code instead of a non root user.

Alan Kis avatar
Alan Kis

Not probably the long answer.

There aren’t any security implication as host encapsulation is preventing privileges escalation. Last but not least, good and encouraged practice, following the principle of the least privilege should be do not run Docker containers as root. And finally, Fargate doesn’t support privileged containers.

RB avatar

yes and duly noted. correct me if I’m wrong.

the original post wasn’t completely clear. my question was exclusively about root user inside the docker container whereas privileged containers is running docker daemon container itself using a root user.

RB avatar

or in other words, you can run a non privileged container using fargate with the root user used inside of the container.

Alan Kis avatar
Alan Kis

Yes, you can run a non-privileged container using Fargate, but as it isn’t a privileged container, and thus completely isolated from the root user on the host machine.

Sometimes, I am having an impression about the root users inside Docker, similar to other taboo themes, everyone is talking about it, no one is practicing.

Again, after tasks that need a root, I would specify the runtime user in Dockerfile.

Maciek Strömich avatar
Maciek Strömich

@ you https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html#container_definition_linuxparameters it changed a little bit with agent version 1.4.0 you can give to Fargate certain linux capabilities

Task definition parameters - Amazon Elastic Container Service

Task definitions are split into separate parts: the task family, the IAM task role, the network mode, container definitions, volumes, task placement constraints, and launch types. The family and container definitions are required in a task definition, while task role, network mode, volumes, task placement constraints, and launch type are optional.

Maciek Strömich avatar
Maciek Strömich

ah for fargate it’s only SYS_PTRACE

1
1

2020-07-07

2020-07-06

michaeljaweed avatar
michaeljaweed

So I recently started with amazon I’m in the noob phase. I’m struggling with terraform and assigning the certificate to my site. When I use cloudposse or acm cert module, it doesn’t load my web application at all. I was able to assign it without using cloudposse module or the other acm terraform module but when I do it it still doesn’t give me the secured up top as I would like. If anyone can help me I’ll greatly appreciate it

imiltchman avatar
imiltchman

Do you have a Route53 host zone setup?

imiltchman avatar
imiltchman

For a domain that you own

michaeljaweed avatar
michaeljaweed

Yeah I do

RB avatar

whats a good way to compare ecs ec2 to ecs fargate cost ?

RB avatar

currently we have a lot of services on shared clusters. the services are tagged appropriately but im afraid this cost doesnt include the ec2 itself.

RB avatar

makes it a bit challenging to compare apples to apples

Zach avatar


im afraid this cost doesnt include the ec2 itself.
The ECS cost is only the EC2 instance itself though

Zach avatar

You aren’t paying anything for the containers running on the cluster

Zach avatar

comparing ECS EC2 to Fargate though is a pain in the ass. You have to figure out your avg/max vCPU/memory needs and then compute your hourly/monthly costs for Fargate off of that

RB avatar

ah yes, so ive been taking a crack at this today. i did exactly that. i compared fargate to ec2 costs. it’s too bad we can’t easily do it in the cost explorer due to shared clusters. unsure what the benefits are of shared clusters…

RB avatar

so we took a single task definition, carved out what the least amount of a shared cluster ec2 that it would use, calculated the cost for that, and then compared it to fargate

RB avatar
RB
08:28:48 PM

here is the sheet for fun. i did not include RIs. if you can poke holes in it, plz let me know.

Zach avatar

You’re showing a savings of moving the task to fargate? That seems suspicious

RB avatar

ya, it does kind of seem like the numbers are wrong lol

RB avatar

but what you dont notice, is that if we didnt rightsize, we would not see a savings to fargate.

RB avatar

the final columns are rightsized with a task count of 2 and a task count of 1 respectively

RB avatar

they had to be rightsized because fargate does not support 32 vcpus which the orig TD requested lol

Zach avatar

well you shouldn’t compare your base cluster size directly to fargate

Zach avatar

you need to take each individual service task mem/cpu requirements and map that instead

Zach avatar

but like as an example, I have go services that run in t3 nanos on ec2. Those cost $8.39 each (including an 8GB EBS) per month if we don’t burn over the cpu credits. Those are 2 cpu and 500mb which doesn’t translate to fargate very well, but this particular service could get by on 0.5-1 cpu I think. Plugging that into fargate, that costs $18 for 0.5cpu/1gb or $32 for 1cpu/1gb using on demand pricing. So its a huge increase and thats why my management doesn’t want to do it

RB avatar


well you shouldn’t compare your base cluster size directly to fargate
ah of course not. that is why i have the ec2 10% as that is the usage of the original task definition. best way to compare, i thought, was to compare fargate maxed out vs ec2 10% costs as those are the most comparable without rightsizing

RB avatar


you need to take each individual service task mem/cpu requirements and map that instead
yes, that is what i’v edone with the 10%. its just not obvious enough i suppose.

Zach avatar

ok, I didn’t know what some of the terms you were using represented. In general t hough, if you’re seeing a savings on fargate you probably have a math error

Zach avatar

there might be a weird edge case where somehow you win, but I’ve never seen anyone show an example of it

RB avatar

we’re on an old ec2-shared instance type too. we’re on m3.2xlarges which are pretty old

RB avatar

if we made the switch to m5, the cost savings might be lower. ill have to triple check the numbers again tho. im a bit worried i made some mistake.

RB avatar

the 2nd to last column, if we compare the 10% ec2 usage vs fargate (non spot, with count=2), we have a cost increase which makes sense

Zach avatar

you should get a price decrease just going from m3 to m5

RB avatar

We definitely would see a decrease. But easier said than done. Lots of tasks run on these. Still were looking at a hybrid approach where we migrate some to fargate and leave the others to migrate to m5

Andreas P avatar
Andreas P

@RB This video explains the pricing differences quite well https://www.youtube.com/watch?v=HoXEyXIf6_U

2020-07-05

Gowiem avatar
Gowiem
08:55:15 PM

Hey for any ECS users here — Any 3rd party tools that you use with that service that you recommend / swear by? I’m giving a talk on ECS this coming week and I’m including a list of recommended tools. Looking for more to fill out the list.

Santiago Campuzano avatar
Santiago Campuzano

docker-compose

RB avatar

This tool that we discussed previously has made our deployment process a lot easier

https://github.com/fabfuel/ecs-deploy

fabfuel/ecs-deploy

Powerful CLI tool to simplify Amazon ECS deployments, rollbacks & scaling - fabfuel/ecs-deploy

RB avatar

Using that you can update a single containers image while retaining the other side cars and all the other configuration.

RB avatar

Oh lol just saw that you already included that

1
Gowiem avatar
Gowiem

@RB Yeah, including that. I had heard of it before but your link in our thread made me look into it again.

Santiago Campuzano avatar
Santiago Campuzano

Docker-compose is the perfect companion when working with ECS

RB avatar

I’ve been using it. It’s great for updating fargate containers without having to update the datadog and fluentbit containers too

:--1:1
Santiago Campuzano avatar
Santiago Campuzano

You can re-use a docker-compose file for deploying to other envs

RB avatar

Ya docker compose is nice but almost a standard now, right?

Gowiem avatar
Gowiem

@Santiago Campuzano Looking for ECS specific tools. Compose is more general purpose across the entire docker ecosystem.

RB avatar

That’s like saying, don’t forget to use git

Edit: sorry i wasnt trying to sound mean! Maybe it’s worth mentioning

1
RB avatar

Are you including your talk or will your talk stream be available?

Santiago Campuzano avatar
Santiago Campuzano
nathanpeck/awesome-ecs

A curated list of awesome ECS guides, development tools, and resources - nathanpeck/awesome-ecs

party_parrot1
RB avatar

You could also mention some cloudposse ecs modules like the alb ingress, alb service task, and the ecs web app ones. That makes it a lot easier.

Gowiem avatar
Gowiem

@RB Talk is here — Will be live streamed / up on Youtube after it’s done. https://www.meetup.com/AWSMeetupGroup/events/270751202/

AWS Elastic Container Service 101 (LIVE-STREAM) attachment image

Wed, Jul 8, 2020, 6:00 PM: LIVE-STREAM: https://youtu.be/pOvV0FypJA0Come learn the basic and not-so-basic details of the AWS Container Orchestration platform: Elastic Container Service (ECS). In this

:--1:1
Gowiem avatar
Gowiem

Ah yeah, I’ll include awesome-ecs — Good call, thanks @Santiago Campuzano

Santiago Campuzano avatar
Santiago Campuzano

RB avatar

Is cost of ec2 vs fargate also a topic?

RB avatar

Also do you know of a tool that will search running containers in a cluster that will return task definitions and then allow you to search those definitions? I’m trying to avoid stringing together a number of aws commands and having to write something… But i think i will have to whip out boto

Gowiem avatar
Gowiem

I briefly touch on pricing in the Launch types slide, but nothing more than a quick example.

Gowiem avatar
Gowiem

Don’t know of a tool that will do that, sorry dude!

np1
Igor Bronovskyi avatar
Igor Bronovskyi

ecs-deploy use in production

Santiago Campuzano avatar
Santiago Campuzano
Introducing AWS Copilot | Amazon Web Services attachment image

The first official command line tool for Amazon Elastic Container Service (Amazon ECS) was created in 2015. In December 2019, we shared the preview release of a new command line experience, redesigned from the ground up to make it easier for you to deploy your applications on Amazon ECS. Today, we are sharing some of […]

:--1:1
Gowiem avatar
Gowiem

@Santiago Campuzano Yeah, we were discussing that over here: https://sweetops.slack.com/archives/CCT1E7JJY/p1594310815461900

quick reminder that the AWS Cloud Container Conference (C3) is going on now from 9am to 6pm PST

https://www.twitch.tv/aws

https://aws.amazon.com/blogs/containers/the-inaugural-aws-cloud-containers-conference/

:--1:2
Santiago Campuzano avatar
Santiago Campuzano

Oh nice !

2020-07-04

Andreas P avatar
Andreas P

Hey guys sorry in advance if I am asking something stupid but I am not very experienced with infrastructure development Can someone describe at very high level how to achieve having an EC2 accessible only through Open VPN in a AWS setup? Currently I have a VPC with a private/public subnet and an EC2 instance which runs open vpn in that public subnet. I can connect fine to the VPN but now I am trying to create a new EC2 instance in the private subnet which is only accessible through the vpn. Any pointers to achieva that? Thank you in advance!

viliam.pucik avatar
viliam.pucik

@ Have you tried adding a second ENI from the private subnet to the public OpenVPN Server EC2? That second ENI could be used as a default gateway for the private subnet as well, to allow connection flow only through OpenVPN Server.

Andreas P avatar
Andreas P

@viliam.pucik Thanks for the response! Just tried now but it seems that is not working either… Maybe something is off in my setup.

viliam.pucik avatar
viliam.pucik

@ Hmm, have you enabled routing between the public and private network on the OpenVPN Server (echo 1 > /proc/sys/net/ipv4/ip_forward)?

Maciek Strömich avatar
Maciek Strömich

The simplest approach is to configure security group of the ec2 instance to allow access only from vpn instance and do a NAT on vpn instance.

With regards forwarding of the packets one thing is the ip_forward and there’s an ec2 source destination check normally set to true which needs to be disabled.

Andreas P avatar
Andreas P

It worked thank you guys! Truly appreciate it !

Andreas P avatar
Andreas P

I had to enable forwarding as @viliam.pucik suggested

:--1:1

2020-07-03

2020-07-02

Joseph Ashwin Kottapurath avatar
Joseph Ashwin Kottapurath

hey everyone, does anyone know a solution for this: https://stackoverflow.com/questions/56259431/persistent-storage-on-elastic-beanstalk-docker-container I have tried searching everywhere it’s almost as if there’s no direct solution to this

Persistent storage on Elastic Beanstalk Docker Container

I’m looking for some way to mount a S3 Storage Bucket (for example) as Docker volume in an Elastic Beanstalk Docker Container. Since i can’t change the application code, i have to configure that p…

Shannon Dunn avatar
Shannon Dunn

I need iam role policy help..

I want to be able to force at least one security group on RunInstances, but then the ability to also select whatever other security group… I was thinking like this, but looks like this is treating as an OR and would want an AND… any ideas on how would accomplish something like this?

        {
            "Sid": "RunInstancePermissionsWithSG",
            "Effect": "Allow",
            "Action": "ec2:RunInstances",
            "Resource": [
                "arn:aws-us-gov:ec2:us-east-1:*****:security-group/sg-1234567889",
                 ### Would want this to be an AND not an OR
                "arn:aws-us-gov:ec2:us-east-1:*****:security-group/*"
            ]
        },
slc avatar

wouldn’t an AND of these two things be the same as using just the first one? not sure what the wildcard gets you on the second resource line.

Shannon Dunn avatar
Shannon Dunn

Yeah. I was hoping it would be allow only if this named security group AND any other security group.

Shannon Dunn avatar
Shannon Dunn

But the role can just use any security group with this policy

Shannon Dunn avatar
Shannon Dunn

:(

RB avatar

Id create a support ticket if no one here can help. I’m interested on how this policy can be configured for this to work

2020-07-01

    keyboard_arrow_up