#aws (2020-07)
Discussion related to Amazon Web Services (AWS)
Discussion related to Amazon Web Services (AWS)
Archive: https://archive.sweetops.com/aws/
2020-07-01
2020-07-02

hey everyone, does anyone know a solution for this: https://stackoverflow.com/questions/56259431/persistent-storage-on-elastic-beanstalk-docker-container I have tried searching everywhere it’s almost as if there’s no direct solution to this
I’m looking for some way to mount a S3 Storage Bucket (for example) as Docker volume in an Elastic Beanstalk Docker Container. Since i can’t change the application code, i have to configure that p…

I need iam role policy help..
I want to be able to force at least one security group on RunInstances, but then the ability to also select whatever other security group… I was thinking like this, but looks like this is treating as an OR and would want an AND… any ideas on how would accomplish something like this?
{
"Sid": "RunInstancePermissionsWithSG",
"Effect": "Allow",
"Action": "ec2:RunInstances",
"Resource": [
"arn:aws-us-gov:ec2:us-east-1:*****:security-group/sg-1234567889",
### Would want this to be an AND not an OR
"arn:aws-us-gov:ec2:us-east-1:*****:security-group/*"
]
},

wouldn’t an AND of these two things be the same as using just the first one? not sure what the wildcard gets you on the second resource line.

Yeah. I was hoping it would be allow only if this named security group AND any other security group.

But the role can just use any security group with this policy

:(

Id create a support ticket if no one here can help. I’m interested on how this policy can be configured for this to work
2020-07-03
2020-07-04

Hey guys sorry in advance if I am asking something stupid but I am not very experienced with infrastructure development Can someone describe at very high level how to achieve having an EC2 accessible only through Open VPN in a AWS setup? Currently I have a VPC with a private/public subnet and an EC2 instance which runs open vpn in that public subnet. I can connect fine to the VPN but now I am trying to create a new EC2 instance in the private subnet which is only accessible through the vpn. Any pointers to achieva that? Thank you in advance!

@Andreas P Have you tried adding a second ENI from the private subnet to the public OpenVPN Server EC2? That second ENI could be used as a default gateway for the private subnet as well, to allow connection flow only through OpenVPN Server.

@viliam.pucik Thanks for the response! Just tried now but it seems that is not working either… Maybe something is off in my setup.

@Andreas P Hmm, have you enabled routing between the public and private network on the OpenVPN Server (echo 1 > /proc/sys/net/ipv4/ip_forward
)?

The simplest approach is to configure security group of the ec2 instance to allow access only from vpn instance and do a NAT on vpn instance.
With regards forwarding of the packets one thing is the ip_forward and there’s an ec2 source destination check normally set to true which needs to be disabled.

It worked thank you guys! Truly appreciate it !

2020-07-05

Hey for any ECS users here — Any 3rd party tools that you use with that service that you recommend / swear by? I’m giving a talk on ECS this coming week and I’m including a list of recommended tools. Looking for more to fill out the list.

docker-compose

This tool that we discussed previously has made our deployment process a lot easier
Powerful CLI tool to simplify Amazon ECS deployments, rollbacks & scaling - fabfuel/ecs-deploy

Using that you can update a single containers image while retaining the other side cars and all the other configuration.


@RB Yeah, including that. I had heard of it before but your link in our thread made me look into it again.

Docker-compose is the perfect companion when working with ECS

I’ve been using it. It’s great for updating fargate containers without having to update the datadog and fluentbit containers too

You can re-use a docker-compose file for deploying to other envs

Ya docker compose is nice but almost a standard now, right?

@Santiago Campuzano Looking for ECS specific tools. Compose is more general purpose across the entire docker ecosystem.

That’s like saying, don’t forget to use git
Edit: sorry i wasnt trying to sound mean! Maybe it’s worth mentioning

Are you including your talk or will your talk stream be available?

A curated list of awesome ECS guides, development tools, and resources - nathanpeck/awesome-ecs


You could also mention some cloudposse ecs modules like the alb ingress, alb service task, and the ecs web app ones. That makes it a lot easier.

@RB Talk is here — Will be live streamed / up on Youtube after it’s done. https://www.meetup.com/AWSMeetupGroup/events/270751202/

Wed, Jul 8, 2020, 6:00 PM: LIVE-STREAM: https://youtu.be/pOvV0FypJA0Come learn the basic and not-so-basic details of the AWS Container Orchestration platform: Elastic Container Service (ECS). In this

Ah yeah, I’ll include awesome-ecs — Good call, thanks @Santiago Campuzano


Is cost of ec2 vs fargate also a topic?

Also do you know of a tool that will search running containers in a cluster that will return task definitions and then allow you to search those definitions? I’m trying to avoid stringing together a number of aws commands and having to write something… But i think i will have to whip out boto

I briefly touch on pricing in the Launch types slide, but nothing more than a quick example.


ecs-deploy use in production


The first official command line tool for Amazon Elastic Container Service (Amazon ECS) was created in 2015. In December 2019, we shared the preview release of a new command line experience, redesigned from the ground up to make it easier for you to deploy your applications on Amazon ECS. Today, we are sharing some of […]

@Santiago Campuzano Yeah, we were discussing that over here: https://sweetops.slack.com/archives/CCT1E7JJY/p1594310815461900
quick reminder that the AWS Cloud Container Conference (C3) is going on now from 9am to 6pm PST
https://aws.amazon.com/blogs/containers/the-inaugural-aws-cloud-containers-conference/

Oh nice !
2020-07-06

So I recently started with amazon I’m in the noob phase. I’m struggling with terraform and assigning the certificate to my site. When I use cloudposse or acm cert module, it doesn’t load my web application at all. I was able to assign it without using cloudposse module or the other acm terraform module but when I do it it still doesn’t give me the secured up top as I would like. If anyone can help me I’ll greatly appreciate it

Do you have a Route53 host zone setup?

For a domain that you own

Yeah I do

whats a good way to compare ecs ec2 to ecs fargate cost ?

currently we have a lot of services on shared clusters. the services are tagged appropriately but im afraid this cost doesnt include the ec2 itself.

makes it a bit challenging to compare apples to apples

im afraid this cost doesnt include the ec2 itself.
The ECS cost is only the EC2 instance itself though

You aren’t paying anything for the containers running on the cluster

comparing ECS EC2 to Fargate though is a pain in the ass. You have to figure out your avg/max vCPU/memory needs and then compute your hourly/monthly costs for Fargate off of that

ah yes, so ive been taking a crack at this today. i did exactly that. i compared fargate to ec2 costs. it’s too bad we can’t easily do it in the cost explorer due to shared clusters. unsure what the benefits are of shared clusters…

so we took a single task definition, carved out what the least amount of a shared cluster ec2 that it would use, calculated the cost for that, and then compared it to fargate

here is the sheet for fun. i did not include RIs. if you can poke holes in it, plz let me know.

You’re showing a savings of moving the task to fargate? That seems suspicious

ya, it does kind of seem like the numbers are wrong lol

but what you dont notice, is that if we didnt rightsize, we would not see a savings to fargate.

the final columns are rightsized with a task count of 2 and a task count of 1 respectively

they had to be rightsized because fargate does not support 32 vcpus which the orig TD requested lol

well you shouldn’t compare your base cluster size directly to fargate

you need to take each individual service task mem/cpu requirements and map that instead

but like as an example, I have go services that run in t3 nanos on ec2. Those cost $8.39 each (including an 8GB EBS) per month if we don’t burn over the cpu credits. Those are 2 cpu and 500mb which doesn’t translate to fargate very well, but this particular service could get by on 0.5-1 cpu I think. Plugging that into fargate, that costs $18 for 0.5cpu/1gb or $32 for 1cpu/1gb using on demand pricing. So its a huge increase and thats why my management doesn’t want to do it

well you shouldn’t compare your base cluster size directly to fargate
ah of course not. that is why i have the ec2 10% as that is the usage of the original task definition. best way to compare, i thought, was to compare fargate maxed out vs ec2 10% costs as those are the most comparable without rightsizing

you need to take each individual service task mem/cpu requirements and map that instead
yes, that is what i’v edone with the 10%. its just not obvious enough i suppose.

ok, I didn’t know what some of the terms you were using represented. In general t hough, if you’re seeing a savings on fargate you probably have a math error

there might be a weird edge case where somehow you win, but I’ve never seen anyone show an example of it

we’re on an old ec2-shared instance type too. we’re on m3.2xlarges which are pretty old

if we made the switch to m5, the cost savings might be lower. ill have to triple check the numbers again tho. im a bit worried i made some mistake.

the 2nd to last column, if we compare the 10% ec2 usage vs fargate (non spot, with count=2), we have a cost increase which makes sense

you should get a price decrease just going from m3 to m5

We definitely would see a decrease. But easier said than done. Lots of tasks run on these. Still were looking at a hybrid approach where we migrate some to fargate and leave the others to migrate to m5

@RB This video explains the pricing differences quite well https://www.youtube.com/watch?v=HoXEyXIf6_U
2020-07-07
2020-07-08

probably a dumb question but what are the cons of running a fargate container as root user instead of a non root user?

thread

The cons are mostly security cons… if the Docker image/container has some sort of vulnerability, it could escalate privileges in the EC2 instance running the container

It could become a Rogue
container

@RB

there is no ec2 instance as it is fargate

i mean, sure there is an ec2 but its managed by amazon. im sure their firecracker encapsulation of my fargate docker container will prevent issues if i run inside my docker container as root, no?

@Santiago Campuzano what do you think?

Right…. it’s Fargate… so Im pretty sure AWS is gonna make sure that your Container won’t do any harm outside


Except giving potential attackers easy means to install whatever shit they need to become part of a botnet, send email spam, used as a jump box to carry other attacks or whatever. If you’re not replacing your containers (e.g when running a legacy system that requires few to no upgrades a year) this potentially means persistent presence in your network especially if you’re not monitoring what processes are running on the systems you’re managing.

That’s a very good point.

I think short answer to your question would be that it would violate “least privilege” principle/best practice.

ya, but i wanted the long answer. i wanted to see what could possibly go wrong if my fargate container was using a root user to run its code instead of a non root user.

Not probably the long answer.
There aren’t any security implication as host encapsulation is preventing privileges escalation. Last but not least, good and encouraged practice, following the principle of the least privilege should be do not run Docker containers as root. And finally, Fargate doesn’t support privileged containers.

yes and duly noted. correct me if I’m wrong.
the original post wasn’t completely clear. my question was exclusively about root user inside the docker container whereas privileged containers is running docker daemon container itself using a root user.

or in other words, you can run a non privileged container using fargate with the root user used inside of the container.

Yes, you can run a non-privileged container using Fargate, but as it isn’t a privileged container, and thus completely isolated from the root user on the host machine.
Sometimes, I am having an impression about the root users inside Docker, similar to other taboo themes, everyone is talking about it, no one is practicing.
Again, after tasks that need a root, I would specify the runtime user in Dockerfile.

@Alan Kis you https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html#container_definition_linuxparameters it changed a little bit with agent version 1.4.0 you can give to Fargate certain linux capabilities
Task definitions are split into separate parts: the task family, the IAM task role, the network mode, container definitions, volumes, task placement constraints, and launch types. The family and container definitions are required in a task definition, while task role, network mode, volumes, task placement constraints, and launch type are optional.

2020-07-09

quick reminder that the AWS Cloud Container Conference (C3) is going on now from 9am to 6pm PST
https://aws.amazon.com/blogs/containers/the-inaugural-aws-cloud-containers-conference/

The AWS Containers team is hosting the inaugural AWS Cloud Containers Conference (C3), a fully virtual event, solely covering Containers at AWS. The initial event will take place on July 9th from 9:00am – 6:00pm PDT and will be a full day of technical deep dives, demos, and workshops. Attendees can look forward to learning […]

is that a new tech ? “Cintainers”

The AWS Containers team is hosting the inaugural AWS Cloud Containers Conference (C3), a fully virtual event, solely covering Containers at AWS. The initial event will take place on July 9th from 9:00am – 6:00pm PDT and will be a full day of technical deep dives, demos, and workshops. Attendees can look forward to learning […]

please do not tell me I need to learn something new now….

lol typo my bad

lol

there is such a thing as a Sintainer. it’s a container that’s been bad

like a container that runs as root user

hahahahahahahahahahah took me a few second to translate that one

but isn’r that a problem of the Sinadmin that set the Sintainer in the first place?

You’re thinking of a Badmin

when he’s hacked he becomes a Sadmin

anyone following this thread will want to kill me lol

I think we are the only too people enjoying the bad humour LOL


anything interesting? any new announcements?

currently going over fargate 1.4 version


containerd is replacing docker. wow.

im sure there will be more performance gains using docker’s core containerd, so thats nice. hoping for cheaper prices soon.


The first official command line tool for Amazon Elastic Container Service (Amazon ECS) was created in 2015. In December 2019, we shared the preview release of a new command line experience, redesigned from the ground up to make it easier for you to deploy your applications on Amazon ECS. Today, we are sharing some of […]

They’ve kind of made something like serverless-cli for ECS

@Joe Niland Huh ecs-cli v2… Don’t know why they gave it a new name. But sadly, I personally am already not a fan —
But later when it’s time to create this architecture, there are a lot more cloud components to think about: VPC subnets, load balancers, deployment pipelines, and durable storage for your application’s stateful data. Copilot handles all these underlying details for you.
This is what bugged me about ecs-cli and is why I stayed away from it… I don’t want AWS to abstract all that stuff away from me because it makes it harder when I need further access to those pieces.

Yeah know what you mean. I’m thinking about who this cli targeted at exactly. Also without plugins / hooks it’s only going to work with pretty common scenarios.

It does look like it does a lot of heavy lifting after looking at the full post. I would possibly use this for small side projects that I couldn’t just host on Amplify but beyond that I think I’d steer clear.

Yes I think it’s pretty cool for Dev/test. But then you need to build out the Terraform modules for production.

I’ve got an app to move off elastic beanstalk. I think I’ll give this a try!

Let us know how it goes!

Sure. So this thing is super similar to Serverless framework. It’s generating CloudFormation and you can extend it by adding your own CloudFormation.


I figured it was using CF under the hood, but figured they wouldn’t expose that.

Seems AWS + Docker worked together on updating compose with the same ecs-cli capability: https://www.docker.com/blog/from-docker-straight-to-aws/
Just about six years ago to the day Docker hit the first milestone for Docker Compose, a simple way to layout your containers and their connections. A talks to B, B talks to C, and C is a database. Fast forward six years and the container ecosystem has become complex. New managed container services have arrived bringing their own runtime environments, CLIs, and configuration languages. This complexity serves the needs of the operations teams who require fine grained control, but carries a high price for developers.

the plot thickens!

Haha yeah… and I find that to be a bigger deal. There is work to make the compose file specification a broader specification so that you can use it to deploy services. And it seems like Docker and ECS have already started on that, which is cool…. If this was a pluggable architecture it would be awesome to build a terraform plugin so you could deploy a compose file with generated terraform.

This is what I’m referring to with the compose spec stuff. Super interesting as this could vastly simplify various cloud setups similar to the way that compose simplified local development setups.
https://www.docker.com/blog/announcing-the-compose-specification/

Docker is pleased to announce that we have created a new open community to develop the Compose Specification.

Does anyone know how reliable Route53 geolocation is? Is it any worse than Lambda@Edge?

we use it and it is mostly reliable — we have < 5% requests that end up routing to the incorrect region

I do not know how it compares or relates to edge lambda, i have only used that for simple slicing on the request URI

we have < 5% requests that end up routing to the incorrect region
so depending on your tolerance for this and the regions you’re deploying in it’s probably good. It’s actually not good enough for us and we’ve written our own routing in order to pin the region — FWIW I notice it’s much flakier in me-east-1 (bahrain) and Asia. very solid in the US regions

We’re using it in the africa region (with our resources deployed in eu-west-1) and it’s working awesomely.

@Igor what’s your issue with lambda@edge

No issue with lambda@edge, just asking which one is better
2020-07-10

I’ve been looking for a nice way to separate environment/account config from cfn templates, so we can ensure stacks are identically configured between accounts. I’ve used terraform previously, but my new team are cfn based. Does anyone have any experience using Sceptre for deploying their stacks as it seems to do what I want (and also have the bonus of handling assume-role with mfa prompt)


What do you mean by environment/account? Sceptre seems much like an overkill.
How are you deploying resource across account/regions using CFN? If using CodePipeline, the pipeline can easily do cross-account/cross-region deployments.
https://docs.aws.amazon.com/codepipeline/latest/userguide/action-reference-CloudFormation.html (TemplateConfiguration).
Executes an operation on an AWS CloudFormation stack. A stack is a collection of AWS resources that you can manage as a single unit. The resources in a stack are defined by the stack’s AWS CloudFormation template. A change set creates a comparison that can be viewed without altering the original stack. For information about the types of AWS CloudFormation actions that can be performed on stacks and change sets, see the

anyone here use cloudflare + global accelerator? do they play nicely together?

global accelerator provides you with static IPs on one side, and connects to ALB, NLB, EIP or EC2 instance on the other side. It should play nicely with any DNS providers, if you are talking about creating A
record in CloudFlare and pointing it to the static IP

will our TCP traffic be routed immediately to global accelerator edge server or through cloudflare edge servers (possibly less ideal?) then hit a global accelerator node way later down the line

curious if someones had the same question and if its even a valid concern , if not i can report back once i get it set up
2020-07-13

Anyone familiar with API gateway? is it possible to have public API gateway endpoint that only allow certain aws accounts/roles to access and deny everyone else ?

this suggest is possible buy it does not work

This page presents a few examples of typical use cases for API Gateway resource policies.

If it is a private API gateway it should work. But for a public one, you’ll need to deal with IP whitelisting

I haven’t looked at this specifically for API gateway (may need to soon), but it is part of AWS design

ohhh so it is a public endpoint you can’t do account/role filtering?

Not that I’m aware of. I’ve always done IP whitelisting with Security Groups or WAF. Since it is public there is no AWS login information

Some services like S3 are an exception to this

But that is because it is an AWS public service. By default all your private traffic goes to the internet to access S3.

if you lean towards private api gateway, read this first: https://st-g.de/2019/07/be-careful-with-aws-private-api-gateway-endpoints

your other option would be some kind of custom authorizer on the public api gateway… https://aws.amazon.com/blogs/compute/introducing-custom-authorizers-in-amazon-api-gateway/

Today Amazon API Gateway is launching custom request authorizers. With custom request authorizers, developers can authorize their APIs using bearer token authorization strategies, such as OAuth using an AWS Lambda function. For each incoming request, API Gateway verifies whether a custom authorizer is configured, and if so, API Gateway calls the Lambda function with the […]

so is a bad idea to use the private api gateway?

this is an internal api endpoint, it should never been accessible from other accounts

that’s not the primary consideration. it’s whether you have other api gateway endpoints that you want to reach from any account where you create a private api gateway endpoint
But in this case, the setting Enable Private DNS Name, which I would certainly want to activate, will prevent all access to any regional/public API Gateway in the whole AWS region!

there are workarounds mentioned in the article also, if you don’t see any way forward but to use a private api gateway

well so we tried today and I call this api gateway policies a bunch of lies

no matter what, you need connectivity to hit the endpoint so you can’t use policies base on account ids on a public endpoint

and the whole idea of all this was so we could avoid doing vpc peering and filter by account id

i’ve not tried the resource policy approach with api gateway. if it seems to not be working, i’d recommend opening a support ticket
2020-07-14

Hi all, I’m looking for a solution to root MFA storage for an enterprise environment where the teams that manage each account are regionally split. Due to the pandemic everyone is now remote first and we would like to have a solution whereby certain members have access to MFA remotely and then a DR emergency backup physical universal 2nd factor device. Does anyone have any experience with a solution like this?

We’ve settled on 1Password for business or enterprise. It supports geofencing and audit trails. It’s one of the few solutions for shared and team managed TOTP.


Other solutions of have seen are private slack channels and totp bots.

Great thanks Erik

I’m looking at using serverless framework to deploy lambda functions with api-gateway and integrating into Microsoft Teams with messaging extensions to try and provide some chatops type solution.
I’m newer to API Gateways and the looking for a 101 on what makes it secure to call from Teams with authentication since I’ll have it do some specific admin functions.
Anyone wrote up something on this or willing to dive into more in a thread or even jump on a zoom call to chat for a few on this? I’m done with using a standard chatbot in Teams as it’s all message queue based, with no activity indicators or anything. I figure Lambda functions with API Gateways will be a better way to go if I can get my head around this.

aws sam, or amplify, or serverless.com make the api gateway and lambda integration really pretty easy

just worked a week-long prototype using aws amplify and came away super impressed… some rough edges, but made it very easy to build out and manage the whole solution… ‘course, if you don’t have a frontend website, it’s not a perfect fit, but it is usable

i’m looking for just api calls so I can have Teams extensions interact with my AWS stuff for chatops “2.0” style interactions

Serverless framework seems great, so I’m starting up on that, but the api gateway stuff is newer to me so hoping someone can give me a quick 101 or dummy version of how it’s secure for this. And yes, I can google it, just been spending too much effort on all the pieces so a quick start on how it’s secure with a token or whatever would be useful to make sure I’m not going down the wrong road

Hi all, is anybody running ECS with autoscaling and capacity providers?

I am testing it now, that there is some cloudformation support, but they way it scale in seems to be very unstable as I always end up with at least one idle server in the cluster

There’s CF support? I’ve tried and found it very lacking. I had to augment our CF runs with some manual calls to the CLI after the fact, and it was really gross.

still missing support to autoscaling group name instead of Arn and if you use Managed termination, ASG lacks support for protecting instance from scale in


The first official command line tool for Amazon Elastic Container Service (Amazon ECS) was created in 2015. In December 2019, we shared the preview release of a new command line experience, redesigned from the ground up to make it easier for you to deploy your applications on Amazon ECS. Today, we are sharing some of […]

Yeah — Pretty cool to bootstrap a small project! We were chatting about this when they first announced over here: https://sweetops.slack.com/archives/CCT1E7JJY/p1594310815461900
quick reminder that the AWS Cloud Container Conference (C3) is going on now from 9am to 6pm PST
https://aws.amazon.com/blogs/containers/the-inaugural-aws-cloud-containers-conference/

Cool.. But.. why don’t they come up with a Heroku-like Paas instead I wonder? A type of Elastic Beanstalk done well.
2020-07-15
2020-07-16

Any happy users of rundeck right now? Thinking of trying it out after waffling between using SSM Docs, Azure Pipelines, and other tools. If I could run in fargate I might be able to demo the benefits and get some buy it, but only want to explore further if it’s really good to help promote more “self-service” activity from teams/devs.
Say:
• runbook — > add me to allow light for specfic group
• Run powershell command to using dbatools and restore a sql server database
• Backup a copy of a database - pause for approval - continue and output the results to teams Etc. Seems like it’s the only common platform for this of it’s type. Is it something I’ll regret or has it made your life better? Not a big fan of ssm docs as they are clunky, hard to stream output and use overall, so not really self-service.

Is it something I’ll regret or has it made your life better?
yes.

We use it, it works. Its not my favorite tool, its a little quirky and it doesn’t see a lot of improvement. They only 1 price tier which is enterprise, and SSO is hidden behind that.

Jobs produce ‘logs’ and not really “output” so its a little annoying if you need to do something like provision something and return some value to the user, they have to dig into the logs to find it.

You can only ACL down to the job itself, nothing within it. ie you can’t restrict certain options from being used by a particular user/group.
The ‘dynamic options’ are poorly documented but are actually pretty powerful once we figured them out. You can point it at a url that returns a specific json structure and rundeck will extract option values from it. This lets you run a lambda behind an API gateway for example and have it lookup complex things for the job. And the dynamic options can ‘stack’ on each other, so that as you select option 1, the values of option 2 can be dynamically computed.

we use it and we are deprecating it

Ooohhh in favor of what

moving to containers

ansible does not make much sense for us anymore

nor any other orchestration tool

Ah I see. We’re using it for not just ‘deployment’ type stuff although that’s the primary use

we do not use AMIs anymore, or a few but they are fading away

We use it, and while I acknowledge there are issues I like it. I do wish there was some competition to push the rundeck team forward a bit.
I get the sense that it may work better for small teams (like mine) than it does for larger groups.

dang.
So what’s an option for a containered (so I could ideally put into fargate) runbook tool so I could plugin in some powershell scripts , python etc… and basically try to get folks away from tickets being done manually?
I want to create a library of simple tasks that folks keep doing manually. I can use azure pipelines, but it’s not really meant for that, so feels like a hack rather than a nice “runbook” to go and use.

I’ve not used it myself, but https://stackstorm.com/ is on my list of things to explore if I ever get a chance.
StackStorm connects all your apps, services, and workflows. Why StackStorm? Get Started Open source and trusted by the enterprise Robust Automation Engine From simple if/then rules to complicated workflows, StackStorm lets you automate DevOps your way. See More Features… Integrates with your Existing Infrastructure No need to change your existing processes or workflows, StackStorm connects…

I’ve looked at StackStorm. It is a hell of a lot more complicated stack than rundeck, and I found it VERY difficult to get the same sort of “self service push button” jobs.

Oof, that’s a shame.

It’s aimed at being an ‘If this then that’ of operations

I think relay.sh is going to be freaking epic. It’s just too early of access I think for me, though maybe it’s what i should focus on

Anything else? I need a free tier, ideally self-hosted so I don’t have to buy anything and “click button” to run something. I need the target to be devs, qa, and some business folks eventually. RIght now just a couple devops team folks

I think rundeck is the closest you get

free, on-prem is why we’re using it too

Spinaker ?

One thing to mention is that while their official SSO is enterprise only, it is possible to use oauth2 with rundeck if you also deploy https://github.com/oauth2-proxy/oauth2-proxy - we’re doing that with gitlab as our auth source since everyone who needs access will also have gitlab access.
A reverse proxy that provides authentication with Google, Github or other providers. - oauth2-proxy/oauth2-proxy


I have our rundeck hooked into Okta via the Okta LDAP Connector

nice

I saw the oauth proxy but was hoping to just not have to run 1 more thing

Believe me, I hear that

So rundeck still for running various powershell/python aws things, or should I just stick with Azure Pipelines (which isn’t quite what I was hoping for)

@sheldonh yes it can do those

Tried running docker version and having issues with default credentials to test it out locally. Hoping i could use prebuilt images with ecs task

our rundeck basically runs a lot of shell/python stuff

^ that’s what we use it for too. mostly shell and a handful of python.

Better than azure pipelines/gitlab etc?
The other last option I can see is spin up a self-hosted gitlab (which might meet with some challenges), and use their interactive runbook feature?
Not exactly a self-service thing but possibly a compromise for at least the devs to run stuff

haven’t ever used Azure so no opinion there

Hmm no price details on Relay.sh?

I haven’t used azure either, but I’m in the middle of moving from github + circleci to gitlab. I would not recommend gitlabci as a general “run this thing” tool.
It could be used that way, but you’d need to have a deep understanding of the pipeline definitions in order to know what variables should be passed in to run the jobs you want to run.

relay.sh is in early access. They told me they will have a free tier regardless. They are working out pricing after they see how things go with users /pipelines etc.

I’m familar with yaml pipelines in azure devops and could pick that up in gitlab pretty quickly I feel.
it’s more about what is the right “interface” experience for making it easy. Azure Pipelines is easy, just feels like a clunky way to use it. Relay is promising but very beta, have to give it access keys and not going to be a quick sell for me at this time

I just got docker rundeck running. Maybe I’ll look at that first

Oh, I have no doubt you could pick up ci gitlab stuff. But if you’re looking for something where others can run jobs themselves, those people would also need a deep understanding of the gitlab pipelines.
Running a pipleline is simple, you just choose a branch, enter some variables (if you need) and hit run.
But knowing what variables are available, what jobs need what variables, etc… that’s where you need to understand the pipelines.

got you

So rundeck or azure pipelines i guess is my main option

the UI for rundeck has so much wasted space

wait until you run a long job and want to view the logs >_<

that’s kinda why i was hoping to use this vs azure pipelines. i think i’m back to just using pipelines for a quick win cause at least i have it and it runs powershell without issues. Even the yaml is preferrable to this workflow builder. Ugggh

I mean if you really want yaml… you can declare a job in yaml and upload the definition to rundeck. You can also set it up to sync with git repos

Updated February 25, 2011

If you’re feeling masochistic, you can even use xml.

Oh yah the rundeck interface for building jobs is AWFUL

you have to hit save in every section, and then at the end. annoying as heck

We found that it has a terraform provider and have been defining jobs that way

woah, that’s a cool idea

nice. I think for now azure pipelines will be the hacky way to provide this, as much as I prefer something designed for it, seems like nothing fits the bill, so might as well stick with something that quick to get running

it also let us get around some issues we had with how it would mangle shell scripts in its own interface

Octopus has free tier with 10 targets included. I’m running mostly just container stuff, so their runbook stuff seems interesting too. Anyone used that?

On the subject of yaml… Not sure this gives me immediate confidence in using that

Can I host rundock in ECS and have it just run docker commands for each task?
Not sure if it’s easy to setup like that as Azure Pipelines has their “hosted” pool and I can pick any ubuntu, mac, windows and go.
I might explore rundeck again in the future if I can easily have it run docker based actions.

Pretty sure you have to pay \(\)$ for Enterprise to get that

At least I think, their docs aren’t really clear on that part

There are docker and k8s plugins for rundeck, so yes you can run docker commands. Of course you can also just run docker commands without the plugins as well.
Also, a quick google turns up a third-party ecs plugin: https://github.com/oyama/rundeck-awsecs-plugin
Rundeck Amazon EC2 Container Service Task execution plugin - oyama/rundeck-awsecs-plugin

I thought the problem was that it relies on Rundeck Clustering which Enterprise only

Which part relies on rundeck clustering?

External nodes like that ecs plugin I thought

Nope, I’ve not used those plugins but I’ve definitely used external nodes with rundeck oss.

Welp ok. I’ll chalk that up their documentation not being very clear then = |

Totally understandable. Clustering is only needed if you want to have multiple rundeck instances, but you can have any number of nodes where jobs are executed without clustering.

@sheldonh just saw this linked and maybe it helps fill your need https://eng.lyft.com/announcing-clutch-the-open-source-platform-for-infrastructure-tooling-143d00de9713
By Daniel Hochman and Derek Schaller

will take a look, thanks!

Got my interest when I read:
Clutch ships as a single binary that contains the frontend and backend, making it trivial to deploy. Many changes can be achieved via configuration rather than recompiling a new binary.

Even more when I saw
Clutch is powered by a Go backend and React frontend.

This component-and-workflow architecture allows a developer with limited frontend experience to replace clunky tooling or command-line scripts with a clear and easy-to-use step-by-step UI in under an hour of development time.

I question that 1 hour based on their “amiibo” example walkthrough

docker run --rm -p 8080:8080 -it lyft/clutch:latest

So far, that’s the first tool like this that run the first time without any issue.

well I’d be curious to see what you do with it, their examples are pretty bare bones

Probably not much right now. No time. Not enough plug and play. But still really cool for when I someday over the rainbow get the time

2020-07-17


Infrastructure as Code (IaC) is a fundamental component of modern DevOps practices because it enables you to deploy any version of your application infrastructure at will, and facilitates the full lifecycle management of all the resources required to run and monitor your application. Organizations who have adopted DevOps practices often deploy hundreds or even thousands […]
2020-07-18

2020-07-19

Is there any framework that would help me add a front end, basic approval , and lambda functions to make a simple “self service portal”?
I’ve moved away from chatops for Microsoft Teams and instead was thinking of simple utilities like
- lookup a row in database based on fuzzy match
- turn a test environment off
- turn a test environment on
- request a backup of a database, but since it’s prod wait for the manual approval from specific set of users
I’m currently doing some of this in Azure Pipelines but it’s not really a “runbook” tool. If I can’t find a better solution I’ll stick with azure Pipelines but wanted to explore a nice “self-service” option if there was one that I could keep entirely private in my VPC. I just don’t want to reinvent and build all the pieces, esp to hopefully leveraging IAM auth for certain actions.
One last option I was considering, was maybe building a simple go CLI interface. Been wanting to do that anyway. Have a couple options in there, and that would solve using Iam auth. I could just wrap up call in the lambda functions. Any approval workflow would be in step functions.
I guess the last option as much as I’m not a big fan of them could be to use AWS ssm automation docs more. Many of the devs don’t have access to the web console so that would be another thing to figure out I guess. SSM automation docs are a bit annoying to troubleshoot too.
Any ideas?

Heh, well I know we’ve discussed it already but you’re basically describing Rundeck (for all its warts)

Sounds like AWS amplify to me

Ha!

I got ya. I surrender.

Kinda was hoping for a serverless/step function interface but nothing out there like that from what I see.

Better for now to stick with azure Pipelines I guess. Takes 50 secs to run a task instead of 5-10 secs but worrying about that is probably premature optimization for all the benefit of starting to add some basic automation anyway.
I wish azure Pipelines had an output view that would immediately come up instead of navigating into sub pages and it wouldn’t be so confusing

I even tried adding Microsoft forms tab to channel …but flow required premium subscription and it didn’t support properly sending runtime parameters
It’s all a plot against me

I hear you, I’ve been trying to find alternatives to rundeck for quite awhile

If i was sticking with slack it would be much easier. Teams is much harder for integrations.

coughrundeckcough

Jeebuz yes. My company recently forced us off slack and onto Teams because thats what the ‘corporate side’ was using. It’s been awful, do not recommend.

I’m kinda biased against rundeck because my first impression was lots of whitespace, overlappy busy UI and not quickly intuitive to run Powershell core container workflows. Not saying it can’t but I’m outta time for anything complicated and was hoping for some quick wins. Don’t think I’m going to get that with rundeck right now and going to limp along with azure pipeline yaml based jobs and use that for now. Will help me raise more awareness of it as well for simple automation.
I just need to figure out templates better and I think DRY will be adequate too.

I was just doing a POC of Azure DevOps over the weekend and I thought the templating was nice, but it also had a lot of verbosity just to use it

Compared to some options it feels super succinct. Don’t forget each level of deploy>job>steps can be ignored to start simple. Can literally run a few lines with just a pool + steps of bash/pwsh and done. It’s only more verbose when you need to add more strategies on it

What’s the ux for someone new to it running a pipeline? Is that something you’re worried about?

The Azure UI was quite nice I thought It feels weird because we run in AWS but are now considering Azure DevOps for build/deploy

The ux for running a pipeline is fine. I meant the UX overall in rundeck made it feel overall busy and complicated vs me just building a yaml pipeline in azure pipelines.

Yah rundeck UI is garbage

I’m pretty picky on UI experience, i know it’s not the most important thing, but I like any tools I use to provide a nice polish when possible

I only ask because while rundeck’s UI isn’t great, it’s really easy for someone who doesn’t know anything about rundeck to login and run a job. Since you mentioned there are a variety of people who will be using this thing (and I have zero experience with azure) I thought it worth asking.

i’ve never before heard anyone say azure devops had a simple interface lol, i can never find anything without at least 20 minutes signing out and back in (trying to find which account owns the resource) and poking through at least a dozen screens

i mean, is this configuration hidden in a page, a tab, or behind an obscure triple-dot context menu? who knows! it’s a mystery!

i literally dread trying to troubleshoot the single azure pipeline we have. we only keep it for the free windows ci. just biding time to cut it over to github actions, or travis-ci, when/if their windows support comes far enough along

Hmm. I had very little issue standing up three Go projects on it on Sunday afternoon. Only thing tricky was its secrets mapping for our private modules

Sorry to hear that Loren. I’ll tell you having used both GitHub actions and azure Pipelines I think that azure Pipelines if you give it a shot is more intuitive, robust, and feature filled . If you get stuck on your build be more than happy to see what’s up with it.
It can be confusing at first but it’s a best in class in my opinion if you can get past some quirks

Hi all, just checking if RunDeck is still a favourite 2 years on… https://sweetops.slack.com/archives/CCT1E7JJY/p1660211217368379
Looking for something to run custom devops scripts from. What do people tend to use ATM? Lamdba/Rundeck/Airflow/ArgoWorkflows? We use AWS with EKS so a k8s solution is an option.

Still using it and haven’t had any reason to look for anything else. Not sure if it qualifies as a “favorite”.

I couldn’t find this yesterday, but it just popped up again on hackernews. If I wasn’t already using rundeck I’d be playing around with this to see if it’d be a good option.

I decided Rundeck was way too complex for my needs. Maybe great for someone going all in. I ended up using AWS SSM Automation runbooks in AWS, and heard good things about Azure notebooks too.
I still think Relay.sh has a lot of promise but not sure it’s been adopted heavily. It’s dockerized workflows like IFTTT for DevOps work. If you are cross cloud then maybe makes sense. Otherwise I’d lean towards using what’s on the platform even if it’s less polished

Windmill is pretty cool! I love the concept. Open standard with lots of building blocks already out there.
The problem is that this solving is that middle ground most don’t seem to know they need.
Nocode/zapier == great, but very limiting to an engineer needing more backend/cloud stuff. Pure lambda/runbooks lots of proprietary snafus I had to work through with AWS Automation runbooks and difficult to debug/develop in general.
We should just all build more Go CLI tools that do everything we need and call it a day

2020-07-20

recently stuck cloudtrail logs into s3 and made them searchable using athena. I’ve found some good queries online by googling them, seeing aws docs, cloudonaut, etc. Any additional queries you folks have found ? We’re doing a hack week at my company and I’d love to add more juicy queries

one of our teammates has been gathering queries here: https://github.com/easttimor/aws-incident-response
Contribute to easttimor/aws-incident-response development by creating an account on GitHub.


we used too, it is awesome

I built a poshbot interaction I was proud of. Poshbot (chatops Powershell bot) could run Athena query that was setup via terraform and return formatted tables or excel files


what are your athena cli tools of choice? im experimenting with this at the moment : https://github.com/dbcli/athenacli
AthenaCLI is a CLI tool for AWS Athena service that can do auto-completion and syntax highlighting. - dbcli/athenacli
2020-07-21

Does anyone know of a tool to export a report (csv,json, etc) on IAM Users? Something with fields such as Name/Username/Group/Console Signin/Last Creds Use/MFA Status etc…

you can do that with aws-cli + jq or awk

get-account-authorization-details
gets most of it but I’d still have to write something to get the rest. I can’t be the first person to do this so hoped there might be an existing tool

@Henry Carter There is an easier way:


IAM Credential Report

nice tip, thanks

Nice, I’d just need to add each users groups to that
2020-07-22

Does anyone know if i can CloudWatch event pattern for SNS in same account. i want to come up with a rule, SNS event activity to trigger RUN Command. Thanks.
2020-07-23

I know UI/UX adoption is a thing, but the new Route53 interface… Feels like Plesk.

I agree 100%


giant icons (like for tablet interface ????)

@maarten Have you tried to change a record? I wanted to change a A (Alias) for a CNAME? I needed 2min to figure out I had to delete the value to have the option to change the record type

No I went straight here to nag about it, it would be better if they drop the whole UI completely now.


I agree. I hate that interface. Almost 3 minutes for creating a new record.

My favorite part is that when you drop into a Hosted Zone, they for some reason don’t set the left-right scroll bar for the records all the way to the left, so you can’t actually see any of the record information until you scroll it over. It only renders correctly if you’re in full screen mode

oh yeah, and i thought the old interface was bad and unintuitive! the new one is waaaay worse! so much clicking!

there’s a new route53 UI?

We are asked to only use API.

AWS is great at many things. UI experience is not one of them. AWS ssm is so slow to improve. You can filter by the presence of a tag but not the value last I checked. Seriously.

Anyone have a solution for sending AWS Config noncompliant alerts to prometheus?
2020-07-24

Also.. is is strange that there exists no Cloudwatch metrics for creating Alarms for AWS Config and AWS State Manager failed associations?

I have used State Manager RUN Command and notifications i have used are RUN Command -> SNS -> grafana CW source Actual Document RUN -> logs to loki
2020-07-26

i’ve always found the limit of 10 attached policies to a role to be a real limiting factor in well designed granular iam policies - does anyone have any patterns or techniques that they use to get around this?
i really want to be able to have policies that are “Allow-This-Specific-Thing” rather than “Allow-These-20-Things-That-This-Specific-Service-Needs”, as I think it’s a much easier pattern to reason with

Terraform module to aggregate multiple IAM policy documents into single policy document. - cloudposse/terraform-aws-iam-policy-document-aggregator

you’ve seen this?

have now

that’s not a bad workaround

You can attach a lot more (unlimited?) inline policies if that helps

That aggregator + the IAM Principal Tag have been really really great for me in reducing IAM pain.
"Condition": {
"StringEquals": {
"ec2:ResourceTag/CostCenter": "${aws:PrincipalTag/CostCenter}"
}
}

Def check out the blog post on aws:PrincipalTag is you haven’t gotten into it yet; TLDR: let say you have an IAM role or User with a tag on it with the k,v being “ec2:ResourceTag/Team_ID” where Team_ID = “Frontend”
in the IAM condition above attached to AWS Resources, the IAM role with the tag “Frontend” can only access that resource if it also has the Tag “Frontend”.
We use it sorta like this in our nonProd Envs: if a user with the Tag for Team_ID == “Data Engineering”, they can access lots of stuff they need to where the AWS Resource has also been tagged with “Data Engineering”.
^^ I’m 2 years late to the game on aws:PrincipalTag - wish I knew about it sooner.
2020-07-27

Hello all, Might be a basic query. is the Resource block Mandatory in Resource Policies? if i’m attaching the resource policy to a bucket i was hoping the only required fields are effect allow and principal as a minimum requirement

it’s mandatory for sure

it is but you can give it a non-existent resource if you need a neutral default

thanks @Issif and @Joe Niland

Has anybody seen this error message?
Error: Error setting SSLNegotiationPolicy: InvalidConfigurationRequest: external-tls can be associated only with a listener with one of HTTPS, SSL as frontend protocol
status code: 409
I haven’t found anything helpful. I opened a case with AWS, but wondering if the genius pool here might come up with something. Thanks!

Yes and no. A little bit context here would be helpful, but on the first it looks like there is a misconfiguration with LB’s security policy (ciphers).

I had been told that some old config for nginx-ingress was no longer needed. I put it back and it started working . it was this config:
rbac:
create: true
controller:
config:
server-tokens: "false"
service:
enableHttp: false
targetPorts:
https: 80
annotations:
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:us-east-2:xxxxxxx:certificate/77a312cb-149e-47c3-b243-19e841a253c3"
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"

Now it makes more sense

I’m glad i remembered that, because i found pretty much zero on my first few searches. thanks for looking, Alan.

How does everyone here create golden amis with toggles ? such as if you want instance X to use AMI1 with datadog and instance Y to use AMI1 without datadog, you wouldn’t build a whole new AMI, you’d have some kind of flag or feature toggle, right?
Would love to here thoughts on this. I’m wondering if we can do something with SSM or tagging on instances to use as toggles.

Does Cognito integrate at all into Cloud Map / Service Discovery? I’m moving some ECS fargate service that are behind an ALB (with Cognito auth) to use Service Discovery, but am unsure how to force beyondcorp auth
2020-07-28

I setup a terraform service account creator for multiple accounts. It uses Cloudposse iam user service user module. That means I have 7-8 plans in Terraform Cloud that any update automatically runs and deploys.
Then I get the benefit of all my jobs running a data source = this plan = get aws access key.
However, I’d prefer to avoid iam keys and use roles when possible. Is this the correct thought process?
- Create home user account with the same approach.
- Create roles in each account for the terraform-service account
- Add trust relationship and allow the iam home account principle user to assume role
- Ensure MFA not set on this particular user
- Done? I’ve have such a time of it in the past with roles I had given up due to time. I’m ready to revisit to setup better, and think this is the correct way to go about it, otherwise I have iam creds for each account/user, which isn’t really preferred.

So these are accounts for external automated systems?

e.g. something like CircleCI

anywhere your service is running inside of AWS, you should avoid the iam user module and instead use assumed service account roles in ECS or EKS, or instance profiles with EC2.

Yes, they are external

Like Terraform Cloud, Azure DevOps pipelines for example. I can setup an agent in one account but honestly trying to avoid where possible as I prefer to use the hosted agents rather than having another server to manage

I think what you describe is the gist of it. It definitely does suck having those keys laying around in SSM, but hard to avoid storing them somewhere.

There’s also a terraform-1password-provider
for storing secrets that should be shared with teams for integrations

So i can use 1 iam user with assume role setup across accounts instead of using 1 unique IAM user per account? That makes sense to you right?

My TF authenticates to multiple accounts, from a main account, using role assumption. I also have roles in each subaccount with reduced permissions, to which I grant groups of users the ability to assume in subaccounts.

@Eric Berg external terraform account right?

Not sure what you’re asking. I have a master account, which has DNS and things loike that in there and for each client, we spin up a new aws account and use TF to create our entire stack in that account. We access these subaccounts using role assumption.

anyone come up with a terraform method of switching between launch_type=EC2
and launch_type=FARGATE
with zero downtime ? looking for a terraform-y way to do this.
im wondering if the code deploy method would work

I think you will need to do that on the Target group level with Code deploy maybe

Target group level seems right. I would make a new target group, add the new Fargate service pointed at that new target group, and then update your ALB Listener to point to the new target group. Then clean up the old ECS launch_type service and target group.
If you wanted to test out that your Fargate service actually works, you could use a Listener Rule where it would use the new service if you pass some special header, otherwise it would forward to the old target group by default

I try this with codedeploy and it worked just fine, I do have the code still and we could make a module out of it

yeah i think this would be super helpful. because we’ve been doing cost comparisons between ecs and fargate, ive been trying to explore a way to switch between those easily with zero downtime

if you have a code already or know of a tf module that already supports this, then id love to use it

my code I think it was for tf 11 but can be easily converted

we can work together on this if you want

yeah that would be awesome
2020-07-29

AWS re:Invent 2020 is officially going virtual and free. re:Invent will be presented across 3 weeks from Nov. 30 - Dec. 18, 2020 https://reinvent.awsevents.com/
2020-07-30

Hello All, I have been working on a CW Alarm solution for a Project, where we are moving all the logs from text files to CW Logsgroups logs. I have to filter a particular line from AWS CW log-groups log and send SNS . I am able to do so via custom metrics.
However I am looking for customisation where I would look to get the complete Message of CW Log Group if there is a match in string? Any help would be appreciated. Thanks Biswajit

Anyone knows AWS SSM Documents can be shared between regions? I can see sharing option with other AWS account only I have tried it, and I can see shared document only in the same region. Should I replicate all my documents in every used regions? Thanks

its like any other AWS resource , you have to create a cross account role and Document policy has to be updated accordingly. Hope this helps.

Aws elb throttling. Any idea how can I avoid it?

What exactly are you referring to as elb throttling?

I just wanted to change the elb ssl policy from default to custom one without tls1 and 1.1. For that I had to get all elb in region. I got throttling for that task when I used ansible elb info module

Aha so if I understand correctly you got throttled by Aws Api. Do you look for a way to slow down Api requests or to retry them when throttled in Ansible? If yes then you can take a look at this issue for some ideas https://github.com/ansible/ansible/issues/30229
ISSUE TYPE bug report COMPONENT NAME ec2_elb module ANSIBLE VERSION Ansible 1.7.2 Boto Version 2.32.1 OS / ENVIRONMENT Redhat Enterprise Linux 6.4 ( Ansible Tower v2.0.0) SUMMARY :** I have a proce…

Just did that. Wait 1 sec:)

Use NLB? :-)

Currently classic is in use