#aws (2020-07)

wouldn’t an AND of these two things be the same as using just the first one? not sure what the wildcard gets you on the second resource line.

Yeah. I was hoping it would be allow only if this named security group AND any other security group.

But the role can just use any security group with this policy

:(

Id create a support ticket if no one here can help. I’m interested on how this policy can be configured for this to work

@Andreas P Have you tried adding a second ENI from the private subnet to the public OpenVPN Server EC2? That second ENI could be used as a default gateway for the private subnet as well, to allow connection flow only through OpenVPN Server.

@viliam.pucik Thanks for the response! Just tried now but it seems that is not working either… Maybe something is off in my setup.

@Andreas P Hmm, have you enabled routing between the public and private network on the OpenVPN Server (echo 1 > /proc/sys/net/ipv4/ip_forward)?

The simplest approach is to configure security group of the ec2 instance to allow access only from vpn instance and do a NAT on vpn instance.

With regards forwarding of the packets one thing is the ip_forward and there’s an ec2 source destination check normally set to true which needs to be disabled.

It worked thank you guys! Truly appreciate it !

I had to enable forwarding as @viliam.pucik suggested

docker-compose

This tool that we discussed previously has made our deployment process a lot easier

https://github.com/fabfuel/ecs-deploy

Using that you can update a single containers image while retaining the other side cars and all the other configuration.

Oh lol just saw that you already included that

@RB Yeah, including that. I had heard of it before but your link in our thread made me look into it again.

Docker-compose is the perfect companion when working with ECS

I’ve been using it. It’s great for updating fargate containers without having to update the datadog and fluentbit containers too

You can re-use a docker-compose file for deploying to other envs

Ya docker compose is nice but almost a standard now, right?

@Santiago Campuzano Looking for ECS specific tools. Compose is more general purpose across the entire docker ecosystem.

That’s like saying, don’t forget to use git

Edit: sorry i wasnt trying to sound mean! Maybe it’s worth mentioning

Are you including your talk or will your talk stream be available?

https://github.com/nathanpeck/awesome-ecs

You could also mention some cloudposse ecs modules like the alb ingress, alb service task, and the ecs web app ones. That makes it a lot easier.

@RB Talk is here — Will be live streamed / up on Youtube after it’s done. https://www.meetup.com/AWSMeetupGroup/events/270751202/

Ah yeah, I’ll include awesome-ecs — Good call, thanks @Santiago Campuzano

Is cost of ec2 vs fargate also a topic?

Also do you know of a tool that will search running containers in a cluster that will return task definitions and then allow you to search those definitions? I’m trying to avoid stringing together a number of aws commands and having to write something… But i think i will have to whip out boto

I briefly touch on pricing in the Launch types slide, but nothing more than a quick example.

Don’t know of a tool that will do that, sorry dude!

ecs-deploy use in production

@Matt Gowie https://aws.amazon.com/blogs/containers/introducing-aws-copilot/

@Santiago Campuzano Yeah, we were discussing that over here: https://sweetops.slack.com/archives/CCT1E7JJY/p1594310815461900

Oh nice !

Do you have a Route53 host zone setup?

For a domain that you own

Yeah I do

currently we have a lot of services on shared clusters. the services are tagged appropriately but im afraid this cost doesnt include the ec2 itself.

makes it a bit challenging to compare apples to apples

im afraid this cost doesnt include the ec2 itself. The ECS cost is only the EC2 instance itself though

You aren’t paying anything for the containers running on the cluster

comparing ECS EC2 to Fargate though is a pain in the ass. You have to figure out your avg/max vCPU/memory needs and then compute your hourly/monthly costs for Fargate off of that

ah yes, so ive been taking a crack at this today. i did exactly that. i compared fargate to ec2 costs. it’s too bad we can’t easily do it in the cost explorer due to shared clusters. unsure what the benefits are of shared clusters…

so we took a single task definition, carved out what the least amount of a shared cluster ec2 that it would use, calculated the cost for that, and then compared it to fargate

here is the sheet for fun. i did not include RIs. if you can poke holes in it, plz let me know.

You’re showing a savings of moving the task to fargate? That seems suspicious

ya, it does kind of seem like the numbers are wrong lol

but what you dont notice, is that if we didnt rightsize, we would not see a savings to fargate.

the final columns are rightsized with a task count of 2 and a task count of 1 respectively

they had to be rightsized because fargate does not support 32 vcpus which the orig TD requested lol

well you shouldn’t compare your base cluster size directly to fargate

you need to take each individual service task mem/cpu requirements and map that instead

but like as an example, I have go services that run in t3 nanos on ec2. Those cost $8.39 each (including an 8GB EBS) per month if we don’t burn over the cpu credits. Those are 2 cpu and 500mb which doesn’t translate to fargate very well, but this particular service could get by on 0.5-1 cpu I think. Plugging that into fargate, that costs $18 for 0.5cpu/1gb or $32 for 1cpu/1gb using on demand pricing. So its a huge increase and thats why my management doesn’t want to do it

well you shouldn’t compare your base cluster size directly to fargate ah of course not. that is why i have the ec2 10% as that is the usage of the original task definition. best way to compare, i thought, was to compare fargate maxed out vs ec2 10% costs as those are the most comparable without rightsizing

you need to take each individual service task mem/cpu requirements and map that instead yes, that is what i’v edone with the 10%. its just not obvious enough i suppose.

ok, I didn’t know what some of the terms you were using represented. In general t hough, if you’re seeing a savings on fargate you probably have a math error

there might be a weird edge case where somehow you win, but I’ve never seen anyone show an example of it

we’re on an old ec2-shared instance type too. we’re on m3.2xlarges which are pretty old

if we made the switch to m5, the cost savings might be lower. ill have to triple check the numbers again tho. im a bit worried i made some mistake.

the 2nd to last column, if we compare the 10% ec2 usage vs fargate (non spot, with count=2), we have a cost increase which makes sense

you should get a price decrease just going from m3 to m5

We definitely would see a decrease. But easier said than done. Lots of tasks run on these. Still were looking at a hybrid approach where we migrate some to fargate and leave the others to migrate to m5

@RB This video explains the pricing differences quite well https://www.youtube.com/watch?v=HoXEyXIf6_U

thread

The cons are mostly security cons… if the Docker image/container has some sort of vulnerability, it could escalate privileges in the EC2 instance running the container

It could become a Rogue container

@RB

there is no ec2 instance as it is fargate

i mean, sure there is an ec2 but its managed by amazon. im sure their firecracker encapsulation of my fargate docker container will prevent issues if i run inside my docker container as root, no?

@Santiago Campuzano what do you think?

Right…. it’s Fargate… so Im pretty sure AWS is gonna make sure that your Container won’t do any harm outside

So I’d say go ahead …. no worries

Except giving potential attackers easy means to install whatever shit they need to become part of a botnet, send email spam, used as a jump box to carry other attacks or whatever. If you’re not replacing your containers (e.g when running a legacy system that requires few to no upgrades a year) this potentially means persistent presence in your network especially if you’re not monitoring what processes are running on the systems you’re managing.

That’s a very good point.

I think short answer to your question would be that it would violate “least privilege” principle/best practice.

ya, but i wanted the long answer. i wanted to see what could possibly go wrong if my fargate container was using a root user to run its code instead of a non root user.

Not probably the long answer.

There aren’t any security implication as host encapsulation is preventing privileges escalation. Last but not least, good and encouraged practice, following the principle of the least privilege should be do not run Docker containers as root. And finally, Fargate doesn’t support privileged containers.

yes and duly noted. correct me if I’m wrong.

the original post wasn’t completely clear. my question was exclusively about root user inside the docker container whereas privileged containers is running docker daemon container itself using a root user.

or in other words, you can run a non privileged container using fargate with the root user used inside of the container.

Yes, you can run a non-privileged container using Fargate, but as it isn’t a privileged container, and thus completely isolated from the root user on the host machine.

Sometimes, I am having an impression about the root users inside Docker, similar to other taboo themes, everyone is talking about it, no one is practicing.

Again, after tasks that need a root, I would specify the runtime user in Dockerfile.

@Alan Kis you https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html#container_definition_linuxparameters it changed a little bit with agent version 1.4.0 you can give to Fargate certain linux capabilities

ah for fargate it’s only SYS_PTRACE

is that a new tech ? “Cintainers”

please do not tell me I need to learn something new now….

lol typo my bad

lol

there is such a thing as a Sintainer. it’s a container that’s been bad

like a container that runs as root user

hahahahahahahahahahah took me a few second to translate that one

but isn’r that a problem of the Sinadmin that set the Sintainer in the first place?

You’re thinking of a Badmin

when he’s hacked he becomes a Sadmin

anyone following this thread will want to kill me lol

I think we are the only too people enjoying the bad humour LOL

we are going to get kicked out from the @contributors channel

anything interesting? any new announcements?

currently going over fargate 1.4 version

@RB well done.

containerd is replacing docker. wow.

im sure there will be more performance gains using docker’s core containerd, so thats nice. hoping for cheaper prices soon.

https://aws.amazon.com/blogs/containers/introducing-aws-copilot/

They’ve kind of made something like serverless-cli for ECS

@Joe Niland Huh ecs-cli v2… Don’t know why they gave it a new name. But sadly, I personally am already not a fan —
But later when it’s time to create this architecture, there are a lot more cloud components to think about: VPC subnets, load balancers, deployment pipelines, and durable storage for your application’s stateful data. Copilot handles all these underlying details for you. This is what bugged me about ecs-cli and is why I stayed away from it… I don’t want AWS to abstract all that stuff away from me because it makes it harder when I need further access to those pieces.

Yeah know what you mean. I’m thinking about who this cli targeted at exactly. Also without plugins / hooks it’s only going to work with pretty common scenarios.

It does look like it does a lot of heavy lifting after looking at the full post. I would possibly use this for small side projects that I couldn’t just host on Amplify but beyond that I think I’d steer clear.

Yes I think it’s pretty cool for Dev/test. But then you need to build out the Terraform modules for production.

I’ve got an app to move off elastic beanstalk. I think I’ll give this a try!

Let us know how it goes!

Sure. So this thing is super similar to Serverless framework. It’s generating CloudFormation and you can extend it by adding your own CloudFormation.

Ah so you’re allowed to extend the CloudFormation. That’s something at least.

I figured it was using CF under the hood, but figured they wouldn’t expose that.

Seems AWS + Docker worked together on updating compose with the same ecs-cli capability: https://www.docker.com/blog/from-docker-straight-to-aws/

the plot thickens!

Haha yeah… and I find that to be a bigger deal. There is work to make the compose file specification a broader specification so that you can use it to deploy services. And it seems like Docker and ECS have already started on that, which is cool…. If this was a pluggable architecture it would be awesome to build a terraform plugin so you could deploy a compose file with generated terraform.

This is what I’m referring to with the compose spec stuff. Super interesting as this could vastly simplify various cloud setups similar to the way that compose simplified local development setups.

https://www.docker.com/blog/announcing-the-compose-specification/

we use it and it is mostly reliable — we have < 5% requests that end up routing to the incorrect region

I do not know how it compares or relates to edge lambda, i have only used that for simple slicing on the request URI

we have < 5% requests that end up routing to the incorrect region so depending on your tolerance for this and the regions you’re deploying in it’s probably good. It’s actually not good enough for us and we’ve written our own routing in order to pin the region — FWIW I notice it’s much flakier in me-east-1 (bahrain) and Asia. very solid in the US regions

We’re using it in the africa region (with our resources deployed in eu-west-1) and it’s working awesomely.

@Igor what’s your issue with lambda@edge

No issue with lambda@edge, just asking which one is better

https://github.com/Sceptre/sceptre

What do you mean by environment/account? Sceptre seems much like an overkill.

How are you deploying resource across account/regions using CFN? If using CodePipeline, the pipeline can easily do cross-account/cross-region deployments.

https://docs.aws.amazon.com/codepipeline/latest/userguide/action-reference-CloudFormation.html (TemplateConfiguration).

global accelerator provides you with static IPs on one side, and connects to ALB, NLB, EIP or EC2 instance on the other side. It should play nicely with any DNS providers, if you are talking about creating A record in CloudFlare and pointing it to the static IP

will our TCP traffic be routed immediately to global accelerator edge server or through cloudflare edge servers (possibly less ideal?) then hit a global accelerator node way later down the line

curious if someones had the same question and if its even a valid concern , if not i can report back once i get it set up

this suggest is possible buy it does not work

https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-resource-policies-examples.html#apigateway-resource-policies-source-vpc-example

If it is a private API gateway it should work. But for a public one, you’ll need to deal with IP whitelisting

I haven’t looked at this specifically for API gateway (may need to soon), but it is part of AWS design

ohhh so it is a public endpoint you can’t do account/role filtering?

Not that I’m aware of. I’ve always done IP whitelisting with Security Groups or WAF. Since it is public there is no AWS login information

Some services like S3 are an exception to this

But that is because it is an AWS public service. By default all your private traffic goes to the internet to access S3.

if you lean towards private api gateway, read this first: https://st-g.de/2019/07/be-careful-with-aws-private-api-gateway-endpoints

your other option would be some kind of custom authorizer on the public api gateway… https://aws.amazon.com/blogs/compute/introducing-custom-authorizers-in-amazon-api-gateway/

so is a bad idea to use the private api gateway?

this is an internal api endpoint, it should never been accessible from other accounts

that’s not the primary consideration. it’s whether you have other api gateway endpoints that you want to reach from any account where you create a private api gateway endpoint
But in this case, the setting Enable Private DNS Name, which I would certainly want to activate, will prevent all access to any regional/public API Gateway in the whole AWS region!

there are workarounds mentioned in the article also, if you don’t see any way forward but to use a private api gateway

well so we tried today and I call this api gateway policies a bunch of lies

no matter what, you need connectivity to hit the endpoint so you can’t use policies base on account ids on a public endpoint

and the whole idea of all this was so we could avoid doing vpc peering and filter by account id

i’ve not tried the resource policy approach with api gateway. if it seems to not be working, i’d recommend opening a support ticket

We’ve settled on 1Password for business or enterprise. It supports geofencing and audit trails. It’s one of the few solutions for shared and team managed TOTP.

Other solutions of have seen are private slack channels and totp bots.

Great thanks Erik

aws sam, or amplify, or serverless.com make the api gateway and lambda integration really pretty easy

just worked a week-long prototype using aws amplify and came away super impressed… some rough edges, but made it very easy to build out and manage the whole solution… ‘course, if you don’t have a frontend website, it’s not a perfect fit, but it is usable

i’m looking for just api calls so I can have Teams extensions interact with my AWS stuff for chatops “2.0” style interactions

Serverless framework seems great, so I’m starting up on that, but the api gateway stuff is newer to me so hoping someone can give me a quick 101 or dummy version of how it’s secure for this. And yes, I can google it, just been spending too much effort on all the pieces so a quick start on how it’s secure with a token or whatever would be useful to make sure I’m not going down the wrong road

I am testing it now, that there is some cloudformation support, but they way it scale in seems to be very unstable as I always end up with at least one idle server in the cluster

There’s CF support? I’ve tried and found it very lacking. I had to augment our CF runs with some manual calls to the CLI after the fact, and it was really gross.

still missing support to autoscaling group name instead of Arn and if you use Managed termination, ASG lacks support for protecting instance from scale in

Yeah — Pretty cool to bootstrap a small project! We were chatting about this when they first announced over here: https://sweetops.slack.com/archives/CCT1E7JJY/p1594310815461900

Cool.. But.. why don’t they come up with a Heroku-like Paas instead I wonder? A type of Elastic Beanstalk done well.

Is it something I’ll regret or has it made your life better? yes.

We use it, it works. Its not my favorite tool, its a little quirky and it doesn’t see a lot of improvement. They only 1 price tier which is enterprise, and SSO is hidden behind that.

Jobs produce ‘logs’ and not really “output” so its a little annoying if you need to do something like provision something and return some value to the user, they have to dig into the logs to find it.

You can only ACL down to the job itself, nothing within it. ie you can’t restrict certain options from being used by a particular user/group.

The ‘dynamic options’ are poorly documented but are actually pretty powerful once we figured them out. You can point it at a url that returns a specific json structure and rundeck will extract option values from it. This lets you run a lambda behind an API gateway for example and have it lookup complex things for the job. And the dynamic options can ‘stack’ on each other, so that as you select option 1, the values of option 2 can be dynamically computed.

we use it and we are deprecating it

Ooohhh in favor of what

moving to containers

ansible does not make much sense for us anymore

nor any other orchestration tool

Ah I see. We’re using it for not just ‘deployment’ type stuff although that’s the primary use

we do not use AMIs anymore, or a few but they are fading away

We use it, and while I acknowledge there are issues I like it. I do wish there was some competition to push the rundeck team forward a bit.

I get the sense that it may work better for small teams (like mine) than it does for larger groups.

dang.

So what’s an option for a containered (so I could ideally put into fargate) runbook tool so I could plugin in some powershell scripts , python etc… and basically try to get folks away from tickets being done manually?

I want to create a library of simple tasks that folks keep doing manually. I can use azure pipelines, but it’s not really meant for that, so feels like a hack rather than a nice “runbook” to go and use.

I’ve not used it myself, but https://stackstorm.com/ is on my list of things to explore if I ever get a chance.

I’ve looked at StackStorm. It is a hell of a lot more complicated stack than rundeck, and I found it VERY difficult to get the same sort of “self service push button” jobs.

Oof, that’s a shame.

It’s aimed at being an ‘If this then that’ of operations

I think relay.sh is going to be freaking epic. It’s just too early of access I think for me, though maybe it’s what i should focus on

Anything else? I need a free tier, ideally self-hosted so I don’t have to buy anything and “click button” to run something. I need the target to be devs, qa, and some business folks eventually. RIght now just a couple devops team folks

I think rundeck is the closest you get

free, on-prem is why we’re using it too

Spinaker ?

One thing to mention is that while their official SSO is enterprise only, it is possible to use oauth2 with rundeck if you also deploy https://github.com/oauth2-proxy/oauth2-proxy - we’re doing that with gitlab as our auth source since everyone who needs access will also have gitlab access.

https://spinnaker.io/

I have our rundeck hooked into Okta via the Okta LDAP Connector

nice

I saw the oauth proxy but was hoping to just not have to run 1 more thing

Believe me, I hear that

So rundeck still for running various powershell/python aws things, or should I just stick with Azure Pipelines (which isn’t quite what I was hoping for)

@sheldonh yes it can do those

Tried running docker version and having issues with default credentials to test it out locally. Hoping i could use prebuilt images with ecs task

our rundeck basically runs a lot of shell/python stuff

^ that’s what we use it for too. mostly shell and a handful of python.

Better than azure pipelines/gitlab etc?

The other last option I can see is spin up a self-hosted gitlab (which might meet with some challenges), and use their interactive runbook feature?

Not exactly a self-service thing but possibly a compromise for at least the devs to run stuff

haven’t ever used Azure so no opinion there

Hmm no price details on Relay.sh?

I haven’t used azure either, but I’m in the middle of moving from github + circleci to gitlab. I would not recommend gitlabci as a general “run this thing” tool.

It could be used that way, but you’d need to have a deep understanding of the pipeline definitions in order to know what variables should be passed in to run the jobs you want to run.

relay.sh is in early access. They told me they will have a free tier regardless. They are working out pricing after they see how things go with users /pipelines etc.

I’m familar with yaml pipelines in azure devops and could pick that up in gitlab pretty quickly I feel.

it’s more about what is the right “interface” experience for making it easy. Azure Pipelines is easy, just feels like a clunky way to use it. Relay is promising but very beta, have to give it access keys and not going to be a quick sell for me at this time

I just got docker rundeck running. Maybe I’ll look at that first

Oh, I have no doubt you could pick up ci gitlab stuff. But if you’re looking for something where others can run jobs themselves, those people would also need a deep understanding of the gitlab pipelines.

Running a pipleline is simple, you just choose a branch, enter some variables (if you need) and hit run.

But knowing what variables are available, what jobs need what variables, etc… that’s where you need to understand the pipelines.

got you

So rundeck or azure pipelines i guess is my main option

the UI for rundeck has so much wasted space

wait until you run a long job and want to view the logs >_<

that’s kinda why i was hoping to use this vs azure pipelines. i think i’m back to just using pipelines for a quick win cause at least i have it and it runs powershell without issues. Even the yaml is preferrable to this workflow builder. Ugggh

I mean if you really want yaml… you can declare a job in yaml and upload the definition to rundeck. You can also set it up to sync with git repos

https://docs.rundeck.com/docs/manual/document-format-reference/job-yaml-v12.html#job-yaml

If you’re feeling masochistic, you can even use xml.

Oh yah the rundeck interface for building jobs is AWFUL

you have to hit save in every section, and then at the end. annoying as heck

We found that it has a terraform provider and have been defining jobs that way

woah, that’s a cool idea

nice. I think for now azure pipelines will be the hacky way to provide this, as much as I prefer something designed for it, seems like nothing fits the bill, so might as well stick with something that quick to get running

it also let us get around some issues we had with how it would mangle shell scripts in its own interface

Octopus has free tier with 10 targets included. I’m running mostly just container stuff, so their runbook stuff seems interesting too. Anyone used that?

On the subject of yaml… Not sure this gives me immediate confidence in using that

Can I host rundock in ECS and have it just run docker commands for each task?

Not sure if it’s easy to setup like that as Azure Pipelines has their “hosted” pool and I can pick any ubuntu, mac, windows and go.

I might explore rundeck again in the future if I can easily have it run docker based actions.

Pretty sure you have to pay \(\)$ for Enterprise to get that

At least I think, their docs aren’t really clear on that part

There are docker and k8s plugins for rundeck, so yes you can run docker commands. Of course you can also just run docker commands without the plugins as well.

Also, a quick google turns up a third-party ecs plugin: https://github.com/oyama/rundeck-awsecs-plugin

I thought the problem was that it relies on Rundeck Clustering which Enterprise only

Which part relies on rundeck clustering?

External nodes like that ecs plugin I thought

Nope, I’ve not used those plugins but I’ve definitely used external nodes with rundeck oss.

Totally understandable. Clustering is only needed if you want to have multiple rundeck instances, but you can have any number of nodes where jobs are executed without clustering.

@sheldonh just saw this linked and maybe it helps fill your need https://eng.lyft.com/announcing-clutch-the-open-source-platform-for-infrastructure-tooling-143d00de9713

will take a look, thanks!

Got my interest when I read:
Clutch ships as a single binary that contains the frontend and backend, making it trivial to deploy. Many changes can be achieved via configuration rather than recompiling a new binary.

Even more when I saw
Clutch is powered by a Go backend and React frontend.

This component-and-workflow architecture allows a developer with limited frontend experience to replace clunky tooling or command-line scripts with a clear and easy-to-use step-by-step UI in under an hour of development time.

I question that 1 hour based on their “amiibo” example walkthrough

So far, that’s the first tool like this that run the first time without any issue.

well I’d be curious to see what you do with it, their examples are pretty bare bones

Probably not much right now. No time. Not enough plug and play. But still really cool for when I someday over the rainbow get the time

Hi

Heh, well I know we’ve discussed it already but you’re basically describing Rundeck (for all its warts)

Sounds like AWS amplify to me

Ha!

I got ya. I surrender.

Kinda was hoping for a serverless/step function interface but nothing out there like that from what I see.

Better for now to stick with azure Pipelines I guess. Takes 50 secs to run a task instead of 5-10 secs but worrying about that is probably premature optimization for all the benefit of starting to add some basic automation anyway.

I wish azure Pipelines had an output view that would immediately come up instead of navigating into sub pages and it wouldn’t be so confusing

I even tried adding Microsoft forms tab to channel …but flow required premium subscription and it didn’t support properly sending runtime parameters

It’s all a plot against me

I hear you, I’ve been trying to find alternatives to rundeck for quite awhile

If i was sticking with slack it would be much easier. Teams is much harder for integrations.

coughrundeckcough

Jeebuz yes. My company recently forced us off slack and onto Teams because thats what the ‘corporate side’ was using. It’s been awful, do not recommend.

I’m kinda biased against rundeck because my first impression was lots of whitespace, overlappy busy UI and not quickly intuitive to run Powershell core container workflows. Not saying it can’t but I’m outta time for anything complicated and was hoping for some quick wins. Don’t think I’m going to get that with rundeck right now and going to limp along with azure pipeline yaml based jobs and use that for now. Will help me raise more awareness of it as well for simple automation.

I just need to figure out templates better and I think DRY will be adequate too.

I was just doing a POC of Azure DevOps over the weekend and I thought the templating was nice, but it also had a lot of verbosity just to use it

Compared to some options it feels super succinct. Don’t forget each level of deploy>job>steps can be ignored to start simple. Can literally run a few lines with just a pool + steps of bash/pwsh and done. It’s only more verbose when you need to add more strategies on it

What’s the ux for someone new to it running a pipeline? Is that something you’re worried about?

The Azure UI was quite nice I thought It feels weird because we run in AWS but are now considering Azure DevOps for build/deploy

The ux for running a pipeline is fine. I meant the UX overall in rundeck made it feel overall busy and complicated vs me just building a yaml pipeline in azure pipelines.

Yah rundeck UI is garbage

I’m pretty picky on UI experience, i know it’s not the most important thing, but I like any tools I use to provide a nice polish when possible

I only ask because while rundeck’s UI isn’t great, it’s really easy for someone who doesn’t know anything about rundeck to login and run a job. Since you mentioned there are a variety of people who will be using this thing (and I have zero experience with azure) I thought it worth asking.

i’ve never before heard anyone say azure devops had a simple interface lol, i can never find anything without at least 20 minutes signing out and back in (trying to find which account owns the resource) and poking through at least a dozen screens

i mean, is this configuration hidden in a page, a tab, or behind an obscure triple-dot context menu? who knows! it’s a mystery!

i literally dread trying to troubleshoot the single azure pipeline we have. we only keep it for the free windows ci. just biding time to cut it over to github actions, or travis-ci, when/if their windows support comes far enough along

Hmm. I had very little issue standing up three Go projects on it on Sunday afternoon. Only thing tricky was its secrets mapping for our private modules

Sorry to hear that Loren. I’ll tell you having used both GitHub actions and azure Pipelines I think that azure Pipelines if you give it a shot is more intuitive, robust, and feature filled . If you get stuck on your build be more than happy to see what’s up with it.

It can be confusing at first but it’s a best in class in my opinion if you can get past some quirks

Hi all, just checking if RunDeck is still a favourite 2 years on… https://sweetops.slack.com/archives/CCT1E7JJY/p1660211217368379

Still using it and haven’t had any reason to look for anything else. Not sure if it qualifies as a “favorite”.

I couldn’t find this yesterday, but it just popped up again on hackernews. If I wasn’t already using rundeck I’d be playing around with this to see if it’d be a good option.

https://docs.windmill.dev/

I decided Rundeck was way too complex for my needs. Maybe great for someone going all in. I ended up using AWS SSM Automation runbooks in AWS, and heard good things about Azure notebooks too.

I still think Relay.sh has a lot of promise but not sure it’s been adopted heavily. It’s dockerized workflows like IFTTT for DevOps work. If you are cross cloud then maybe makes sense. Otherwise I’d lean towards using what’s on the platform even if it’s less polished

Windmill is pretty cool! I love the concept. Open standard with lots of building blocks already out there.

The problem is that this solving is that middle ground most don’t seem to know they need.

Nocode/zapier == great, but very limiting to an engineer needing more backend/cloud stuff. Pure lambda/runbooks lots of proprietary snafus I had to work through with AWS Automation runbooks and difficult to debug/develop in general.

We should just all build more Go CLI tools that do everything we need and call it a day

one of our teammates has been gathering queries here: https://github.com/easttimor/aws-incident-response

we used too, it is awesome

I built a poshbot interaction I was proud of. Poshbot (chatops Powershell bot) could run Athena query that was setup via terraform and return formatted tables or excel files

you can do that with aws-cli + jq or awk

get-account-authorization-details gets most of it but I’d still have to write something to get the rest. I can’t be the first person to do this so hoped there might be an existing tool

@Henry Carter There is an easier way:

https://console.aws.amazon.com/iam/home?region=us-east-1#/credential_report

IAM Credential Report

nice tip, thanks

Nice, I’d just need to add each users groups to that

I agree 100%

all new AWS interfaces are dumb, slow, you can’t sort columns

giant icons (like for tablet interface ????)

@maarten Have you tried to change a record? I wanted to change a A (Alias) for a CNAME? I needed 2min to figure out I had to delete the value to have the option to change the record type

No I went straight here to nag about it, it would be better if they drop the whole UI completely now.

I agree. I hate that interface. Almost 3 minutes for creating a new record.

My favorite part is that when you drop into a Hosted Zone, they for some reason don’t set the left-right scroll bar for the records all the way to the left, so you can’t actually see any of the record information until you scroll it over. It only renders correctly if you’re in full screen mode

oh yeah, and i thought the old interface was bad and unintuitive! the new one is waaaay worse! so much clicking!

there’s a new route53 UI?

We are asked to only use API.

AWS is great at many things. UI experience is not one of them. AWS ssm is so slow to improve. You can filter by the presence of a tag but not the value last I checked. Seriously.

I have used State Manager RUN Command and notifications i have used are RUN Command -> SNS -> grafana CW source Actual Document RUN -> logs to loki

https://github.com/cloudposse/terraform-aws-iam-policy-document-aggregator

you’ve seen this?

have now

that’s not a bad workaround

You can attach a lot more (unlimited?) inline policies if that helps

That aggregator + the IAM Principal Tag have been really really great for me in reducing IAM pain.

"Condition": {
                "StringEquals": {
                    "ec2:ResourceTag/CostCenter": "${aws:PrincipalTag/CostCenter}"
                }
            }

Def check out the blog post on aws:PrincipalTag is you haven’t gotten into it yet; TLDR: let say you have an IAM role or User with a tag on it with the k,v being “ec2:ResourceTag/Team_ID” where Team_ID = “Frontend”

in the IAM condition above attached to AWS Resources, the IAM role with the tag “Frontend” can only access that resource if it also has the Tag “Frontend”.

We use it sorta like this in our nonProd Envs: if a user with the Tag for Team_ID == “Data Engineering”, they can access lots of stuff they need to where the AWS Resource has also been tagged with “Data Engineering”.

^^ I’m 2 years late to the game on aws:PrincipalTag - wish I knew about it sooner.

it’s mandatory for sure

it is but you can give it a non-existent resource if you need a neutral default

thanks @Issif and @Joe Niland

Yes and no. A little bit context here would be helpful, but on the first it looks like there is a misconfiguration with LB’s security policy (ciphers).

I had been told that some old config for nginx-ingress was no longer needed. I put it back and it started working . it was this config:

                rbac:
                  create: true
                controller:
                  config:
                    server-tokens: "false"
                  service:
                    enableHttp: false
                    targetPorts:
                      https: 80
                    annotations:
                      service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:us-east-2:xxxxxxx:certificate/77a312cb-149e-47c3-b243-19e841a253c3"
                      service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
                      service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
                      service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"

Now it makes more sense

I’m glad i remembered that, because i found pretty much zero on my first few searches. thanks for looking, Alan.

So these are accounts for external automated systems?

e.g. something like CircleCI

anywhere your service is running inside of AWS, you should avoid the iam user module and instead use assumed service account roles in ECS or EKS, or instance profiles with EC2.

Yes, they are external

Like Terraform Cloud, Azure DevOps pipelines for example. I can setup an agent in one account but honestly trying to avoid where possible as I prefer to use the hosted agents rather than having another server to manage

I think what you describe is the gist of it. It definitely does suck having those keys laying around in SSM, but hard to avoid storing them somewhere.

There’s also a terraform-1password-provider for storing secrets that should be shared with teams for integrations

So i can use 1 iam user with assume role setup across accounts instead of using 1 unique IAM user per account? That makes sense to you right?

My TF authenticates to multiple accounts, from a main account, using role assumption. I also have roles in each subaccount with reduced permissions, to which I grant groups of users the ability to assume in subaccounts.

@Eric Berg external terraform account right?

Not sure what you’re asking. I have a master account, which has DNS and things loike that in there and for each client, we spin up a new aws account and use TF to create our entire stack in that account. We access these subaccounts using role assumption.

I think you will need to do that on the Target group level with Code deploy maybe

Target group level seems right. I would make a new target group, add the new Fargate service pointed at that new target group, and then update your ALB Listener to point to the new target group. Then clean up the old ECS launch_type service and target group.

If you wanted to test out that your Fargate service actually works, you could use a Listener Rule where it would use the new service if you pass some special header, otherwise it would forward to the old target group by default

I try this with codedeploy and it worked just fine, I do have the code still and we could make a module out of it

yeah i think this would be super helpful. because we’ve been doing cost comparisons between ecs and fargate, ive been trying to explore a way to switch between those easily with zero downtime

if you have a code already or know of a tf module that already supports this, then id love to use it

my code I think it was for tf 11 but can be easily converted

we can work together on this if you want

yeah that would be awesome

its like any other AWS resource , you have to create a cross account role and Document policy has to be updated accordingly. Hope this helps.

What exactly are you referring to as elb throttling?

I just wanted to change the elb ssl policy from default to custom one without tls1 and 1.1. For that I had to get all elb in region. I got throttling for that task when I used ansible elb info module

Aha so if I understand correctly you got throttled by Aws Api. Do you look for a way to slow down Api requests or to retry them when throttled in Ansible? If yes then you can take a look at this issue for some ideas https://github.com/ansible/ansible/issues/30229

Just did that. Wait 1 sec:)

Currently classic is in use