#aws (2020-06)

@Prasad ACM cert management and IAM cert management are two different things. You’re correct that it’s confusing, but I would suggest you investigate the two solutions.

ACM is definitely the suggested route, but I believe it’s not available in all regions.

if i need to upload a internal custom signed cert and assign to a Internal LB which one do we prefer ..? Internet is restricted for us and white listing is being done only when its absolutely necessary. Can we use Endpoint for acm service and perform this? i believe aws iam has only public endpoints..Have you come across any pointers that explains the difference between the 2…Thanks again!

@jose.amengual what broke or what was difficult ?

you need to do that

on the cli

is an account wide setting you could I think AWs organizations to do that

right because its region specific. i thought it was an account wide setting too

it is pretty stupid

so wait after you enabled it, did you see any issues with the new arn format?

awsvpcTrunking

and containerInsights is not needed for the ID thing

not issues at all

I did this in a very old prod account

nothing happened

awesome! that makes me feel a lot better

seems easy enough to opt out anyway

They were supposed to make these the default settings earlier this year, but then they backed out. It’s a real pain.

don’t mess with your cookies?

To my knowledge I haven’t been

I would start here https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_BestPractices.html

we have a new one from terraform called git_repo which is dynamic based on where the terraform was applied

The https://github.com/cloudposse/terraform-null-label also has namespace , stage, and attributes tags

what is the value of Billing tag ?

It depends… normally it’s gonna be the team/system that;s gonna pay for that cloud service

So we can charge back and split the AWS bill based on that

we have a similar tag. out accounting provided cost allocation numbers for teams and we use those to tag resources in shared accounts. if an application is using a separate account (e.g. aall production environments are separated) it’s automatically attached to a certain number and we don’t have to do it (but we still do to simplify our cf templates)

thats pretty cool

i also found this nice doc on best practices

https://d1.awsstatic.com/whitepapers/aws-tagging-best-practices.pdf

does anyone use tags for access control ?

We do…

We use AWS Session Manager

ah interesting. found this example for that.

https://docs.aws.amazon.com/systems-manager/latest/userguide/getting-started-restrict-access-examples.html#restrict-access-example-instance-tags

You can put 10 policies in one group, so I’ve honestly never needed to associate a user with more than 100 policies.

we hit that limit already

we have users in more than 10 groups

and more than 10 policies per group

AWS limit that you can request to be bumped then?

We are looking at ways to use Service policies for the ORG

not, it can’t be bumped, that is what they told us

it is a hard limit

So you have users who you want to put in more than 10 groups and add more than 10 policies to those groups is what you’re saying?

mmmm no I think I’m mistaken, we have more than 10 team/groups and some users need to belong to more than 10 groups so we attache policies directly to the users

it is a mess

you can’t add users to more than 10 teams

Ah that is a mess. So @David’s is valid and you’ve already run up against. That sucks.

Can’t you create new pseudo groups for those that need to be in more than 10 groups? You might require some automation to manage the pseudo groups.

For example the automation would take two groups and make a new Group1_Group2 group with the policies from both. Would need to listen for updates to Group1 or Group2 to sync the policies over.

That sounds like hell @jose.amengual - Perhaps you need to simplify your users / groups configuration or move to something like Okta / SSO and externalise user / group management instead allowing users to assume roles based on groups in the Identity Provider.

It’s been a long time since I’ve had any AWS accounts that contain actual AWS users in them.

We’re using assumed roles only, same deal. I have a few ‘legacy IAM Users’ that I’ve been giving a countdown to for their access to be removed

it is a mess, it has been done for a log time by hand and now we have a team of people starting to automate, finally

Ah… the joys… How are your groups organised? Around business function or capability? I’m guessing all the users use a single account?

For the users with 10 groups you might be better off either consolidating their permissions into more powerful groups or creating roles that have the policies attached and allow those users to assume those roles.

I think that anything you do will be coding around the problem and your account(s) is probably due a bit of an IAM refactor. I’ve done similar things with various places migrating from IAM users / groups to IAM users / roles to SSO / roles with permissions boundaries.

These things are always scary but can be done gradually allowing experimentation and learning. You can enable SSO and run it alongside plain old IAM users. There are a few good articles out there on the subject (https://segment.com/blog/secure-access-to-100-aws-accounts/) and lots of tooling to help your users (aws-vault, aws-azure-login, aws-okta). Maybe it’s worth a chat with your friendly local AWS TAM to get some ideas?

That said, I don’t know how big the real problem is for you or if it warrants the work needed to refactor your accounts / IAM. It would be satisfying for you once it’s “done”

we have groups that we attach roles to it so it is not soooo bad but there is a mix of user with policies attached, users attached only to groups or direct attached policies

so we need to cleanup and take one approach

you can use “default” security group for that. all machines with this group can see each other in internal communication

what about services like rds

same with rds, it had security groups too

so you are saying default one have special permission to talk with in vpc

well, not a permission, but an inbound rule that says: allow all traffic from the same group (must be in same region)

It is bad practice to use the default vpc, usually default vpc is fully open, inbound and outbound

in our case I think we delete it or at least change the rules to not allow anything and then we create SG for each resource

and you always need an SG to open ports to talk to other services inside AWS

Yep, the fluentbit approach is working like a charm. The only downside is that you should have sidecar (which needs resources) for every workload that need to ship logs to datadog

ah fantastic. i was planning to do the following, add a datadog sidecar container, use the fluent configuration from that document, and is there any additional configuration ?

do i have to manually open the tcp port on the agent or can i just add a port mapping for that ?

Nope, sidecar can talk to the main container in the task and vice versa without any additional rules or port openings. And obviously the task should be able to communicate with datagod’s intake for logs (i.e. have egress rule in sg)

perfect!

so the firelens and fluent portion, firelens is built into aws, but for fluent, do i need to install fluent ?

Firelens is a log driver that allows to route logs and fluentbit (in a sidecar container) receives them and forward to the destination (datadog in your case). You don’t need to install anything, you should only change your task definition as suggested in the docs you’ve mentioned

ah so it sounds like i need fluentbit sidecar with my app as a sidecar for sending logs

i figured if i wanted apm, i also needed the ddagent as a sidecar too

so then ddagent, fluentbit, and my app

Not necessarily, AFAIK you can also have ddagent deployed as a service and configure your app to send metrics and traces to this service. So sidecar for ddagent in your app’s definition is not a must.

AWS has a fluent image design to work with firelens. Makes it easy to setup. You’ll need to add datadog plugin to it or route all traffic through an aggregator that has the plugin

interesting so perhaps to make our fargate more scalable, it would be best to deploy the ddagent as a service in my cluster, and then have all my fargate services (app + fluent with firelens config) configured

so that way ddagent will get apm for the fargate services and each fluent sidecar with firelens config will send the logs to datadog

yes

we have multiple regions, multiple clusters. would every cluster in a region need its own ddagent service ?

Yes, each instance of ddagent would monitor the cluster it’s deployed to.

i got fluent to work appropriately using a fluent sidecar

running into this now. hoping someone from this thread might be able to weigh in

https://sweetops.slack.com/archives/CCT1E7JJY/p1592504977306200

You get credits that exceed the membership cost

I think is better to use parameter store or secret manager

Doesn’t that cost a lot more compared to s3? I do like the idea of secret rotation with secrets manager

parameter store is free, unless you need the “advanced” parameter feature

secrets manager does seem rather pricey in comparson

personally i see nothing wrong with kms-encrypted s3 objects. i think it is a fantastic option. to me, really the only difference between the three are their APIs, and any integration with other aws services (for automatically retrieving/decrypting values, e.g. cloudformation + parameter store)

Parameter store and Secret manager have version for secret, use KMS for the encryption etc

SM is 3x more expensive than Parameter store

s3 buckets can be set public very easily so having secrets could be a risk

Also, Parameter Store have size restrictions so you’re often forced to use S3 anyway.

The file attribute in the config file supports wildcards I’m pretty sure. Have you tried that?

yes, didn’t seem to work for me

Do you get any errors in eb-activity.log?

no

the service runs fine

The logs just aren’t being streamed to CloudWatch

Yeah, I do remember when this happens awslogs agent should notify if the file match pattern doesn’t match. Perhaps it was in its own log file.

It’s been a while since I looked at this

Turns out it’s a bug, AWS is working on it.

Wow, ok. Thanks for updating. Is it a bug with the agent or in the EB processes?

I think EB

the way it configures the Docker env

Ok I see.

we are using, but w/o firehose. if you’re just shipping logs to HEC, wondering the need for it

How are you sending the logs to directly to hec from cloud watch?

lambda

just don’t see what you’re using firehose for

so if the logs arent being processed then they can send it to s3 in firehose

can we do that with lambda?

Figured this out. AWS is a PITA… They show / provide the private IP of the Storage Gateway instance in the console and corresponding file share mounting commands. That led me to believe AWS was only exposing the File Gateway privately inside my VPC, but that’s not the case. Using the Elastic IP that is associated with the storage gateway instance did the trick.

Not an exact answer to your question… but I’m in a similar situation using a YubiKey for programatic/cli access, and I do use it for IAM auth with RDS.

Specifically I’m using aws-vault and the YubiKey with OATH-TOTP support. More details on setting it up here: https://github.com/99designs/aws-vault/blob/master/USAGE.md#using-a-yubikey

You can do the same setup without using aws-vault as well.

For console access, I use Alfred on Mac to run the ykman cli command to generate the token and paste it in the MFA prompt.

So essentially, the easy way is to use the YubiKey with OATH-TOTP and not U2F.

that is interesting

we use aws-vault all the time

this specific user is on windows so I hope this works for him

It looks like the ykman cli is installed along with the GUI of YubiKey Manager for Windows (details), that should be all that is needed. Good luck!

I will report back

if everything else in your environment is terraform - use terraform.

if not then the tool seem solid

It’s a new setup, using variant2 for the orchestration

that seems brave. be interested to see how you go with that

Brave or something else, a fine line. I will report my experiences

hahaha good luck

TY

It’s great, don’t use cloudformation, it sucks..

eksctl is great?

Yes, if you are new comer of AWS, it’s very easy to use eksctl to create AWS networking components such as VPC and subnets.

Maybe I’m a bit late to the party, but eksctl it’s great. We’re using it for our EKS and I love it.

We use AWS extend switch roles Chrome extension

we too

and a config file for the cli

I do not however manage 100 accounts, so don’t know what the experience is at that level.

We use assume-role at the CLI

And same Chrome extension as @Tyrone Meijn

Using aws-vault exec or login depending on the need

SSO w/ all the accounts under an org master

At a previous role I used different Chrome profiles for the different IAM authentication accounts (2) and then different AWS Extend Role Switcher Config for the two profiles. You could also generate the AWS Extend Role Switcher config using a NibleText template or some other templating tool so that you can switch the config between different groups of accounts.

SSM secured string uses the aws managed KMS.

There is an option to encrypt environment variable on lambda itself. You can use KMS key to encrypt those lambda variables

why are you using shell in this particular context? try removing that part

yeah it’s just to execute the docker login command. But your approach was right. I just had to replace the

$(shell command)

with

`command |tr -cd "[:print:]"

`

the piped tr is there to cut all non printable chars from the output before execution

was a shot in the dark, but glad it helped!

sometimes the simplest solutions are hiding from us because of “experience”

What @Chris Fowles said, NACL’s provide a good way of blocking IP addresses VPC wide. Security Groups are the instrument to do the actual firewalling with.

Thanks for the input, guys, but my question was whether NACLS actually even apply to ingress on load balancers.

I have heard and like the idea of keeping NACLs to a minimum.

and a simple listener for port 80 redirected to target group on port 80
target group(instances are same but on different port 80 and 443)
instances are serving webapp on port 80 This seems mixed up, not sure if you just typed it incorrectly?

What you probably meant/want is 1 ALB rule that redirects port 80 to 443 on the ALB 1 ALB rule that forwards 443 to your targetgroup (etiher on 80 or 443 at that point, whatever the group is configured to listen on)

shuold i forword alb 443 listner to 80

now i have simple config listener 443 to target ec2 port443 listener 80 to target ec2 port 80

You want both 443 and 80 to reach the instance?

if so, yes thats the config

most setups though you redirect port 80 on the ALB to 443 so that it establishes the TLS

@Zach Thnx man its working now

Yah might be odd. Maybe this is just deployment command rather than something in the state.

https://github.com/terraform-providers/terraform-provider-aws/issues/13785

I kinda like the name. I can’t think of a better one off the top of my head.

I think “Instance Rollout” would be my name. Not saying that is much better, but “instance refresh” makes me think of updating static instances. Not that it is swapping out old instances with new ones.

Ahh, I guess I just see your point. I’m bet Product Marketing said Refresh sounds better, so they went with that.

Haha yeah likely.

Yes, if you have one cluster to manage

Yes that’s what people tended to due, or just replace the ASG entirely. The downside of those methods is that they take awhile to execute, and you double (or more) your number of running instances during the swap - which could run into account limits or temporary AWS capacity shortages.

I’ve run into this problem a few times. It’s always turned into “Data Engineering Team owns EMR”. We might try to bootstrap something for Security and watch IAM and SGs and the normal stuff, but I haven’t really found a good way to bring EMR clusters into a Central DevOps/SRE team.

End up easier that that team just hired their own DevOps/Infra person and managed stuff in that space more loosely coupled than the central team.

Yah that is what I’m trending toward. They were comfortable writing some Ansible/boto stuff to create the EMR, but we took it off their hands because we didn’t want to give them all these resource creation permissions for the SGs and IAM. That then turned into a constant “hey can you guys update the json in terraform with X”. But maybe I just let them do that and turn it into an automation job they can run

Hmmm…maybe do a Jenkins job with a bunch of build parameters to launch the cluster and me okay as it morphs away from code

I dunno. I’m my older years, I really believe in decentralized DevOps squads now, esp. in my current role. It’s super expensive to have teams with their own SRE though. Only big successful orgs can do it.

yah we’re a small company, I don’t have time to micro manage this stuff

the constant json updates reminds me of this: https://info.acloud.guru/resources/why-central-cloud-teams-fail-and-how-to-save-yours

We initially tried it as this HUGE parameterized terraform module, with a Rundeck job for it, where they could specify overrides and stuff, but it was just too much for them to figure out how certain settings mapped into what they saw in the execution. For whatever reason they understand the boto reprsentation of the Pipeline better than how it actually ends up rendered in the AWS side

Yah I dunno. I’m happy to remove it from my terraform repo and just tell them to make some python to create their pipeline and we’ll run it from a job

that might be a fair idea. create a repo they can PR too, devops approves the merge and the jenkins job fires off the apply

if they get python templating and don’t get TF, it’ll make them more productive, you still get security review with the PR review and jenkins maintains IAM applies/SG permissions

Right, mostly just need to see that they didn’t stick in “super_secret_admin_role”

there doesn’t seem to be a terraform list-entities-for-policy data source to tie into this

It’s the SSM-Agent running on those instances. It comes with Amazon Linux 2 AMIs.

You can install it yourself on other linux distros if that is what you’re looking to do:

sudo yum install -y <https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm>
sudo systemctl enable amazon-ssm-agent
sudo systemctl start amazon-ssm-agent

Ok, so just having the instance on the box is enough?

You mean the agent? Yeah. You may also need an EC2 Instance Profile with correct permissions… I forget actually

Here are the docs: https://docs.aws.amazon.com/systems-manager/latest/userguide/ssm-agent.html#sysman-install-ssm-agent

Pretty sure I’m wrong about the Instance Profile — You don’t need that. I have a module written around ssm-agents, so I should remember these things haha.

Ah, I think I’ve cracked it - some of my instances didn’t have the correct IAM policy.

Ah what was the IAM Policy they needed? I was looking back at my module and thinking they didn’t need anything…

I’m using it with cloudwatch so I needed both of these

"arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore",   "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy",

Aha yes — AmazonSSMManagedInstanceCore

Beware the AmazonSSMManagedInstanceCore policy, it gives access to all SSM parameters.

Is that not what I need to use SSM with those instances?

@Harry It is. @randomy is just saying it’s pretty permissive which can be a bad thing. If that sounds scary to you (probably if you’re in a large org) then you can look at the underlying policy and pull it apart. I’d only do that if you’re worried about security / POLP / etc within your org / team.

Ah ok, good to be aware of but I’m definitely at the scale where I’m not going to worry too much about that. I’m the only one doing any ops work here.

yep, that’s what i meant, thanks. a colleague of mine found that this policy worked: https://gitlab.com/claranet-pcp/terraform/aws/tf-aws-iam-instance-profile/-/blob/master/ssm.tf#L42-68

plus this if you want to use session manager: https://gitlab.com/claranet-pcp/terraform/aws/tf-aws-iam-instance-profile/-/blob/master/ssm.tf#L109-124

holy smokes, why on earth does ManagedInstanceCore have GetParameter[s] ? argh… sometimes i hate the aws managed policies so so much

I’m hoping I won’t have to create my own fluent container

Id like to use an environment variable if possible

Worst case scenario, I have to use a custom fluentbit configuration which I should be able to store in S3 and retrieve easily upon starting a new task

Ref: https://github.com/aws/aws-for-fluent-bit/issues/45

Are you importing your logs as JSON?

yessir

I believe that is done by using this in the firelens config

"config-file-type": "file", 
"config-file-value": "/fluent-bit/configs/parse-json.conf"

If you have it coming in as JSON, I think you need to remap the values in the log config in

In my case I had to run a simple grok pattern that parsed the log as JSON. When you do that, DD will identify all the properties of the log as metrics you can add as columns in the log explorer

I believe that is done by using this in the firelens config Yep, I was thinking about using an s3 conf file if I cannot use an env var to remap the key

Once they are separate metrics you can remap them in DD

If you have it coming in as JSON, I think you need to remap the values in the log config in Have not considered this. Would this affect all services ? I wonder if it would be better to do it in fluentbit or in datadog itself.

In my case I had to run a simple grok pattern that parsed the log as JSON. When you do that, DD will identify all the properties of the log as metrics you can add as columns in the log explorer
Once they are separate metrics you can remap them in DD that’s really good to know. I will investigate further

This is the message mapper ui. Go to Logs > Config then create a pipeline (filtering for your apps logs). All logs the filter catches will run through the pipeline and be remapped

No experience with Client VPN, and this will most likely kill the way you setup security; I think you can create a NAT instance in a routable transit gw subnet, in which you MASQ the VPN traffic through.

I think you’re looking for aws s3api put-bucket-logging

Thanks @bradym !!!!

Happy to help.

Interesting / odd that there is no way for reversing that

’Cause I want to enable logging for a couple of weeks and then disable all the logging

You use the same command. From the manpage aws s3api put-bucket-logging help:

To enable logging, you use LoggingEnabled and its children request elements. To disable logging, you use an empty BucketLoggingStatus request
element:
    <BucketLoggingStatus xmlns="<http://doc.s3.amazonaws.com/2006-03-01>" />

Hmmm interesting

Thanks again !

No implied limits other than physical limits of the networking infrastructure.

Actually I tried to pump more than 150K packets per second(pps) with 128Bytes frame size to the instance type c5a.4xlarge(supported up to 10Gbps speed) . I got only around 100K pps inside the aws instance.

we’re using a ddagent and fluentbit sidecar for each of our apps

we’re constrained by the task definition mapping of fargate services, such as 1024 cpu has to use minimum of 2048 mem

I will say they stupid answer you are not hopping for but use ECS+EC2

ah crap. ya thats not what im looking for as we’re migrating off of that haha

we’re simultaneously “rightsizing” these tasks to reduce costs

we calculated that fargate when it gets close to the limit of cpu unit and memory it is 3 times more expensive than ecs+ec2

oh wow

we run docker with 300 GB of memory

we’re in the middle of cost comparing now and willing to pay extra to forego managing our own cluster instances

Even with the recent-ish price changes?

maybe now is about 2x

I keep looking at it because I’d rather not have to manage ECS but my VP would slam the door if I told him it was 2x the cost of our EC2 instances

do your math, I could be wrong too, I did this last year

as a test with real data, you could spin up your most expensive ECS service in both fargate and ec2 and have 50% of production traffic hit 1 and hit 2

then use cost explorer to see which costs more (provided each have different tagging)

Back in Jan 2019 when they did the last price cut AWS blogged this, which is why I keep looking at it now
If your application is currently running on large EC2 instances that peak at 10-20% CPU utilization, consider migrating to containers in AWS Fargate.

sure, if it uses less than 16GB of ram and whatever is the limit of cpu units

if you are using ebs or efs need to watch that out

they just rolled out EFS support

but I do not know about EBS, I do not think is possible

Oh yah, in my case I’m using a lot of t3 nano/mico/small instances for our EC2 apps

its my goal to get the company into containers but its slow and the price of the managed services makes my bosses unhappy

Prices have gone down a couple times in last year. But new options have been added as well. Savings plans discounts apply to fargate and spot for fargate is available. We just cut $50K by moving services from fargate to fargate spot

interesting

how does one enable fargate spot with terraform ?

ah i see it https://github.com/terraform-providers/terraform-provider-aws/issues/11134

yah kind of a secret squirrel definition … their docs don’t mention what the potential names for the capacity provider are

aws_ecs_cluster has a default_capacity_provider_strategy and aws_ecs_service has a capacity_provider_strategy that can set the capacity_provider to FARGATE_SPOTcutting costs further.

yah kind of a secret squirrel definition … their docs don’t mention what the potential names for the capacity provider are yep. looks a bit confusing from the docs but that issue makes it look pretty easy

We had to create new ECS clusters to get the fargate spot capacity provider defined. Don’t know if they fixed that. But easy to setup services to use it

have you all noticed any issues betw switching from fargate back to ec2 ?

we were thinking perhaps it would be as easy as changing the launch type from fargate back to ec2

this was thought of as a failsafe if the few services that we’re POCing have a significant cost over the EC2 launch type.

There are a few changes needed. How many depend on your config on the EC2 side. But it is all straight forward and consistent. Assuming EC2 side is consistent

one thing that comes up is that before going to fargate, our ec2 containers would log to cloudwatch whereas our fargate containers use the ddagent and fluentbit containers to send logs to datadog so we’d have to remove those too

You can have fargate log to cloudwatch or use fluentbit sidecar. Again easy to change in task def. If you’re doing ddagent as a sidecar, that will also be easy. If you have it in your app image, this just got outside of terraform’s complete control

ooh, does anyone have a pointer to a KISS implementation of a elasticsearch/kibana logging sidecar?

@RB if you want a code sample for enabling Fargate Spot I can provide

Yes please! @Joe Niland

@RB

Cluster example:

resource "aws_ecs_cluster" "fargate_cluster" {
  count = var.enabled ? 1 : 0
  name  = var.cluster_name

  capacity_providers = ["FARGATE_SPOT", "FARGATE"]

  default_capacity_provider_strategy {
    base              = 0
    capacity_provider = var.default_capacity_provider
    weight            = 1
  }

  setting {
    name  = "containerInsights"
    value = var.container_insights_enabled ? "enabled" : "disabled"
  }

  lifecycle {
    create_before_destroy = true
  }
}

cluster .tfvars:

default_capacity_provider   = "FARGATE_SPOT"
container_insights_enabled  = false

ECS Service example (from https://github.com/cloudposse/terraform-aws-ecs-alb-service-task/blob/master/main.tf#L253)

# this is within resource "aws_ecs_service" "a_service" {

dynamic "capacity_provider_strategy" {
    for_each = var.capacity_provider_strategies
    content {
      capacity_provider = capacity_provider_strategy.value.capacity_provider
      weight            = capacity_provider_strategy.value.weight
      base              = lookup(capacity_provider_strategy.value, "base", null)
    }
  }

Service .tfvars:

capacity_provider_strategies = [
    {
      capacity_provider = "FARGATE_SPOT",
      weight            = 1,
      base              = 1
    }
  ]

base is minimum tasks per provider weight is a proportion, so you could put 1 for FARGATE and 1 for FARGATE SPOT and 50% of your tasks will run on each.

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cluster-capacity-providers.html

Why don’t you want to use a load balancer ?

To support a crazy architecture

But actually, we came to a different conclusion, so this is not needed anymore

(where we are putting load balancers in front, haha)

sanity prevails

Are you using a managed instance? If so then it makes sense that you wouldn’t be full admin on that DB instance.

@Joe Niland If you get a chance, check out this project. Would appreciate your thoughts if you end up getting to use it.

Not suited for your node project because why not just invoke RunTask through the native AWS JS SDK but it might help you out elsewhere!

Thanks I am having a look!

That’s a great project. I’m thinking of various ways to use it.

Awesome! Let me know how it goes and if you have any feedback!

As in update the image tag on the container def? So you don’t need to point to latest?

Right. If we point to the latest in the task definition and the latest image is updated, would it auto update the running task?

One thing we’d lose is the ability to know what version of our container is currently running. Its a bit hard to decipher with latest tag, no?

Nope — You need to invoke update-service (I think that is the API name).

And the latest tag is an anti-pattern anyway. It’s a bad practice.

Yeah, latest tag is crap. Don’t use that

Ya we dont use the latest tag

Not sure if were speaking about the same issue

So you need something to invoke after you push the image that updates the task def to use the newest image tag and then invoking update-service to update your currently running tasks.

Ah yes. I suppose we could continue using the makefile to build and push the container then replace the fabfile with the update service command to update our running task

Yeah. You still do need to update the task def to point at the new image tag. Invoking update-service is just the piece that will actually deploy that new revision of your task def.

lol then back to square one

current process

1. Makefile to create new build of docker and upload it
2. copied and pasted fab file that registers a new task definition by recreating it and then runs update service to trigger an update of the task

new process that would be cool

1. Makefile to create new build of docker and upload it
2. community maintained script that will take an existing task definition, update a specific containers container image, recreate that task definition, and run update service to update the running task

the #2 from the new process would replace our wonky, drift ridden, copy pasta fabfile

it would also allow us to maintain our task definition in terraform while the separate script can reuse its params while only replacing the specific container image

@Matt Gowie I hope this makes sense

Yeah makes sense for sure. That’d be an interesting project — Basically a tool to do a ECS deploy given a task def + service. I’ve written a bash script around that too, but just never wanted to abstract it away from reuse.

so ive been thinking more about this….

why shouldn’t we use a latest tag for the docker image in our task definition? if we did, then to update the service would be as easy as running the update-service command with --force-new-deployment

source: https://stackoverflow.com/a/48572274/2965993

we could update the docker labels in the Dockerfile itself for a git ref versioning so that could be a way to figure out what version hash is actually running

besides hitting the /status/version endpoint across out APIs

Others can explain better than I can: https://vsupalov.com/docker-latest-tag/ https://blog.benhall.me.uk/2015/01/dockerfile-latest-tag-anti-pattern/

ah ok, so i skimmed through this. so instead of calling the tag latest, we could call the tag by it’s env name like production, no ?

I just learned it the hard way on a client project where I was using latest. It was a stupid mistake on my end, but had my CI / CD pipeline for production building my image, pushing it to the registry, asking for approval to deploy to prod, and then doing deploy to prod if approved.

We had a release candidate waiting for approval and during that time, our containers had died and restarted. That inadvertently triggered the release as the task def was pointed at latest and therefore the build that was a release candidate was deployed which the client wasn’t ready for and caused problems.

would that still be regarded as an antipattern if this tagged was used in the task definition ?

ah ok i see how that can be problematic

I think that’d still be a pattern as my above problem still would’ve exposed itself.

Yeah and then rollbacks require re-tagging the old image as latest or doing a full build of the old code and pushing that as latest

Kinda weak.

i wonder if using a different tag than latest and having strict controls in place to prevent pushing those tags can allow my proposed setup to work without causing headaches of accidental deploys

Possible. One of the problems with latest is that it’s implicit. I believe images get tagged that way in most repos regardless if you explicitly tag it that way or not.

right so we can easily solve that, at the very least, by renaming it to the env like production

Or docker tags newly built images implicitly with latest. That is the issue.

then if we can put in a policy for all engineer roles to be unable to overwrite this tag, then create another iam role used by cicd that does overwrite this image tag, then it should be safer, no ?

Yeah that solves some problems. But still gives you problems with rollbacks. Means you either need to tag your rollback image with production and push it or you need to rebuild with your old code possibly.

we could tag our images twice. once with the first 7 chars of the git ref or a version number and once with the env production and push both up

that would allow us to rollback fairly easily

Yeah, and you should definitely do that regardless.

we do

i wonder if there is a project out there that creates skeleton repositories for these kinds of sane defaults so you dont have to always reinvent the wheel. someone has certainly solved this before.

I would bring this up as a #office-hours question. I think it’s a good one. I’m sure Erik and crew have opinions / war stories with this

cool. ya i havent joined the office hours yet.

The first I heard of latest being an anti-pattern was from Zach L sometime a bit back during office hours. Then it bit me.

Link this convo over there and we’ll chat through it next week. Will probably spark some interesting conversation as we obviously just chatted for a bit about it.

If you can’t join the office hours, you can always listen / watch after the fact.

@Erik Osterman (Cloud Posse) and team suggested this tool

https://github.com/fabfuel/ecs-deploy#deploy-a-new-image

and the following approach

1. setup a tag input for the terraform module to pass to the container definition
2. when building a new docker image, continue to only tag it with something unique like the first X chars of the git log ref
3. copy that new tag and pass that to the module using terraform apply -auto-approve that should be good.

we could use that tool in lieu of terraform with this command as it will only update a specific container’s image in our task definition

ecs deploy my-cluster my-service --image webserver nginx:1.11.8

Why is fargate a problem?

Is the task being killed at some point?

Hey PePe - sorry meant to say - the task will be replaced during a deployment based on the standard ECS deployment controller behaviour

also we’re using Fargate Spot in staging so that can have an impact too I think

Fargate also has a 120 second max stop timeout after which it kills the container AFAIK

So the issue is that you’re doing a service deployment and it’s killing your running jobs, which you don’t want? Or is it that you’re using Fargate Spot and the tasks are being killed because they’re spot instances and AWS is reclaiming them? Both?

Yes, both, but mainly the deployment. BTW, I am now realising that we won’t use Fargate Spot for this particular ECS service!

If you invoke via the RunTask API then it should live outside of the Service lifecycle and a deployment shouldn’t affect it.

And I wouldn’t think Fargate Spot would be the best choice for jobs that you’re hoping to keep.

How are the jobs invoked? Through a single service or are they scheduled?

It’s a node.js service managed by PM2 inside the container. The jobs are managed by Agenda and stored in Mongo. This architecture preexists AWS hosting.

Thinking about how we could orchestrate RunTask in this case.

Huh got it. When a new job gets posted is agenda just scaling a service or is it invoking RunTask?

Or is agenda a long running service and it’s invoking jobs internally — Something similar to celery or sidekiq?

Yes that

it’s just running in a loop inside the node process

I don’t think I can move away from using it within this project scope

Got it got it. Okay — This might be hard to implement but I wonder if you could spin up a new service each time, let the old service finish its currently running jobs, launch a new service which is then the only one allowed to read from Mongo / launch new jobs, and then when the old service completes its jobs it shuts itself down.

Not sure if there is an ECS native way to deal with it.

Interesting. I had a similar thought using cloudwatch events to schedule the runs. It would run and pick up whatever jobs are waiting.

From what I can see your idea (or close) could be done with external deployment controllers

AWS support sent an example of doing blue/green with Jenkins - https://github.com/aws-samples/ecs-bg-external-controller/blob/master/Jenkinsfile

Huh. I mean yeah, depending on how complex things are you could similarly update all your jobs to invoke RunTask and then they’ll spin up new tasks for each job and the jobs on the agenda server will complete quickly so you won’t have to worry about interrupting them when you do a service deploy.

I like that idea. I will look into whether we can integrate that somehow. Appreciate the input!

I think your idea is good because we don’t have to mess around with deployment controllers, which doesn’t look simple at all

Yeah that groovy code is non trivial. Ugh Jenkins, I’m glad I have no work with Jenkins right now

Hahah agreed

It’s been a while for me but I am talking to a client who may need some Jenkins pipeline updates

we have super long running job inside ECS as sidecards, not fargate but we used to set the Minimum healthy percent 100 and Maximum percent 200 and that way you can have running two versions of a task

we have sidecards as cron job basically

@jose.amengual how are you deploying the job containers?

yes

we deploy 3 containers at the same time

1 is the initial sync tas that sync s3 and then dies ( they run in order)

then we have the cron, that stays running for ever

and then the app that runs until a new deployment happens

I see and are you using the runtask API or codedeploy or something else?

basically a Go app we build that calls the standard api

it does not do any magic, it basically fires a deploy

the trick was on the deployment %

Ah I see

@jose.amengual Does your team primarily use Go?

no, we are a Java shop

I am thinking I need to do something similar because the standard ECS deploy switches the primary task as soon as the new one becomes healthy

you could have task in the same service and deploy independently ?

on service, two tasks same app

Yes we could. Sorry, I said task but I think I mean https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_Deployment.html.

I am going to play with the percentages

Thanks PePe

thank Us if it works

Actually where do you run your go app from? CI process?

jenkins yes

I ended up not using ECS deployments for this service. Instead, within CodeBuild, I downloaded the latest task definition, and used jq to modify the image name/tag. Then registered this new task definition and used run-task to create new tasks. To kill off the old tasks, my client modified the worker code to use stop-task to shut the oldest tasks down when they see that there are more tasks than MAX_TASKS (which we stored in param store.) Hack? probably, but it is working

that is not a hack

fwiw, support for this is totally separate from tf 0.13, just a new resource/feature in the aws provider. Here’s the issue to track for the release… https://github.com/terraform-providers/terraform-provider-aws/issues/13785

you’re completely right about the provider, good point

thanks for linking the issue

just set the password before removing NOPASSWD

Don’t I need to know the existing password before I can do that?

Oh.. I can sudo passwd. I had forgotten.

https://aws.amazon.com/aws-cost-management/faqs/

Q: How frequently is the AWS Cost & Usage Report updated?

The AWS Cost & Usage Report is updated at least once per day. An updated version of the report is delivered to your S3 bucket each time an update is completed.

thx @Maciek Strömich . This relates to AWS Cost & Usage Report. Do You know if the 24h also apply to AWS Cost Explorer API?

AWS Cost and Usage Report Service is part of Cost Explorer API.

my guess is that both follow the same data freshness principles

I thought:

• AWS Cost & Usage Report - reports stored on S3 - more static

• AWS Cost Explorer API - more real time HTTP API

AWS wants 0.01$ per AWS Cost Explorer API request, so I thought one could expect more

nah, dw about it

according to https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-quotas.html

Revisions per task definition family is 1,000,000

Excellent. Thanks.

since it’s a default quota you could probably request an increase. this is per TD tho so you’d have to be really abusing task definitions to have to increase it lol

Haha, yeah. :)

our choice is what we’re currently using.

we use buildkite. basically if a pr passes tests/linting and is approved by github CODEOWNERS, once it’s merged, the code is built into a container, and deployed on to ecs with zero downtime.

@nileshsharma.0311 Make a pilot project. Evaluate in a month. You need real world experience to decide. Other people’s experience can provide a guide but it is your skills that matter.

@David Medinets thanks for this one

im proficient with it. been using it for a couple years

but i do fall into snags where i need to hit up their gitter

or stackoverflow/reddit/etc

it works but it can be a pain in the ass to configure and setup

So the deployment part is kinda cumbersome

is that a question or statement

No just thinking out loud , obviously it depends on the skill level

Can you provide some input to my question , it’s in the channel, just above yours , it would be helpful

not sure how that related to cloud custodian but kk

¯_(ツ)_/¯

custodian isn’t hard to deploy. that’s the easy part. it’s hard to configure.

Gotcha

Yeah it wasn’t related but just needed some feedback

I ran it in a few minutes from a docker command without issue. Is there something more complex to it? Couldn’t you just set it to run periodically in fargate container and be done?

the deployment to a lambda or running it from the command line is dead simple

how would you rate cloudcustodian vs the services from aws like config, control-tower etc?

custodian doesnt do control tower

custodian does leverage aws config tho

yes, I know - but the tools kind of do the same

so you could roll your own with CC or go with AWS services ($$)

kindof. custodian will configure aws config for you using the yaml file. it doesn’t touch upon control tower as its not in its purview.

here are the different modes where you can configure custodian https://cloudcustodian.io/docs/aws/aws-modes.html

the only way you can use cloud custodian without aws specific services like lambda is by running it in a cron job like jenkins from the cli but you give up a lot of functionality

so you can configure your own lambdas to emulate custodian or create your own rules in aws config manually, but it’s a lot more convenient with cloud custodian because of its simplified yaml language

@David J. M. Karlsen thoughts ?

thanks for the info. I might take a closer look at CC now.