#aws (2020-03)

did you see the complete working example https://github.com/cloudposse/terraform-aws-eks-fargate-profile/tree/master/examples/complete

terratest https://github.com/cloudposse/terraform-aws-eks-fargate-profile/blob/master/test/src/examples_complete_test.go

take a look at this https://github.com/cloudposse/terraform-aws-eks-fargate-profile/blob/master/test/src/examples_complete_test.go#L109

before you deploy any k8s resource to the fargate nodes (into the namespace for which the fargate profile was created), the nodes will be in Pending state

@Manuel Pirez ^

Thanks @Andriy Knysh (Cloud Posse)

I liked Salt, you could render states with jinja (unlike Ansible when i used it) and since Jinja can almost run like normal Python code and definitely can build data structures, for advanced uses you could use the json filter to output some dynamically built code that would parse as YAML.

Of course, as someone who dabbled with Clojure, I find this whole “let’s render data with text interpolation” thing shameful to our industry. Any expression-oriented dynamically typed programming language with rich data literals may be used to render heterogeneous data.

interestingly, even terraform acknowledges that: https://www.terraform.io/docs/configuration/functions/templatefile.html#generating-json-or-yaml-from-a-template

Hi, why do you need to install ruby ?

I don’t, OpsWorks agent wants

I’d have thought that would be all done in an omnibus package or similar

Been many years since used Chef (never used opsworks)

Opsworks with spot instances for k8s workers, am I reading this right ?

Agree with others here, wouldn’t go near Opsworks especially for your use case. ^^

You will loose so much time with opsworks, and it’s from a time docker wasn’t really there yet.

Why are you using Opsworks when you clearly don’t want to?

I agree with you all, I wouldn’t touch OpsWorks too

but ?

But someone decided to use OpsWorks to manage people’s SSH access while I was busy working in another team porting a particularly blobby bit of software to Python 3.

Ah ok, but then it’s a good moment to communicate that to this person that this might not be the best way forward, copy paste the slack to this person when needed.

and start again

include me in the screenshot!

OPSWORKS

Also why is ssh access needed to begin with ?

most of the time it’s unnecessary but sometimes it’s useful to SSH to strace something in a container (yes, there is probably a tool for that), or do other sorts of debugging

I don’t really SSH there most of the time

For executing inside the container kubectl exec can be used, ssh is not needed there.

you cannot really strace this or other containers from there

IIRC even if you run as the root, you cannot [normally?] strace from docker because that would circumvent security

yeah, you need --cap-add sys_ptrace

Ok but in that case you can also run telepresence from a host with trace capabilities and do it there. https://www.telepresence.io

Might look complicated, but it’s far less complicated than maintaining Opsworks

i currently can ssh, except it’s done by a 10 lines long script executed on instance init

looks like it will remain so for now because I’ve got real work to do

and i’ll later probably try out https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-connect-methods.html

https://docs.aws.amazon.com/systems-manager/latest/userguide/ssm-agent.html is worth a look too. It can be used for SSH access (look up SSM Session Manager) and also to run commands on multiple instances and get back the results (look up SSM Run Commands)

here an example of a bastion using the ssm agent

https://github.com/Flaconi/terraform-aws-bastion-ssm-iam

looks better

OpWorks is a steaming pile of trash - I’ve had to do demos of it for AWS and even in that tightly constrained scenario it rarely worked

SSM instance connect is actually worth a look though - the rest of the SSM space is pretty questionable as far as I’m concerned (except param store) but the instance connect takes away a lot of pain pretty simply - gives you iam controlled, audited connectivity from either console or cli.

I have two templates - one defining the execution role and the other the assumed role. I need to link these accounts in a trust relationship. How do I import the value of the execution role into the template for the assumed role? I had created an Outputs section in the executionrole.template, hoping to use SSM but I feel that it will get exported to the SSM in the master account while the value will be needed in the child account. So I need to pass the execution role ARN from one account to a stack running in a different account

not sure 100% but it seems that this CF feature might help you https://aws.amazon.com/about-aws/whats-new/2020/02/aws-cloudformation-stacksets-introduces-automatic-deployments-across-accounts-and-regions-through-aws-organizations/

don’t use periods in bucket names?

lol. let me just get in my time machine

have you tried using cloudfront as a workaround? you can use cloudfront to front an s3 bucket for https access

we haven’t. for now, we were waiting to hear back from amazon to see what they say while we use the old url

though the article says they are trying to come up with a solution for bucket names with periods…
Bucket Names with Dots – It is important to note that bucket names with “.” characters are perfectly valid for website hosting and other use cases. However, there are some known issues with TLS and with SSL certificates. We are hard at work on a plan to support virtual-host requests to these buckets, and will share the details well ahead of September 30, 2020.

they did say they would share the details well ahead of the sep 30, 2020 timeframe so looking for the official reply.

i quoted that same bit in the OP

reading is essential!

ive asked my TAM and am waiting to hear back

they’ve adjusted the plan once already. i’m half-expecting the date to push out. with these blog posts being a scare-tactic to get customers to come forward with issues and use cases, and start folks changing behavior earlier so there are fewer issues after the real, eventual change

my brain just exploded

lol

we also depend heavily on other company’s s3 urls and they use periods in their buckets

now we have to nag them to get them to add an access point? ridiculous…

yep

well, it sounded from the blog that if the bucket already exists then you’ll be fine. just new buckets after 30 september get impacted
Revised Plan – Support for the path-style model continues for buckets created on or before September 30, 2020. Buckets created after that date must be referenced using the virtual-hosted model.

^ maybe in the #terraform channel

We sometimes release a bug fix for earlier versions of the module for 0.11

Issue is the version went backwards I think

Also, one heads up on this module: make sure the lifecycle rules are right for you

we’re going to disable them by default in an upcoming PR, but still expose the functionality

We use IAM roles for service accounts, and associate each service account with the pods that need it. E.g. we create an IAM role with the requisite polices for external-dns then associate that with a service account that we pass to the external-dns deployment/pods.

We then do this for every other service we deploy

like cert-manager

I see so you dont use kiam or kube2iam but this newly announced feature of AWS Eks https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/

Yes, for EKS we use the new service accounts. for kops we still use kiam- haven’t explored it yet.

Service accounts are much better once you have it working.

Yep to service accounts. For multi tenant if you need hard boundaries I’d consider a separate cluster. Otherwise namespace network policy, service accounts, pod security policy, OPA on admission, etc.

I think cloudtrail would be best suited to answer both questions

So i’ve been looking into using cloudtrail for the latter. For the former I was hopeful that I wouldn’t have to do something like query all cloudtrail logs for the last time GetParameter(s) was called on each secret

You can deliver to S3 and pull the data from there into redshift.

Yeah, thought of that work around this morning before seeing the message. Seems to be the way to do it.

Hi @Alex Siegman . I think that you can achieve this using aws vpc endpoint. With Redshift in the VPC, it should be able to communicate with Firehose through the endpoint without leaving the vpc(aws backbone). I haven’t used this solution but it may work for you.

https://docs.aws.amazon.com/vpc/latest/userguide/vpc-endpoints.html

my impression was that was for outbound connectivity to services, if redshift was “Reading from” firehose, not the public firehose service delivering to redshift. Could be wrong. Might try it in a test account

Have you tried/seen kubeflow?

@Marcin Brański does kubeflow do ETL?

but yeah for ML side at least it seems good, but havent tried

Please avoid cross posting, but if you do, link to your original post. #terraform is the correct channel

got it

thanks

no ec2 instance restart should be required, it should be near immediate

I guess there are some exceptions, but still not on the EC2 side. Like if you’re using BGP on a VPN and the routes are being propagated from your edge-device, then that device may need to push updates?

To be honest I’m not super clear on those VPN / Gateway scenarios. But if you’re just talking about entirely within the context of a single VPC, you should be good

Thanks a lot for the clarification @androogle !

Wouldn’t you prefer to use task role instead of instance role?

Its been a while since I’ve messed with perf tuning Aurora but I would look at:

• instance classes you’re using since that directly impacts network throughput

• provisioned iops

• memory buffers (in cases of mysql / pgsql)

• any development usage of the replica (data-warehouse / etl / jobs)

Isn’t aurora replication done at I/O layer, have you contacted support ?

we haven’t contacted support because no degradation has been seen. this was just unnerving to see from our developers and i was wondering if any other people have noticed issues and successfully lowered replication lag

according to aws, it looks like replication lag is correlated well with write operations and we can see that side by side in the metrics

for now, we’re raising our alert from 0.2 sec to 1.5 sec since with 1 sec lag we’re not seeing issues

We removed READ replicas due to the lag on my project. The lag can mount to few seconds on heavy write operations.

@Meb so you didn’t need high availability?

We were having data state inconsistencies that blew up our workflow, so we end up removing the replica and up sizing the main cluster

I mean for READ replica’s

It was a pain the read replica’s in aurora… too much lag

I could be wrong but I don’t think you’ll find anything “standard” in this use-case for the following reasons:

• it predicates the instance is using a role (which isn’t required in general for launching an instance)

• the roles policy permissions would be pretty use-case specific

• sadly there’s no meta-data available for assigned tg’s (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-categories.html)

It might help if you provided more context around the use-case. I’d wonder why not conditionally assign the instance in whatever process is doing that

Thanks for the reply @androogle ! . We have an instance role with the proper permissions for querying the Load Balancers and Target Groups… and registering and de registering from them

what is the event that you’d like to trigger the removing of the instance from the target group?

We have a super big application with around 20 different Target Groups on Different AWS zones… so I want to create a small Python/bash script that would be executed by the Java JVM whenever there is a critical condition, so the instance can de register itself from the TG

So that script needs to know what is the TG the instance is attached to

Using the AWS CLI

https://docs.aws.amazon.com/cli/latest/reference/elbv2/deregister-targets.html

got it. I’m a python person so I’d probably do it with a mixture of metadata to get the instance-id and boto3 to get the target-group

Yep… the Instance ID is a piece of cake … getting the Target Group is more challenging … ’case we have 20+ TargetGroups

I mean.. it’s doable on either Python or Bash… I just wanted to avoid rewriting the wheel.. in case someone have already created it

yeah I follow. I was just saying because of those initial bullet-points it’d be hard for anyone to make a standardized approach to that. Its a pretty unique use-case

Yep… seems like

Out of curiosity, what happens with the instances that are failing?

Java internal issues …. OutOfMemoryExceptions

like is it possible to configure the java app to return an http code that would cause the health-check to fail and remove it from the TG ?

and other conditions …. so we want to take the instance of out the load balancer so we can perform post-mortem

Otherwise the instance is terminated by the ASG

ahhh I gotcha

Anyways … @androogle thanks for much for your interest and for the tips !

Good luck

TY … I will post the resulting script..

do you mean spot or just in-general you couldn’t launch a c5.xlarge ?

in general

I’ve never run into that. hm I wonder if there’s anything in the API

https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeInstanceTypeOfferings.html

maybe that?

hmm. is it presently happening? I don’t know if that metric is life or “general” availability

aws ec2 describe-instance-type-offerings  --region=eu-central-1 | grep -B1 -A3 c5.xlarge
        {
            "InstanceType": "c5.xlarge",
            "LocationType": "region",
            "Location": "eu-central-1"
        },

no happened mostly last two days in the mornings

but that still doesn’t narrow it down to zone

I’ve had issues where I can get certain instance types in us-east-1a but not like us-east-1e

next time it happens you could try: aws ec2 describe-instance-type-offerings --region=eu-central-1 --location-type availability-zone | grep -B1 -A3 c5.xlarge

if it doesn’t return anything for the zone its erroring on then you know you can use that call

I will take a look thanks @androogle

are these running as a service or a 1 off task?

This means they are running as a service right?

yup!

have you looked at the service / task logs when they stop?

also when you look at the stopped tasks, does it give a reason?

This is the event log in the service itself. To me it reads like ECS itself decided to drain connections and stop the containers, rather than an unexpected error occurring in the containers.

is the service mapped to a target-group?

if the target in the target-group fails health-checks, it will de-register the task

Also I’d check if the service is logging to cloudwatch (or anywhere) and check the logs of the api itself

but to your question, no thats not normal / expected

happy to help debug it with you

is the service mapped to a target-group? Yes, indeed!
if the target in the target-group fails health-checks, it will de-register the task I see in the ALB logs that there were HTTP 503 codes during the same time!

happy to help debug it with you Thank you, really appreciate that! For your information, I’m quite junior in Ops, but still the most experienced in our company.

cool np at all What I would try and do if I were in that position is find the api logs and corollate them to the 503 response codes if you can

see if its an app issue, environment issue or combo of both

if the services have been setup to use the aws logger, you may find logs in cloudwatch that are for that api service

Thanks I found the problem: from the api we ping the postgres database to see if it’s still available. The client from node-postgres had an unhandled error, leading to a crash. Seems like the database drops connections active longer than 24 hours, but I’ll have to investigate more.

yeah postgres is probably doing a schedule backup or maintenance

if you’re using a HA setup of postgres (RDS with replicas) I’d ping the cluster url, not the individual instance

hopefully soon AWS RDS Proxy (assuming you’re using RDS) will become GA and make issues like this less prevalent

We are using Aurora Serverless and I think that one is using a proxy too already. However, in de api, I make a connection with the database and perform a simple query (SELECT true ).

do you know if you’re using the cluster-url vs the ro endpoint?

that may make a difference

when maintenance period happens or a scale-up, the resource for the replicas may change, but to be honest I don’t have a lot of experience with Aurora Serverless

this is the url: bouw-staging-db.cluster-xxxxxxxxxxxx.eu-west-1.rds.amazonaws.com

hm ok yeah that looks like a cluster-url format

since this is a staging environment, do you leverage Aurora for a cost savings measure and have it turn-off after a certain period?

though if the app is constantly doing a select 1 it should reset the countdown timer to shutdown, so thats probably not it. hmm

Hmmm, it happened happened exactly 24 hours after container startup, in different envs (staging and prod).

hehe, yeah indeed.

I guess I’d poke around in Cloudwatch Metrics for that Aurora instance. Look at connection count and other things to see if it drops all connections. See if the Aurora instance has events in its event log about restarting or applying queued maintenance

has this been recurring or just a one time thing per environment?

has this been recurring or just a one time thing per environment? We deployed yesterday, looking in the logs now to see if it happened somewhere before the deployment.

I’d like to thank you for your patience and suggestions in the meantime, appreciate it a lot

np One thing I’d say to keep in mind, recently AWS updated their RDS CA certificate for secure connections to RDS

and gave time for everyone to switch over, its possible that time has passed and anyone who hadn’t switched to using the new CA had it “forced” on them

causing a restart

hm that was supposed to be March 5th, so maybe it wasn’t that

I found one entry on staging staging having the unhandled error, but it didn’t down all the tasks it seems.

for the target-group this service is registered in. Does the health-check allow for multiple failures before considered unhealthy?

or just a single one? Technically the cluster-url should be 100% available but its hard to calculate what possible edge-cases there could be

Not much, but it should be enough for a recovery.

hmm ok so if this were on the Aurora side, it’d have to be unavailable for 30+ seconds to trigger this

Uhhmm, our app is not robust in that regard: one of those unhandled errors causes the container to crash, so when the connection gets terminated, it’s an unhandled, causing a container crash.

ah ok. so if the db was unavailable at the time of check the process exits.

I’ve had similar issues, in a lot of cases we used supervisord in our containers

and that would be the “watchdog” of the process. but we also had APM and other reporting so we knew when the process crashed

SELECT pg_terminate_backend(pid) FROM
pg_stat_activity WHERE application_name = 'api-health-check';

Just ran this, killing all connections from the api and they all crashed.

ouch

gotcha so error handling and retries might be a good takeaway

haha yep Still wondering why it happened at the 24 hour mark though, but could be connection cleanup or something like that…

I would look at the Logs & Events Tab in the Aurora Serverless Instance details

and see if there were any maintenance or restart events

Cannot see anything there happening around that time. Also nothing weird in the postgres logs

I find APM’s priceless in triaging most issues like this. Depending on your companies appetite there are some reasonable options out there ranging from really cheap DIY, to arm and leg hosted in cloud

NewRelic / DataDog are pretty common

I’m a fan of Elastic APM as its simple but effective

Thanks for the suggestions, unfortunately i’m not given the time (yet…..) to look into those solutions..

np, been there. Good luck Feel free to reach out

Thanks again! Will keep that in mind

Looks like it’s related to this issue: https://github.com/segmentio/chamber/issues/251

Answering my own question, we found an Okta blog post that showed us how to pass additional Principal Tags through the SAML response that AWS can use, which allows us to get the exact username from Okta.

, means that you don’t have to create a user for database, instead use the real user who signed using Okta SAML. Using a tag are you able to control which user should have an access to which database?

No you still have to create the user on the DB, and grant it the rds_iam flag so that it can use the IAM auth. What I was then seeing was that our ‘userid’ comes in as <roleid>:<oktaname> which was super user unfriendly as a login name. Eventually we found a blog from okta that said we could enable a beta feature on our account that would let us pass extra SAML tags into AWS. Note though - this required a change to ALL trust policies for the SAML, it actually broke some of our logins for awhile

thanks for sharing it!

FWIW, I had a similar scenario and I accomplished it with SES forwarder

it required a lambda and s3 and some policies but it works (for the most part)

there’s the caveat that gmail won’t show attached .eml’s inline

but thats more of a gmail issue

so you have to download it and open it

https://aws.amazon.com/blogs/messaging-and-targeting/forward-incoming-email-to-an-external-destination/ if you’re interested

I went that route since workmail is a lot more expensive

cool…that was my other concern…thanks

ill check that out

That blog post has comments where people complain about it, and link to a github project with a lot of issues. I ended up making an SES Lambda forwarder in Python that has seen heavy use and works for the most part. I can try to open source it if there’s interest. (it’s a Terraform module, it also uses Step Functions for retries)

“heavy use” = forwarding 10-15k emails per month

does the Ningxia region have cognito? Maybe setup an identity pool and point to the AD in VA as the User source?

more of an SSO than directory replication

Hmmm let me dig into that - I appreciate the quick response!

np, I was thinking something like this - https://medium.com/@zippicoder/setup-aws-cognito-user-pool-with-an-azure-ad-identity-provider-to-perform-single-sign-on-sso-7ff5aa36fc2a

with exception of AD being in AWS instead of Azure

I do NOT believe that AWS Workspaces supports Cognito as an IdP. You need to back the “workspaces” instances with a “Directory Service”.

ahh ok workspaces. gotcha

Yeah - here are my options. -AWS Managed Microsoft AD -Simple AD -AD Connector

I’m looking to back one of these directories with an LDAP server in Virginia if possible. I think I need to open a public IP of sorts…or create a VPN between the two, but that sounds illegal…

I’m not familiar with the restrictions of that region, does it not allow things like VPN / VPC peering etc?

I don’t even see a Ningxia region in my console

Oh, the AWS China partition (Ningxia and Beijing) is completely separate from the AWS that you’re (probably) currently using. It’s a completely different login and console with limited services.

ah ok. Well routing issues aside, I suppose you could setup something like FreeIPA in China region

and have that be a peer / slave to your VA region, assuming you can solve the connectivity issue

I think you could use AD Connector w/that? I’m not sure

I’ve never used FreeIPA - I am trying to determine if that would be better than just spinning up another Windows based domain controller in the China AWS region, instead of learning and testing this new tech (FreeIPA).

yeah just regular windows might be better if you have the experience there. FreeIPA is great but there’s a curve. I wasn’t sure what the availability was for MS licensing on that region

Oh goodness…if they don’t have MS licensing then we’ve got bigger problems lol

When you say IP Whitelist, does that mean outbound from the client -> ALB ?

or inversely whitelist the clients coming over a VPN to the ALB

I think you might be able to do this with a WAF header based rule, using the host header?

I’m saying whitelisting client IPs so they’re the only ones who can access that domain.

WAF header rule is interesting. I already have a WAF ACL setup against the CF Distribution for the top 10… Will add that into my research

Yeah you could do a webacl at the ALB level, set it as a regular rule with two statements. The first to match the host-header, and the second to match the IP

whitelist by default and then blacklist if not matching that pattern maybe

or vice versa

I’ll check that out. Haha it’s funny that using the WAF for typical firewall usage didn’t come to me.

@androogle Was able to get that working via WAF rules. Thanks for the idea man!

Np!

Cluster auto scaler works great. Could set it up in a GPU node pool