#aws (2020-02)

aws Discussion related to Amazon Web Services (AWS)

aws Discussion related to Amazon Web Services (AWS) Archive: https://archive.sweetops.com/aws/

2020-02-29

2020-02-28

joshmyers avatar
joshmyers

As in finding out what permissions a CFN template is gonna require to allow you to run it? No, if you find one :–1:

chrism avatar
chrism

Anyone know when AWS are going to add K8 1.15 to EKS

chrism avatar
chrism

nvm im just slow on the go

kskewes avatar
kskewes

Mad slow. :( They must have hit a serious problem updating clusters in place. Any tooling improvements to update in future would surely have been time boxed to get 1.15 out. 1.15 is almost EOL too.

chrism avatar
chrism
[EKS]: Support for Kubernetes 1.15 · Issue #380 · aws/containers-roadmap

Tell us about your request Support for Kubernetes 1.15 in Amazon EKS Upstream Release timing Changelog

Gowiem avatar
Gowiem

Hey folks — new to CloudFront usage here… I have a Django application that has 3 concerns in regards to CDN caching:

  1. CMS Uploaded Media Files — All Stored in a S3 bucket.
  2. Javascript / CSS Static Files — Served by the Django application via whitenoise (CDN cache management package).
  3. Basic HTML caching. Will have a blacklist for HTML paths that shouldn’t be cached. Now for the actual question at hand: Should I have two CloudFront distributions or one?

The two CDNs would work where all S3 content is served by one CF distribution and the other CF distribution has the application as it’s origin and it serves the Static + HTML files.

The one CDN option would serve content from both the Application and the S3 bucket.

I think one CDN would be ideal as it’s less to manage, but I’m actually just confused on if this is possible since CF has such a wide footprint in terms of configuration / usage.

Any thoughts / suggestions on a path forward here? Gonna start reading more into CF, but figured someone here would have a quick: “Yes do the second option” or “No that’s not possible, you’ll need two”.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

it could be (and we usually do it like this) one CF distribution with two Origins (one from the load balancer for the Python app, the other is for the S3 bucket), and a few behaviors. e.g. one behavior could be for the S3 origin and to cache everything, another could be for the load balancer origin for the static files (and caching), another for HTML files with possible diff caching rules, and a behavior for the blacklist path with no caching

:--1:1
Gowiem avatar
Gowiem

@Andriy Knysh (Cloud Posse) Cool — Yeah digging around and I’m starting to understand that’s possible and likely the correct route to go down. Thanks for weighing in!

2020-02-27

Eugene avatar
Eugene

Good day, community, are there any best practices for naming convention for internal hosted zones in AWS world with Route53 with consideration of ACM certificates (use public domains to approve private names ) and environments like dev or prod, regions & multiple aws accounts, clouds?

squidfunk avatar
squidfunk

Hey guys, I have a pretty annoying problem with ECS. I’m using AWS Batch which manages an ECS cluster that uses regular EC2 instances. Once a day I run ~50 batch jobs (i.e. ECS tasks) in parallel. Everything is fine, except when the container exits the task fails with CannotInspectContainerError: Could not transition to inspecting; timed out after waiting 30s. My research has brought me to believe it may be related to exhausted IOPS, but after increasing IOPS the errors keep coming in. Has anybody experienced the same?

Maciek Strömich avatar
Maciek Strömich

have you considered running those jobs on larger number of smaller instances?

squidfunk avatar
squidfunk

Not yet, but that’s a good direction to explore, thanks!

PePe avatar

I have seen that error

PePe avatar

and I think is some sort of hardware exhaustion

PePe avatar

resource exhaustion

PePe avatar

in my case was the memory soft setting in my task def

PePe avatar

in ecs +ec2

squidfunk avatar
squidfunk

Thanks @PePe , I’ll check that, too!

imiltchman avatar
imiltchman

Since we are on the topic of ECS, I got a confirmation from AWS Support today that Fargate performance is not guaranteed. We have noticed significant differences in cpu performance between regions, and Fargate also seems to be frequently outperformed by T2 instances. AWS recommendation was to switch to using EC2-backed ECS setup.

:100:1
:--1:1
imiltchman avatar
imiltchman

Good to keep in mind, especially since now Fargate is available for EKS as well

rohit avatar
rohit

does anyone know what’s the best way to develop,test and deploy AWS lambda functions ?

RB avatar

i enjoy developing in python cause its easy and i do it in vscode. i dont have to worry about memory or fast execution.

bradym avatar
bradym

I don’t know about “best” but I’m a fan of https://github.com/aws/chalice - makes it easy.

aws/chalice

Python Serverless Microframework for AWS. Contribute to aws/chalice development by creating an account on GitHub.

:--1:1
rohit avatar
rohit

i am planning to develop nodejs app

loren avatar
loren

I also use terraform with the claranet module. I think it’s way more powerful and less magical than serverless

loren avatar
loren

You can use whatever language you want, the claranet module should work with anything lambda supports, but it has some special packaging logic built-in for python. You can provide your own packaging script to the module though

loren avatar
loren

Or you can hit the easy button and just commit everything in node_modules, which is what I did here… https://github.com/plus3it/terraform-aws-slack-notifier

plus3it/terraform-aws-slack-notifier

Terraform module that builds and deploys a lamdbda function for the aws-to-slack package. - plus3it/terraform-aws-slack-notifier

randomy avatar
randomy

Cool, I wrote most of that Claranet module. I’m glad people like it. I made this one more recently which I think is better for many cases. https://github.com/raymondbutcher/terraform-aws-lambda-builder

raymondbutcher/terraform-aws-lambda-builder

Terraform module to build Lambda functions in Lambda - raymondbutcher/terraform-aws-lambda-builder

randomy avatar
randomy

It can pip/npm install remotely inside another lambda function to build your lambda package, so less setup is required for the machine running terraform.

randomy avatar
randomy

(Look at the nodejs and numpy tests to see how)

RB avatar

if the build is inside the lambda, doesn’t that take longer for the lambda to execute, which then increases your lambda costs?

randomy avatar
randomy

Nope it’s a separate lambda function that builds your one

1
randomy avatar
randomy

It builds it once (until you make changes)

randomy avatar
randomy

It’s magic

loren avatar
loren

Nifty! Didn’t realize you were in this slack @randomy!

randomy avatar
randomy

Hi, I just joined today

1
rohit avatar
rohit

what does “npm install remotely inside another lambda function” mean ?

randomy avatar
randomy

@rohit It’s explained here https://github.com/raymondbutcher/terraform-aws-lambda-builder#lambda-build-mode The module creates a 2nd “builder” lambda func using the same runtime as the one you’ve specified, then runs your build script (npm install) inside there to make your final lambda package, and stores that in s3. It then makes the actual intended lambda func using the zip in s3 that the builder func made.

raymondbutcher/terraform-aws-lambda-builder

Terraform module to build Lambda functions in Lambda - raymondbutcher/terraform-aws-lambda-builder

rohit avatar
rohit

@randomy Thanks.

rohit avatar
rohit

If i have my lambda function in a separate repository on github, how can i use your module ?

rohit avatar
rohit

I am new to Lambda functions so i don’t understand the complete picture

randomy avatar
randomy

There’s not really any way to pull in the Lambda source code from an external repo. If you can, turn that repo itself into a terraform module (put .tf files in the root dir of it).

randomy avatar
randomy

Example using the claranet lambda module, same approach works for the lambda builder module too. You don’t have to have the source in a subdir like this if you don’t want to.

https://github.com/claranet/terraform-aws-asg-instance-replacement

claranet/terraform-aws-asg-instance-replacement

Terraform module for AWS ASG instance replacement. Contribute to claranet/terraform-aws-asg-instance-replacement development by creating an account on GitHub.

nishgupta29 avatar
nishgupta29

Anyone have any experience with setting up cross account cloudwatch logging in aws ? Like sending cloudwatch logs of one account to another

Chris Fowles avatar
Chris Fowles

i don’t suppose anyone knows of a magical tool to inspect cloudformation templates and then spit out an iam policy for creating/update/deleting those resources?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I’d seen a cli/script at one point that did this

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think it worked by looking at the cloudtrail logs

2020-02-26

aaratn avatar
aaratn

Does anyone has experience setting up codepipeline for all the branches ? I want to run default pipeline for any branch the code is pushed to / created new branch

Gowiem avatar
Gowiem

Hey folks — Just realizing having 4 NAT Gateways (one for each private subnet) across 2 VPCs is costing a client 120 bucks a month (0.045/hour * 24 * 30 * 4). That’s almost a 1/3 of their bill at the moment as this is a small application that is still in Development…

That seems outrageous, but I do understand that’s nice to not have to manage a NAT instance. For those of you that have to deal with cost a lot — Do you just eat the cost or is there any cost mitigation tactic on that front?

roth.andy avatar
roth.andy

You can just run 1 NAT gateway and have all the subnets use it. The point of running extras is for high availability.

grv avatar

Exactly. moreover, do they have EIP’s assigned to NAT gateways as well? Cz they will be costing extra too

roth.andy avatar
roth.andy

The 2 VPCs thing is a bit of a curveball. Maybe run 1 NAT gateway per VPC?

Gowiem avatar
Gowiem

Hm yeah, routing both private subnet outgoing traffic through one seems like a reasonable change.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

If you are using the cloudposse modules we support a flag to use NAT instances instead which are a tiny fraction of the cost

Gowiem avatar
Gowiem

I’ll have to lookup what the EIP being assigned to each NAT is costing me. Didn’t notice that when looking through billing.

grv avatar

Yea that cost is additional

Gowiem avatar
Gowiem

@Erik Osterman (Cloud Posse) Yeah — I am using the cp modules. Was wondering if that would be recommended.

:--1:1
Gowiem avatar
Gowiem

For Dev / Stage environments… that should be fine. Don’t need the costly high availability of the Gateway.

Gowiem avatar
Gowiem

Thanks for weighing in folks!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Ya worth doing in dev/staging if cost is a concern

2020-02-25

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
AWS has launched the Activate Founders package for Startups :rocket: | Amazon Web Services attachment image

Are you in a Startup? As of today, AWS has launched the Activate Founders package for Startups! This package unlocks a new set of benefits. If your startup isn’t affiliated with a venture capital firm, accelerator, or incubator, then your startup can now apply to receive $1,000 in AWS Activate Credits (valid for 2 years) […]

2
Maciek Strömich avatar
Maciek Strömich

if you’re part of APN you can get funding via Innovation Sandbox: https://aws.amazon.com/partners/funding/

Hemanth avatar
Hemanth

looking for strategies to use in bash scripting (hope it’s fine asking here as it uses aws-cli) I got this far using jq from the json output of aws cloudwatch get-metric-statistics

Instance-ABC
19.514049550078127  
12.721997782508938  
13.318820949213313  
15.994192991030545  
18.13096421299414 

Instance-BCD
19.5140495 
12.7219977 
13.3188209
15.9941929  
18.1309642
13.3188209
15.9941929  
18.1309642 

I want to achieve

Instance above 70% 
Instance-ABC
Instance-BCD

Instances below 20% 
Instance-EFG
Instance-HIJ
Zachary Loeber avatar
Zachary Loeber

lets see your current query

Hemanth avatar
Hemanth
for i in $(aws ec2 describe-instances | jq -r '.["Reservations"]|.[]|.Instances|.[]| .InstanceId' | sort -n) do echo "Instance $i" aws cloudwatch get-metric-statistics --metric-name CPUUtilization --start-time 2019-02-20T15:00:00T --end-time 2019-02-20T18:00:00 --period 60 --namespace AWS/EC2 --extended-statistics p80 --dimensions Name=InstanceId,Value=$i | jq '.Datapoints[].ExtendedStatistics[]' done 
Zachary Loeber avatar
Zachary Loeber

maybe something like this?

Zachary Loeber avatar
Zachary Loeber
jq ‘.Datapoints[] select(.ExtendedStatistics CPUUtilization>=70)’
Zachary Loeber avatar
Zachary Loeber

sorry, not currently on aws so cannot test easily

Hemanth avatar
Hemanth

Np, thanks for your input

Zachary Loeber avatar
Zachary Loeber

did it work?

Zachary Loeber avatar
Zachary Loeber

(or was I at least close? lol)

Hemanth avatar
Hemanth

it errored, trying to figure out how to use it

2020-02-24

Pierre Humberdroz avatar
Pierre Humberdroz

sorry for the rant but why does it suck so hard to deploy an eks cluster on aws with terraform? This is the most unusable provider I have seen so far.

joshmyers avatar
joshmyers

Please elaborate

Nikola Velkovski avatar
Nikola Velkovski

I think the right answer would be, “Please vent more”

Pierre Humberdroz avatar
Pierre Humberdroz

It takes 18 different resources to deploy a single eks cluster.. With Digital Ocean I get the same in 11 lines https://github.com/helm-notifier/Terraform-Infrastructure/blob/master/01-base/digitalOceanK8s.tf with azure I can get the same in 30 lines .. I do not need autoscaling groups to get started..

helm-notifier/Terraform-Infrastructure

Contribute to helm-notifier/Terraform-Infrastructure development by creating an account on GitHub.

fast_parrot1
Pierre Humberdroz avatar
Pierre Humberdroz

Like I get that AWS is hard to get into but this is just .. not fun?

Pierre Humberdroz avatar
Pierre Humberdroz
terraform-providers/terraform-provider-aws

Terraform AWS provider. Contribute to terraform-providers/terraform-provider-aws development by creating an account on GitHub.

Nikola Velkovski avatar
Nikola Velkovski

Pierre, SaaS is always more fun than IaaS , you just have to select your poison

Nikola Velkovski avatar
Nikola Velkovski

wait till you see ECS

:100:3
Pierre Humberdroz avatar
Pierre Humberdroz

puuuuuuuuuuuuh.

Pierre Humberdroz avatar
Pierre Humberdroz

Well anyways I got it working..

Pierre Humberdroz avatar
Pierre Humberdroz

but it took me like 6 hours

Pierre Humberdroz avatar
Pierre Humberdroz

I just needed to vent I am sorry..

1
Nikola Velkovski avatar
Nikola Velkovski

2020-02-22

Maciek Strömich avatar
Maciek Strömich
How to define least-privileged permissions for actions called by AWS services | Amazon Web Services attachment image

February 21, 2020: We fixed a missing comma in a policy example. When you perform certain actions in AWS, the service you called sometimes takes additional actions in other AWS services on your behalf. AWS Identity and Access Management (IAM) now includes condition keys to make it easier to grant only the minimum level of […]

2020-02-21

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@scorebot help keep tabs!

scorebot avatar
scorebot
05:45:53 PM

@scorebot has joined the channel

scorebot avatar
scorebot
05:45:53 PM

Thanks for adding me emojis used in this channel are now worth points.

scorebot avatar
scorebot
05:45:55 PM

Wondering what I can do? try @scorebot help

2020-02-19

Michel avatar
Michel

Hey someone are already use “Provisioned Concurrency Lambda” to avoid cold start ? I have always on X-RAY some request with “Initiliazation” on Lambda…. The article : https://aws.amazon.com/fr/blogs/aws/new-provisioned-concurrency-for-lambda-functions/

New – Provisioned Concurrency for Lambda Functions | Amazon Web Services attachment image

It’s really true that time flies, especially when you don’t have to think about servers: AWS Lambda just turned 5 years old and the team is always looking for new ways to help customers build and run applications in an easier way. As more mission critical applications move to serverless, customers need more control over the performance […]

2020-02-18

2020-02-14

Hemanth avatar
Hemanth

Looking for some free tool/options/advice to generate Reports based on CPU utilization of EC2 instances, primarily to check if they cross above 80% and 10 % between certain period ? Thoughts ?

imiltchman avatar
imiltchman

Does CloudWatch+Excel not meet your use case?

Hemanth avatar
Hemanth

looking to make it automated by sending reports in an email

imiltchman avatar
imiltchman

Does it have to be a report; the alerts generated from CloudWatch are not sufficient?

2020-02-13

2020-02-12

Hemanth avatar
Hemanth

Hello, I am trying to attach an CSV file from local to email as attachment AWS CLI (SES) First i tried

cat <<EOF > ./message.json
{
  "Data": "From: [[email protected]>\nTo: <mailto:[email protected]|[email protected]](mailto:[email protected]\.com)\nSubject: Report\nMIME-Version: 1.0\nContent-type: Multipart/Mixed; boundary=\"NextPart\"\n\n--NextPart\nContent-Type: text/plain\n\nReports: report\n\n--NextPart\nContent-Type: text/csv;\nContent-Disposition: attachment; filename=\"report.csv\";\npath=\"report.csv\"\n;Content-Transfer-Encoding: base64;\n--NextPart--"
  }
EOF
cat message.json
aws ses send-raw-email --raw-message <file://message.json>

i also tried modifying

  {
"Data": "From: [[email protected]>\nTo: <mailto:[email protected]|[email protected]](mailto:[email protected]\.com)\nSubject: [Subject]\nMIME-Version: 1.0\nContent-type: Multipart/Mixed; boundary=\"NextPart\"\n\n--NextPart\nContent-Type: text/plain\n\n[Body]\n\n--NextPart\nContent-Type: text/comma-separated-values;\nContent-Disposition: attachment;\nContent-Transfer-Encoding: base64; filename=\"report.csv\";\npath=\"report.csv\";--NextPart--"
}

Both the methods didn’t work for me, not sure how to modify next to achieve what i am trying to do ?

rms1000watt avatar
rms1000watt

@Erik Osterman (Cloud Posse) does the maintainer of ssm-diff live in this slack? rofl https://github.com/runtheops/ssm-diff/pull/27

Configurable overwrite argument by rms1000watt · Pull Request #27 · runtheops/ssm-diff

This PR lets you run the command like: ssm-diff –overwrite false apply To prevent overwrites in special cases.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Doesn’t look like it!

Configurable overwrite argument by rms1000watt · Pull Request #27 · runtheops/ssm-diff

This PR lets you run the command like: ssm-diff –overwrite false apply To prevent overwrites in special cases.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I’ll be happy to reach out to him and invite to slack, if you can DM me his email

rms1000watt avatar
rms1000watt

I was trying to find his email, but then google actually pointed me back to the setup.py

rms1000watt avatar
rms1000watt

lol

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(ok, gonna delete that to avoid spam)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

invite sent!

2020-02-11

David avatar
David

I am trying to replace a bastion host that was used for port forwarding an RDS db to localhost, and replace it with an EC2 with SSM permissions. The command we used before was ssh -i ${sshPrivateKeyPath} -L ${localPort}:${remoteDbUri} -Nf ${publicBastionUri}

remoteDbUri would be something like db.private:5432, with both a private DNS name and a port.

In SSM, I found the AWS-StartPortForwardingSession document, but that won’t let me specify the db.private part I need.

Anyone know how I can do this?

maarten avatar
maarten

I battled with the same problem and created a terraform project + scripts for it. Take a look here: https://github.com/Flaconi/terraform-aws-bastion-ssm-iam

Flaconi/terraform-aws-bastion-ssm-iam

AWS Bastion server which can reside in the private subnet utilizing Systems Manager Sessions - Flaconi/terraform-aws-bastion-ssm-iam

David avatar
David

Incredible, thank you! Your module (plus looking at https://www.reddit.com/r/aws/comments/df6uip/ssm_tunnelling_ec2_what_about_rds/fhcm3e1/?context=3) got me to where I needed.

Thank you so much @maarten!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
AWS CLI v2 is now generally available | Amazon Web Services attachment image

We’re excited to announce the v2.0.0 GA release of the AWS CLI version 2 (v2). AWS CLI v2 builds on AWS CLI v1 and includes a number of features and enhancements based on community feedback. New Features The AWS CLI v2 offers several new features including improved installers, new configuration options such as AWS Single […]

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Finally an updated cli with binary distribution and built in SSO support

loren avatar
loren

Well, AWS SSO support… At least there is credential_process for others I guess

Chris Fowles avatar
Chris Fowles

This topic describes the changes in behavior between AWS CLI version 1 and AWS CLI version 2. It covers some backward-compatibility concerns and other items that might require script changes.

Chris Fowles avatar
Chris Fowles

ecr get-login has been removed and replaced with ecr get-login-password

Chris Fowles avatar
Chris Fowles

or actually

Chris Fowles avatar
Chris Fowles

RTFM The older aws ecr get-login command is still available in the AWS CLI version 1 for backward compatibility.

Chris Fowles avatar
Chris Fowles

ignore me

2020-02-10

David avatar
David

For those who use CircleCi, how do you manage rotating the IAM User credentials you supply to your CI workflows?

JJ Ferman avatar
JJ Ferman

Right now I just use the CircleCI Environment Variables for the repo

David avatar
David

Gotcha, thanks! I’m not sure that would work for my use case as we don’t want devs to be able to get the key values ever, but it’s good to know all solutions that exist

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I have thoughts on this - but haven’t implemented it

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

What I’ve wanted to do is setup a cron job (in codefresh parlance) that calls the STS API to get the short lived credentials. Then update the shared secrets on codefresh. That way if credentials leak, their validity is limited.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I assume something similar could be done on circle

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Of course, the ideal way is to have something like a runner that runs on-prem

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

in codefresh, this is venona

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

If have the runners on prem, those can run as a pod and assume a temporary role with credentials

JJ Ferman avatar
JJ Ferman

@David When you store an env var in circle CI it’s a write only operation. So you can’t see the variables once their set.

David avatar
David

My understanding was that devs could ssh onto CircleCi servers and run env to see env values. Or they could make a workflow that prints env values to the circle output

JJ Ferman avatar
JJ Ferman

ah yes, that’s true. But’s there’s nothing stopping them from doing that in the code either right?

David avatar
David

well if you use CircleCi Contexts, it ensures that devs can’t see the values.

But the sad part is that Circle has an API for updating standard env vars, but not env vars managed using contexts

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

That sucks

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

GitHub actions also doesn’t support setting secrets via API. We were trusted by that since we want to programmatically update all of our repos.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

#codefresh supports it though

:--1:1
curious deviant avatar
curious deviant

Hello .. Would you be open to using the AWS Secrets Manager or the Parameter Store ? There’s an orb that CircleCI provides. I haven’t spiked it out yet but is on my list :)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think the problem though we’re trying to solve here is how to have “short lived credentials” for AWS users

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I guess one could write short lived credentials to ASM and SSM

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but then you still need long lived credentials in Circle to access the short lived ones

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I don’t think it’s optimal

curious deviant avatar
curious deviant

I agree.. you are right

2020-02-03

Maciek Strömich avatar
Maciek Strömich

Hey, does anyone seen a boto3 behaviour where 1st request fails with unable to find credentials exception? I’ve recently started to observe this behaviour on an multi-docker elastic beanstalk environment (which until ~thursday was running flawlessly). For me it happens only when a new instance is brough up by autoscaling and worker starts to send records to firehose. What’s even more strange is that when debug was enabled to have a littlebit more verbose output in the logs for boto problem perished.

Maciek Strömich avatar
Maciek Strömich

BTW firehose credentials are supposed to be obtained from task IAM role

    keyboard_arrow_up