#sre (2019-10)

Loki released 0.4.0, ~we can now do alerting on logs~i class=”em em—1”></i>

Hello everyone! I am happy to see you and I would like to ask about CloudWatch: Is there something tool for streaming/pushing logs from s3 bucket to CloudWatch? We stream logs to s3 from WAF and ALB, but I have not found any tools for streaming the logs to CW Thanks

You might consider using kinesis

Stream the logs first there

Then from there stream them to cloud watch and s3

Erik, thanks for your answer. So, we already use Kinesis for streaming logs to s3 but Kinesis does not have the ability to stream logs from s3 I think we should use some Lambda function for this purpose, the same way as DataDog uses for getting logs from customer’s s3 buckets But anyway thanks a lot

No, you don’t stream from s3

But you can output to both s3 and cloud watch logs at the same time

Which is I think what you wanted to achieve

Yes, that is I want, but looks like kinesis can not stream to CloudWatch

Hrmm

Look into AWS glue?

@Andriy Knysh (Cloud Posse) might know more

He worked on something similar

I want to implement something similar. Stream to s3 and elastic search

Yeah, I saw this manual https://aws.amazon.com/ru/blogs/security/enabling-serverless-security-analytics-using-aws-waf-full-logs/ But it brings several additional tools and looks like very overloaded

I have found this lambda function https://github.com/miztiik/serverless-s3-to-elasticsearch-ingester So, pyton code very similar to the lambda function that datadog uses) I am going to update that codebase and add additional function - streaming to CloudWatch

why do you need to move it from S3 to CloudWatch? What’s the use case?

Hi, Andriy. Nice to see you We are using AWS ALB + WAF. They can streams their logs only to s3 bucket. Also we are using CloudWatch as one point for monitoring and logging our applications and infrastructure components. So, we need some tools that can get logs from s3 and send them to CloudWatch