#monitoring (2019-10)

Prometheus, Prometheus Operator, Grafana, Kubernetes

Archive: https://archive.sweetops.com/monitoring/

2019-10-31

s2504s avatar
s2504s

Hello everyone! I am happy to see you and I would like to ask about CloudWatch: Is there something tool for streaming/pushing logs from s3 bucket to CloudWatch? We stream logs to s3 from WAF and ALB, but I have not found any tools for streaming the logs to CW Thanks

Erik Osterman avatar
Erik Osterman

You might consider using kinesis

Erik Osterman avatar
Erik Osterman

Stream the logs first there

Erik Osterman avatar
Erik Osterman

Then from there stream them to cloud watch and s3

s2504s avatar
s2504s

Erik, thanks for your answer. So, we already use Kinesis for streaming logs to s3 but Kinesis does not have the ability to stream logs from s3 I think we should use some Lambda function for this purpose, the same way as DataDog uses for getting logs from customer’s s3 buckets But anyway thanks a lot

Erik Osterman avatar
Erik Osterman

No, you don’t stream from s3

Erik Osterman avatar
Erik Osterman

But you can output to both s3 and cloud watch logs at the same time

Erik Osterman avatar
Erik Osterman

Which is I think what you wanted to achieve

s2504s avatar
s2504s

Yes, that is I want, but looks like kinesis can not stream to CloudWatch

Erik Osterman avatar
Erik Osterman

Hrmm

Erik Osterman avatar
Erik Osterman

Look into AWS glue?

Erik Osterman avatar
Erik Osterman

@aknysh might know more

Erik Osterman avatar
Erik Osterman

He worked on something similar

Erik Osterman avatar
Erik Osterman

I want to implement something similar. Stream to s3 and elastic search

aknysh avatar
aknysh

https://aws.amazon.com/kinesis/data-firehose/ can stream to S3 and Elasticsearch

Streaming Data Firehose - Amazon Kinesis - AWS

Process and load data streams into AWS data stores and analytics tools. Learn more here.

s2504s avatar
s2504s

Yeah, I saw this manual https://aws.amazon.com/ru/blogs/security/enabling-serverless-security-analytics-using-aws-waf-full-logs/ But it brings several additional tools and looks like very overloaded

Enabling serverless security analytics using AWS WAF full logs, Amazon Athena, and Amazon QuickSight | Amazon Web Services attachment image

Traditionally, analyzing data logs required you to extract, transform, and load your data before using a number of data warehouse and business intelligence tools to derive business intelligence from that data—on top of maintaining the servers that ran behind these tools. This blog post will show you how to analyze AWS Web Application Firewall (AWS […]

s2504s avatar
s2504s

I have found this lambda function https://github.com/miztiik/serverless-s3-to-elasticsearch-ingester So, pyton code very similar to the lambda function that datadog uses) I am going to update that codebase and add additional function - streaming to CloudWatch

miztiik/serverless-s3-to-elasticsearch-ingester

AWS Lambda function to ingest application logs from S3 Buckets into ElasticSearch for indexing - miztiik/serverless-s3-to-elasticsearch-ingester

aknysh avatar
aknysh

why do you need to move it from S3 to CloudWatch? What’s the use case?

s2504s avatar
s2504s

Hi, Andriy. Nice to see you We are using AWS ALB + WAF. They can streams their logs only to s3 bucket. Also we are using CloudWatch as one point for monitoring and logging our applications and infrastructure components. So, we need some tools that can get logs from s3 and send them to CloudWatch

2019-10-25

Jean-Michael Cyr avatar
Jean-Michael Cyr

Loki released 0.4.0, we can now do alerting on logs:–1:

    keyboard_arrow_up