#monitoring (2019-10)

Prometheus, Prometheus Operator, Grafana, Kubernetes

Archive: https://archive.sweetops.com/monitoring/

2019-10-31

s2504s avatar
s2504s

Hello everyone! I am happy to see you and I would like to ask about CloudWatch: Is there something tool for streaming/pushing logs from s3 bucket to CloudWatch? We stream logs to s3 from WAF and ALB, but I have not found any tools for streaming the logs to CW Thanks

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

You might consider using kinesis

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Stream the logs first there

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Then from there stream them to cloud watch and s3

s2504s avatar
s2504s

Erik, thanks for your answer. So, we already use Kinesis for streaming logs to s3 but Kinesis does not have the ability to stream logs from s3 I think we should use some Lambda function for this purpose, the same way as DataDog uses for getting logs from customer’s s3 buckets But anyway thanks a lot

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

No, you don’t stream from s3

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

But you can output to both s3 and cloud watch logs at the same time

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Which is I think what you wanted to achieve

s2504s avatar
s2504s

Yes, that is I want, but looks like kinesis can not stream to CloudWatch

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Hrmm

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Look into AWS glue?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Andriy Knysh (Cloud Posse) might know more

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

He worked on something similar

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I want to implement something similar. Stream to s3 and elastic search

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

https://aws.amazon.com/kinesis/data-firehose/ can stream to S3 and Elasticsearch

Streaming Data Firehose - Amazon Kinesis - AWS

Process and load data streams into AWS data stores and analytics tools. Learn more here.

s2504s avatar
s2504s

Yeah, I saw this manual https://aws.amazon.com/ru/blogs/security/enabling-serverless-security-analytics-using-aws-waf-full-logs/ But it brings several additional tools and looks like very overloaded

Enabling serverless security analytics using AWS WAF full logs, Amazon Athena, and Amazon QuickSight | Amazon Web Services attachment image

Traditionally, analyzing data logs required you to extract, transform, and load your data before using a number of data warehouse and business intelligence tools to derive business intelligence from that data—on top of maintaining the servers that ran behind these tools. This blog post will show you how to analyze AWS Web Application Firewall (AWS […]

s2504s avatar
s2504s

I have found this lambda function https://github.com/miztiik/serverless-s3-to-elasticsearch-ingester So, pyton code very similar to the lambda function that datadog uses) I am going to update that codebase and add additional function - streaming to CloudWatch

miztiik/serverless-s3-to-elasticsearch-ingester

AWS Lambda function to ingest application logs from S3 Buckets into ElasticSearch for indexing - miztiik/serverless-s3-to-elasticsearch-ingester

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

why do you need to move it from S3 to CloudWatch? What’s the use case?

s2504s avatar
s2504s

Hi, Andriy. Nice to see you We are using AWS ALB + WAF. They can streams their logs only to s3 bucket. Also we are using CloudWatch as one point for monitoring and logging our applications and infrastructure components. So, we need some tools that can get logs from s3 and send them to CloudWatch

2019-10-25

JMC avatar

Loki released 0.4.0, we can now do alerting on logs:–1:

    keyboard_arrow_up