#monitoring (2020-1)

Prometheus, Prometheus Operator, Grafana, Kubernetes

Archive: https://archive.sweetops.com/monitoring/

2020-01-17

2020-01-16

2020-01-15

Christopher

Sorry, newbie question… If I wanted to diagnose where a memory usage is going in a PHP application, is that something a tool like New Relic APM can do for me?

Pierre Humberdroz

New Relic APM does not really show these kind of things but it gives you time spans for how long it takes for certain things to load.

Christopher

Ahh, okay.. Thanks, that probably won’t help me then. I found another [blackfire.io> which looks like it might help instead </i](http://blackfire.io)

Pierre Humberdroz

Google’s Stackdriver is the only tool that I know that does profiling on the fly of your applications and this would still mean it only works if you have a single request in a timespan to check.

Christopher

I see. I might have to come about this a different way then .. I have no idea where the problem is coming from, so cannot currently isolate it to a single request.

Pierre Humberdroz

maybe you can explain what is happening and I get an idea how you could tackle this issue

Christopher

Essentially, we have a WordPress website running on a VPS… And, every day, the log files are full of loads of “exhausted memory” errors in the logs. It only happens in production (probably because of the number of requests, or work load). There is very little to indicate what has caused it, the only mention is a file called wp-db.php which is a PHP class to interact with MySQL.

Increasing the memory limit for the process does not solve it, it just eats up all of that too.

Christopher

I’m mostly a front end developer, so I’m quite far out of my depth here, so sorry about that haha

Pierre Humberdroz

did you verify that the memory limit is higher ? with phpinfo or the likes?

Christopher

Yep

Christopher

It was previous 512M, upped it to 1G

Erik Osterman

so wp-db.php is the ORM for WP. Wouldn’t surprise me if some query is trying to load all records into memory.

Erik Osterman

How large is the blog? Are we truly certain that 1gb is enough for a poorly written query?

Erik Osterman

You mention that it only happens in production. Does staging have an equal dataset?

Christopher

The amount of content between staging & production is quite similar. It’s an e-commerce store using WooCommerce. There’s about 500 orders per day. With a total of about 180,000 orders on the site.

It’s possible that 1G is not big enough. I could up this to 4G and see if this help. I can’t imagine it needs to be more than 4G. I’ll see if that helps the situation shortly. I’ll also look to see if I can find any rogue queries loading everything in.

Thanks everyone btw

Pierre Humberdroz

I ran into a similar issue there was a Cronjob trying to generate some kind of report across all orders but this was 5 years ago

Christopher

Alright, 4G wasn’t enough either!

I’ll see if I can find some information around the queries. Perhaps I can find a pattern

Pierre Humberdroz

are you on nginx or apache2

Christopher

Litespeed

Pierre Humberdroz

does it put out logs compatible to nginx or apache ?

Christopher

Honestly, I have no idea. If I paste you one of the lines from the log file, would that help?

Christopher
2020-01-16T10:39:19+00:00 CRITICAL Out of memory (allocated 1556828160) (tried to allocate 4096 bytes) in /home/website/public_html/releases/1579104310/web/wp/wp-includes/wp-db.php on line 2007

2020-01-16T10:41:19+00:00 CRITICAL Out of memory (allocated 1561346048) (tried to allocate 58720264 bytes) in /home/website/public_html/releases/1579104310/web/wp/wp-includes/wp-db.php on line 2007
Pierre Humberdroz

looks like apache logs

Pierre Humberdroz

did you ever look at this with something like kibana and analyse the looks that way?

Christopher

That’s getting way beyond my knowledge of this, and what’s setup.

I inherited this project, mostly do front-end development work, and only get to spend 2 days a month maximum on it.

Christopher

I mostly hoped there was an easy solution if I am honest.

I appreciate all your help by the way, i’m learning a lot of new stuff based on your comments.

Erik Osterman

By the looks of it raising the memory limit did not help either because it didn’t apply (there are many ways to set it in php), it was unset or changed by something else, or the server simply doesn’t have enough ram.

Erik Osterman

Because it crashed at 1.5gb total requested, which is lower than the limit you set

Roderik van der Veer

New Relic has a PHP agent which is actually very useful. It will also show you slow queries etc: https://docs.newrelic.com/docs/agents/php-agent/getting-started/introduction-new-relic-php

Introduction to New Relic for PHP | New Relic Documentation

For an overview of New Relic’s PHP agent (compatibility, requirements, installation, configuration, troubleshooting, known issues), start here.

Roderik van der Veer

Switched technologies and i’m still missing this level of tracing in nodejs

Pierre Humberdroz

Elastic APM has the level of tracing @Roderik van der Veer for nodejs

Roderik van der Veer

It has the same limitations as the NR one. In PHP, it can show you for each request, what call (written by you, a dependency or, and this is important, php built in funcion) takes how long. The nodejs APM solutions can show you, the route, a database call. but not for example a call to nodejs crypto which takes forever.

Roderik van der Veer

but wil give it a go, because it does look nice TBH

Pierre Humberdroz

you can define custom spans in that case.

Christopher

@Erik Osterman oh, sorry i just posted an old line from the logs as an example of what they look like. It definitely did increase to 4GB as I have some entries in the log that exhausted all of that. Sorry for the confusion.

:--1:1

2020-01-02

    keyboard_arrow_up