Archive for November, 2007

Checking what is eating your memory….

We have several servers running > 50.000 sites. On those servers there is always 2 gb of memory free. Except sometimes when memory suddenly disappears in seconds and the system starts swapping, load jumps to > 400 in seconds after that. This happens once every few hours for a few days now.

When watching top, we see that memory of Apache2 grows to > 50%. So what site is doing that out of 50.000+? I made a script for checking that;



`mkdir memstrace`;
open(F, ">mem.log");
while(1) {
@r=`ps auxwhww|awk '{print $4,$2,$11,$12,$13,$15}'|sort -n|tail -n 5`;
foreach(@r) {
/(.*?) (.*?) (.*)/;
if ($1 > $maxmem) {
next if -f "memstrace/$2";
print F "$_n";
if(!fork()) {
`strace -f -o memstrace/$2 -p $2`
sleep 1;
close F;