Archive for November, 2007

Simple stuff to use and remember; one liner grep and replace and when the time on your server is not ok

Grep and replace over all files in the current dit and the dirs below that;


perl -e '@f=`find . -type f`;foreach(@f){chomp;$s=`cat $_`;$s=~s/php/jsp/isgm;open(F,">$_");print F $s; close F;}';

When the time on the server is not ok, even after running ntpdate, probably your timezone is wrong. Run:

tzconfig

and set your timezone even when it looks ok already.

Preventing bugs in PHP : stripslashes

To prevent code injection, most PHP code has magic_quotes_gpc = On. For more than one reason this bad and introduces a lot of (difficult to debug) problems in large code bases. So it is better to switch that off from the get go.

If you do not have permission to switch that off (like on a lot of free hosting services), you can call the following code before doing anything else.


if(get_magic_quotes_gpc()) {
foreach($_REQUEST as $k=>$v) {
$_REQUEST[$k] = stripslashes($v);
}
foreach($_POST as $k=>$v) {
$_POST[$k] = stripslashes($v);
}
foreach($_GET as $k=>$v) {
$_GET[$k] = stripslashes($v);
}
}

Ofcourse do not forget to escape stuff that goes into your database!

Checking what is eating your memory…. Part III

Finally I found out what it was AND it was / is (using this script) a major flaw in the (gz) output handler; a user on the server uses in his code;

ob_start(“ob_gzhandler”, 9);

something();

ob_flush();

If something() generates a certain amounts of data, the ob_gzhandler will get stuck in a loop, eating all memory.

In the user his particular case, because his database was not working, some pages always have this ‘magic size’ (probably some factor of 9 in this case).

I tried this on the latest stable of PHP 5 and the bug is still there; you can crash any php hosting machine on the net with this 🙁

As to prevent this kind of thing from happening, I put the following code on the machines;


#!/usr/bin/perl

$maxmem=10;

while(1) {
@r=`ps auxwhww|awk '{print $4,$2,$11,$12,$13,$15}'|grep httpd-users|grep -v grep`;
foreach(@r) {
chomp;
/(.*?) (.*?) (.*)/;
if ($1>$maxmem) {
print "$_ killedn";
`kill -9 $2`;
}
sleep 1;
}

Update: Only PHP5.x is affected, not lower (tested); didn’t try higher. I have code to reproduce it every run.

Enterprise workflow…………….

Today the server of an important client went down. Was not reachable. Not that interesting as, with the enormous amounts of servers we are managing these days, that happens more often.

What was nice about it is the process surrounding this issue. The server was the mail relay server. So the client could no longer send mail from his website to the outside world, as this mail goes via the relay server.

Unfortunately the admin was, for a brief moment not reachable to use remote powerboot to fix it, so I uttered the, in hindsight, dangerous words;

this is a mail relay, why don’t we just set another relay for the moment?

Apparently the 10 minutes (even less…) required to change the relay in Postfix, were translated into the company (that hired me) workflow to:

1. ask John to create an issue in Jira
2. assign the issue to the person in charge of the admin work load distribution, Frank
3. Frank will check the roster of the admins and assign one; Paul
4. Paul will estimate the time needed to change the relay (a whopping 8 hours!) and assigns the issue back to Frank
5. Frank fixes the estimate by Paul by setting the estimate to estimate*PI and assigns back to John
6. John assigns the issue to sales
7. Sales talks to the client and asks for estimate*PI*120%*$100 ~ $3000
8. After approval the issue is assigned back to John
9. John assigns to Frank
10. Frank assigns to someone other than Paul (as that is how it goes)

During step 2 the admin came back and rebooted the server.

Unfortunately the above is not strange; any person working at a bigger company knows that this is actually not too bad compared to even bigger companies. In fact; Frank or Paul in steps 2,3,4 should have sent the issue back to John for lack of a clear RFP. And the estimate is actually quite low, considering someone has to write an RFP, FO and TO for this problem. And no, I am not kidding, as I see it happen every day.

Enterprises are weird entities that spend massive amounts of $ on nothing at all. Wondering if it is possible to run one without all that crud… It is a disease and I am waiting for 10 years already for IT to kick in and make at least the almost enterprises or middle sized companies a lot leaner than this. Hasn’t happen though. People still buy ERP systems for millions that are unusable, they still hire ‘enterprise Java’ guys for $200/hour and consultants for $2500/day to tell them how to work Websphere and so on. A sick, sick world.

(the names have been changed to protect the innocent)

Checking what is eating your memory…. Part II

The experiment of yesterday was showing me only apache2 doing this; I was hoping the strace would show me the path to the offending code (as I am almost 100% sure it is PHP doing it). Instead it showed me an entire file full of mmap2 which are taking away memory. No paths as the strace only kicks in when the system is already dying, so too late.

So I had to make some code to check this further:


#!/usr/bin/perl

$maxmem=10;

`mkdir memstrace`;
`rm memstrace/*`;

%detected=();

open(F, ">mem.log");
while(1) {
@r=`ps auxwhww|awk '{print $4,$2,$11,$12,$13,$15}'|grep httpd-users|grep -v grep`;
foreach(@r) {
chomp;
/(.*?) (.*?) (.*)/;
if ($1>$maxmem and !$detected{$2}) {
print F "$_n";
$detected{$2} = 1;
}
next if -f "memstrace/$2";
if (!fork()) {
`strace -f -o memstrace/$2 -p $2`;
exit;
}
}
sleep 1;
}
close F;

Which should store all Apache syscalls including the offending and then log which one is offending when it becomes that.