Archive for April, 2008

Using tcpproxy for http forwarding? Add X-Forwarded-For!

TCPProxy is one of the most beautiful tools I know of; it was nicely programmed, but above all; it has no bugs. None. We are routing millions of pageviews a day of hundreds of thousands of unique users through it and it has never let me down. It is fast, uses almost no CPU and is very very robust (I miss FTP support, but further it is totally perfect).

This unlike Apache (proxy pass) or Squid or the many other, less known, proxies I tried. Apache is especially crappy; proxy pass is riddled with bugs; under load it simply is unusable. And most of them are feature-heavy; too feature heavy; they are a pain to set up, but trivial things like not binding to 0.0.0.0 etc or simple config files they have not.

We have the necessity to route web traffic via different machines in different places and ofcourse I wanted to use tcpproxy for the task, but it has no x-forwarded-for and so we were pissing our users (with forums, blogs etc) off. So I decided to hack tcpproxy for this. And it works. Serving the millions of pages through it. With x-forwarded-for.

This was tested on 1.1.9;

In tcpproxy.c, search for these lines;


int proxy_request(config_t *x, int cfd)
{
int sfd, rc, bytes;

and change to:


int sfd, rc, bytes, first=1, i;
char *hp, *hp1;
char tmp[4096];

then search for;


else if (write(sfd, buffer, bytes) != bytes) {
syslog(LOG_NOTICE, "client write() error");

and change that to;


else {
if (first && (
(buffer[0] == 'G' && buffer[1] == 'E' && buffer[2] == 'T') ||
(buffer[0] == 'P' && buffer[1] == 'O' && buffer[2] == 'S' && buffer[3] == 'T') ||
(buffer[0] == 'P' && buffer[1] == 'U' && buffer[2] == 'T')
)) {

hp = &buffer;
*(hp+bytes)='';
hp = strstr(buffer, "rnrn");
if (hp) {
hp1 = &buffer;
i = abs(hp1-hp);
*hp = '';
memcpy(tmp, buffer, bytes+1);
strupr(buffer);
if (!strstr(buffer, "X-FORWARDED-FOR")) {
sprintf(buffer, "%srnX-Forwarded-For: %sr", tmp, x->client_ip);
hp = &tmp;
hp += i + 1;
i = bytes - i - 1;
bytes = i + strlen(buffer);
hp1 = &buffer;
hp1 += strlen(buffer);
memcpy(hp1, hp, i);
}
}
}
first = 0;

if (write(sfd, buffer, bytes) != bytes) {
syslog(LOG_NOTICE, "client write() error");

break;
}

}

Compile it and it’ll add X-Forwarded-For.

Debootstrap must be the most brilliant thing ever conceived

Are you lazy?
Are you tired of installing the same shit across Linux servers?
Are you sick of compiling the latest & greatest bag of mplayer/ffmpeg/lame on all your systems?
Are you becoming suicidal when someone asks you to send you a ‘working out of the box version’?

You don’t need to be 🙂 Install once, run anywhere (on Linux ofcourse, but what else would you run anyway huh);

Install debootstrap;

apt-get install debootstrap

I personally find it nicest to have everything in a file (so you can easily copy one file and leave it at that);

First make a loopdevice (3 gb file);

dd if=/dev/zero of=/myETCH bs=1M count=3000

Set up kernel loopdevice if you didn’t;

insmod loop.o

Set up the loop device;

losetup /dev/loop0 /myETCH

Format the loopdevice with ext3;

mkfs.ext3 /dev/loop0
mount /dev/loop0 /myETCH

then remove everything again;

umount /dev/loop0
losetup -d /dev/loop0

Your loopdevice is now ready for use!

mount it with;

mount -t ext3 /myEtch /etchinstall -o loop=/dev/loop0

If you do not want a loopback device, simply run;

mkdir /etchinstall

After this you can proceed with installing Debian etch;

debootstrap etch /etchinstall

Wait for it to be done.

Then mount proc;

mount -t proc proc /etchinstall/proc

then you can chroot it;

chroot /etchinstall

And there you are; your own nice, contained version of Debian Etch.

Install whatever you want into it. Provide your friends with the loop file /myEtch when you want to have 100% the environment as you!