BTW - I did spend a couple days trying out the i386 686 BigMem with 32-bit user land as well as the x86_64 install (which is very similar to the overall CentOS 5 64-bit install.)

With disk based activity the x86_64 and pure AMD64 systems faired about the same in overall speed; however with regard to memory and especially network speeds - the pure AMD64 install was about 25 to 33 percent faster. Plus the x86_64 installation 'seemed' to be stuck/glued to a single CPU -- like it didn't want to distribute the load among all the CPUs.

Moving Tar'GZipped files between the systems, packing and unpacking and re-packing them, and sending them back between the systems was a lot better (maybe I'm biased) when using the pure AMD64 installation.

Speaking of speed; the following are what performance parameters I decided upon after a lot of testing (up to terabyte file sizes, etc.)
# 3ware SATA II 9550SX Controller Settings -
# echo cfq > /sys/block/sda/queue/scheduler \
# Set via Kernel option "elevator=xxx" (in /boot/grub/menu.lst)
# These 3 below are set inside /etc/init.d/rc.local
echo 128 > /sys/block/sda/queue/max_sectors_kb
echo 512 > /sys/block/sda/queue/nr_requests
blockdev --setra 16384 /dev/sda

Executing sync; hdparm -t -T /dev/sda1 ; a couple of times yields these results:

/dev/sda1:
Timing cached reads: 1814 MB in 2.00 seconds = 907.13 MB/sec
Timing buffered disk reads: 442 MB in 3.01 seconds = 146.94 MB/sec

/dev/sda1:
Timing cached reads: 1814 MB in 2.00 seconds = 907.08 MB/sec
Timing buffered disk reads: 434 MB in 3.01 seconds = 144.17 MB/sec
... which is about 50% faster that using the Debian defaults, and about 15% faster than 3Ware's recommended 'deadline' scheduler...

Also please see Section 1.