system swaps before the memory is full
One thing to take into account for the huge numpy array is that you are not touching it. Memory is allocated lazily by default by the kernel. Try writing some values in that huge array and then check for swapping behaviour.
My program plots a large number of lines (~200k) with matplotlib which is pretty greedy for memory. I usually have about 1.5G of free memory before plotting. When I show the figures, the system starts swapping heavily when there's still about 600-800M of free RAM. This behavior is not observed when, say, creating a huge numpy array, it just takes all the available memory instantaneously. It would be nice to figure out whether this is a matplotlib or system problem.
I'm using 64-bit Arch Linux. 
UPD: The swapiness level is set to 10. Tried setting it to 0, as DoctororDrive suggested, but same thing. However, other programs seem to be ok with filling almost all the memory before the swap is used.