Does mysql "lock" (no other process can take from it) the memory allocated to it?
If you check kernel log, you should see OoM Killed messages.. Kernel will usually kill the largest offender which is MySQL in ourcase.