Some environment details first:

**Hardware:**  

Intel Server Board S2600GZ  

2 x Intel Xeon CPU E5-2620  

64GB DDR3 RAM  

Intel RAID Controller RS2BL (LSI SAS2108) with 4TB LVM Volume made of SAS disks

**Software:**  

Ubuntu 12.04.4 LTS / Linux 3.11.0-24-generic x86\_64 (with latest updates)  

qemu/KVM (libvirt) with 6 VM's (running with no problems in spite of the situation)  

glusterfs server 3.4.5 (seems to work normally too)  

some other lightweght soft (e.g. bind9, keepalived, openvpn etc.)  

*NO* custom / experimental / homegrown software!

For already a long time we have a *very strange* problem with one of our Ubuntu servers: periodically it floods the syslog with "allocation failure" messages like these:

```
Aug 28 07:00:18 srvname kernel: [4210234.157335] irqbalance: page allocation failure: order:4, mode:0x1040d0
Aug 28 07:00:19 srvname kernel: [4210234.711173] zabbix_agentd: page allocation failure: order:4, mode:0x1040d0
Aug 28 07:00:20 srvname kernel: [4210235.938599] zabbix_agentd: page allocation failure: order:4, mode:0x1040d0
Aug 28 07:00:34 srvname kernel: [4210250.307283] zabbix_agentd: page allocation failure: order:4, mode:0x1040d0
Aug 28 07:00:51 srvname kernel: [4210267.170359] irqbalance: page allocation failure: order:4, mode:0x1040d0
Aug 28 07:01:02 srvname kernel: [4210278.625530] zabbix_agentd: page allocation failure: order:4, mode:0x1040d0
Aug 28 07:01:19 srvname kernel: [4210295.671569] zabbix_agentd: page allocation failure: order:4, mode:0x1040d0

```

The messages are logged approximately every 30 secs and they do reflect the real picture: the processes shown in this log snippet are really failing (e.g. zabbix agent fails to transmit data to zabbix server). But it's just the tip of the iceberg. While this memory exhaustion is going on *any process* that needs to read the `/proc` directory (e.g. `ps`, `top`, `mpstat` etc.) crashes right after start because it fails to read it (`/proc` also can't be listed manually with `ls`) and this event is immediately logged to syslog with the same order 4 allocation failure error. 

With all this, there's enough of free RAM (1/4 of the total size), but if we check it out by blocks - the blocks of order 4 are really exhausted. *BUT*, what I really can't understand is *WHY* these processes actually do request such large blocks? We have another, almost identical (by hardware and software) server - its blocks of order 4 are exhausted too - and it feels fine, no order 4 allocation failures!
 Moreover, this identical server is under *MUCH* heavier load.

I've searched the web deeply many times for "(high order) allocation failures" symptoms, but nothing seems to be relevant. We've tried to play with various sysctl parameters (e.g. `vm.min_free_kbytes`, `vm.vfs_cache_pressure` etc., as suggested by some articles), but nothing helps. Eventually we're rolled back all these changes and now most of sysctl settings are system default. We've also tried `echo`ing to `/proc/sys/vm/compact_memory` and `/proc/sys/vm/drop_caches` without any obvious (or prolonged) effect. After some long period of exhaustion, all of a sudden and by itself, everything becomes normal (it seems like memory gets defragmented and order 4 blocks become available, `/proc` becomes available too), but not for long - after some short period all starts over again. A reboot helps for a longer period (due to the totally unfragmented memory), but eventually everything ends up the same... 

In general, the only real troubles (that we're *aware* of) caused by the described behavior is the inablity to monitor and manage server resources, neither remotely (zabbix), nor locally (`ps`, `top`, `mpstat` etc.).

As far as I understand, the lack of order 4 blocks is a *normal regular state* of memory under Linux. It's just that the processes shouldn't generally request such blocks (especially processes that *DON'T* do it on other servers). If anyone has any idea about what could be the cause of such a behavior, what can we check or where to dig - we would really appreciate it! We're ready to supply any additional info on demand.