Why are zero fee transactions not being included in blocks that are not full?
It likely has to do with the minimum relay fee set on the nodes that are mining. There's also coin age that is taken into consideration on low or no fee transactions. 

Relay and Mining: Priority transactions

Bitcoin Core has a heuristic ‘priority’ based on coin value and age. This calculation is used for relaying of transactions which do not pay the minimum relay fee, and can be used as an alternative way of sorting transactions for mined blocks. Bitcoin Core will relay transactions with insufficient fees depending on the setting of -limitfreerelay= (default: r=15 kB per minute) and -blockprioritysize=.

Soure