Mac seems to ignore the result of local DNS for local hosts
This has been driving me mad for ages, anyway, I think that I have stumbled on the solution.

The way I had set up my dynamic DNS was that serv2.njj.chickenkiller.com was an alias for njj.chickenkiller.com(probably not the best way of doing this, but it meant that serv2 handled all the requests for njj.chickenkiller.com. 

So when you query the server:

beethoven:~ nick$ host serv2
serv2.njj.chickenkiller.com has address 192.168.2.4
serv2.njj.chickenkiller.com is an alias for njj.chickenkiller.com.


So what I think is happening is that mDNSresponder was clever enough to cache njj.chickenkiller.com as an alias (and I assume that this doesn't happen on windows and linux), so when it did a subsequent DNS lookup for serv2, it tried to save time by sending a query for njj.chickenkiller.com (not serv2...), which it has cached as the alias. My local DNS server didn't recognize njj.chickenkiller.com as part of the local domain and responded by giving the external IP address.

To solve this I added njj.chickenkiller.com as an alias for serv2 in the hosts file used by dnsmasq:

# Fixed IP machines
192.168.2.1 gateway
192.168.2.2 wifi1
192.168.2.3 wifi2
192.168.2.4 serv2   njj.chickenkiller.com