WiFi high packet loss at home network
Have you scanned the WLAN frequencies to see how strong is your own AP in relation to the other ones? I used Android program called Wifi Analyzer that does that; it shows you visible APs and their channels, and how badly their signal leaks into other channels. You can also see signal strength of you AP from that.

The mobile phone often has a lot crappier Wifi antenna than laptop, just because it needs to be smaller and lighter. Also, the orientation of the device varies more.

If you problem is signal strength, or other networks overlapping yours, changing channels or installing better antenna can help a lot. In this case I direct you to look at a sister site.

And, to me your setup sounds pretty geek-normal! I, too, use a Linux home server as AP, with PCI-E and USB Wifi adapters and hostapd.