Fast cryptographic hash function for short inputs
Did you try SipHash, especially the reduced-round version SipHash-1-3?
It was explicitly designed for short input, doesn’t require key expansion, is fast on pretty much all kind of architectures, and can output 64 or 128 bit.
The name might be confusing, though: a key is required, but since you mentioned that it wasn’t an issue for your use case, give it a try.