Can I use ChaCha20-Poly1305 as my KDF?
Since you have ChaCha20 as the cipher, it shouldn't occupy too much resource to add a BLAKE2 hash function next to it.
The BLAKE2 hash function comes in 2 variants: BLAKE2b, and BLAKE2s. ChaCha20 uses 32-bit words, which is more compatible with BLAKE2s. The only major difference between the hash and the cipher are the shift/rotate amounts, and the HAIFA hash compression function mode of operation.
While @SAI_Peregrinus have a good suggestion, I find it a little bit radical, so if you prefer a conservative approach, you can implement an additional BLAKE2 hash function to be used in any hash-based KDF you prefer.
As for your original ideal, I wouldn't recommend it as the authenticator is based on universal hashing in a biased modular integer arithmetic.