Is it safe to use fast/small hash for key identity?
In general, you never want to use CRC/weak checksum for any computations on secret material (like keys). CRC is a linear function and by showing CRC of a key, you reveal a lot of equations that hold among the key bits. This is equivalent to showing the same number of bits of the key as the length of the checksum.

The proper way of doing it has been suggested by @puzzlepalace: Take a cryptographic hash function (like SHA-2), compute the hash of the key and take only 32-bits of the digest.

Key fingerprint of 32-bits may be too small: if you have ~2^16 keys, you will get different keys that give the same fingerprint. You need to take this into account when designing your system.