Lispy code? Prime factors of a number
Your code is perfectly fine, except for a few nitpicks.

Use push instead of (setf x (conx y x)) for readability.

Use floor instead of rem + / for efficiency.

1st and 3rd loops are identical, so they should be abstracted into a local flet function.

Use nreverse instead of reverse for the return value.  Yes, you can avoid that by using collect combined with nconc but it makes little sense in your specific case.