Can a small gauge wire be used in a fractal antenna?
Here are the effects that I would expect from switching between the coat hanger wire and the wire in CAT5:


(Obvious) The wire in CAT5 will not retain its shape as well as a coat hanger wire. It will also not be as rigid. If you were planning on fastening the whole thing to a board, this wouldn't matter.
Thinner wire will have a narrower bandwidth than thicker wire. The closer you get to an ideal wire (zero diameter, zero ohms), the higher the "Q" of the antenna. This is more significant when you are using really thick wire (or a birdcage dipole structure) and really thin wire. Additionally, I don't think your topology is very Q sensitive, so impact should be minimal.
Thinner wire will have higher resistance. Since you are receiving, impact will be minimal.


There are several other very minor effects that you might expect, but ultimately you won't notice an electrical difference, and it would be difficult to measure the difference. The 300-75 ohm transformer is only needed if the antenna is designed to be 300 ohms. Since you didn't post any documentation, I can't comment on that.