How to tighten big wires into circuit breaker holes?
No guessing is allowed. 

The breaker's labeling or specification sheet will call out a specific torque you must use.  Torque it to that spec.  And no other.  If you lack the equipment to do so, get it. 

I would say look closely inside there while you turn the screw on an empty one, so you have a clear understanding of how the mechanism works.  The wire flaring out like that suggests it may be missing the mechanism. 

Also, as Programmer66 suggests, make sure that particular hole isn't broken; try another hole.