My circular old work electrical box is too small for the light fixture mounting bracket
Edit: Sorry, trim the screws

In electrical, screws are not sacrosanct.  If one is too long, you simply trim it.  That's why your stripper-crimper multi-tool has thread shears around the center pivot.  

Wrong holes

This box has 2 sets of holes at 2 different spacings.  

 The inner (closer) set of holes sit on inward sticking nubs.  Those are the ones you are trying to line up on.  

 The outer (farther) set of holes sit pretty much on the rim of the box. They are 90 degrees from the other holes.  Try those. 



The reason for the two spacings is the two sizes of steel octagon box (pictured). 


The older, now largely obsolete 3-1/2" trade size (3-1/2" across corners, 3" across flats) had 2-3/4" screw spacing. 
The current 4" trade size (4" across corners, 3-1/2" across flats) has 3-1/4" screw spacing.