Trying to find correct bolt for ceiling fan electrical box
As Ed said, it's almost certainly an 8-32, with a slim chance of a 10-32. My guess is the screws aren't working because the threads got damaged by the old incorrect screws. This is most likely why it was suggested you replace the box. If the threads are damaged I would probably replace the box. It's a chore but is the best option.
The alternative (which may violate the UL listing and/or NEC which are there for your safety) would be to re-tap the holes at a larger size than the damage. This could be done with self forming screws or with a drill and tap set. However, this will almost certainly result in a weaker connection and given this is an overhead box holding something up above people's heads and someone could come along and do something later like install a ceiling fan I would not recommend it.