What is my bicycle flange nut size?
The thread pitch looks too coarse to be metric. Since a 3/8-16 (3/8 inch diameter with 16 threads per inch) was too large I would speculate its' a 5/16-18 (5/16 inch diameter with 18 threads per inch). The standard wrench used with this size nut is 1/2 inch. This is a common size on older American made bikes.