How close to beam edge can I put light fixture nails?
The NEC code says that a box should be adequately supported.  This means the installer should read and follow directions provided by the box manufacturer.  

In your example the box was secured adequately just not properly.  Proper installation would only allow the necessary overhang to flush up with the ceiling.  

As for nails being too close to the edge, this should never be an issue if the installer were to follow the directions, know the NEC codes on luminare box support, and the structural integrity of the object they are attaching the box to.