Round hole, rectangular junction box
Why not use a standard round junction box? They come in several widths and depths, for example



If the space is very shallow, there are round pancake boxes as well



Two safety issues: 


It is imperative that a fixture be solidly mounted, so, if it attaches to the box itself, the box must be attached to framing members or an adequate brace.
Small round boxes have limited space, so you must be careful about overfilling them with wires and attachment devices. If there is a single cable with two or three wires, you will usually be okay, but pancakes generally cannot handle several cables.


SUPPLEMENT: As to the question about a square box in a round hole, NEC Section 314.21 says


  Repairing Noncombustible Surfaces. Noncombustible surfaces that are broken or incomplete around boxes employing a flush type cover or faceplate shall be repaired so that there will be no gaps or open spaces greater than 3mm (1/8 in.) at the edge of the box.


Section 314.19 also says that flush fixtures have to be have to be completely enclosed at back and sides, so gaps around a box behind the fixture would probably be considered to violate that as well.

That said, it is routine to apply spackle and other noncombustible fillers around junction boxes that have gaps where they meet the wall. But it is critical to ensure that whatever box is used, it is securely mounted to underlying structure.

Finally, cites are to NEC which may or may not be applicable in your jurisdiction, but is generally viewed in North America as standard practice.


  Images and links are for illustration only, not an endorsement of
  goods or sources.