Is there any benefit to sweep back on an ultralight?
There are two primary benefits of wing sweep at ultralight speeds. The first is that it acts as variable dihedral -- the higher your angle of attack, the more dihedral effect you get from sweep.

Secondly, when you see ultralights (or hang gliders) with a lot of sweep, it's generally for a tailless design that needs to get the tip fins/rudders and twisted tips and/or elevons back behind the CG location so they have a useful moment arm.

The benefits of sweep in transonic flight (delay of Mach effects onset) obviously don't apply at ultralight aircraft speeds.