What should I do to level an uneven floor with a potential load bearing wall above?
In general, load bearing walls are placed above and parallel to (with as little offset as possible) the beams below them. 

I would span another beam or mini beam parallel to the first one, to support the joists mid span.  I would start out with screw jacks (raising a new beam) and some form of force plate over your slab (steel over plywood to support and distribute the bottom of the jacks), and S l o w l y elevate.  2 inches is a lot of movement.  This elevation process should be done over months.  You may find that stabilization is all that is necessary, (or prudent). If you add 1/2 to 3/4 at your lowest point, you might be happy.

Monitor what is happening at the original beam.  You may find a gap developing there.. If that is the case, you may have to raise the original beam slightly or shim the beam to joist connection with tapered shims.   

Once you're happy with the final leveling, permanent columns (and footings below them) are the proper conclusion.