Do I leave this insulation up?
Remove it unless you can verify that that plastic covering is vapor-permeable. Vapor barriers have no place in basement walls. Terrible idea. Those exterior walls should be insulated with rigid foam boards air sealed to the floor and rim joist. Then you can cover the boards up with drywall or an insulated stud wall.