Why is there dirt in my wall (by the tub)?
The dirt you have removed is probably from around the tubs drain pipe.  When the slab floors are poured all the plumbing has already been installed. When pipes protrude from the floor the contractor builds a walled surround that keeps the concrete away from the pipes. This is due to: 1) it is easier to have space to position a pipes' final connection and 2) depending on the alloy's used the alkalis' in the cement can corrode the pipe. If you enlarge the wall opening and using a flashlight you should see the tubs overflow and drain pipe going straight down into this dirt you excavated. Another possibility is the one you mentioned; it was fill dirt used to level the cast tub. Building codes aren't specific regarding what material to use when leveling a cast tub, as long as it is leveled and stabile. If the soil isn't in contact with any framing members and isn't annoying you leave it. If you feel more comfortable without it inside your walls than open the wall and remove it in 5 gallon buckets. Don't remove any dirt from under and close to the tub bottom. Watch for plumbing if you excavate.