If someone changes their number and you never stopped sharing your location, would the new person with that number see your location?
That information is actually not passed through the network using the phone number as the identifier. It's run through iCloud and is tied to Apple ID. The new person with the same phone number won't see your location.