What type of material is this and do I have a leak?
I would be more concerned about what appears to be a low spot the toilet sets in according to the first picture. It may be the floor is bulged there, and it makes it look like the toilet is sunk, but there is an issue there. It may be the root of the problem. The red stuff may be the adhesive layer of the peel and stick tile that you have now, but I don't think it's that. I would think it is the color of the next old floor down. The picture does't show it all, but it is curious how that red layer looks like it is coming up as well as the tile above it, like the joints in the tile are lined up.

Your answer to 2 and 3, is the toilet leaking swelling the floor, in my opinion.
and 4, the floor needs to come up, down to the subfloor to expose any rot, so it can be repaired.

5, Since your house is old enough to have asbestos tile and there may be many layers on the floor, yes the possibility is there. The are asbestos remediation companies out there that test and remove the asbestos if any is found. These companies are pricey in what they do, IMHO.