Why does an acid exist when dissolved in water?
Your doubts are legitimates, but there are some aspects that you should take in account:
1-Not all acid are fully dissociated
Not all acids fully dissociate, so if you take acetic acid you have in fact an equilibrium with some acetic acid molecules not dissociated. So your question holds only for strong acid like sulfuric acid you mentioned.
2-You are using pseudo-Arrhenius definition, why don't you use Lewis acid definition?
If you think the acid as a 'substance that that releases $H^{+}$' this is still an acid even if it dissociate, and even here you should notice that acidity is linked with $H^{+}$. However if you want to dig deeper there are many definitions of acid if you use Lewis acid theory and so you think the acid as a specie that accepts electrons you can easily see that in fact that the acidity is linked mainly with $H^{+}$!