What limits the size of fish compared to marine mammals?
I'd suggest that one factor could be the difficulties of breathing water - it is highly inefficient. The following is a summary of this response to a TREE article.
Fish spend 10-30% of their energy on breathing because it is so inefficient to use gills because diffusion rates are much slower and oxygen is less abundant. Gills grow in proportionate scales to body size, generally for every 100% increase in body size there is a 50-90% increase in gill size. Thus the larger fish species are working on a lesser body mass/gill size ratio so they can't take up oxygen very easily.
From this I'd suggest it is difficult for fish to grow gill systems much larger than those of a whale shark as it would become to inefficient for them thrive. By comparison whales are air breathers so, despite &quot;breathing&quot; (collecting fresh air) less frequently they can get much more oxygen from it.
I'm sure there is more to it but at least this is a starting idea. If anyone has more then feel free to add to/edit this answer, I merely suggest this as one possible theory and it is not my field.