Was mammal body better suited for some type of the marine environment than the body of fish?
By asking So how did they succeed in the competition with fish? you are making the false assumption that they had somehow to succeed at some form of competition. What lead you to this false assumption is I think either of the following two mistakes..


A misunderstanding of the principle of competitive exclusion.
A false assumption about the existing overlap between marine mammals ecological niche and fishes ecological niches


Competitive exclusion

The principle of competitive exclusion states that two species occupying exactly the same ecological niche cannot stably coexist. One must either become extinct or shifts its ecological niche (process that is known as character displacement).

A consequence of the above is that if two species have ecological niches that partially overlap, they can still stably co-exist.

Ecological niche of marine mammals

There are two independent lineages of mammals that evolve marine life (cetacea and sirenia). I don't know which one you have in mind when asking the question and a sensitive complete answer would depend on the lineage of interest. However, in any case, there is such a diversity of ecological niches in fishes that it is hard to really fully address the question.

Marine mammals occupy different ecological niches. There is no question of whether one has an advantage over the other as the species do not compete for the same ressources.

A typically important reason for why marine mammals and most fishes occupy different ecological niches relate to their size differences. Simply put, mammals to be quite big! For example a orca eat fishes (and other things) that had few predators before. There is therefore not much competition involved.

Just to reiterate... in your comment you say so the main advantage was the size?. Size was one of the biggest reason why they would be able to occupy previously unoccupied ecological niches. But it is not an advantage toward competition with fishes as you seem to be thinking as there probably little competition.