Why isn't the biological niche occupied by whales instead occupied by fishes?
The "niche" of whales comprises much more than just being big. I would actually argue that this is just a byproduct of the marine lifestyle, since endotherms need to maintain their body temperature which is easier when you're big (lower surface/volume ratio). Some reasons for whales being very successful are:

1) Higher oxygen use efficiency:
AMR already mentioned the advantages of the mammal metabolism in the terms of oxygen intake. This allows whales to forage in great depths. Note that no fish could withstand such rapid pressure changes, since they need to dynamically adapt their swim bladder.

2) Intelligence:
Mammals are much more intelligent and therefore have access to resources that are not available for most other marine animals. E.g. hunting in groups requires a fair amount of intelligence and is much more efficient than hunting alone.

3) Acces to cold environments:
In contrast to fish, mammals can maintain high activity regardless of the ambient temperature. This gives them almost exclusive access to highly productive ecosystems like the antarctic and southern oceans.

4) Echolocation:
Toothed whales including dolphins use echolocation to communicate and sense objects. This is a very powerful ability for locating and tracking prey.