How to define the force at the base of an oceanic crust
Your approach of adding all the forces together is correct. You have a mistake in in the solution of the $F_{oc}$ integral, though.
You forgot to integrate the terms $\rho_{w}gh_{w}$ and $-\rho_{oc}gh_{w}$

It should read:

$F_{oc} = ... = \rho_{w}gh_{w}(h_{w}+h_{oc}) - \rho_{w}gh_{w}h_{w} + 0.5\rho_{oc}g(h_{w}+h_{oc})^{2} - 0.5\rho_{oc}gh_{w}^{2} - \rho_{oc}gh_{w}(h_{w}+h_{oc}) + \rho_{oc}gh_{w}h_{w} $

You have to keep this in mind for $F_{M}$ as well.