Born in 1772 at Eastry, Kent to Captain Harvey and Judith Harvey neé Wise, Harvey was raised with his brothers at home and in the 1780s joined his uncle Captain Henry Harvey's ship HMS Rose off the North American station to train as a midshipman.
Thanks to family influence Harvey gained a prime commission in January 1795, serving aboard his uncle's flagship the second-rate HMS Prince of Wales as captain.