Species with reproduction barriers that can both reproduce with a third species
How are such species are defined, and at what point dogs stop being
  dogs anymore?


This is a bit like the is-Pluto-a-planet-discussion. A group of scientists have to come together and hold a big conference.
You have a few principles that you want to adhere to and then it's big groups of people making decisions.