Why does the sex of the parent species determines the species of a hybrid offspring
I thought that the only difference between the male and female reproductive cell's DNA code is the X-Y chromosome that determines gender, not differences that can define the species of the offspring. Why does this happen?


Well that is not the only difference, there are certain characters that are only transmitted by one of the parents.

First of all you have to think that the zygote (the cell deriving from the fusion of two gametes) contains the nucleus of the egg and of the sperm but the cytoplasm of the egg only (or at least in great part).
One thing that is present in the cytoplasm are mithocondria, which contain DNA, called mtDNA. mtDNA is therefore transmitted only by the mother (at least, in most species). Therefore a hinny will have donkey mtDNA, while a mule will have horse mtDNA.

Furthermore, cytoplasm contains transcription factors and various proteins that will influence the subsequent development of the zygote into an embryo and, again, these are maternally derived.

Another thing that changes is that there are certain genes on nuclear DNA which are expressed in a single-parent manner. That is, you have 2 copies of these genes, called imprinted genes, one from the mother and one from the father but only one of these copies -that of the mother or of the father, depending on the gene- will be expressed.