Neanderthal/Human Genetics
The 2-4% of Neanderthal genes refer to alleles, as you guess.  If, for a super simplified example, a human genome has

ALLELE_1__ALLELE_2__ALLELE_3__ALLELE_4__ALLELE_5__ALLELE_6__ALLELE_7

and the Neanderthal equivalent had

ALLeLE_1__ALLELE_2__ALLELe_3__ALLELE_4__ALLELE_5__ALLELE_6__ALLELE_7

(note the two lower-case 'e's), then they are 97.1% identical (67/69 characters), but if we sequenced a human and found

ALLELE_1__ALLELE_2__ALLELe_3__ALLELE_4__ALLELE_5__ALLELE_6__ALLELE_7

then we might infer that "ALLELe_3" came from Neanderthals and that that genomic region was 14.2% Neanderthal (1 of 7 alleles).  

(Obviously with only 7 alleles there would be other interpretations, but I don't feel like typing out 10,000 alleles right now.  You can see how you'd get enough information in a longer stretch of DNA.)

This is exactly the same as when you say "Half of my DNA is from my father and half if from my mother."  Obviously (one hopes) your parents were both human, and their genomes were virtually identical, but it's still possible to identify -- through allelic variation -- which segments came from which side.