What is the name for when genetic deviation results in an organism that doesn't work?
If you are referring to the mutation
I think the word you are looking for is lethal.
A lethal mutation is a mutation that causes the death of its carrier. Now there are recessive lethal and dominant lethal. For dominant lethal mutations, all heterozygotes are inviable. For recessive lethal, only double mutant homozygotes are inviable.
If you are referring to the individual
An individual that is not able to survive and reproduce is said to be inviable (I already used the term above). However, the reason for it is not necessarily genetic.