Difference in reptile inner ear and otosclerosis
Good question. To understand it perfectly, you'll need a good reference to a text on detailed explanation of sound conduction by the ear ossicles in reptiles and humans. I couldn't find anything better than these here and here.  

But these are partly inadequate in addressing your questions and are a bit involved in the physics used.
But, if we leave the physical details aside below is the possible reason as to why a single bone in case of reptile conducts just fine but in humans fails to conduct.
 The reptile ear has evolved such that it uses a single bone for conduction, while the human ear is functioning with three transmitting ossicles (the only example i could think of). This is similar to the working of a watch. You can create a time-keeping device by using a combination of three gears (old play watches) or a combination of a large number of gears (my Timex Retrogade). Both have the same function, but if we fuse several gears in the second case to equal the number in the first (hypothetically only), it will obviously stop working. In our analogy, the ear has evolved such that the sound is optimally transmitted due to several effects like, resonance, impedance, transformation and simple amplification along the ossicles, and all these functions depend very specifically on the shape and orientation of the various conducting elements of the system. And therefore, even a slight disfigurement might severely impair the conduction of sound. Therefore, had the human ear system evolved to work with one ossicle, it would have evaded osteosclerosis by being able to conduct with one ossicle also.