How large are the inner ear hairs?
Those are just ordinary hairs growing in your ear, like the kind you find in your nose and, if you are fortunate, on the top of your head. The "hair cells" of the inner ear aren't really "hairs" at all, they are totally different, and you would never find them falling out of your ear.

Hairs like on your head are protein filaments produced by follicles in the skin.

Hair cells in the inner ear, on the other hand, are sensory epithelial cells with stereocilia, which are protrusions of the cell membrane. Those membrane protrusions have mechanically gated channels which are coupled to nearby protrusions; when the whole cell is pushed one way or the other, that influences the mechanical stress on those channels and they open and close, which produces an electrical signal that can be sensed by the auditory nerve.

These special 'hair cells' are way way deep in your ear, past your ear drum, past three little bones that conduct sound into your cochlea. If you are touching those, you are in bad shape indeed - this is very very unlikely under normal circumstances.