Why are propeller engines uncommon on airliners?
Well, first let's clear up a few terms: 
When you say "rotary" engine I'm assuming you're referring to radial engines, a type of piston engine that used to be pretty common on aircraft. (These days opposed piston engines are what you typically find on piston-powered aircraft, rotary engines are yet another design, but their usage died out around the end of World War I.)



The TU-95 is not actually a piston-powered aircraft.  It's a turboprop – basically a turbine engine similar to what you'd find in a jet, only rigged up to turn a propeller rather than produce "jet thrust" directly.

From an efficiency standpoint, turbine engines are usually more fuel efficient than their piston counterparts, and jet fuel is denser and less refined than aviation gasoline and is consequently cheaper for operators to procure. Turbine engines also offer more reliability than piston engines, and the maintenance on a turboprop engine is also largely similar to a jet engine with a few extra components, which is an advantage for a company operating a fleet of jet and propeller driven aircraft.

The differences in operating efficiency and reliability are the major reason why gasoline-powered piston engines have basically disappeared from scheduled airline service.



So why don't we see more turboprops? Actually we see a lot of them, if you look in the right places.

Jets and Turboprops are good at different things: broadly simplifying, a turboprop is more efficient at lower altitudes and airspeeds while a jet engine is more efficient at higher altitudes and airspeeds.  

As a result we see turboprop aircraft like ATR 72s in use for short-haul "commuter" service, but for trans-continental or trans-oceanic flights where they spend a long time cruising at high altitude jets dominate the sky.

Since most people are flying to go relatively long distances there are comparatively more jets in scheduled airline service than turboprops.

Noise is probably also a factor: fast turboprops like the Bear are LOUD not due to the engine, but due to the propeller.  The tips of the spinning propeller on a TU-95 can approach supersonic speeds, which causes quite a bit of noise. The TU-95's contra-rotating propellers (which help produce thrust more efficiently) also contribute to a louder noise footprint.  In the case of the TU-95 this doesn't matter – it's a military plane, and the Russian air force doesn't care if people complain as the aircraft has a mission to complete and that's more important than a few noise complaints.  If United Airlines were to operate a TU-95 out of Kennedy departing over people's houses I suspect they would quickly reconsider their choice of equipment when the noise complaints started coming in....