Why is there still a preference for turboprop airliners over the new regional jets?
The simple answer is turboprops are more profitable on short stage lengths, say up to 400 miles or so, where the limited ceiling and slower speed are less of a detriment and the turboprop's better seat mile fuel burn is a bigger factor.  This allows lower ticket prices. Once you are getting out toward 1000 miles or more, it becomes worthwhile to go higher for both efficiency and weather avoidance ability, and jets start to become more profitable.

The -8 400 extends this somewhat and can almost match jets on moderate stage lengths on total trip time, so it bridges the gap between TPs and jets to a degree.  It's noise and vibration cancellation system helps this a bit by making the trip a bit less unpleasant.

There are other complications.  Depending on the market, pax may be willing to pay more for jet service on short trips, especially in affluent markets.  There are RJ operators in the US that run CRJ200s or 900s on very short runs that would be way cheaper served by TPs because their customer base demands jets and it's worth running those routes at razor thin margins to keep the customer base.

Other factors include things like maintenance sophistication.  Older TPs require less electronics-savvy technical expertise and a small operator having great difficulty retaining mechanics trained to figure out software based control systems will be attracted to simplicity.   

One of the biggest drawbacks to the -8 400, and which in truth is killing it vs its ATR counterpart in sales, is the level of electronics sophistication is high compared to other turboprops, nearly jet-like.  When Canadian regional Air Canada Jazz acquired the -8 400, their -8 technical team was warned by their RJ technical team to get ready because the airplane was closer to a CRJ900 maintenance wise (systems run by black boxes vs hydromechanical components that any wrench turner can figure out) than an older -8.  The training level of the technicians becomes critical in achieving decent dispatch reliability.

TP vs C Series (soon to be renamed the Airbus somethingorother)?  Same thing. Depends on the mission requirement.  The C Series' competition is other jets, not turboprops.