How does fast-charge work? How does Fast Charge work? 
Does it actually charge your device faster, and if so, 
Is there a downside to using it?
Fast Charging can be achieved in two ways


Hardware based Charging


Qualcomm-  Qualcomm Quick Charge 2.0 and Qualcomm quick Charge 3.0. Also see Fast Charging by Qualcomm 3.0- impact on battery?. This is the most popular, adopted by OEMs under various brand names
Type C USB Charging- USB Type C Charging explained. Also see Android phone “charging slowly”: How to make it faster
By OPPO- VOOC (Voltage Open Loop Multi-step Constant-Current Charging) and Super VOOC  

Kernels (including kernel tweaks) capable of increasing charge rate whether it is USB or wall outlet charging


First part of your question How does fast charging work? Does it actually charge your device faster

This has been covered at length here (linking it rather than copy-paste) Will plugging the Samsung fast charger into a outlet rated 10W/2.1AMP slow down the charge rate?. 

To summarise, normal charging is done at 0.5 to 0.8 C (C is charge rate) and fast charging done at 1 to 3 C. Please note that the exact or indicative charging rate is not revealed in any OEM documentation despite extensive search and it is unlikely that OEMs will reveal such  details, deeming it "proprietary"

Coming to the second part of your question Is there a downside to it (Fast Charging)?

Yes, in two ways:

A common feature while using fast chargers is raise in temperature of battery (since more current is being pumped in and power (read heat) is proportional to square of current) and this is inimical to good health of battery, which is extensively  covered here. 


Short Term (Battery capacity after charging) Battery is not really fully charged as it does not undergo charging spending enough time in all stages, compared to conventional charging. In simple words battery may deliver typically 85% of performance compared to regular charging. Details on this are available here
Long Term (Battery life span) Battery life is specified in Cycles or discharge cycles. OEMs/ Battery resellers may specify this number, for example, it is 550 Cycles for Samsung Note 2


Quoting from BU-501: Basics About Discharging


  What Constitutes a Discharge Cycle?
  
  A discharge/charge cycle is commonly understood as the full discharge of a charged battery with subsequent recharge, but this is not always the case. Batteries are seldom fully discharged and manufacturers often use the 80 percent depth-of-discharge (DoD) formula to rate a battery, meaning that only 80 percent of the available energy is delivered and 20 percent remains in reserve. Cycling a battery at less than full discharge increases service life and manufacturers argue that this is closer to a field representation because batteries are commonly recharged with some spare capacity left.
  
  There is no standard definition as to what constitutes a discharge cycle.


This does not mean that every time you charge you are consuming a cycle


  For example, if you discharge a battery to 50 percent one day, charge it back to 100 percent, then discharge it 50 percent again the next day, that is counted as one "cycle" of the battery. So shallow discharges, in all these regards, are ideal for a Li-ion battery.


Now that battery life in cycles has been explained, let us see how fast charging adversely affects long term battery life

Figure 2 from BU-401a: Fast and Ultra-fast Chargers graphically shows deterioration in long term battery life. Note deterioration in charge holding capacity and life of battery are both affected adversely



Figure 2: Cycle performance of Li-ion with 1C, 2C and 3C charge and discharge
Charging and discharging Li-ion above 1C reduces service life. Use a slower charge and discharge if possible. This rule applies to most batteries.

Bottom Line: Use fast charging when you "have to" due to time constraints else if possible stick to normal charging if you care for your battery( if you have read this lengthy answer, it's because you care:). But a relevant question is what can be done if your cell is equipped only with a fast charger? Answer would be to use a battery pack for normal use and fast charging when you really need it. This way, you could end up spending more but wouldn't need to bother about battery health

Perhaps realizing that ill effects of fast charging would be negative, Samsung has come up with user selectable enable/disable fast charge option



See FoneArena Labs : Best Fast Charging Standard in the market for performance comparison of different charging techniques