When should I use a transistor, mosfet, and when should I use a relay?
Transistors (such as Bipolar Junction Transistors (BJT)) are solid state switchers, that is, they don't have any moving parts. Metal-Oxide-Semiconductor Field-Effect Transistors (MOSFET) are also transistors, but are voltage controlled, whereas classical BJTs are current-controlled. Relays are more expensive, but they have moving parts.
BJTs and MOSFETs produce a lot of heat (with high currents). They both have three pins (usually) called the base, collector and emitter (in the case of BJT) or gate, source, drain (in the case of MOSFET). More info here
Relays use a magnet to control the position of a metal switch. They generally take more power than an Arduino pin can deliver, and therefore they are often controlled by a transistor. Relays can usually switch more current than either a transistor or a MOSFET.
There are also Solid State Relays (SSR), which have a triac (a device similar to a transistor) in them. These can't switch as much as a normal relay, but they generally have better ratings than MOSFETs and BJT. example