Why does capacitor polarity matter?
Not all capacitors are polarized. Electrolytics and tantalums have polarity but ceramic and film capacitors for example don't. This is because of the chemistry of the electrolyte materials used - something I don't understand sufficiently to delve further into. Film and ceramic capacitors don't use electrolytes so they don't have polarity requirements.

So why should we use polarized types when we can use non-polarized? Because both tantalums and electrolytic capacitors have more capacity for their size and can have much larger capacitance values without taking up too much PCB real estate.