How do I look at all the footprints in a KiCAD library?
In recent versions of KiCad there is a Footprint Library Browser, allowing exactly what you asked for.
It can be opened from inside PcbNew via the menu option "View >> Library Browser".