What exactly is the APN field used for, and why do some providers require it while others don't?
APN settings are necessary for data service on GSM and related networks (UMTS, LTE) to work.  It defines what network to connect to and the configuration of the network (e.g. the gateway GPRS support node address, which is the interface between the mobile network and the public internet).

APN settings are always required for data service on GSM and related networks.  The reason why you didn't need to enter them before would be either that you purchased the device from the carrier, in which case it would have been pre-configured with their APN settings, that the ROM (either stock or custom) included your previous carrier in their APN database, or your phone was able to accept automatic configuration from the carrier.

Most custom ROMs and some stock ones (in the case of direct-from-the-manufacturer devices) include an extensive database of different carrier's APNs and Android can automatically choose the correct one (Based on information from the SIM card) so you don't have to enter stuff manually.