Why does pin numbering and placement (in general) appears to be done so haphazardly to the uninformed person (me)?
I assume pin placement has some design factors in it, such as trace length and EMF, am I correct on this?

Yes, but sometimes it is more about internal routing than external signals. Whether it be tracks on a PCB, or metallization layers on the surface of a monolithic IC, the shorter the route and less vias needed the better. The more compact the layout the harder it is to get signals to pins without being blocked by other tracks, so the designer may decide to change the pin layout instead. For example the Z80 has scrambled data pins for no apparent reason, but makes sense once you understand how it is laid out internally.
But wouldn't ease of use be more important than having the shortest possible route inside the IC or module? In many cases no, because the same problem occurs on the PCB the device is put on. That nice orderly pin sequence may get messed up as soon as you try to route the signals to other places. Modern PCB layout programs take care of ensuring that signals go to the correct places, so you don't need to worry about physical pin assignments, just ensure they are correct on the schematic (where you can arrange the symbol's pin order any way you like).
MCU GPIO pins are often used for 'random' functions rather than arranged in groups, so the pin order (or even which port it is on) isn't an issue. Some MCUs have programmable pin assignments for special functions as well, allowing even more flexibility in pin routing. The PCB designer can then route signals to whichever 'random' MCU pins produce the best layout, and do the assignments in software.

For instance, I would think it would make more sense to put the VIN
next to the 5V, next to the 3.3V, next to the ground on the Arduino

Putting all the power pins right next to each is good for EMI, but bad for safety. The Arduino was designed for hobbyists with little electronics experience who are likely to make mistakes such as accidentally shorting adjacent pins while working on it. It might be better to have power pins separated by others that don't mind being shorted to them. For example if the Arduino's VIN pin was right next to the +5V pin, shorting them together would apply high voltage to the MCU which would probably destroy it.

When it comes to ESP development boards, why are the GPIO pins
usually all over the place?

Most likely for some of the reasons stated above. To keep the module size down the chip pins will most likely be routed to the nearest module pins. IC pin ordering will be influenced by signal routing on the chip itself.