What method does ATC use to calculate distance between two aircraft?
The things you mention would matter, if you wanted to know the distance to 6 or more significant digits. But ATC does not need that precision. They need to be able to tell whether it's 4 nmi (too close), 5 nmi (still sufficient, but getting in trouble) or 10 nmi (no worry, but should not be heading directly towards each other). That is one significant digit.

The controller does not use any formula. They just look at the distance of the targets on the radar scope (which is drawn properly to scale). And possibly use the grid that is drawn over it or compares to the leader lines (indicating velocity) or history trail. The controller doesn't need precision, they need to look and see. And they are not getting any precision; on centre station (controller handling en-route traffic) the radar target blob is about a mile wide.

Until recently, aircraft did not broadcast position at all. Many still don't; not all aircraft are equipped with ADS-B. For those that don't, the interrogator may have precision similar to DME (~ 0.1 nmi) in range and a degree or two in angle. Not great precision either. The radar also has single rotating antena that takes couple of seconds to scan the whole circle, so the position of the aircraft is only updated once every few seconds. Jet airliner in cruise moves some 700 m in 3 s.

On such radar, the scope most likely transforms directly from the angle and distance reported by the radar to the screen coordinates. Altitude was usually not taken into account (so higher targets appear further than they really are!) in the older systems.

New radars with ADS-B support (less busy areas don't need it) need to combine the radar data with coordinates reported by ADS-B, which are in WGS84, so WGS84 is usually used as intermediate format for the collected data. But shortcuts can still be taken in the software.

I don't know what which radar system manufacturer uses, but I work on navigations and we tested the precision. By assuming Earth is flat and the coordinate system is orthogonal and using simple Pythagorean formula, you can still get 3-4 significant digits (unless you are working around north or south pole) for points tens of miles apart, which is more than enough for the radar screen. So the system can simply multiply longitude and latitude with appropriate factors (that depend on latitude of the sector centre) and get sufficient precision.