How is a near miss defined?
First I'd just like to comment that "near miss" is a bad term (as falstro also noticed). It seems to say that the planes "nearly missed", implying they didn't miss. The FAA uses the terms "Near Midair Collision" (NMAC) for serious incidences, and "loss of separation" for less serious ones. As such, terms like "near miss", "close call", and "narrowly escaped disaster" are not well defined.

ATC does usually have radar. The data is recorded and can be analyzed later. Of course, not all areas have radar coverage. In these cases it is up to pilots to report if they feel safety was compromised. They can use the Aviation Safety Reporting System, which is part of the Aviation Safety Information Analysis and Sharing system.

Especially with commercial flights, Traffic Collision Avoidance System (TCAS) is also used. If TCAS issues a Resolution Advisory (RA), which commands pilots to take action to avoid a collision, the pilots are usually required to report the incident.

A "loss of separation" would be anything less than the required separation.

According the the FAA:


  A near midair collision is defined as an incident associated with the
  operation of an aircraft in which a possibility of collision occurs as
  a result of proximity of less than 500 feet to another aircraft, or a
  report is received from a pilot or a flight crew member stating that a
  collision hazard existed between two or more aircraft.


That and more information about reporting incidents can be found in Section 6 of the AIM.