Which trigonometric algorithm does an FMS use to determine distance between waypoints?
Short answer

Aeronautical data (fixes, waypoints, navaids, instrument procedures) are delivered to aircraft operators and avionics manufacturers referenced to the WGS-84 ellipsoid. 

To make is simple the ellipsoid is the volume enclosed by the revolution of an ellipse which best approximates the shape of the Earth, generally in a given region only. WGS ellipsoid is a global one which approximates the whole Earth shape.



FAA provides rules for designing procedures, and algorithms to use to compute various elements, including the distance between two fixes and their relative azimuths. This is described in Order 8260.54 The United States Standard for Area Navigation (RNAV), Appendix 2. TERPS Standard Formulas for Geodetic Calculations:


  1.0 Purpose
  
  The ellipsoidal formulas contained in this document must be used in
  determining  RNAV flight path (GPS, RNP, WAAS, LAAS) fixes,
  courses, and distance between fixes. Notes: Algorithms and methods
  are described for calculating geodetic locations (latitudes and 
  longitudes) on the World Geodetic System of 1984 (WGS-84)
  ellipsoid, resulting from intersections of geodesic and non-geodesic 
  paths. These algorithms utilize existing  distance and azimuth
  calculation methods to compute intersections and tangent points 
  needed for area navigation procedure construction. The methods apply
  corrections to  an initial spherical approximation until the error is
  less than the maximum allowable error, as specified by the user.


[...]


  3.4 Direct and Inverse Algorithms.
  
  The Direct and Inverse cases utilize formulae from T. Vincenty’s,
  Survey Review XXIII, No. 176, April 1975: Direct and Inverse Solutions
  of Geodesics on the Ellipsoid with Application of Nested Equations.


FMS use ellipsoid geometry to compute distances and angles, in order to match aeronautical data. 

A visible consequence is that, except for meridians and equator, prlongating a geodesic on an ellipsoid doesn't draw a closed curve, as if it were a great circle:





Detailed answer: Need for WGS-84

New modes of navigation have emerged with a significantly better accuracy than using VOR-DME and inertial:


RNAV (area navigation) generally uses GNSS (global navigation satellite system, one GNSS being the US GPS),
RNP (required navigation performance),
and more generally performance-based navigation (PBN).


At the same time these new direct modes of navigation (compared to a navaid-to-navaid paths) can define paths extending to thousands of NM, and commonly crossing different geodetic reference systems where latitude, longitude, height and vertical can have significantly different values.

As an illustration, the measure of "height" used to be referenced to local MSL (mean sea level):


(source)

But a GPS receiver, which is now the main instrument to locate a point, can only provide the height above an ellipsoidal simplified Earth model. Latest receivers also provide the height above the simplified sea level model (geoid), but only as the result of a transformation of the ellipsoidal height, etc.


(source)

As visible above, the direction of the local vertical is impacted by the transformation, creating another problem for airborne vehicles navigation.

This can quickly become a nightmare with differences for the same point up to 100m depending on the reference used:

For these reasons, it appeared very early that all elements involved in mapping, navigation and ATC must use the same datum, in all countries. ICAO has selected WGS-84.

ICAO Doc9674:


  The Council [...] on 28 February 1994, adopted Amendment 28 to Annex 15,
  introducing the provisions concerning the promulgation of WGS-84-related
  geographical coordinates. [...]


WGS-84 on Wikipedia:


  The WGS 84 datum surface is an oblate spheroid (ellipsoid) with major
  (equatorial) radius a = 6378137 m at the equator and flattening f =
  1/298.257223563. The polar semi-minor axis b then equals a times
  (1−f), or 6356752.3142 m.


Current country compliance with ICAO selected datum: See Jeppesen WGS-84 Status Report.

Aeronautical databases

FMS uses aeronautical data (fixes, procedures) in the ARINC 424 format, where legs are WGS-84 paths.



Processing of aeronautical data

I've had difficulty to get the exact authoritative document making mandatory the use of ellipsoid geometry in a FMS, however, from FAA Order 8260.52 related to RNP, FMS are part of the so called GPN means which are referenced to WGS-84:


  Geographic Positioning Navigation (GPN).
  
  Navigation based on geodetic
  calculation of geographic position referenced to the WGS-84 ellipsoid.
  Global positioning system (GPS), wide area augentation system  (WAAS),
  local area augmentation system (LAAS), flight management system (FMS),
  RNP, and area navigation (RNAV) are examples of GPN.


When calculating distances between fixes, or angles, a FMS computes geodesics which are shortest paths between fixes. On a sphere this would be a great circle. On an ellipsoid this has to be adjusted, Vincenty's formulas are one of the methods that can be used for approximation.

Additional details on how geodesic calculations are done can be found in FAA Order 8260.58 related to Performance Based Navigation.