Combinatorial Optimization: Shortest distance given sets of drivers and riders
These kinds of scheduling problems tend to be hard (e.g., NP-hard to solve exactly; and messy because of all of the different constraints you have).

My favorite "big hammer" for this kind of scheduling problem is to use integer linear programming, as that has been used successfully for a number of different scheduling and operations research tasks.

The best way to formulate this as an ILP problem may depend on the parameters (how many drivers and riders you have), but let me describe one illustrative approach.  Suppose each car can have up to 5 riders.  Then consider all possible ways to choose a combination $S$ of 1 driver and up to 5 riders.  For each such, compute the total distance $d_S$ of the shortest route that involves that driver picking up all riders and taking them to the destination (this might involve $5!=120$ shortest-path computations).  Now introduce a zero-or-one variable $x_S$ for each combination $S$, with the intended meaning that $x_S=1$ means we select that combination.  We require that each participant is selected exactly once; thus, for each participant $p$, we add the equality $\sum_S x_S = 1$, where $S$ ranges over all combinations with $p \in S$.  Then, we minimize the linear objective $\sum_S d_S x_S$, where $S$ ranges over all combinations.  This ensures we minimize the total distance, while ensuring every participant reaches the destination.

Then, apply an off-the-shelf ILP solver.  If the number of riders and drivers is small enough, it should find an optimal solution.  If it is taking too long, many ILP solvers will let you terminate them early and report the best solution found so far, which may be effective at finding a good but not optimal solution.

This particular formulation might work well if the number of riders is not too large.  If the number of riders is very large, you might need to consider other formulations.