Delaunay triangulation when some triangles have been pre-specified?
Just insert sufficient number of points along the edges of the triangle you want - if you insert enough points, the edges of the triangle you want would be the union of edges of the triangulation. It is not pretty but it would work. This the basic idea behind many of the constrained triangulation meshing algorithms. Here is a nice paper on the topic, which contains more information:

https://people.eecs.berkeley.edu/~jrs/papers/2dj.pdf