How to map square texture to triangle?
This is achieved via Barycentric Interpolation.

First, we find the barycentric coordinates of $P$. Barycentric coordinates represent how much weight each vertex contributes to the point, and can be used to interpolate any value which is known at the vertices across the face of a triangle.

Consider the 3 inner triangles $ABP$, $PBC$ and $PCA$.



We can say that the barycentric coordinate, or weight of the vertex $A$ on the point $P$ is proportional to the ratio of the area of inner triangle $PBC$ to the area of the whole triangle $ABC$.

This is intuitively evident if we consider that as $P$ approaches $A$ the triangle $PBC$ grows larger and the other two become smaller. 

Also intuitively evident should be that the sum of the barycentric coordinates of a point inside a triangle always equals $1$. So, it is enough to find only two of the coordinates to derive the 3rd.

The method for computing the barycentric coordinates is:

$$\begin{aligned}
{Bary}_A &amp;= \frac{(B_y-C_y)(P_x-C_x) + (C_x-B_x)(P_y-C_y)}{(B_y-C_y)(A_x-C_x) + (C_x-B_x)(A_y-C_y)}\\
{Bary}_B &amp;= \frac{(C_y-A_y)(P_x-C_x) + (A_x-C_x)(P_y-C_y)}{(B_y-C_y)(A_x-C_x) + (C_x-B_x)(A_y-C_y)}\\
{Bary}_C &amp;= 1 - {Bary}_A - {Bary}_B
\end{aligned}$$

The derivation and reasoning is explained in the wikipedia article.

Once you have coordinates, you can determine the texture coordinates of $P$ by interpolating the values at the vertices using the barycentric coordinates as weights:

$$P_{uv} = {Bary}_A \cdot A_{uv} + {Bary}_B \cdot B_{uv} + {Bary}_C \cdot C_{uv}$$

The reasoning is also explained very nicely in this presentation.

Also see this question for efficient methods of computation.