An algorithm to find the area of intersection between a convex polygon and a 3D polyhedron?
There are different numerical approximations you could use:

A simple solution is to use brute-force Monte Carlo integration. Distribute $N$ random points on the polygon and calculate the number of points inside the 3D polyhedron $N_i$ using ray tracing. The area inside the polyhedron is $A_i=A\frac{N_i}{N}$, where $A$ is the area of the polygon.

To improve convergence of the above algorithm, you can distribute the points on grid on the polygon and jitter the sample positions within each grid cell. This is called stratified sampling.

Further improvement would be to perform the MC integration in 1D instead of 2D. Pick a sample line $L$ that's on the polygon plane and a sample direction $D$ that's on the plane perpendicular to $L$. Select sample line segment $L_s$ that bounds the polygon within $D$-directed lines (min &amp; max of the polygon projection on $L$ for tight fit). For each sample on $L_s$ first calculate the line segment $S$ that intersects the polygon in direction $D$. Next calculate the length of $S$ that's inside the polyhedron $S_i$. You can calculate $S_i$ using ray tracing, by starting from one end of $S$ and marching along $S$ to intersection points of the segment with the polyhedron, summing up the segments of $S$ inside the polyhedron. More specifically, you need to first check if the start point on $S$ is within polyhedron and set the "inside" state accordingly. Then for each intersection flip the state and add the length from previous intersection to $S_i$ if the state changes from inside to outside. The approximation of the polygon area inside the 3D polyhedron is:
$$A_i=\frac{|L_s|}{N}\sum_{i=1}^N S_i$$