An algorithm to find the area of intersection between a convex polygon and a 3D polyhedron?
You have a 2D convex polygon $G$, and a 3D polyhedron $H$.  Let $P_G$ denote the plane that the polygon is contained in.  The following should work:

Their intersection is a 2D polygon.  You can find the edges of the intersection as follows:


For each triangle of $H$:


If the triangle intersects $G$, output this intersection.



This procedure outputs the edges of a 2D polygon, and now you want to compute its area.  To do this, compute a triangulation of it (which apparently can be done in linear time), then sum up the area of the resulting triangles.

You can test whether a triangle intersects $G$ as follows: compute the intersection of the triangle with $P_G$; call the result $R$.  Typically, $R$ will be a point or line.  $R$ can be computed efficiently.  Then, compute the intersection of $R$ with $G$.  This is purely a 2D problem.  There are standard algorithms for solving this; see, e.g., https://en.wikipedia.org/wiki/Line_clipping or http://geomalgorithms.com/a13-_intersect-4.html.