Is there an algorithm to find all vertices that are inside of a shape?
It seems to me that the problem is not about graph theory, but computational geometry. Phrased slightly differently, your input is a set of two-dimensional points $P$ (corresponding to the coordinates of the vertices of some graph drawing) and a polygon $Q$. The task to determine a subset $P'$ of the points $P$ that are contained within $Q$.

One possible algorithm is to iterate over each point $p$ in $P$, and check whether $p$ is inside $Q$. This classic point in polygon problem has many simple and efficient algorithms.