Scatter plot: calculate box where 80 % of the points are
Your question seems to imply that there will always be just a single cluster of points.  If this is the case, I would go with a fairly simple heuristic like the following-

Calculate the mean and variance of the point positions in both the x and y directions.  The mean is the center point of your box, and you can use the ratio of the x/y variances to determine in which directions the box should grow faster.  For instance, if the x variance is "1" and the y variance is "2", I would have the box expand twice as fast in the y direction.

Now simply expand the box until you have at least 80% of the points contained.

If there is ever more than one cluster of points you may need to use a more sophisticated solution that uses clustering algorithms to identify the clusters and a box for each cluster.