Project Euler 91 (via HackerRank): Right triangles with integer coordinates
righttri violates the single responsibility principle: it tests the triangle, and (possibly) appends it in the done list. Appending doesn't belong here.
righttri is surely working too hard: it calculates square roots just to compare them being squared back. Besides losing performance with unnecessary math, it also losing precision. With problems like this it is much better to stay within the integer domain.
Algorithm inspects every pair of dots on the grid, that is a time complexity is \$O(N^4)\$. That explains the timeouts. There are just too many pairs which do not form the right triangle.

A better algorithm would enumerate only right triangles via the Euclid's formula.