Why is the result of this product operation wrong?
Integer overflow. You're multiplying an int by an int then assigning it to a long.  Instead if you multiply a long by a long, or an int by a long, or a long by an int, it should work.

unsigned long res = 775L * 775;


The L suffix tells the compiler to treat it as a long, not an int.