Threshed and Malachi'd: Sieve of Eratosthenes
This line here is horrible, terrible, and miserable:


if (number % i == 0) { primes.Remove(number); }



That one line is basically (behind the scenes) doing the following:


scan all the data from the beginning until you find the member with the value number
if you find that value:

for every remaining value, shift it one back (replace each n with the value at n + 1).
shrink the size of the List by 1.



The above is an \$O(n)\$ operation.

Using an indexed method to remove the value would be much faster, and a data structure like a Linked list with a \$O(1)\$ remove, would make a huge difference.