Sum of primes less than 2,000,000
Some other things that you can try, short of trying a totally different algorithm:


rename all_numbers to prime_candidates and remove composite numbers from it. (list_of_nonprimes.Add(i * j); -> prime_candidates.Remove(i*j). Avoiding the Except at the end. 
You can also hold which numbers are non primes in an array of bools and lose all the HashSets. Changing:

list_of_nonprimes.Add(i * j)


to 

nonprimes[i*j] = true;


Thus avoiding a bunch of hash lookups. After the loops sum up any n s.t. nonprime[n]==false
You can also use a BitArray instead of a bool array. Because it is more space efficient it might reduce cache misses.


Of course, you can only be sure if any of these actually makes any improvement after you try.