Is 500,000 files in my home directory too much
As pointed out in a comment, you're using a command that is locating files within directories below the directory in question.    While you may only be able to count to about 20 (fingers and toes) OS's are much much better at dealing with large numbers of files, and this is really an issue you shouldn't worry about. 

Now if your computer is slow, you might want to look into what is using memory.