Pentaploid genotype frequencies
Ploidy number vs number of alleles

You seem to be confounding number of alleles with ploidy numbers. You rightly figured the number of possible genotypes for a diploid individuals when there are 3 possible allelic states.  When you ask for a pentaploid, are you many alleles are you wiling to consider?

Pentaploids are very rare

If I am not mistaken, species with an odd ploidy number are very rare. Tetraploids and hexaploids, while still rare are much more common than pentaploids.

General case

It is indeed a simple math problem. Let's answer to the question in the most general case. If the ploidy number is $P$ and there are $S$ possible alleles, then the number of possible genotypes is

$${P+S-1 \choose S-1} = {P+S-1 \choose P}$$

, where ${\choose }$ refers to the binomial coefficient. For a pentaploid organism, with 3 possible alleles, there are therefore 21 possibilities.

More information

You can have a look at this webpage that offers much more info about the combinatorics of genotypes.