What determines the number of chromosomes an organism carries?
Practically, there is an obvious barrier to changes in chromosome number: haploid cells produced by diploids must contain exactly one copy of each chromosome (or chromosome section, or locus). Otherwise over- or under- expression of genes can lead to dosage problems. Across all of diploids some large groups show strong conservation of number of chromosomes but others seem fairly free to change the number. 

An example of conservation are the Drosophilids (fruit flies) where for approximately 100 million years they have maintained five or six chromosomes (search for Muller elements). Incidentally, with a huge amount of intra-chromosomal shuffling of genes "Drosophila 12 Genomes Consortium", 2007, Nature. 

An extreme example of changes in chromosome number can be seen between similar species of Muntjac deer . Where in a relatively short time period (of the order of a million years or less (?) - my guess), the number of chromosomes has reduced from 46 to six (Wurster, D. H., Benirschke, K., Jun. 1970). Someone may correct me but I think amongst vertebrates, mammals show particularly fast evolution of chromosome number relative to, say, birds (though more birds also have micro-chromosomes...).

The above changes (or lack of change in Drosophilids) appear to be independent of genome size - so that was a long way round to saying "It's not genome size". 

It could be repetitive DNA. The centrosomes (where mitotic spindles attach) of mammals are full of repetitive DNA and for this reason have been hard to sequence accurately (Graphodatsky, A. S., Trifonov, V. A., Stanyon, R., 2011. ). These repeats evolve (in sequence and copy number) and may promote more or fewer spindle attachments. Drosophilids (in general) have fewer repeats but I don't know if this is a diploid-wide pattern [Research anyone?].