Does muscle get bigger by increase in size of individual cells or increase in number?
The "back of your mind" is correct: "if the muscle gets bigger, it's simply because individual cells get bigger."

Growth of muscle can occur in three ways: 


by an increase in muscle cell numbers
by an increase in muscle fiber diameter
by an increase in fiber length. 


However, growth in cell numbers is limited to the prenatal and immediately postnatal period, with the animals and man being born with or soon reaching their full complement of muscle cells. 


  [G]rowth occurs by either hypertrophy of the existing muscle fibers by adding additional myofibrils to increase the muscle mass or by adding new sarcomeres to the ends of the existing muscle fibers to increase their length. Both of these mechanisms occur during the growth process. Growth in the girth of the muscle fibers... may be stimulated by development of stress creating an unequal pressure with splitting at the Z-band and development of additional SR and T-tubule systems. This adds to the diameter or girth of myofibers without any hyperplasia. The growth in length occurs at either end of the fibers and results in addition of new sarcomeres. In both cases, new myofibrillar protein must be synthesized and deposited in the muscle cells. 


Adding or removing sarcomeres and myofibrils is determined by exercise, that is, the degree of force a muscle can generate which is in turn dependent on the degree of overlap of the thick and thin filaments. Thus, the amount of tension would control the number of in-series sarcomeres and the number of myofibrils in a single muscle fiber. 

Nutrition is also known to play an important role in muscle growth, but growth (that is, increase in girth) cannot occur without exercise. 

The exact roles of GF and IGF-I are not completely clear.

Muscle size does not increase by the addition of new muscle cells:


  The results show that the increase in muscle cross-sectional area from childhood to adult age is caused by an increase in mean fiber size. This is accompanied by a functional development of the fiber population: the proportion of type 2 fibers increases significantly from the age of 5 (approx. 35%) to the age of 20 (approx. 50%), which, in the absence of any discernible effect on the total number of fibers, is most likely caused by a transformation of type 1 to type 2 fibers.


Activation of satellite cells, a claim made by body builders, is mis-represented by them. The satellite cells seem only to aid severely damaged muscle cells to heal, not to increase in number.

Muscle growth and exercise
Muscle hypertrophy
Growth and development of human muscle: A quantitative morphological study of whole vastus lateralis from childhood to adult age
Regulation of muscle mass by growth hormone and IGF-I