If a chemical formula for a mineral contains a comma, are there limits on the amounts of each element?
Actinolite is a mineral that is part of a so-called solid-solution series, which means that you get different but related minerals depending on the amounts of (in this case) iron and magnesium that are present. The magnesium-rich end member is tremolite; the iron-rich end member is ferro-actinolite; and what lies between is actinolite.

According to Wikipedia's article on actinolite, the formula is more precisely:
$$\rm Ca_{2}(Mg_{4.5-2.5}Fe^{2+}_{0.5-2.5})Si_{8}O_{22}(OH)_{2}$$

So, for your question:


  is it possible for the amount of magnesium to be less than iron and still be actinolite, or must magnesium always have the greatest amount?


the answer is no: at best, Mg and Fe can be present in equal amounts for the mineral to still be called actinolite, but note that in that particular case you are on the "border" between actinolite and ferro-actinolite, so perhaps what you choose to call it is up to you. And as mentioned above, if magnesium is dominant, the mineral is tremolite.

In the case where iron is the dominant atom, the mineral is called ferro-actinolite with the formula:
$$\rm Ca_{2}(Mg_{2.5-0.0}Fe^{2+}_{2.5-5.0})Si_{8}O_{22}(OH)_{2}$$