Do all conversion factors have an infinite number of significant figures?
Your intuition is correct. There are two types of conversion factor.
One kind of conversion factor is based on definitions. These conversions are exact (infinite significant figures).
In addition to your examples of the specific heat of water (in calories), the kilogram/gram conversion, and the quart/gallon conversion, other conversions are exact, including:

The speed of light - $c=299,792,458\ \mathrm{m/s}$ the length of the meter is then derived from this constant.
The price of gasoline/petrol.
The inch is defined as exactly $25.4\ \mathrm{mm}$, and thus the conversions from all imperial to metric lengths (including your  are exact as well.
The specific heat of water in joules - the relationship between the calorie and the joule is defined exactly as $\mathrm{1\ cal = 4.184\ J}$

Note that most conversions of this type are unit conversions. Many units have been defined exactly in terms of SI-units.
Other conversion factors are based on measurements. One common example is atomic mass. Atomic masses are valuable conversion factors that are based on measurements. However, some unit conversions live in this category.
The electronvolt is based on the measurement of the elementary charge of the electron (which is not exact). Thus the electronvolt is $\mathrm{1\ eV=1.6021766208(98)\times 10^{−19}\ J}$.