How do I determine the molecular vibrations of linear molecules?
As Tyberius noted, the projection formula does not work for infinite order groups (this is because the Hermitian form on characters is defined to be G-invariant by averaging over all elements in a group. That is, it works because it is possible to hit all the elements of the group in some order).

How can one work in the infinite dimensional groups then? One property that still holds for these particular infinite groups (though not in general 1) is Maschke's Theorem, that every representation is the direct sum of irreducible representations. So we can still determine what irreps make up a rep by adding them up.

First we need to find the representation we wish to reduce. I find that the easiest way to do this is to start with just the atoms (instead of the vectors) and then include the vector contribution afterwards. For the sake of a new example, I will do the linear $\ce{I4^{2-}}$ anion instead of $\ce{CO2}$.

$$\small \begin{array}{c|cccccccc} \hline
D_{\infty\mathrm{h}} &amp; E &amp; 2C_\infty^\phi &amp; \cdots &amp; \infty\sigma_\mathrm{v} &amp; i &amp; 2S_\infty^\phi &amp; \cdots &amp; \infty C_2 &amp; \\ \hline
\Gamma_\text{atoms} &amp; 4 &amp; 4 &amp; \cdots &amp; 4 &amp; 0 &amp; 0 &amp; \cdots &amp; 0 &amp;\\
\end{array}$$

Now, we multiply by the vector's contributions to the character.
$$\begin{array}{c|c}E&amp;3\\\hline C_2&amp;-1\\\hline \sigma&amp;1\\\hline i&amp;-3\\\hline C_n&amp;1+2\cos(\frac{2\pi}{n})=1+2\cos\theta\\\hline S_n&amp;-1+2\cos(\frac{2\pi}{n})=-1+2\cos\theta\end{array}$$

These come from taking the trace of the matrix representations of each symmetry operation. For example, a rotation about the z-axis by an angle $\theta$ is given by:
$$\left[\begin{array}{ccc}\cos\theta&amp;\sin\theta&amp;0\\-\sin\theta&amp;\cos\theta&amp;0\\0&amp;0&amp;1\end{array}\right]$$

$$\small \begin{array}{c|cccccccc} \hline
D_{\infty\mathrm{h}} &amp; E &amp; 2C_\infty^\phi &amp; \cdots &amp; \infty\sigma_\mathrm{v} &amp; i &amp; 2S_\infty^\phi &amp; \cdots &amp; \infty C_2 &amp; \\ \hline
\Gamma_\text{xyz} &amp; 12 &amp; 4+8\cos\theta &amp; \cdots &amp; 4 &amp; 0 &amp; 0 &amp; \cdots &amp; 0 &amp;\\
\end{array}$$

Now, we are only interested in the vibrational modes so we must subtract off the translational and rotational degrees of freedom. We can check the character table for these (they are the ones which correspond to $z$, $(x,y)$ and $(R_x, R_y)$, $R_z$ doesn't count because we define the molecular axis to be the $z$-axis.
$$\small \begin{array}{c|cccccccc} \hline
D_{\infty\mathrm{h}} &amp; E &amp; 2C_\infty^\phi &amp; \cdots &amp; \infty\sigma_\mathrm{v} &amp; i &amp; 2S_\infty^\phi &amp; \cdots &amp; \infty C_2 &amp; \\ \hline
\Gamma_\text{xyz} &amp; 12 &amp; 4+8\cos\theta &amp; \cdots &amp; 4 &amp; 0 &amp; 0 &amp; \cdots &amp; 0 &amp;\\
-\Gamma_\text{rot} &amp; 2 &amp; 2\cos\theta&amp;\cdots&amp;0 &amp; 2 &amp; -2\cos\theta&amp;\cdots&amp;0 \\
-\Gamma_\text{trans} &amp; 3 &amp; 1+2\cos\theta&amp;\cdots&amp; 1 &amp; -3 &amp; -1+2\cos\theta&amp;\cdots&amp;-1 \\
\hline\Gamma_\text{vib} &amp; 7 &amp; 3+4\cos\theta&amp;\cdots&amp; 3 &amp; 1 &amp; 1&amp;\cdots&amp;1 \\
\end{array}$$

So now we have the actual representation of the vibrations. How do we go about decomposing it? We want to eventually subtract off all of the irreps until we have the zero vector so we consider what effect each irrep has and try to move our remaining rep closer to zero. The first thing to note is that only $\Pi_g$ and $\Pi_u$ irreps have $2\cos\theta$. $\Pi_u$ adds $2\cos\theta$ under both proper and improper rotations, while $\Pi_g$ adds to proper and subtracts from improper. We note that in our rep, we only have $\cos\theta$ terms under proper rotations and so we need to subtract off enough $\Pi_u$'s until there are the same number of $\cos\theta$ (with different signs) under proper and improper rotations, at which point we should subtract off $\Pi_g$s until there are no $\cos\theta$ terms.

$$\small \begin{array}{c|cccccccc} \hline
D_{\infty\mathrm{h}} &amp; E &amp; 2C_\infty^\phi &amp; \cdots &amp; \infty\sigma_\mathrm{v} &amp; i &amp; 2S_\infty^\phi &amp; \cdots &amp; \infty C_2 &amp; \\ \hline
\Gamma_\text{vib} &amp; 7 &amp; 3+4\cos\theta&amp;\cdots&amp; 3 &amp; 1 &amp; 1&amp;\cdots&amp;1 \\
-\Pi_u &amp; 2 &amp; 2\cos\theta&amp;\cdots&amp;0 &amp; -2 &amp; 2\cos\theta&amp;\cdots&amp;1 \\
-\Pi_g &amp; 2 &amp; 2\cos\theta&amp;\cdots&amp;0 &amp; 2 &amp; -2\cos\theta&amp;\cdots&amp;1 \\
\hline\Gamma &amp; 3 &amp; 3 &amp;\cdots&amp; 3 &amp; 1 &amp; 1&amp;\cdots&amp;1 \\
\end{array}$$

Similar to the $\Pi_g/\Pi_u$, the $\Sigma_g^+$ shift adds 1 to all positions, while $\Sigma_u^+$ adds 1 to all positions before $i$ and subtracts from all after. We subtract enough $\Sigma_g^+$ until all the positions before and after $i$ have equal magnitude.
$$\small \begin{array}{c|cccccccc} \hline
D_{\infty\mathrm{h}} &amp; E &amp; 2C_\infty^\phi &amp; \cdots &amp; \infty\sigma_\mathrm{v} &amp; i &amp; 2S_\infty^\phi &amp; \cdots &amp; \infty C_2 &amp; \\ \hline
\Gamma &amp; 3 &amp; 3 &amp;\cdots&amp; 3 &amp; 1 &amp; 1&amp;\cdots&amp;1 \\
-2\Sigma_g^+ &amp; 2 &amp; 2 &amp;\cdots&amp;2 &amp; 2 &amp; 2&amp;\cdots&amp;2 \\
-\Sigma_u^+ &amp; 1 &amp; 1 &amp;\cdots&amp;1 &amp; -1 &amp; -1&amp;\cdots&amp;-1 \\
\hline \Gamma&amp; 0 &amp; 0 &amp;\cdots&amp;0 &amp; 0 &amp; 0&amp;\cdots&amp;0
\end{array}$$

Thus, we have found that the vibrational modes of $\ce{I_4^{2-}}$ to be $2\Sigma_g^+ +\Sigma_u^+ + \Pi_u + \Pi_g$.

We can check this with the formulas given in Group Theory And Chemistry by David M. Bishop as noted by Feodoran. One can also follow Tyberius's suggestion and work in $D_{2h}$ which will give the decomposition $2A_g+B_{1u}+B_{2u}+B_{3u}+B_{2g}+B_{3g}$.

It is important to note that correlation from the $D_{2h}$ point group to $D_{\infty h}$ works fine for vibrations because you cannot have terms higher than $\Pi$ and so there is no ambiguity. For things like electronic states, though, in which direct products must be decomposed, you will run into some problems. If the $D_{2h}$ result includes $A_g+B_{1g}$ does that correlate with a $\Sigma_g^+ +\Sigma_g^-$ or a $\Delta_g$ state in $D_{\infty h}$? Working in the parent infinite group may be more difficult without the reduction formula, but these problems are avoided.

The reason going to lower symmetry works is because all characters under the infinitely many proper and improper rotations are the same for each irrep (namely $2\cos n\theta$) and so we can represent the behavior of all of them by the behavior of one of them. Hence, we pick $\theta=\pi$ and ignore all the others and this produces the $D_{2h}$ point group (the two improper rotation axes are equivalent to $i$ and the two proper axes are equivalent to one $C_2$ about $z$). This is origin of the ambiguity. In $D_{\infty h}$, $\Sigma_g^+ + \Sigma_g^-$ are equal to $\Delta_g$ at $\theta=\pi$, but not in general.

Having now identified the origin of the ambiguity, one can tune how fast these ambiguities occur though. The problem occurs when the $\theta$ chosen to represent all $\theta$ has the property $1=\cos n\theta$ where $n$ represents the period of the ambiguity. So for $D_{2h}$, $\theta=\pi$ and $1=\cos2\theta$ and the period of the ambiguity is 2. That is, every other term will be ambiguously identical ($\Sigma, \Delta, \Gamma, \ldots$ are the same and $\Pi,\Phi,\ldots$ are the same). By dropping to $D_{4h}$ instead, $\theta=\pi/2$ and $1=\cos4\theta$. The ambiguity period is longer and we can distinguish $\Delta$ from $\Sigma$ by correlation from the $D_{4h}$ point group. We can thus select the point group to descend to based on how much resolving power we expect to need in the parent point group (though we should stick to even numbers to maintain inversion symmetry for $D_{\infty h}$). Of course we could also stay in the parent point group and never have a problem, but also have no reduction formula.