Why is carbon monoxide not a greenhouse gas?
$\ce{CO}$ is not considered a primary (or significant) greenhouse gas due to the weak (but non-zero) absorption of energy in the infrared. However, it does increase global warming by reacting with certain chemical species in the atmosphere which in turn lead to an increase in concentration of primary greenhouse gasses, most notably methane and ozone (see final paragraphs cited below). Thus, it is not completely accurate to dismiss it as a contributing factor.
From the National Oceanic and Atmospheric Administration's page on $\ce{CO}$ as a greenhouse gas:

Carbon monoxide (CO) is not considered a direct greenhouse gas, mostly because it does not absorb terrestrial thermal IR energy strongly enough. However, CO is able to modulate the production of methane and tropospheric ozone.

Carbon monoxide does contribute to global warming indirectly, as noted by the American Chemical Society's page entitled &quot;Which Gases Are Greenhouse Gasses?&quot;:

Several other gases can have indirect effects on atmospheric warming. This occurs when chemical reactions in the atmosphere produce or destroy greenhouse gases, including tropospheric ozone. Indirect effects also occur when a gas influences atmospheric lifetimes of other gases or affects atmospheric processes like cloud formation that alter Earth’s radiative energy balance by increasing Earth’s albedo. Gases that can cause these indirect effects include carbon monoxide ($\ce{CO}$), oxides of nitrogen ($\ce{NO_{\rm x}}$) and non-methane volatile organic compounds (NMVOC).

The above is elaborated in the text Global Climate Change Linkages: Acid Rain, Air Quality, and Stratospheric Ozone (James C. White, Editor. Elsevier (1989), p. 106):

As has been noted, ozone also initiates reactions that generate the highly chemically active oxidants, including hydroxyl. The lifetime of radiatively active methane in the atmosphere is determined by hydroxyl abundance. Carbon monoxide and methane are both sinks for hydroxyl, $\ce{CO}$ being the more reactive of the two. Greater concentrations of carbon monoxide in the atmosphere lead to lesser concentrations of $\ce{HO\cdot}$, which in turn lead to higher concentrations of methane.
Carbon monoxide and ozone are linked in the non-urban atmosphere that is low in hydrocarbons through the interaction with $\ce{HO\cdot}$, as described above, and $\ce{NO2}$. Hydroxyl is destroyed by $\ce{CO}$, increasing the ambient concentration of $\ce{NO2}$. The resulting increased proportion of $\ce{NO2}$ to $\ce{NO}$ leads to a higher steady state concentration of ozone. Carbon monoxide, a non-radiatively active species, enhances the greenhouse warming by raising the levels of methane and ozone.

In summary: $\ce{CO}$ is not a direct greenhouse gas due to low absorption in the infrared but is an indirect greenhouse gas because it increases the concentration of direct greenhouse gases (methane and ozone most prominently) by reacting with atmospheric $\ce{HO\cdot}$, which otherwise would decrease the concentration of those gasses. The toxicity part is not germane to the greenhouse gas issue.