The somewhat confusing specific epithet olivacea ("the olive-colored one") was based on a female or immature specimen rather than erythromelas ("the red-and-black one"), which authors attempted to ascribe to the species throughout the 19th century (older scientific names always takes precedence, however).
Female, immature and non-breeding males may be distinguished from the same ages and sexes in summer tanagers, which are more brownish overall, and western tanagers, which always have bold white bars and more yellowish undersides than scarlet tanagers.