Are all carcinogens mutagens?
How can a non-mutagenic agent be carcinogenic?
An agent that causes overexpression of oncogenes or inhibition of tumor suppressors, would be carcinogenic but not mutagenic.
HPV, for instance, produces proteins that cause inactivation and degradation of tumor suppressors, p53 and pRb[1].
Regarding alcohol. As you guessed, alcohol is metabolized to form acetaldehyde which is mutagenic as well as carcinogenic. Moreover, alcohol metabolism can lead to production of ROS which can also be mutagenic. Overall, alcohol is considered a potential carcinogen by IARC due to the observed effects. These classifications are not based on molecular mechanisms.
More recently, a study by Riva et al (2020) has found that only three out of the twenty carcinogens they tested had a mutagenic effect.
Thus, to conclude, not all carcinogens are mutagens.

Reference:
 [1] IARC Working Group on the Evaluation of Carcinogenic Risk to Humans. Human Papillomaviruses. Lyon (FR): International Agency for Research on Cancer; 2007. (IARC Monographs on the Evaluation of Carcinogenic Risks to Humans, No. 90.) 4, Molecular Mechanisms of HPV-induced Carcinogenesis.