Entropy change and heat transfer in non-ideal heat engines
So apparently the definition of entropy change in a system is $\Delta S = \int_\mathrm{rev} \frac{\mathrm d Q}{T}$ and this is a state variable.

This definition is for the surrounding if it is boring, e.g. a temperature bath. The $_\mathrm{rev}$ subscript is just for the temperature bath, i.e. there could be irreversible processes elsewhere (e.g. in the system).

Yet in my professor's notes it says $$\mathrm dS = \oint \frac{\mathrm dQ}{T} &lt; 0 \text{ for irreversible processes }$$

The inequality is correct. Equating it to $\mathrm dS$ is not. The entropy change for a cyclic process is zero because entropy is a state function.

[OP...] elsewhere, talking about the Sterling process, it also says
$$ \oint_\mathrm{irrev} \frac{\mathrm dQ}{T} &lt; 0 \text{ , so the entropy of the machine (not the total entropy) decreases } $$

This is the Clausius inequality. If a heat engine is not ideal, it will not extract the maximal energy possible from the heat transfer from hot to cold reservoir. The claim about the entropy of the machine is incorrect. Indeed, the entropy of the machine stays the same (use your argument of taking a reversible path from one cycle to the next, i.e. do nothing). Overall, the entropy increases because the entropy loss in the hot reservoir is no longer matched by the entropy gain of the cold reservoir. Thus, we are not extracting the maximal work in the process.
I found this explanation helpful. They used $\delta Q$ instead of $\mathrm d Q$ to remind us that there is a path-dependency.