OP_RETURN marks transaction or output as invalid
Both.

Transaction output scripts are only executed at the time they are attempted to be spent.

If you try to spend an OP_RETURN output, the spending transaction is invalid, as this instruction immediately fails the execution.

As a result, OP_RETURN outputs (not the transactions they are in) are unspendable.