Transactional replication upgrade from SQL Server 2017
Rolling back an in place upgrade on a distributor is pretty similar to any failed in place upgrade. Which is to say, it's not usually quick and simple.
If the failure is during the install, you'll need to uninstall, reinstall, and revert to backups. With replication, msdb is very important, in addition to the distribution database, in order to restore all of the jobs. VM-level snapshots can be very useful here, as that can make reverting easier, without having to do messy uninstall/reinstall. Depending on your VM configuration, VM snapshots might save you from having to restore databases, too.
If the failure/reason to rollback happens after upgrade, when you've re-enabled any of the agents, then restoring from backup won't be enough. Data that is being distributed to subscribers or read from the publisher logs after upgrade, before rollback would cause data inconsistencies. In that case, you'll also need to re-snapshot your publications on rollback.