Database Compatibility Requirement for Transactional Replication
Does the subscriber "within two version requirement" for transactional
  replication from this MSDN article apply to the SQL Server Instance,
  Database Compatibility Level, or Both?


Yes. No. No.

The database compatibility level is irrelevant, and you would never set the database compatibility on the distribution database, as it's a system database.

A SQL Server 2014 publisher/distributor can have a SQL Server 2017 subscriber.

SQL Server 2017 supports compatibility levels back to 100, so you can set the subscriber database to 110 compatibility level if you want to.  And you might need to if you publish views, stored procedures, stored procedure executions, or have triggers on subscriber tables that aren't NOT FOR REPLICATION.