Does read-only activity on synchronous secondary replica slow down primary replica in Availability Group?
usually, there is no impact on primary replica in performance point of view as the resource utilization is completely on secondary replica. i.e. (CPU, Memory, IO), and read operation would takes place on secondary copy of the database.

I believe, your assumption of performance impact considering the log processing workers, but that (AG worker connections) would anyway constantly active (which can be identified with select * from sys.dm_exec_connections where protocol_type = 'Database Mirroring' ) on primary replica regardless weather the secondary read-only operation or not. If it's really a concern you can use dedicated Ethernet for Database mirroring end-pints. 

In summary, there is no performance impact but we should consider about Data Latency as the active transactions from Primary replica cannot be available in Secondary replica's select query until and unless the transaction committed by Primary replica and secondary replica has processed particular transaction.