Prevent read access on primary AG replica, but allow it on the readable secondary
It seems we basically need to allow a login connect permission on the primary replica [...]


That's exactly right.  The login being used to connect has to be able to connect to the primary.

When you using ApplicationIntent=ReadOnly in a connection string that's pointed at an AG listener, the driver initially connects to the primary instance.  This is so that SQL Server can review the list of replicas and see if there are any available, online readable replicas to send you to.  This is documented here:

Read-Only Application Intent and Read-Only Routing


  The primary database of the availability group processes the incoming read-only routing request and attempts to locate an online, read-only replica that is joined to the primary replica and is configured for read-only routing. The client receives back connection information from the primary replica server and connects to the identified read-only replica.


There's also a good summary of the process laid out by Top Microsoft Man Sean Gallardy on the PFE blog here:

Finding Which Connections Have Been Read Only Routed


  
  We create a connection through the driver
  The driver connects to SQL Server through the listener and passes the relevant information
  SQL Server notices that we have read only routing setup and checks the configuration
  SQL server reports back to the driver that we have a read only secondary that can be used
  SQL Server sends the new secondary information to the driver
  The driver creates a new connection to the secondary with the given information
  


Regarding the other item you noted in the question:


  [...] while denying it read access to the database in the AG on the primary, but give that login permission to read the database on the readable secondary replica.


Database level permissions are going to be replicated across the AG - so if you remove the user's read-access on the primary, they will lose read-access on the secondary as well.

I'm a little confused about the goal of such an endeavor, since the data should, in theory, be the same (or close to the same).  Maybe you just don't want certain logins running queries on the primary at all for performance reasons.  Expressing Read-Only intent and connecting to the listener is really the most straightforward approach to solving this.

Other options would involve separate hardware, or custom coding in your app that deals with connecting directly to the right replica (and retry logic if a failover has occurred, and scheduled jobs on each replica to disable / enable the logins in the event of a failover, etc).