Limit connection numbers with pgbouncer
You don't have to "specify pool_size for my_db, too". 

But your config won't limit the number of connections though. You need to limit max_db_connections for that like here:

[databases]
my_db = host=10.10.10.10
my_db_with_conn_limit = host=10.10.10.10 dbname=my_db pool_size=55  max_db_connections=55


Otherwise you limit the number of sessions in pool, but not the number of connections... 

You can also just set different limits per database+user pair instead (which I believe is more common practice).