Can one trace a criminal with his Pubkey? Say FBI arrested a guy who bought some drugs using bitcoin. Then, is it possible for FBI to trace a drug dealer using the arrested guy’s PrivateKey and his transaction histories?

If my understanding of bitcoin algorithm is correct, this is impossible since PublicKey only shows that “there is a user with this public key” and it does not give any more information about a user who owns that PublicKey.

Moreover, transaction that the dealer and the arrested guy made only shows public keys.

Can one trace someone using past trasactions?
Knowledge about his addresses and transaction history will be sufficient to discover addresses with which he made deals. His Private key will not help. This is the consequence of public ledger.

The different case is with deterministic wallets. When you know master private key/ mnemonic seed you can check all derived addresses, even the ones you have no previous knowledge.