Why does an exchange of bitcoin transaction not need to be over a secure connection? I apologise for my rather ignorant question and maybe my question is completely wrong. But my impression about one of the advantages of bitcoin (digital currencies) is that when a customer pays a client, that payment gateway does not need to be over a secure protocol merely because of the way the bitcoin works.

If this is true, can I have a more technical explanation of why this is the case?
Conventional banking relies on you and your bank both knowing some sort of secret, like a PIN or your mother's maiden name. This requires encryption, because otherwise someone listening in on the connection can capture the information and impersonate you.

Bitcoin uses only public key cryptography to identify accounts. It is possible to prove that the creator of a transaction had access to a private key without revealing what that private key is. See also How does the ECDSA verification algorithm work during transaction?

Also, although Bitcoin connections are usually not encrypted, there are circumstances where they probably should be, like if you want to make sure that the administrator of the local network doesn't know which Bitcoins you're spending.