Public Key encryption /Assymetric encryption For a start, I 've got a question regarding the Public key encryption.
I know the public key encryption overcomes the Key distribution issue with the symmetric encryption and that in the case of communication between 2 parties, there is a total of 4 keys (each party has a public and a private key )

But here's what's been confusing me.
If Bob wants to send a message to Alice, then he will encrypt the message using Alice's public key. For Alice to understand that message, she would decrypt the message using her private key.  The problem is, isn't it like they both have the same keys? Because if Alice were to decrypt the message using her private key and no one else knows what that key is but her, won't she get a different message than the one sent by Bob?
The part you're missing here is that Public Key encryption is asymmetric - that is, data can only be encrypted with the public key, and can only be decrypted with the private key. Public key encryption systems are created in such a way that the encryption is (mostly) irreversible, but there is a mathematical 'trapdoor' that allows the private key to be used decrypt the data.

Compare this with a symmetric encryption scheme, where there is only one secret key (so both Alice and Bob would have to share the key to use such a scheme).

i.e. once the data is encrypted by Bob using Alice's public key, noone holding Alice's public key can decrypt it. Only Alice, using the private key and the different decryption algorithm can decrypt it.

The answers to How does asymmetric encryption work? give more detail as to how this works.