What is the need for Diffie-Hellman key exchange when we have asymmetric encryption? I am new to cryptography and I am having a hard time to understand what is the need for DHKE procedure. I think I fully understand the principle in which it happens and also the mathematics behind it however I feel it is not needed (most probably I am wrong that's why I asking you guys here).
Let's say a client wants to communicate with a server - everything makes perfect sense with the first part of the handshake:
The client sends: &quot;Hello server, this is my TCP max version, this and that, etc.&quot;

Then the server responds: &quot;Hello, I am the Foo server. This is my public key (go check my certificate) this is my signature (so you can be sure it is me and not a 'middle man')&quot;

Now due to the properties of the public-private key encryption, the client can send the server HIS public key (we don't care how, it's public anyway, but still the message can be encrypted with the server's public key) and then the secure communication can begin by using asymmetric encryption - when the client send something to the server it will be encrypted with the server's public key (so only the server can decrypt it) and vice versa (when server has to send something to the client it will be encrypted with the client's public key).
If my understanding is correct the DHKE procedure brings nothing new to the table in terms of security and I don't understand the need for using it (I doubt that the symmetric key encryption/decryption is much more efficient in terms of computing power).
Thanks in advance!
when the client send something to the server it will be encrypted with the server's public key (so only the server can decrypt it) and vice versa (when server has to send something to the client it will be encrypted with the client's public key)

As I understand it, you are suggesting not to bother using symmetric encryption; instead, have both sides exchange messages using public key encryption.
That would be horribly expensive (that is, slow); symmetric schemes are literally thousands of times faster than asymmetric schemes.
For example, on my test machine (using OpenSSL), it would 0.4 microseconds to encrypt a 64 byte message (size selected because that would fit within a single RSA encryption); in contrast, it took 614 microseconds (that is, 1500 times as long) to RSA decrypt the same amount using 2048 bit RSA.
Performance varies a lot depending on the processor, however this sort of huge performance is typical.

What is the need for Diffie-Hellman key exchange when we have assymmetric key exchange?

Well, one could use assymmetric key exchanges to share symmetric keys, and use those symmetric keys to encrypt the actual messages - that certainly can be done (for example, in some ciphersuites in older TLS versions).  It wasn't great when dealing with server compromise (which is why newer TLS versions abandoned it), but it did work.