How encryption in SQL login authentication works
how are the password encrypted and how does sql server decrypt it? What kind of encryption is involved in it?


Super high level, if you have encryption setup for the instance then the certificate that is configured can be used. If you don't have it enabled, the self-signed certificate created on SQL Server startup will be used. 

Regardless of which is used, the SSL/TLS setup for this is done during the pre-login negotation phase. If, however, encryption is either not set or enforced, the TLS/SSL portion will only be active for the length of the credential transfer and will immediately go back to plain text (no SSL/TLS) once this is completed for the life of the connection.

SQL Server itself doesn't encrypt or decrypt the data, it's encrypted and decrypted in transit only. Depending on your client driver and application, this may or may not exist in plain text in your process memory space. Additionally, SQL Server uses hashes to check the passwords and doesn't store the original plain text password for SQL logins (unless you want to be pedantic about proxy accounts and linked servers).