Which encryption algorithm is used for password protecting a file
This is Traditional PKWARE Encryption as documented in the PKWARE ZIP specification (see Section 6.0). It's an old stream cipher which suffers from a known-plaintext attack (see Biham and Stay publications) and has been replaced with AES in newer ZIP archive utilities.

See a lengthy answer here regarding the possibility of ZIP file password recovery when this cipher is in use.

Various people have implemented the known-plaintext attack and code is available (pkcrack and bkcrack).