How effective is Serpent-256 in CTR mode? I downloaded a compression program known as PeaZip which offers encryption and 2-way authentication also.

PeaZip has an option to encrypt compressed file with variety of ciphers and one of them is Serpent.

It gives option of: 


Serpent 128 CTR OMAC
Serpent 256 CTR OMAC


I want to know how strong any cipher, specifically Serpent, is in CTR mode. Assuming a strong password or truly random key file of appropriate size is used.

I am not aware of how effectively this cipher is handled in PeaZip, but that is a different matter.

Ideally, in which mode Serpent is more effective.

-- Edited --

The details of crypto library which PeaZip uses are here.

Author of this library says about Serpent:


  Serpent (designed by Ross Anderson, Eli Biham, and Lars Knudsen) was one of the AES finalists; it is a 128-bit block cipher with key sizes of 128, 192, or 256 bits. Please note that although my code supports OMAC and EAX, these modes are not tested against public test vectors (the usual C libraries do either not implement Serpent or OMAC/EAX).
Any block cipher within CTR operates the same way; CTR security is only dependent on the block size. Serpent was one of the candidates to become AES. It is thought to have a high security margin (and no weak keys etc). None of the currently known attacks seem to have done much damage to the security claims.

All AES candidates have a block size of 128 bit and a key size of 128, 192 and 256 bits (at least). In your case you just list two key sizes, 128 bit or 256 bit as difference. The higher key size is more secure. As the number of rounds of Serpent doesn't change the greater sized key should be preferred, although the strength of the password is much more important.

In the end, an algorithm is effective when it handles your specific requirements. And in general, the security of the system is very likely not much influenced by the choice of Serpent as block cipher (especially if timing attacks are hard to do such as with file encryption, hardware implementations of AES such as AES-NI may have a small advantage with regards to timing based attacks, besides the obvious performance benefit).