How effective is Serpent-256 in CTR mode?
Any block cipher within CTR operates the same way; CTR security is only dependent on the block size. Serpent was one of the candidates to become AES. It is thought to have a high security margin (and no weak keys etc). None of the currently known attacks seem to have done much damage to the security claims.

All AES candidates have a block size of 128 bit and a key size of 128, 192 and 256 bits (at least). In your case you just list two key sizes, 128 bit or 256 bit as difference. The higher key size is more secure. As the number of rounds of Serpent doesn't change the greater sized key should be preferred, although the strength of the password is much more important.

In the end, an algorithm is effective when it handles your specific requirements. And in general, the security of the system is very likely not much influenced by the choice of Serpent as block cipher (especially if timing attacks are hard to do such as with file encryption, hardware implementations of AES such as AES-NI may have a small advantage with regards to timing based attacks, besides the obvious performance benefit).