Why IV does not have to be secret yet has to be random The IV used in schemes such as CBC has to be random and unpredictable. But at the same time it does not have to be kept secret.

If the IV does not have to be secret, why does it have to be random then? I fail to make sense out of these seemingly contradicting requirements.

I have seen descriptions of attacks which exploit the non-randomness. So, I would understand why randomness is needed. However, things get confusing when the requirements specify that the IV does not have to be secret !This seems to defeat the whole purpose of randomness.

Somebody help clarify this please.
OK! This won't be the best answer I can give, but since I know a fair bit about this subject, CSPRNGs (Cryptographically Secure Pseudo-Random Number Generators), also known as Dertemistic Random Bit Generators (DRBGs).

Now your question asked about CBC, so I'll come onto that second, first I want to give a slightly less technical/broad answer, I'll give you an example of where the importance of IVs are crucial.

Let's assume we had a STREAM CIPHER (asin RC4, which is in itself full of problems, but that's not what you asked). In WEP, RC4 was using IVs extremely badly, but let's assume there's a bunch of alterations you can make to RC4 to make it significantly stronger (albeit, less elegant).

In this circumstance, because we have a stream cipher, the vulnrability to plain-text-attacks is very high, that's because if we use the same key more than once, then the relationship between any two texts using that same key is known , this means that if we can guess the plain text, this is often the case in practical everyday transmissions, then firstly we know EXACTLY what the key stream is and from there, finding out the initial key is pretty trivial.

So we NEVER want to use the same key twice for any encryption... Block Ciphers, Stream Ciphers, whatever.

OK So an IV can be used in many ways, the reason I bought up WEP was say you were using a 64-bit key, then to make sure no two "keys" were the same, the IV would act like a nonce, that is, the effective key used for key-scheduling would be (IV|Secret), now as I said, this particular implementation was awful but here's the point.

WE NEVER EVER WANT TO HAVE THE SAME KEY TWICE... There's plenty of ways of achieving this.

Now in WEP implementation of RC4, using (IV|Secret) as the initial key where "Secret" is 40 bits and "IV" is 24 bits, this means there's only 2^64 possible keys, however more importantly if we use the same "secret", then there's only 2^24 (16,777,216) different keys you can create from the same secret... since the 40 bit key is just 5 random bytes chosen or derived from the user. 16,777,216 is a laughably small number and infact, we will find two instances of (IV|Secret) that match half the time inside about 4300 key-streams, providing the "secret" doesn't change. So that was one example of using an IV failing spectacularly.

So if we imagine that our key for any encryption is HMAC-SHA256(IV,Secret), then we can afford for the IV to be the same size as the Secret, as there will be 2^128, that is over 3 * 10^38 and there will also be this many possible IVs, so even if we KNOW the IV and it's being used more or less as a nonce, although HMAC-HASH(IV,Secret) is better than simply using HASH(IV|Secret)... once we have done this, we can PUBLICLY REVEAL what the IV was. Presuming we use a good CSPRNG and a good HASH function, then to attempt a known-plain-text-attack, we would not take a single guess and reveal a fair amount of info, instead we would have to use over 3*10^38 guesses.

This isn't ideal but the point is.... the IV does not to be secret because the IV is there to make sure realistically known-plain-text attacks are significantly more complicated... in my example it becomes 2^128 times more complex to launch a known-plain-text attack OR partially-known-plain-text attack.

In a nutshell one of the primary functions of IVs is to make sure no two encryptions are the same or at very least, finding two the same is exponentially more difficult.

If you want more clarity, just ask. This isn't exactly a well written essay, but I thought I'd have a crack at shedding some light on this question. If needed, I could write a much more technical answer.