What is the purely mathematical definition of encryption strength? An encryption function can be used to obfuscate pieces of information and later on retrieve them.

What is the yardstick for measuring how well the function encrypts the information ? Given any invertible function, which are the quantified parameters which measure how effective the function is to encrypt a message ?

I am assuming a finite domain for the function. Any other assumption may be stated.



Motivation: Is there any mathematical/set theoretic formulation at all... I ask this specially because I don't see this approach ever being used as has been rightly observed. My motivation is to find out weather the notion of 'encryption' can be captured fully in a mathematical formalism.-- or that we require other kind of 'concepts' to describe it fully. –
There isn't just one, because there are many different scenarios where you'd use such a function, where the attacker has lesser or greater powers, or variably stringent success goals—different attack models.  For example:


Does the attacker know any plaintext/ciphertext pairs encrypted with the same key?  (Known plaintext attack)
Is the attacker able to trick the defender into encrypting plaintexts of the attacker's choice, and observe the corresponding ciphertexts?  (Chosen plaintext attack)
In the latter case, is the attacker able to observe the ciphertexts in real time, and use this to make smarter choices of plaintext to submit?  (Adaptive chosen plaintext attack)
Is the attacker's goal to decrypt a message, or something weaker like telling true encrypted messages apart from random bits?  (The latter is what's called a "distinguishing attack," and strengths are often expressed in terms of that.)


So it's common for algorithms to have different security levels in different scenarios. If you look for example at the "Known attacks" section in Wikipedia's entry on AES, you'll see a variety of figures, all qualified by the relevant attack model.

Another detail is that strengths are often expressed not as one number but as a function of the resources that the attacker expends in some model (e.g., in a chosen plaintext attack, the number queries, i.e. the number of plaintexts that they submit to the defender).  Such functions generally give the probability that a randomized attacker will succeed if they expend that many resources.

And yet another detail is that often for some algorithms like block cipher modes of operation, which are parametrized by one or more primitives, the strength is often given as a function of the strengths of the primitives.  If you're math-inclined or even just math-curious, two papers I found very useful to understand this are:


Bellare, Kilian and Rogaway's "The Security of the Cipher Block Chaining
Message Authentication Code"
Bellare, Desai, Jokipii and Rogaway's "A Concrete Security Treatment of Symmetric Encryption"


These two papers are perhaps overwhelmingly detailed, but even just skimming the major points might be instructive.