RSA Protocol behind Yaksha Security System
Can Alice obtain the session key due to the multiplicative properties of the modulus function and the basis of which RSA is built on?


Essentially, yes.  One way of looking why RSA works (that is, why the encryption and decryption are inverses of each other) is because of two mathematical identities:

$$(M^a \bmod N)^b \bmod N = M^{a \cdot b} \bmod N$$

$$\text{If}\  a \cdot b \equiv 1 \pmod{\phi(N)}\  \text{then}\  M^{a\cdot b} \equiv M \pmod{N}$$

(Note: in the second one, you can replace $\pmod{\phi(N)}$ with $lcm(p-1,q-1)$, which gives you a stronger identity; however what I have written is good enough for now).

For normal RSA, this is sufficient; however it is easy to extend these identities to three exponents:

$$((M^a \bmod N)^b \bmod N)^c \bmod N = M^{a \cdot b \cdot c} \bmod N$$

$$\text{If}\  a \cdot b \cdot c \equiv 1 \pmod{\phi(N)}\  \text{then}\  M^{a\cdot b \cdot c} \equiv M \pmod{N}$$

This is what Yaksha is relying on; the have $a$ = the public exponent (eAlice), $b$ = the private key known to the server (dAliceY), and $c$ = Alice's private key (dAliceA).

We can see that the relationship between eAlice, dAliceY and dAliceA holds, based on the fact that:


  $dAliceA \cdot dAliceY \equiv dAlice \pmod{ \phi(mod nAlice)} $


If we multiply both sides by eAlice (and rely on the fact that $dAlice \cdot eAlice \equiv 1 \pmod{ \phi(modnAlice)}$, we get:

$$dAliceA \cdot dAliceY \cdot eAlice \equiv 1 \pmod{ \phi(mod nAlice)} $$

which is precisely the relationship we needed to assume.