Proof of concept for RSA malleability
NAND is not the appropriate combiner for RSA malleability , and can causes the issue observed by making what's produced larger than the public modulus $N$.

RSA malleability applies to textbook RSA, where encryption goes $$x\to \text{Enc}_{(N,e)}(x)=x^e\bmod N$$ and malleability is the property $$\text{Enc}_{(N,e)}(x_1\cdot x_2\bmod N)=\text{Enc}_{(N,e)}(x_1)\cdot\text{Enc}_{(N,e)}(x_2)\bmod N$$ where $\cdot$ stands for integer multiplication (just nothing to do with NAND).

Verifying this with plaintext consisting of files with bytes containing 1 and 2 requires extracting $N$ from the public key, and large-integer arithmetic modulo $N$.