Creating a PKCS#12 file where public key doesn't match the private key
You could try portecle. Import your keystore.p12 store, using the correct password of course or it will (silently) fail. Delete the certificate entry, and import a new "trusted" certificate entry.

Now it could be that some PKCS#12 defined attributes are different (e.g. friendlyName, which is visible as alias in portecle. Normally the key and certificate are linked through the modulus value however, so this should not make a difference for anybody trying to use the private key and certificate. It's probably best to copy the alias of the certificate just to be sure.

Portecle is written to use the Java KeyStore class, which supports PKCS#12 out of the box. The Java keytool utility uses the same class, but it seems to automatically delete certificates and keys together so it is not much use here.