Can an adversary distinguish a private key from a pseudo-random string of the same length? Apologies if this is a dumb question but allow me to describe the dilemma I have:

Suppose that I am protecting a private key on a device using a password &amp; PBKDF2. The obvious attack would be an offline bruteforce by which an attacker would know the password used to protect the private key and would also have access to the private key.

For a very important &amp; undisclosable reason, the password used to protect this private key has a small search space i.e. it is not complex and is of a short length.

The controls I have in my mind are:


Have the key pair generated on the device but send the public key to the backend. The public key will not be stored on the same device as the private key. This will mitigate the bruteforce attack as the attacker would not know if the decrypted blob is the private key without being able to correlate it to it's corresponding known public key. (Hope that makes sense)
Encrypt just the private key using PBKDF2 and provide no "Known Plaintext" to the adversary like -----BEGIN RSA PRIVATE KEY----- or -----END RSA PRIVATE KEY-----
The private key is used to generate a signature which is verified by the back end and this will have limited attempts thereby mitigating an offline attack
I will be using a salt and a slow derivation function


My hope is (and I could be wrong) is that the attacker would not be able to distinguish such a private key from other pseudorandom blobs received after decrypting it using various password attempts in a bruteforce attack. Am I correct in my understanding? Am I missing some fundamental mathematical understanding in the RSA cryptosystem or any asymmetric cryptosystem that debunks the above controls? I wish to understand if the distinguishability of a private key from a pseudorandom string (of the same length) is close to negligible.
Usually choosing a safe password and standard parameter for the PBKDF2 key derivation would be enough protect your cipher. If PBKDF2 is correctly used, the symmetric key you get as output is well generated and attacking the ciphertext is infeasible.

Protecting a private key as you're doing is a standard operation, usually the password is used (in PBKDF2) to generate a symmetric key which is used with a symetric algorithm (let's say AES) to protect your precious private key. 
AES is supposed to be secure even if an attacker is given access to a decryption oracle (i.e.: it can get decryption of every ciphertext he wish but yours, the challenge). So modifiing the plaintext (stripping the header) would not improve the security level.

After your edit.
Let's imagine that someone saw almost all your password while you were typing it or know that it is included in a relatively small set.

You said that you would strip the "-----BEGIN RSA PRIVATE KEY-----" string, that means that the key is in PKCS#1 standard, so what you keep is a base64 encoded string corresponding the following DER structure:


RSAPrivateKey ::= SEQUENCE {
  version           Version,
  modulus           INTEGER,  -- n
  publicExponent    INTEGER,  -- e
  privateExponent   INTEGER,  -- d
  prime1            INTEGER,  -- p
  prime2            INTEGER,  -- q
  exponent1         INTEGER,  -- d mod (p1)
  exponent2         INTEGER,  -- d mod (q-1)
  coefficient       INTEGER,  -- (inverse of q) mod p
  otherPrimeInfos   OtherPrimeInfos OPTIONAL
}



(source)

So, an attacker could try to check for the value of $e$ (usually a standard value among 2 or 3) or check if $p \times q = n$, or ... 
So even if, in theory an RSA key is just a couple of integers, it is usually stored in a well known structure helping the attacker to recognize what he has found.