Which characters to take into account when calculating unicity distance? I need to calculate the unicity distance of a block cipher. Assuming I know that the plaintext is an English message I considered using 53 as the size of the alphabet, since the message could contain lowercase ans uppercase letters from the english alphabet, as well as whitespace.
Would this be the right way to approach this calculation?

Thanks in advance.
The alphabet size you need to consider, when calculating the unicity distance, is the size of the ciphertext alphabet.  This follows from the definition of unicity distance, as the amount of ciphertext needed to rule out all but one of the keys.*


For a classical cipher that only encrypts letters to letters, and ignores spaces and case distinctions, the alphabet size would be 26 (or possibly less, e.g. if I and J are merged together).
For a cipher that uses exactly the alphabet you described (upper- and lowercase letters and spaces) for both plaintext and ciphertext, the alphabet size would be 2 &times; 26 + 1 = 53.
For a modern cipher that outputs one 8-bit byte for each plaintext character, the alphabet size would be 256.
For a hypothetical cipher that only outputs 7-bit bytes, the alphabet size would be 128.


Note that, in the formula $U = H_k / D$ for the unicity distance, the per-character redundancy $D$ should really be interpreted as the number of bits of information revealed by one additional ciphertext character.

This can be calculated as the number of bits encoded by one ciphertext character (i.e. as log2(alphabet size), if we assume the ciphertext to have no redundancy), minus the number of bits of entropy per plaintext character, assuming that each ciphertext character decrypts to one plaintext character.  For ciphers where this is not always the case, like the VIC cipher or other fractionating ciphers, things get more complicated.

(Another example of a fractionating cipher would be a modern block cipher in a mode like ECB or CBC that always encrypts or decrypts one full block at a time.  For such ciphers, it's arguably more meaningful to calculate the unicity distance in units of blocks rather than characters.  Of course, you can then convert the result back to, say, bytes, and typically get the same result as if you'd assumed a one-byte block size to begin with.  But you still can't really decrypt a partial block with such a cipher &mdash; except by brute force iterating over the missing part of the block, which makes your already slow brute force attack even slower.)

It's also worth noting that the whole concept of a constant "plaintext entropy per character" is somewhat nonsensical, since it's not really fixed &mdash; typically, the more plaintext you already have, the easier the next character is to predict (and, thus, the less entropy it carries).  The plaintext entropy also depends strongly on the assumed distribution of plaintexts, and, in practice, on how good the attacker is at recognizing valid plaintexts.  Thus, reasonable estimates for the unicity distance of the same cipher can in fact vary widely, depending what assumptions one makes.



*) In fact, this "casual" definition I gave has a subtle error: for most ciphers, there is no amount of ciphertext that can be guaranteed to rule out all keys but the correct one.  Rather, the usual formula for calculating the unicity distance gives approximately the amount of ciphertext needed to reduce the expected number of false positives (i.e. wrong keys that randomly yield a plausible decryption) to one.

This means that, if you have exactly one unicity distance worth of ciphertext, and try to decrypt it with all keys by brute force, you will, on average, end up with two keys that give a plausible plaintext &mdash; one correct key, and, on average, one false positive.  To actually be reasonably sure of getting no false positives at all, you'll need more than one unicity distance of ciphertext.