Classify ciphertext vs. plaintext I'm attempting a rather simple exercise in machine learning and trying to classify samples of text as either plaintext or ciphertext (encrypted).

Here are two samples:


Plaintext: This is a sentence in plaintext which any human person can read
Ciphertext: 5oXbLiEZbMUgOOdYy+q4+rsDaqUngBrrUbpVeuu2ggvP6hHObC4GgTLhq


The specific encryption used doesn't have any special attributes I can use to classify (for example, the ciphertext isn't guaranteed to be significantly longer than plaintext), so the task is all about figuring out what text is indistinguishable from random characters and what text is readable plaintext.

My current heuristic involves counting whitespaces and assuming anything with a whitespace ratio above a certain threshold is plaintext, but I'm trying to find a better robust algorithm.
Simply counting the frequency of characters ought to easily distinguish between English language and ciphertext, because they're so obviously different.

You can just count the frequency of characters in a big corpus of English, and a big corpus of ciphertext, and apply a chi-squared test to each to figure out which one matches the counts in a new chunk of text.

Or if you can assume ciphertext has a roughly uniform distribution over characters, that alone lets you construct a good test for whether new text is unlikely to be ciphertext.

I did a short blog post on something similar. https://blog.cloudera.com/blog/2016/09/solving-real-life-mysteries-with-big-data-and-apache-spark/