I was reading this interesting question:

[Is my developer's home-brew password security right or wrong, and why?](https://security.stackexchange.com/questions/25585/is-my-developers-home-brew-password-security-right-or-wrong-and-why)

It shows a weak home-brew algorithm developed by "Dave", and the answers discuss why this is a bad idea. (Actually hashing algorithm rather than encryption, but my question applies to both.)

It makes sense to me that a home-brew algorithm is a very bad idea, but there's one thing I'm not understanding.

Assume I'm an attacker, and I am faced with an weak-but-unknown encryption algorithm developed by "Dave". How would I crack it? I wouldn't even know where to begin. It would be a seemingly meaningless string of characters.

For example, say that the home-brew algorithm is like this:

* Use a weak well-known encryption algorithm on the original data, then:
* Do a bitwise-negative on any byte whose serial number in the file has a repeated digit sum which is prime. (Or any other such mathematical manipulation, this is just an example.)

How would one hack a file produced by such an algorithm *without* knowing it in advance?

**Edit:** Everybody, please don't try to convince me of how hard it is to keep an algorithm secret. Please answer this question on the assumption that the algorithm is kept completely secret, despite of how difficult that is to achieve in real life.

Also, assume that I have no access at all to the algorithm, only to the resulting data.