Does brute force attack use the program that created the ciphertext? I've read several articles about brute force cryptanalytic attacks, but none explicitly say what algorithm is being run for each attempt, nor what criteria is used to declare an attempt a success or a failure. If one is to try every possible every possible key value, what is one feeding those keys into? A brute force attack on a block of ciphertext implies that the cryptanalyst treats the cipher as a black box. Does the cryptanalyst have the block box and are they able to drive the black box?

Say, for example, that someone uses a GUI application that takes a file and a password as input and produces an encrypted file as output. The intended use would be for the receiver to use the encrypted file and the password to get the cleartext file as output. Is this same GUI application somehow driven by the cryptanalyst's brute force application to try every possible password? Most GUI programs are not fast enough to run billions of times in a reasonable period. Add to that the fact that most encryption is compute-intensive and can be made slower to prevent such an attack.
Brute force attacks are usually implemented in custom hardware or software for maximum performance.

Obviously the method of encryption must be determined before the attack can be implemented. The encryption method can be determined in several ways:


It's publicly available e.g. open source code, or open specification
It's secret and gets leaked by someone with access to insider information
It's reverse-engineered e.g by diassembling the application