How to test an encryption against different attacks
Cryptograph Network Security by William Stallings is pretty decent read I had to read in my crypto class. Each encryption method is different, the way you can test its effectiveness as a encryption method is by what they call avalanche effects , where by changing one bit in decryption it changes a lot of other bits throughout the process. 

Also I recommend checking out capture the flags online that have cryptography categories to use for real world practice. Just a simple google search will find you free sites online to practice, and you can even see write ups for the challenges to get better ideas how to attack the encrypted flag. 

Hope this helps, cryptography just takes some time then you will understand it really easy. Good luck