What is the branch of Computer Science that studies how Anti Virus programs work?
There is a subarea of computer security called Computer Virology. The Journal of Computer Virology is devoted to the topic. Studying how anti-virus software works only scratches the surface of what the area is about. 

For instance,  there is even some work applying logic to malware: A General Definition of Malware by S. Kramer and J.C. Bradfield. Journal of Computer Virology (6) 2010.