How to write test cases for custom ajax functionality using simpletest in drupal7
You cannot test custom javascript with simpletest because it simply ignores them. Simpletest can simulate the ajax behavior (via xpath) only if you add that to the form with the #ajax property. http://api.drupal.org/api/drupal/developer!topics!forms_api_reference.html/7#ajax 

To test custom js you might want to have a look at Selenium.