[Tests] test only for fast test as a default
Test failure is not related to change in test file.

Test only for one config on circle ci to speed up testing. Add all config test as a slow test. 
@mariamabarham @thomwolf 