Transfer apps and data from a rooted phone to a phone without root
I tried your command and in the logcat an error was given "no backup packages supplied and neither -shared nor -all given" so I assume you are missing some parameters. 

I tried the following command "adb backup -all" and the backup window successfully popped up. My advise is to check your logcat or modify your adb parameters. As per this post the command 


  adb backup -apk -shared -all -f c:\backup20111230.ab


should work fine.