adb backup command on non-rooted device creates an empty backup file?
The workaround was to set the desktop backup password under Developer settings. It seems there is a bug in backup process. If you didn't set a desktop backup password under the developer options settings the above problems occur and your backup will eventually be incomplete(empty backup file). However, I couldn't find an official source for this bug report that setting the desktop backup password solves adb backup issue.

Got this information based on the XDA Forum [GUIDE] Full Phone Backup without Unlock or Root


  Note: There seems to be a bug in which backup and restore operations will
  fail unless a desktop backup password is set under Developer Options.
  It will not work with a blank password.


After I set my desktop backup password under 

Settings &gt; Developer Options &gt; Desktop backup password

adb backup -apk -shared -all -f D:/backup.ab

I am now able to run the above adb backup commands successfully. Note that I left the -system parameter as it is default one and the backup.ab file is almost 6GB now.