Why this challenge in Yu-Gi-Oh! 5D's Tag Force wasn't completed?
Maybe I'm wrong, but, why the challenge wasn't recognized?


Perhaps this could be a bug on that game... but I suspect the reason was the following: you actually did 8000 points of damage, not 10600.

The first hit your Marauding Captain did caused 5300 damage; that is true. Your opponent's LP went from 8000 to 2700 thanks to that attack.

Now, on your second attack you actually did... 2700 damage, and not 5300. Why? Because on YuGiOh Lifepoints can't go negative. From the rules on LP:


  
  If a player's LP would drop below 0, they become 0 rather than negative.
  If a player's LP become 0, they lose the Duel. 
  


This means that you did a net amount of damage of 8000, and not 10600 as it may seem.

In order to achieve the challenge, perhaps you will have to fight an opponent that gains LP, so when you win you would have made more than 8000 damage (and perhaps 10k to complete the challenge).