Does Sugar remember who she had turned into a toy?
There's a way she can remember about her victims. It's mentioned in the manga chapter 738: 


  "Sugar is able to make a contract with her victims and control them,
  leaving them with no control over their own bodies."


Read about her abilities here:  One Piece Wikia - Sugar