Canasta and the rules on invalid melding
There are two different rules that pertain to this and it will differ based on the exact rules being played at the time. Some rules list for the first meld you would take it back but have a penalty for next attempt. Others say you would take back the invalid meld and pay a point penalty.

The main point of having to leave the cards face up on the table if you expose them is to prevent partners from sharing information for their benefit.

Rules


  If a player exposes any cards from his hand, but does not meld them, he must leave them face-up on the table, and discard them one at a time in subsequent turns. However, if these cards can be validly melded before they are discarded, he may do so, rather than discard them.    
  
  If a player makes an invalid meld by adding more than three wild cards to a meld, he may either transfer the extra wild card(s) to another valid meld, discard them, or return them to his hand and take a 100-point penalty. (This penalty also applies to the initial meld.)


More Rules


  If you put down an initial meld that doesn’t meet the minimum point requirement, you can put down additional melds on that same turn to make up the required total without penalty. You can also take back your attempt, but the minimum total you have to meet the next time you try goes up by 10 points — quite a severe penalty.