Hide summary if empty (no trimming)
Check the Rewrite the value of this field-option of the body field, 
Fill out the following token in the text area: [body-summary]


Now it will only display text from the summary, and nothing when that very summary is empty. So it won't use the trimmed text of the full content either. Perfect.