Boolean field: hide (in display) if unchecked
OK, so it is possible and actually rather simple. Here's how I did it in the end:

Edit the field's settings:
"value if checked" -> use the text you want to see displayed when checked
"value if unchecked" -> leave empty

And then under Content type > Manage Display, hide the label for this field.

This way, only the value of this field will be shown.
Since the value when unchecked is empty, nothing will be shown when the field is not checked.