How to make "Time" part of a date field not required?
I'm not sure this is possible without some re-engineering of how the date module works, or at least writing your own custom date field. 

The date_field_validate() function calls DateObject::validGranularity() which checks to make sure there are sufficient date parts for the granularity you've chosen. If the field instance is set to be required, it's necessary for that check to pass before you can save the field values. Obviously this check won't pass if you haven't provided all of the relevant date parts.

A simple way around the problem might be to use a field collection, to which you can add two fields: one for the date and one for the time. There is a Timefield module out there, I recommend the dev version as the alpha version has some weird problems.

That method would also help with your display issue as the date and time are now separate fields.