CiviVolunteer - Log Hours (requires not zero value)
You're not doing anything wrong -- that's just the way the form was designed.

I think it's time this form got an overhaul. Conditional validation would be helpful in this case as well as some others. As an example, I seem to recall someone else pointed out that if you mark a volunteer as a no-show, you still have to enter a non-zero value for the actual duration. Clearly, that's nonsensical.

Full disclosure: CiviVolunteer is primarily maintained by a small development shop of which I am a part. Unfortunately we are not in a position to do unfunded work, but we'd be happy to improve this feature with funding or to help whatever developers you might have at your disposal, provided they contribute the improvements back to the project.