Create Taxonomy term with added text and date fields, then display them in "add content"
Am not familiar with any contributed module which will server your requirement.
But i can help you to achieve this functionality.

Add the Date and Notes field to the specific content type using the 'Add Existing field' option. Since you have already created the fields for taxonomy term, its not necessary to create a new fields.

While adding a new content of that particular content type, mention the category , date and notes in the respective fields. 

Now what we need to do is to hack-into the node submit flow and update the specific taxonomy term with the newly entered date and notes values.

Easily do this by using a single hook : hook_node_presave($node)

You can using this hook in any of the custom module. The '$node' variable will contain the date and notes values. We can easily load the selected taxonomy and update the date and notes of the taxonomy via DB queries

Thanks,
Vineeth