How to obtain the average of the number of a field according to the term of taxonomy?
I'm not aware of a configuration that can do that out of the box on the Content type Views. But if it must be that exact Taxonomy Terms Views you can do it with the help of an additional module (check out Views Calc and Views Aggregator Plus). 

But if you only want the result, to show the Average number on Term pages, then you can create a different View, of type Taxonomy term and display it as a Block on term pages. This type will work without additional modules. 


Create Term Views type 
Filter to your Vocabulary, filter Note field to be Not empty 
Add the Taxonomy Name field, just to test, you'll see terms listed in separate Views rows
Add the Relationship to all content using the field Content using Note, (Relate each Content with a Note set to the term.) You can require this Relationship
DO NOT add any Sort criteria, sorting often messes up Aggregation 
Add the Note Field, it will show separate Nodes' Note numbers in separate Views rows
Turn ON Aggregation 
Edit the Note Field Aggregation and set it to Average, Group column Value  

You should now again only see individual terms in rows, and the Note will show the average number calculated
Check the Note field Formatter, the Default might round the numbers not to show decimal places, Unformatted might show more 
If this is all working the last thing to do is to add a Contextual filter to filter according to the term page you're viewing: Add the Contextual for the field relating to your relevant Taxonomy terms and configure it:
 When the filter value is NOT available
Provide default value
Taxonomy term ID from URL
Load default filter from term page
When the filter value IS available or a default is provided
Specify validation criteria
Taxonomy term ID  

Now you can enter a term ID into Preview field and you'll see what the block will display on that term's taxonomy page. 


Place the block in a region, configure the block, limit the Visibility to term pages etc.