Getting total vote results for several nodes in a view (Drupal 7, Views 3)
Create a view:


Use Agregation. 
Use fields.


Add fields in this order remove everything else:


Taxonomy or (Category) as you call it. (Aggregation settings = Group results).
field_rating (or your five star field). (Aggregation settings = Average).


Filters.


Published (YES).
Content type = (restaurants).


What you need to do in resume, is to call all the nodes of type "restaurants" group the results the taxonomy field by Aggregation and AVERAGE all the ratings.