Simple average calculating formula
You can try the following code

{% set ratingsSum = 0 %}
{% for order in pastOrders %}
    {% set ratingsSum = ratingsSum + order.orderDeliveryTime.value %}
{% endfor %}

{% set ratingAverage = ratingsSum / pastOrders | length %}
AverageTime:{{ratingAverage}}


Difference between yours and my code is i have added .value with order.orderDeliveryTime