Meaning of the number of days in an average age?
age() returns intervals. In SQL-92 "an Interval is an unanchored directional duration of the time line" [1]. They have two types due to different month lengths in Gregorian calendar (year to month and day to second). Oracle gives you an error message, if you try to do what you did.
Intervals can be positive and negative. They need to be anchored to have a precise meaning, like date + interval = date2. Taking averages of these values is ill-defined in general and trying it for age-values might still give you unexpected results (as it does). 

So what is the meaning of a 3 month 32 days in Postgres? Well only the code can tell (or the one who has written it) for sure. I guess it means "advance one month, then 32 days". One can't convert days to months or vice versa. 

How can it happen to be there? Average is sensitive to outliers, so if some rather large value is in there for days, it will have an influence. How are null values handled? Did some users specify dates in the future or gave unreasonable values? Is there some implicit conversion going on? Have Postgres developers created a special average function for intervals?

For your problem at hand I recommend using (sorry if this doesn't work, I don't have a database at hand):

select avg(extract(epoch from now()) - extract(epoch from birthday)) from "user";

This works for users of reasonable age, but if you had users from the 1700th,the answer would also depend on location, as the new calendar was adopted later in some countries.

Read the following book to get to know other quirks with dates.

[1] "Developing time-oriented database applications in SQL", Richard T Snodgrass, Morgan-Kaufmann, 1999. See his homepage, Pages 30-32.