What does pg_stat_activity.xact_start mean for autovacuum threads?
The xact_start field should be before or equal to query_start, covering the entire duration of the vacuuming of the table.  

Your vacuum is slow because your setting of vacuum_cost_limit is very conservative.  It is throttling itself, not blocking on other things.