What are the downsides of using arXiv references instead of conference/journal references for the same paper?
Using arXiv instead of a conference / journal reference has an obvious disadvantage: readers won't know that the paper eventually passed peer review and was published.  This is important context, especially if they want to cite the paper themselves later.

Using arXiv in addition to a conference / journal reference might make sense, but some might consider it redundant.