Can we think of an impedance or an admittance as a case for a transfer function?
But wouldn't it be better to think the other way around, as the
  resulting potential produced by a flow through the system?


No of course not; engineers usually think of the current flowing as a result of a potential being applied. However, it all depends on what your primary source is (more often than not voltage).


  Can we think of an impedance or an admittance as a "special" case for
  a transfer function?


Not a special but an actual case.