How is a Particle Filter used to Estimate Parameters of a State Transition Function?
I suppose I can treat the parameters of the state transition function as the space in which I want to generate the particles ... Is this how it's done with a particle filter?

More or less, yes.  You'll find the richest sources of information on this if you look for books on system identification and on adaptive control (of which one important branch is doing system identification on-line, then adapting one's controller to match the identified system).
I can't recommend a good system identification book, but Karl Åström and Björn Wittenmark's &quot;Adaptive Control&quot; is a pretty good book on that topic.