do we only use pull-up resistor or pull-down resistors when dealing with switches?
It would help if you give more specifics as to where have you "heard" that and in which type of circuit. I can only say that while with a switch you have a floating input, with other components you have a leakage current which pulls the node up or down, so a pull-up or pull-down resistor serves to counter the effects of that leakage current.
In short, we don't use pull-up or pull-down resistors only with switches because of their floating state, but also with other components (including, but not limited to, electrolytic capacitors and transistors) because of their leakage currents.