How can i extract transfer function an unknown nonlinear system?
You can definitely try System Identification Toolbox of Matlab. Official page says

You can use time-domain and frequency-domain input-output data to
identify continuous-time and discrete-time transfer functions, process
models, and state-space models.

Which is what you are looking for.