How to measure the frequency response of an acoustic filter by input and output signals
tfestimate is a good approach. But it seems you're using the wrong kind of input to test the filter. If your input signal only contains lower frequencies tfestimate has not enough information to compute the transfer for higher frequencies. It's sort of a division by zero situation, then.