smd vs toroidal RF filter coils: best choice and for which frequency ranges in receivers
Toroids, with a closed magnetic path, will have much lower stray magnetic fields to be picked up by other inductors, as well as picking up much less from other inductors.

So one possible difference between the two filter implementations is coupling between the separate inductors.

Test this by isolating each inductor on the SMD filter (removing capacitors, cutting tracks etc), applying signal to one inductor and measuring what (amplitude and phase) each other inductor picks up. 
Repeat for each inductor in the filter.

If it reveals nothing, you have eliminated this hypothesis and move on to the next answer.

Otherwise, this will give you some idea of the coupling (stray inductances) between components - what do you do next?


Add inductors to your simulation model, reflecting the worst few culprits, and see if that explains most of the measured discrepancies.
Rotate inductances while measuring, you may be able to arrive at a component placement that eliminates most of the coupling by locating inductors normal to each other.
Experiment with screening (cut thin copper sheet to make walls between inductances or screening cans around the worst offenders
Replace one or more with toroids (or "screened" versions of the same inductor)
Possibly add further components to add out-of-phase versions of the coupled signal to cancel (aka "neutralize") some of the coupling.


until you have achieved a satisfactory compromise between performance, cost and difficulty of assembly.