Why the magnitude response of my low-pass passive filter is not flat at some frequencies?
The Agilent design uses inductor and capacitor values that you can actually buy. The NuHertz design (at least the first one) uses values you'd have to have custom wound and carefully trimmed. 

Of course component values also vary from part to part and when the operating temperature changes. The Agilent designer likely determined that given the requirements of the application, the off-the-shelf values are close enough to give adequate performance. For your design, you should simulate the response with different values for the components depending on the precision of parts you're going to use, and verify that performance is adequate over the expected range of variation of all the components.