How to avoid input noise?
From a quick look at the two datasheets:


You should expect the 2981 to have very sensitive inputs - these are voltage driven logic inputs, which take a few hundred uA max to turn them on.  They're expecting to be driven by a source which is confident about what voltage it's driving.  If you disconnect them, you can expect them to wander around.
The segment outputs of the 7219 are current source outputs, which are expecting to supply current (mA) to the anode of an LED - instead you've connected them to a high-impedance input which basically draws no current at all.


I'm not really clear what the 2981 is bringing to the party here, but I guess you want more segment drive.  If that's the case, you should convert the SEGx current outputs to voltage outputs, by arranging a pull-down resistor on each one.  You will need to chose the current programming for the SEGx drives, and the resistor value, to give you sensible input voltages for the 2981.