16 Segment LED, 4 Digit, Common Anode (4 sets)
I'd recommend using an MCU (and some external driver circuitry) for this. Each of your displays should be mounted onto a tiny board containing the 4-digit display and one MCU + circuitry. The block diagram of functions would look something like this:

You need 64 bits of RAM for the (4) 16-seg display. That's easy to find in any MCU.
I have added a special &quot;control register.&quot; I'd recommend supporting at least one 3-bit control word to specify the desired brightness level (handled using PWM.) The intensity levels will be 100%, 53%, 40%, 27%, 20% 13%, 6.6%, and 0%. You could add bits for features like &quot;blinking&quot; or other ideas. Up to you. But if you don't provide for a special control register, you'll probably regret it.
I think you might consider using a 9-bit serial data word size. (Maybe 10-bit.) If you are willing to break up the 16 bits required for a digit into four 4-bit transmissions, then the idea is that you'd transmit the 2-bit digit-select, the 2-bit section-select (there are four 4-bit parts making up the full 16 bits required for a digit), and the 4 bits of data. In addition, since you need to be able to write to the control word, the upper (or lower) bit of the 9-bit serial data word would specify whether this was a control-word write or a 4-bit write to some part of a digit.
I've included the idea of a \$\text{CLKSEL}\$ line. If this is high, then the MCU would generate a master clock as an output on the \$\text{CLK I/O}\$ line. If it is low, then the MCU would use the observed clock seen on \$\text{CLK I/O}\$ (configured then as an input. ) Thus, you can use this concept to combine two or more 4-digit displays into a single concept. The only thing you'd otherwise need is a way to select the appropriate \$\overline{\text{LOAD}}\$ with an external decoder so that when writing data you could make sure the right 4-digit module is addressed.
The \$\overline{\text{RESET}}\$ line doesn't have to be brought back to where you control the full system, if you don't want to.
This brings things down to about the four lines you wanted, I think.
You'd need to select an MCU with sufficient I/O lines or else add more circuitry. I'd probably look for one with enough I/O.
Just a thought.
Other than this, you may be able to find an existing LED display controller IC for your needs. (Such as the boutique Maxim MAX6954.) But shopping questions are off-topic. So I'm not going to bother looking one up for you.