Multiplex data and LED pin
You would not be able to do this without an extra ic or two. The simplest would be a Port Expander. An I2C or SPI port expander would be simple to code, and provide both outputs and inputs. A single i2c 8bit port expander could be used for the shift registers and the encoder's outputs, using only 2 pins, if you use polling to check the encoder interrupts. Three pins if you use a port expander with an interrupt pin. The other three pins on your ATTiny could handle the encoder's led pwm directly.

Or you could add an i2c rgb led driver to handle the led pwm. Or use a 16bit expander to handle everything.

The extra overhead for i2c or spi would not be much on a ATTiny85.