How can i connect 01 ESP8266 to a shift register? I am connecting this shift register to my esp8266 01 for more GPIO options.
it appears there is a GPIO 16 but it's tied to a RESET pin. I checked both datasheets :
i checked the esp8266 01 datasheet 
http://www.microchip.ua/wireless/esp01.pdf

i checked the datasheet for the shift register for more information on what kinds of pins can be hooked to each of the input pins. 
https://www.sparkfun.com/datasheets/IC/SN74HC595.pdf

also thinking whether it's viable to connect one of the other available pins of the esp8266 and use it for this connection with the shift register. (either RX or TX)

My understanding is need to connect 3 pins of the ESP8266 directly to the shift register for its latch, clock , serial pin. 

also wondering if can connect the serial pin of the ESP8266 to the serial pin of the shift register?
While I do believe you could use GPIO2, TX, and RX for your connections, I don't think the clock is explicitly exposed on the ESP-01. That means your code would have to "emulate" a clock via GPIO2 (most likely) with that pin going to your shift register's clock pin. That said, it's really hackish and I'd recommend other ESP dev board that has a pin for the onboard clock.