Analog inputs on ATtiny85 don't work
Any type of communication protocol uses pull-up resistors (not 100% sure that all of them do but still, it's a rule of thumb) so using the serial trick on the attiny enables the internal pull-up resistors on PB3 and PB4. I can bet the ADC value is near 1023 isn't it? Disabling the serial communication should in theory fix your problem.