Correct model for interference from an unterminated transmission line branch
Answering my own question with help from EEVBlog and Wikipedia:
The impedance looking into an unterminated piece of transmission line is:
$$
Z_{in} = -j Z_0 \cot(\beta l) 
$$
Where \$\beta = 2 \pi / \lambda\$ is the wavenumber, \$Z_0\$ is the characteristic impedance of the coax, and \$l\$ is the length of unterminated transmission line.
This reduces the problem of calculating the voltage seen by the oscilloscope to a simple voltage divider with bottom half being a parallel pair of impedances:
$$
V_o = \frac{V_{in} Z_l || Z_{in}}{Z_l || Z_{in} + Z_s}
$$
Below is some python code implementing the above equation and normalising to the base (non-interference) case.
import numpy as np

def interference_attenuation(frequency, length):
    c = 3e8
    v_rel = 0.659  # velocity in coax relative to c - from RG58 spec sheet
    v = c * v_rel

    wavelength = v / frequency

    Zs = 50 # source impedance
    Z0 = 50 # characteristic impedance of the coax (for both pieces)
    Zl = 50 # terminator impedance
    
    wavenumber = 2 * np.pi / wavelength

    # impedance of an open ended transmission line
    # https://en.wikipedia.org/wiki/Transmission_line#Open
    Zin = 0 + 1j * (-Z0 / np.tan(wavenumber * length)) 

    ZlparZin = Zin * Zl / (Zin + Zl) # normal parallel formula
    non_inteference_amplitude = Zl / (Zs + Zl)

    # voltage divider in complex domain, then take magnitude with abs()
    # and make it relatrive to the base case
    # web calculator here:
    # http://hyperphysics.phy-astr.gsu.edu/hbase/electric/vdivac.html
    return abs(ZlparZin / (Zs + ZlparZin)) / non_inteference_amplitude


This model agrees very well with experimental results (for a 50cm unterminated stub here):

It was also pointed out that the reflected wave will see \$50\Omega || 50\Omega = 25\Omega\$ and therefore a secondary reflection will occur. In practice this is normally avoided by not using a terminator at the oscilloscope and relying purely on the waveform generator source impedance to dampen all reflections.
In that case the transfer function would simply be:
$$
V_o = \frac{V_{in} Z_{in}}{Z_{in} + Z_s}
$$