Temporal kernel filtering on video frames coming in a loop
Maybe not the answer you're looking for, but I focused on that loop:

for j in range(1, NewDepth) :  # iterate from oldest to newest frame
    temp = temp + model['TempKern'][j]*Buffer[sindex]
    sindex = (sindex+1) % NewDepth


To speed this up you could:


create a variable to avoid accessing model['TempKern'] (dict access)
use in-place adition for temp


improved:

tk = model['TempKern']
for j in range(1, NewDepth) :  # iterate from oldest to newest frame
    temp += tk[j]*Buffer[sindex]
    sindex = (sindex+1) % NewDepth


Aside: outside the loop you do:

temp=np.zeros(tuple(clip_size),dtype=float)
temp = model['TempKern'][0]*Buffer[sindex]


so you can remove the first line, as it has no effect, temp is overwritten at the next line.