What is causing my ANC LMS update to diverge?
Turns out it's just a scaling issue. Because the struck.unpack() gives out a signed 16-bit value, the step size is too large for it. Two ways to go around this, you scale down your input and scale back up when outputting, or you just adjust the step size mu.
I used a mu value of 10e-8 and seems like this value works differently in different environments, reasons unknown. With jupyter-notebook is around 10e-7, python idle is 10e-8, and Matlab is 10e-9. Just tweak around that range with the code I posted above and should give you a converged result.
Hope this helps whoever is reading.
edit; for MatLab, I scaled the input by 32768 so the mu value is 10e-9, if max amplitude is 1, use 0.1.