Are there any analytic approaches to the design of quadratic polynomial filters?
Not really an answer (hence this is community wiki), but I thought we should capture @Mohammad's code:

%Mohammad Z

%Teager-Kaiser Non-Linear Energy Operator. 
function [out] = TKEO(x)
    N = length(x);
    x = x(:).';
    temp = x(2:N-1).^2 - x(3:N).*x(1:N-2);
    out = [temp(1) temp temp(end)];    
end