How to implement a $j$-level $M$-band wavelet transform of an image?
[EDIT: some code made available]  The  notion of $M$-band wavelet transform, $M\ge2$, generalizes the standard $2$-band wavelet. The theory is provides, for instance, in Theory Of Regular M-Band Wavelet Bases, P. Steffen et al., 1993. Remember that with $2$-band wavelets, one cannot obtain wavelets that are real-valued, symmetric, orthogonal and with finite support, except for the Haar wavelet (not very regular). For $M&gt;2$, such designs can be obtained with sufficient regularity, and $M$-band wavelets are sometimes believed to be more efficient for image analysis. 



Alas, regular $M$-band filterbank are generally harder to design, and $M$-band wavelets have been used more rarely. Resultingly, they are less implemented in standard software. 

On $M$-band implementations: I am not aware of build-in Matlab functions. However, there exist $1$D versions, that you can built upon for your applications:


LTFAT (Large Time-Frequency Analysis Toolbox) : WFILT_CMBAND - Generates $$M$-Band cosine modulated wavelet filters, from On cosine-modulated wavelet orthonormal bases, 1995
M-band dual-tree wavelet: Shannon &amp; Meyer wavelets, and Modulated Lapped Transforms for specific $M$s, and a $4$-band filter from Design of Efficient $M$-Band Coders with Linear-Phase and Perfect-Reconstruction Properties, 1995, Alkin and Caglar. They  come with their Hiilbert transform, and are detailed in Image Analysis Using a Dual-Tree M-Band Wavelet Transform, 2006; Noise Covariance Properties in Dual-Tree Wavelet Decompositions, 2007; A Nonlinear Stein Based Estimator for Multichannel Image Denoising, 2008. Below, 2D oriented $4$-band wavelets from Alkin and Caglar:




A $2$-D version is now embedded in the Matlab toolbox:  $M$-band $2$D dual-tree (Hilbert) wavelet multicomponent image denoising. Other sources to grab coefficients from:

-An algebraic construction of orthonormal $M$-band wavelets with perfect reconstruction

And one day, I shall rewrite my own codes to make them much cleaner.