1. A method of binaural audio rendering performed by an audio playback system, the method comprising: extracting direction-dependent segments of left and right binaural room impulse response (BRIR) filters, wherein: the left BRIR filter comprises a left residual room response segment, the right BRIR filter comprises a right residual room response segment, each of the left and right BRIR filters comprises one of the direction-dependent segments, wherein a filter response for each of the direction-dependent segments depends on a location of a virtual speaker; applying a rendering matrix to transform a left matrix and a right matrix to left and right filter matrices in a spherical harmonic domain respectively, the left matrix and the right matrix including the extracted direction-dependent segments of the left and right BRIR filters; combining the left residual room response segment and the right residual room response segment to produce a left common residual room response segment and a right common residual room response segment; convolving the left filter matrix and spherical harmonic coefficients (SHCs) to produce a left filtered SHC channel, wherein the SHCs describe a sound field; convolving the right filter matrix and the SHCs to produce a right filtered SHC channel; computing a fast convolution of the left common residual room response segment and at least one channel of the SHCs to produce a left residual room signal; computing a fast convolution of the right common residual room response segment and at least one channel of the SHCs to produce a right residual room signal; combining the left residual room signal and the left filtered SHC channel to produce a left binaural output signal; and combining the right residual room signal and the right filtered SHC channel to produce a right binaural output signal.