Question about symbol detection in ISI channels (use of APP decoder in Matlab) I am studying the problem of digital symbols detection in ISI channel. In the figure below, the output symbol is represented as a weighted sum of input symbols. 



I am trying to use MAP decoder comm.APPDecoder to estimate the transmit symbols. To construct the trellis of such figure, I am using the Matlab function poly2trellis. The function can be used as follows:

trellis = poly2trellis(ConstraintLength,CodeGenerator)


My question is how can I enter (or consider) the real-valued coefficients $\Gamma_{-L_2}, ..., \Gamma_{L_1}$ in the second input argument CodeGenerator. If these coefficients are binary, then it is straightforward to convert this binary sequence to octal. But, I have no clue with real-valued coefficients. Any hints?
As you mentioned, applying linear equalization (e.g. filtering with the inverse of the channel) will not yield the optimal performance. Though, what you are searching is not the comm.APPDecoder (which is solely for convolutional codes, i.e. bits with binary $\Gamma$), but you are looking for maximum likelihood sequence estimation (MLSE). 

Here, MATLAB offers the comm.MLSEEqualizer object, which you can e.g. use like this:

Constellation = [1+1j, 1-1j, -1+1j, -1-1j];
Channel = [1 0.7, 0.3, 0.2]';
sigma2 = 0.3;

tx = Constellation(randi(4, [100,1])).';

noise = sqrt(sigma2/2)*complex(randn(size(tx)), randn(size(tx)));
rx = filter(Channel, 1, tx) + noise;

mlse = comm.MLSEEqualizer('Channel',Channel,'Constellation',Constellation);

estimated = step(mlse, rx);

[estimated tx abs(estimated-tx)]


program output (only a few rows of it):

   1.0000 - 1.0000i   1.0000 - 1.0000i        0          
  -1.0000 + 1.0000i  -1.0000 + 1.0000i        0          
   1.0000 + 1.0000i   1.0000 + 1.0000i        0          
   1.0000 + 1.0000i   1.0000 + 1.0000i        0          
   1.0000 + 1.0000i   1.0000 + 1.0000i        0          
  -1.0000 - 1.0000i  -1.0000 - 1.0000i        0          
  -1.0000 - 1.0000i  -1.0000 - 1.0000i        0          
   1.0000 + 1.0000i   1.0000 + 1.0000i        0  


as you see, the MLSE equalizer completely recovers the transmitted sequence.