Question about symbol detection in ISI channels (use of APP decoder in Matlab)
As you mentioned, applying linear equalization (e.g. filtering with the inverse of the channel) will not yield the optimal performance. Though, what you are searching is not the comm.APPDecoder (which is solely for convolutional codes, i.e. bits with binary $\Gamma$), but you are looking for maximum likelihood sequence estimation (MLSE). 

Here, MATLAB offers the comm.MLSEEqualizer object, which you can e.g. use like this:

Constellation = [1+1j, 1-1j, -1+1j, -1-1j];
Channel = [1 0.7, 0.3, 0.2]';
sigma2 = 0.3;

tx = Constellation(randi(4, [100,1])).';

noise = sqrt(sigma2/2)*complex(randn(size(tx)), randn(size(tx)));
rx = filter(Channel, 1, tx) + noise;

mlse = comm.MLSEEqualizer('Channel',Channel,'Constellation',Constellation);

estimated = step(mlse, rx);

[estimated tx abs(estimated-tx)]


program output (only a few rows of it):

   1.0000 - 1.0000i   1.0000 - 1.0000i        0          
  -1.0000 + 1.0000i  -1.0000 + 1.0000i        0          
   1.0000 + 1.0000i   1.0000 + 1.0000i        0          
   1.0000 + 1.0000i   1.0000 + 1.0000i        0          
   1.0000 + 1.0000i   1.0000 + 1.0000i        0          
  -1.0000 - 1.0000i  -1.0000 - 1.0000i        0          
  -1.0000 - 1.0000i  -1.0000 - 1.0000i        0          
   1.0000 + 1.0000i   1.0000 + 1.0000i        0  


as you see, the MLSE equalizer completely recovers the transmitted sequence.