Why does raised cosine pulse through Butterworth filter introduce (significant) ISI? Here is a simple MATLAB experiment: 

alpha = 1;
R = 10;
[b,a] = butter(7,alpha*1/R);
x = resample(sign(randn(100,1)),R,1,0); % zero-order-hold
y = filter(b,a,x);
eyediagram(y, 2*R);


With channel twice the bandwidth than the pulses (alpha=2) the eye diagram looks clean. The first thing I am concerned that the eye diagram at the critical bandwidth (alpha=1, i.e. channel bandwidth is bitrate/2) still looks reasonably clean (little bit ISI but not too much):



I would have expected significant ISI at the critical bandwidth.

Now instead of using rectangular pulses, I use a raised cosine filter:

 hcos = rcosdesign(0, 30, R);
 x = upfirdn(sign(randn(1000,1)), hcos, R);


Now the eye diagram is significantly worse:



I would expect the result to be significantly better. I played with all possible rolloff factors 0...1 and number of symbols. I also tried cutting the signal at beginning/end, thinking it may be the initial transient. No changes.

What is going on here?

(Additional info: I do not want to simulate TX/RX with raised cosine but assess a channel without having to use an equalizer - in simulation. One way to do this is using an equalizer or just using a pulse shape that works close to the theoretical bandwidth - that should be raised cosine with a low beta).
That's because Matlab's rcosdesign creates a root-raised-cos filter, not a raised cos filter. If you want no ISI (ie meet Nyquist's criterion), you need to do the match filtering of the reception part : 

%%
alpha = 1;
R = 10;
hcos = rcosdesign(0.25, 30, R);
x = upfirdn(sign(randn(1000,1)), hcos, R);
y = upfirdn(x, hcos, 1, R);
eyediagram(x, 2*R);
hold on
eyediagram(y, 2*R);




eye of x



eye of y