Why does raised cosine pulse through Butterworth filter introduce (significant) ISI?
That's because Matlab's rcosdesign creates a root-raised-cos filter, not a raised cos filter. If you want no ISI (ie meet Nyquist's criterion), you need to do the match filtering of the reception part : 

%%
alpha = 1;
R = 10;
hcos = rcosdesign(0.25, 30, R);
x = upfirdn(sign(randn(1000,1)), hcos, R);
y = upfirdn(x, hcos, 1, R);
eyediagram(x, 2*R);
hold on
eyediagram(y, 2*R);




eye of x



eye of y