Sparse Channel Estimation in OFDM System
Are you still there, I've seen this question right now. 
First let me correct your question, 

Y(k)=H(k)X(k) === > Y(k) received signal in frequency domain

That's true, but it's not necessary that signal in frequency domain in order to be used in compressed sensing, it might be in time domain also. 

Now, regarding your questions, 
the representation of h channel can be sparse too but it's not always, it can be also in other forms .. taking an example of multi-path environment, the received signal can be sparse, and h channel can be either sparse or not, that's not important. 

how to design the dictionary,  that's the goal of the compresses sensing, it's how can you design it,  you can check online, you will get many dictionary. 

Estimation is different about using compressed sensing. you may use pilot or not, that's different topic. 

reference for compresses sensing, is An Introduction to Compressive Sensing
Collection Editors:
Richard Baraniuk
Mark A. Davenport
Marco F. Duarte
Chinmay Hegde

Thanks