What is playing on Shinji's walkman?
Take a look at this cut from Episode 4 (via the official storyboards), and also the Shinji Ikari character page notes that the two songs that play are:


  The two songs usually played on Shinji's SDAT are: Track 25 for "You are the only one" and Track 26 for "Blue Legend", respectively. The songs are from "Lilia from Ys", an image album for the "Ys" RPG series. The Vocalist is Kotono Mitsuishi, Misato's seiyu. However, Episode 02 has two Eva BGM pieces playing, while Episode 23 has a different vocal song for Track 25 instead. 


Also on Youtube:

Aoi Legend, You are the only one