Cannot get rid of the yellow lines
Ctrl + Tab is your friend here. Got into edit mode and switch to Vertex selection. Select the vertices you want to delete, hit x, and select "Delete Vertex." Delete has several options, and some of them are selective. My guess is that you have been selected "Delete only faces" or something like that. 

Another trick: select 1 vertex of the item you want to remove and hit the L key, which will selected all linked geometry. Usually it'll select the whole object you're wanting to delete. 

Also make sure that you've applied or removed any modifiers when you have mesh geometry issues like this—will save you a headache later on.