What screws should I have used for drywall? (And can I fix this situation?)
I expect you want some kind of wall plug. You would take the table down, install the wall plugs (which generally need a bigger hole than screws so your exiting screwholes shouldn't be an issue) and then screw the shelf to the wall plugs. 

Here in the UK I would look at something like this, but i'm not sure how american drywall compares to our plasterboard (also not sure if that one is the right size, I generally judge sizes of such things by eye rather than measurements).