Do I need to install drainage medium for my sump pump to work well in an old basement?
Sump pumps can't do their job effectively without a suitable route for water to reach the pail. You don't have that. You may not need to remove the entire slab, though. It's possible that evacuation routes around the perimeter would do the job. It's hard to say with the small window into the problem we have. 

Note that it's not "gravel" that you'd install--it's washed rock and possibly drain tile (perforated, corrugated plastic pipe), which may have a fabric sleeve over it. If you're going to all the effort and expense, be sure you have a solid plan.