Drum sander attachment for drill keeps coming loose
Your Drill Chuck is actually the problem here. 

You can try a few different things.


Buy a new chuck for that drill.
Buy a new high quality drill.
Since your spindle is damaged already, you might want to put a groove in it for your chuck to bite into it. 


I have no relation or affiliation with any of the following - I am supplying the links so you can see some different ways to get new chucks.

In No particular order of quality, usefulness or reputation; Links for Replacement Chucks Follow:

Toolots

Jacobs Chucks

At Amazon

Craftsman