When would a paint-roller be considered "worn out"?
If you can't get all the paint out then that's a good time to think about replacing it.

The problems you might get are some paint flecks from the old paint coming off onto the newly painted surface - which will ruin the finish, or some of the old colour leaking into the new.

I wouldn't worry about the colour of the roller material itself and it is a good sign that you can still get a nap on the roller.