Relevance of computer science degree to pursue a career in cognitive science
Think further ahead: what do you want to do after you finish the Cognitive Science degree?

Do want to go into academia and work as a professor (probably after some research postdocs) in the field? If that's your ultimate goal, then that's the most straightforward career path.

If it's something else, figure out what the prerequisites for that thing are. There are too many people who got into grad school pursuing a degree in something because they thought was cool only to discover that after they graduate, they have nowhere to apply their degree.

Whether it's worth it depends on you. Weigh the costs to you against the benefits to you.