I mistakenly chose the CC BY 4.0 license on arxiv. Can anyone take my paper and publish it in a journal or conference?
Can anyone publish my paper in a journal or conference?

They'd need to attribute the paper to you.  Most journals will only accept papers written by the person who is submitting the paper.

Can a journal publish my paper on its own?

Yes.  You already gave permission to do so.  A legitimate journal probably won't do so.
Possibly a predatory journal might do so if they wanted to pad their journal.

Is there any way for changing the license?

Effectively, the answer is no.  The CC BY 4.0 license is irrevocable.  If you release the paper under a different license, people can just take the copy they received under CC BY 4.0 and distribute that.

What are the consequences?

Anyone can take your paper and publish it wherever they want, even if you don't provide individual permission.  However, they will have to cite you.  https://creativecommons.org/licenses/by/4.0/ provides an overview of these permissions.

How bad is the situation for me?

It's not that bad for you.  CC BY 4.0 still requires attribution.  Unless you're trying to make money selling your paper, there's usually no reason to care if people freely distribute your paper.  Of course, many journals will refuse to publish a CC BY 4.0 paper (e.g., because they sell access to the paper or because they want an exclusive license).
Edit to add more details in reply to comments

Is it legal for someone to publish my paper without asking for explicit permission?

Yes.  Anyone is free to &quot;redistribute the material in any medium or format.&quot; Mind you, they still have to attribute the work to you.

Can they do so without contacting me?

Yes.

Someone I dislike published my work.  Do I have any recourse?

Not usually.  Many licenses, including CC-BY 4.0 are specifically designed to prevent authors from having recourse, except if the license terms were violated.

Why would someone design a license that blocks authors from exercising control over how their work is used?

By disclaiming control over your work, licensees are protected from rogue licensors who later retract permission.  Similarly, licensees can spread the work without working with the licensee.  Licensors wishing to spread their work as widely as possible may wish to avoid placing any barriers on distribution of their work.