Recent PhD graduate, can I change fields and get back into it?
First, I'm assuming that field X and Y are not totally unrelated. (e.g. Greek literature and high energy physics.)

If you are working in academia this isn't usually a problem. In fact, it's great to have two areas where you can contribute, so that you continue to publish, etc, in area X while starting to do work in area Y. This keeps your productivity up while you get up to speed in area Y.

If you are working for a research lab this can also work, but may be a bit tougher. It depends on how strongly the lab wants you to contribute to the core work of the company. If there is a research lab that is doing work in both X and Y then you can probably find opportunities to collaborate in both areas. If they have no interest in X you need to see if they have flex time or if you can find some connection that would interest them.

But, in the end, the best option is to ask them. You never know what other areas they want to expand into, or what connections they may be aware of that you aren't.