How difficult is it to switch majors going from a BA to MA/Phd?
In the US there is normally little problem in chaining fields after a BA/BS degree. It is very common. This is partly due to the general nature of the baccalaureate in the US. For example, a math or science major also has quite a few courses in the other as well as in the social sciences and humanities. And someone majoring in the humanities has an even broader view of that and the social sciences. 

However, note that you may be missing a few things when you begin graduate study. This is normally covered by the advanced courses that you take starting out in either a masters or doctoral program. I think a change such as you desire wouldn't be a block at all. 

But you will need to measure up on all of the other things such as GPA and letters of recommendation. Doing well in related courses will probably be considered. 

But the only way to know for sure is to apply to a few programs and see what feedback you get. 

Having been out of school for a while isn't necessarily a block, but if you have gotten rusty in your field, then you will have a harder time getting back into the groove. And letters may be harder to obtain, so try to reestablish contacts with professors before you apply. They can also give you advice. 

Your statement of purpose in returning to study will probably be important.