How to get academic search engines only search in open access journals?
I'm not sure, whether you really meant open access journals, because when I'm searching at home and I'm too lazy to use the VPN of our university, I'm usually interested in publications I can access.

I wouldn't rely on Google Scholar alone, because I'm not sure whether they filter the results. When you study the advanced search of google a bit, you'll see, that some of the switches are really nice. One really important switch is the -filetype option. If you search for instance for image segmentation in Google Scholar you get instantly only results with a downloadable pdf file if you use

image segmentation filetype:pdf


Even more handy becomes this approach in a two step method. As you know, when you "put something in quotes" in google, you search for the exact string. This can be used to do the following:


find some preview of the article you want to download. 
choose a very specific part of a sentence there
search for this part and use the filetype option


Non-scientific example: Sometimes I'm searching for piano notes of some popular songs and since everyone tries to make money with this, you often won't find a free, downloadable version. Therefore, I first search for something like "sheet music evanescence my immortal" and look at one of the first links, where you usually can see one sample page. Then I see, what's on the page and I use this information to adjust my search

"My immortal" "I'm so tired of" "Slowly and freely" filetype:pdf


and the first hit is a high-quality pdf with all pages.

The same approach sometimes works for scientific publications, because most scientists put their publication on their website. So while you have to pay on the journal website, you may get it for free somewhere else.