Is it very difficult for international applicants to get a Visiting Assistant Professor position in the US?
I'll mention a few important issues here:


There are relatively few visiting positions in mathematics with a certain term of more than one year.  Rather, most of these positions have a term of one year, and may be continued for another year or two if the need for the visiting position continues.  Visiting positions like this are typically created to fill the hole left by a faculty member on sabbatical leave or as a temporary replacement for a faculty member who has resigned, retired, or is too ill to work.  Generally, the plan is to get a tenure track faculty member hired during the one year term of the visiting position.  There are certainly some circumstances in which a visiting position is setup to last more than one year, but these are very desirable positions and the competition for them is intense.  Thus your chances of finding a visiting position with a term of more than one year are limited.  
In order to work in the United States in a visiting position, you will need to have an appropriate visa and the types of visa that are appropriate here (most likely a J1 visa for a temporary position, but perhaps an H-1B visa) require employer sponsorship.  Your potential employer must pay fees to apply for such a visa and typically will need to retain a lawyer to assist in the process.  This can cost thousands of dollars.  Furthermore, there's a substantial risk that the visa might not be granted or might be delayed.  Because of these issues many academic departments are hesitant to hire applicants who must be sponsored for a visa.  In my experience, many universities and academic departments will undertake to sponsor newly hired tenure track faculty but don't want to do this for a visiting position.  
Most visiting positions in the US are focused primarily on teaching.  The hiring committee will be concerned if you have not had experience teaching in the United States, because there are substantial and important cultural differences between the US system and higher education in other parts of the world.  The hiring committee will also be concerned if English is not your native language.  
There will almost certainly be many (tens if not hundreds) of well qualified US citizens and permanent residents with appropriate teaching experience in the US who will apply for the position.   


If you have a strong record of research and would otherwise be qualified for a tenure track position, then you might have a better chance of getting hired into a tenure track position than into a visiting position.