What type of foam to fill this pipe hole?
Great Stuff can be used, but I would not fill it with anything especially if I did not know what it was - its purpose. 

You really should find out what the purpose of that pipe is. Is the pipe part of an old drain left after a bad remodel, a vent pipe? 

If the pipe is PVC or Metal you can just cap it off - they make caps for these pipes less than $5 USA.