Add test for core.nanops and fix several bugs in nanops
This pull request accomplishes two main things:
1. It adds unit tests for `nanops`
2. It fixes several bugs identified by the unit tests

related #7352 
related #7353 
related #7354 
related #7357 
