SQL Server Memory Performance
I configured the max memory with the same number on each instance. Total 36 Gigs and I gave 17 to each


This is a poor configuration, change is ASAP. You have allocated 34 GB to SQL Server and just 2 GB to OS, in such scenario you are bound to starve OS from memory resulting in paging or memory trimming of SQL Server processes.

If you want to fix the memory for both the instances start with 15 GB for each instance and leave 6 GB for OS. This is considering that you do not have any other services being used like SSAS/RS/IS, if they are their, you need to give 2-3 GB more to OS so that these extra services can use it.


  Why the target memory keep coming up and down on one instance? (moves from 17 to 12 or between)


Target memory can change as per the load on the system, nothing alarming but, having said that you cannot just rely on one counter you have to look at lot of other counters related to memory.