Should I avoid working on my newly installed patio stones?
If you're careful it won't matter too much if you do the patio first. What you do first depends a lot on how close the fence line is to the patio. Digging post holes and pouring concrete can make a mess and you'll want to keep access open to bring material for the patio into the worksite.
Having done a few of these jobs I would suggest that you first dig your post holes and concrete in your fence posts. This will get the digging and concrete pouring out of the way beforehand. I would then lay the patio. It will be easier not having the fence in your way as you bring in pavers, sand, etc. After the patio is done finish the fence.
Good luck.