Install a single handle with latch on closet door
If you put a conventional door handle latch system on the pantry door you will want to have a working knob on the inside in the event someone gets shut inside the pantry!!

If the pantry is as you say a closet that no-one could get into then you certainly can install what is known as a "one sided door knob". Google it - you will find a plethora of ideas.