DQN exploration strategy for large grid-world environment
What does your Q-network look like?

If you're using Convolutional Neural Networks (CNNs), I would advise against using pooling layers since the spatial translation data matters in this application (i.e. a cell with '1' top right of the agent is not the same as a cell with '1' below).


  1) How I might encourage exploration that will not always "cancel out" to only explore a very local region to my starting location?


This is referred to as exploration vs exploitation dilemma.

One way to add tackle this is by adding extra logic to the way you're choosing random actions. To elaborate, the chosen random action shouldn't be the same as the opposite of the previous action: if the agent went LEFT, the next random shouldn't be RIGHT as that is not exploring.

Another method is to use different action selection strategies such as Boltzmann Policy rather than epsilon-greedy. Click here for a quick crash course on the mainstream strategies by Arthur Juliani.


  2) Is this approach a good way to accomplish this task (assuming I want to use RL)?


RL is well suited for this application. Try different parameters for your network. I'd start with a small network and add more layers/neurons as necessary if it does not converge. However, keep in mind that it might take a while (hours or days) to converge so don't get hasty.


  3) I would think that attempting to work with a "continuous" action space (model outputs indexes of "1" elements) would be more difficult to achieve convergence. Is it wise to always try to use discrete action spaces?


I'm not sure I fully understand your question. Mind elaborating? What do you mean by the indexes of "1" elements that the model outputs?