Is it hazardous to leave a hole in the wall for a week or two?
Breathing in the dust isn't healthy, but you've basically already created all there is to create. You should vacuum* it up and fill the hole whenever you get around to doing it. No rush.

If you patch the hole properly, you will probably create more dust patching the hole than you did in making it.

If you want to scare yourself look up the effects of breathing in silica dust.

*a shopvac with a fine particle bag or hepa filter is best for vacuuming, as drywall dust is very fine