Is this damaged aluminium frame unsafe to use?
If that's a chainstay/seatstay, then it really is unsafe.

With regards to @DavidRicherby and @ChrisH 's comments, I agree that you should go consult your LBS or any local frame builder. If you're lucky enough, it might just be salvaged by bridging/filling that hole by welding.

If you do plan to ride it in rough areas then I personally suggest you get a new bike frame instead. Fixed or not, the chain/seatstay is already probably compromised enough that I'd get off and lift that bike up every time I encounter a pothole or speed bump.