How can I abort a calculation that is hanging in Numbers for Mac?
If you have a Time Machine backup, you can get the file from an earlier state.



From Apple Support:

Restore from a Time Machine backup

Restore specific files:  


Choose Enter Time Machine from the Time Machine menu Time Machine icon, or click Time Machine in the Dock.





Find the files to restore:


Use the timeline on the edge of the screen to see the files in your Time Machine backup as they were at that date and time. The timeline may also include local snapshots.
Use the onscreen up and down arrows to jump to the last time the contents of the window changed. You can also use the search field in a window to find a file, then move through time while focused on changes to that file.
Select a file and press Space Bar to preview the file and make sure it's the one you want. 

Click Restore to restore the selected file, or Control-click the file for other options.


This should get you back to before you made the errant formula.  Provided you have a recent Time Machine backup.