How to know when its time to quit grad school, due to my own failings
The results my codes are calculating show that my method makes no
  improvement to current methodology and sometimes its actually worse.


For a master's degree, there is usually no requirement that your research produce a successful result. One can make a contribution to science by showing others what doesn't work. But you may find that your new method is an improvement under certain circumstances. Or you may get an idea (after you get some sleep!) for improving your method. That's exactly what happened to me.

Given your lack of sleep and your state of mind, I'm not sure you should trust your impression that your advisor is disappointed in you. I recommend you have a heart-to-heart talk with your advisor, being as open as you can about all your concerns and fears, including that you are stressed out and not sleeping. It's very common for students to feel like failures at some point during a master's degree, and your advisor should be ready to help.


  I wrote the first few pages of my thesis and basically everything was
  wrong: formatting, writing style, poor graphs and figures.


Academic writing is a very different beast than the types of writing you've done before in school. No one is born knowing how to do it. Expect to have to do a lot of rewriting as you learn this skill.