How to pass an entry's Matrix fields as JSON?
I'd strongly recommend looking at the Element API plugin for this.

If you're dead set against that, there are other examples of how to pull out section data in Craft for rolling your own JSON.

For Matrix it'd be no different that if your were pulling Matrix data out to be output in HTML instead of JSON (looping through blocks, grabbing fields, etc). Lots of examples in the Matrix templating docs.

The error you're getting is because when you simply output entry.myMatrixHandle, you're technically given back an ElementCriteriaModel object and the Element API plugin tries to convert that to a string to display in the JSON data and it doesn't know how to convert that object to a string.