Does Couchbase/CouchDB allow me to move / skip the CRUD middle man from the server?
I took a look at the REST API endpoint list on the Couchbase docs site.

I think it might be hypothetically possible. You could use the N1QL REST API to do inserts, selects, updates, and deletes.

I can't say for sure whether this is going to help scaling and load or not, but I do think you're going to have to address a number of other challenges: validation, authentication, authorization, business logic, and more.