Views with contextual filter ignores sort order and number of nodes per page
I tried your example (with Drupal 7.39 and Views 7.x.3.11) and was not able to reproduce the issue. In short, I followed the following steps:


Created a taxonomy vocabulary, called "Tutorial Vocabulary"
Created a content type, called "Tutorial Type" containing two additional fields:


"Tutorial term" (Entity Reference to Taxonomy Terms of type "Tutorial Vocabulary") and 
"Weight" (Integer).

Created some content.
Created a View for "Content" type "Tutorial type", added the "sorted by Weight" criterium, removed the default "sorted by creation date" criterium, and configured the default pager to display only "1 item"
Created a contextual filter for the "Tutorial term" field, specifying "Show page not found" when the filter value is not in the URL, and leaving everything by default.


The contextual filter seems to work as expected when I pass the tid of the taxonomy terms: Only the nodes associated to a term are displayed, they are sorted by weight, and only one node per page.

Maybe you are following different steps from the ones I am listing above. If so, please edit your post to specify them, it might be useful in order to find a solution to your question.