How to review a software-tool paper?
One perspective is from the Journal of Open Source Software, which explicitly establishes reviewer guidelines that cover these questions (http://joss.theoj.org/about#reviewer_guidelines).

A (slightly edited) quote of the demands:


  Software license
  There should be an OSI approved license included in the repository.[...]
  
  Documentation
  There should be sufficient documentation for you, the reviewer to understand the core functionality of the software under review.
  
  A statement of need
  The authors should clearly state what problems the software is designed to solve and who the target audience is.
  
  Installation instructions
  There should be a clearly-stated list of dependencies. Ideally these should be  handled with an automated package management solution.
  
  
  Good: A package management file such as a Gemfile or package.json or > equivalent
     OK: A list of dependencies to install
     Bad (not acceptable): Reliance on other software not listed by the authors
  
  
  Example usage
  The authors should include examples of how to use the software (ideally to solve real-world analysis problems).
  
  API documentation
  Reviewers should check that the software API is documented to a suitable level. This decision is left largely to the discretion of the reviewer and their experience of evaluating the software. [...]
  
  Tests
  Authors are strongly encouraged to include an automated test suite covering the core functionality of their software.
  
  
     Good: An automated test suite hooked up to an external service such as Travis-CI or similar
     OK: Documented manual steps that can be followed to check the expected functionality of the software (e.g. a sample input file to assert behaviour)
     Bad (not acceptable): No way for you the reviewer to check whether the software works
  
  
  Community guidelines
  There should be clear guidelines for third-parties wishing to: Contribute to the software, report issues or problems with the software, or seek support


This is a valuable way of thinking about reviewing software, and I think we should move closer to this. But it is not yet accepted! In my field, most "methods" papers don't even include software - and those that do would almost certainly fail these tests. (I'd never heard of continuous integration until recently, for instance.)