What is the threshold for a software paper to be publishable?
There is no bright line threshold for such things and the answer should be the same as for any scientific paper, hinging on questions of novelty and extension of knowledge. If it doesn't have that, then it probably isn't a good candidate, though the standards of different journals vary widely. 

But you seem to describe a small-if-any advance with little novelty and you also seem to have made up your mind. We can't help you with the judgement and you just need to take a risk and call it. Others may disagree with you, but that is always the case in reviewing. 

I don't really understand your point 5 and might disagree with your emphasis on it. Why is work done within a group less valuable that work that crosses institutional lines? Lots of papers are done by one or a few people within a research group. However, I might agree more if you mean all of their cited papers come from those same people. That isn't necessarily a red flag, but might be.