Issue tracking when writing a paper
It sounds like you want to use a version control system, combined with a bug tracker. Luckily, Mercurial combined with an online bitbucket.org repository provides a nice workflow. When remarks or notes appear, you add them to the issue tracker at your bitbucket page (note that bitbucket provides private repositories). See this example from one of my software projects. 

When working on the paper you can select one of the issues, fix it, add to your repository, with a mention of the issue number in the bug tracker. This sounds like it could provide you with two important things: managing of the growth of your paper, and organizing your workflow of handeling the interaction with the other authors.