Maintaining search indices with binary trees
if I change a single index the whole file will need to be rewritten


So don't change indices. When you add data, remember what words already existed, and store subsequent words afterward. For each word, instead of storing one index in the index file, store several.

There is a lot to optimize on top of that. But I don't think this is a productive use of your time. There are existing programs that can do this (and far more complex use cases, too) very efficiently and without raising a sweat: databases. Use a database engine (Berkeley DB, SQlite, NoSQL, MySQL, TSQL, …).

If you want to understand how to store this kind of data efficiently, read about implementations of database engines. B-trees will be involved at some point.