How to index molecules in computer programs?
A molecule is a graph, where nodes have labels (chemical element type), and edges are chemical bonds.


while fairly common and acceptable for an initial approach, this models fails to catch many types of isomers and tautomers. There is no well-established general approach because the basic definitions are kinda wonky and the subjects itself is extremely complex. For example, monomethylbulvalene would for all intents and purposes work as a single compound yet a set of different molecules would be present in all reasonably analyzed samples.


  By extension, how to efficiently search a set of molecules and, given a particular molecule, find a set of the closest structural matches? 


That's a part of QSAR big problem: to identify "similar" molecules. It doesn't work reliably. 


  What characteristics of the molecules should be memorized that would allow such search to be efficient?


There are specialized graph databased that aim to solve the problem of efficient search on graph-like structures, but see above.