Searching text for (potentially) tens of thousands of tokens
<p>I am maintaining a simple php-based in-house cms. I'd like to search the text of articles as they are saved into the system for what will eventually be tens of thousands of different tokens, in order to automatically apply links to those tokens and also to establish a relationship in an association table between the article and the entity the token represents.</p>

<p>What is the best way to do this? Is there a faster/more efficient way to do it than to retrieve a list of all of the tokens and their relevant entity/id every time an article is saved?</p>

<p>I'm less interested in the replacement of the tokens than the best way to establish the list of tokens to search - they will come from several different tables, and I would think that on a per-request basis the data set which needs to be queried would be quite a burden on both the DB and the memory load of the script</p>

<p>Edit: I think I've posed the question incorrectly.</p>

<p>Consider the following text:</p>

<p>Steve McMuffin ate seventeen Fabulous Furry Fajitas at The Stinking Bean, while Johnson Fatlumps ate thirty-two.</p>

<p>I've got two people in there who are both in the 'person' table, one restaurant which is in the 'restaurant' table and one restaurant menu item which is in the 'restaurant_menu_item' table.</p>

<p>I want to know the best way, after that text is saved, to automatically go through and identify what is a person, what is a restaurant, and what is a restaurant menu item <em>without</em> resorting to custom markup as the intended audience has virtually no chance of ever getting that right.</p>
