Building a search tool and classifying text using NLP and ML Im a newbie in information Retrieval. Currently Im reading a book entitled "An Introduction to Information Retreival" by Christopher D. Manning and Prabhakar Raghavan. Im trying to implement an ai based search tool to search for some relevant information from a private dataset. ( say a chemical or mathematical dataset that invloves more number of numericals and unstructed messy representation of unit of measure)

Following is a row in my dataset in JSON format where keys are the coloumn name and value are the values corresponding to it. 

  "ABC Project": {
    "In/Out diameter": " Both in and out are 1”  ",

    "Design Pressure (barG)": {
      "Max": "116 psiG (7.99 barG)",
      "Minimum": "79.7 psiG (5.49 barG)",
      "Design": "174 psiG (11.99 barG)"
    },

    "C02 %": "0.671",
    "MW": "16.68 kg/kmol",
    "TITLE": null,
    "Mothiram": "There is  very dense forest and their lived a king. The name of the king was Pandidhurai. He was very brave ",
    "Thooval delivery material": "- thooval delivery material is panam patta 316/316L  "


The sample search query that will fetch the above mentioned row from my dataset would be  

" The project in which molecular wight is nearly 16 kg/kmol and Thooval delivery material panam patta 316l is used in which Mothiram is King Pandidhurai "

What I have done.

Preprocessing 

Read each row from table(say $T$) as $R_{i}$ and search query as $Q$ do  the following preprocessing 


convert_lower_case(data)
remove_punctuation(data)
remove_apostrophe(data)
remove_single_characters(data)
convert_numbers(data)
remove_stop_words(data)
stemming(data)
remove_punctuation(data)
Convert Word to Vector for coloumn $C_i$ and do it for each corresponding row in table $R_i$
Build Vector Space Model for coloumn $C_i$ and do it for each corresponding row in table $R_i$
Build a vector space model for query $Q$.
Compute cosine score to select Row having high score


Challenges I'm facing 


When I do 'remove_punctuation(data)' Im loosing vital informations (say  $"$ denotes inches in the diameter coloumn) 
A way to interpret unit of measure in the data
Unable to keep the relation between value ( say 16.78 kg/k mol  is stored in multi diamentions and unable to find it is related to MW). 
I think it could be solved by building a ML classifier and train it to identify the entity MW. Say &lt; 16.78 kg/k mol , MW > and using the unit of measure as a feature. But there are values like percentage composition of different chemical components, IN and OUT diameter , Inside and Outside Temperature. etc
Difficulty in interpreting scientific terms and mapping to one root form ( say MW, mol wt, molwt etc are all different ways of representing molecular weight)
Difficulty in finding the close proximity of neumerical values


Question


  
  Please suggest me a step by step approach to build a search tool using this dataset (Please also suggest me an apt algorithm that would
  be useful in each step)
  



I have read from here that some documents can be tagged as &lt; document, class > and train a Navie Base Classifier Model for better search performance. Is it possible in this case , if so what could be class label you would be suggesting ? UPDATE : I think Navie Base cannot be used as the coloumn or key would  be greater than 1000. Im looking for a scalable approach to the problem.



  
  Is there any better approach to solve this problem than using a vector space model and  computing cosine similarity ?
Here are a few general thoughts about your project:

As far as I understand, you're trying to extract very specific information from a semi-structured database using free-form natural language queries, correct? If yes it's important that you realize that this is a quite ambitious project, reaching a decent stage of quality is probably going to take a lot of work, and the performance is unlikely to be perfect.

Apparently numerical values and units are important pieces of information for matching the query. In this case you should probably implement a special process for those, because standard text processing is not going to work very well.


Detection: if there is only a small number of possible ways in which these values are written, it's probably more efficient to use ad-hoc regular expressions. If not, you could try to train a custom NER model.
Representation: that's the tricky part imho. For general text vectors are fine, but with vectors it's difficult/impossible to represent special values like those. Given that you have a semi-structured database, you might want to try a more semantic representation adapted to your data: that could involve techniques from semantic role labeling, relationship extraction, etc.
Matching: the advantage of a semantic representation is that you could convert the query to a semantic representation and then apply a detailed matching procedure suited to your data (in particular you can compare corresponding numerical values and use a threshold, or return the difference to represent how far they are).