BACKGROUND OF THE INVENTION 
       [0001]    1. Field of the Invention 
         [0002]    The present invention relates generally to an improved data processing system, and in particular to a computer implemented method, an apparatus and a computer program product for unsupervised stemming schema learning and lexicon acquisition from corpora. 
         [0003]    2. Description of the Related Art 
         [0004]    A stem is the base or root of a word. The stem may be combined with any derivational affixes, to which inflectional affixes may be added to form a final form of a word. A stem consists, at a minimum, of a root, but may also be analyzable into a root form and additional derivational morphemes. A stem may require an inflectional operation, such as adding a prefix or a suffix to make the stem into a fully understandable word. If a stem does not occur by itself in a meaningful way in a language, it is referred to as a bound morpheme. 
         [0005]    For example, in the English language, the simple verb “tie” is a stem and cannot be further reduced. Adding the suffix “s” to the stem forms “ties” and adding a prefix “un” forms “unties.” An affix is a morpheme added to a word to change its function or meaning. Two typical ways to do this include use of a prefix, adding a morpheme to the beginning of a word and suffix, and adding morpheme to the end of a word as just shown. 
         [0006]    Stemming is a process of reducing a word to a primitive base form, by peeling away instances of prefixes and suffix usage. Morphological analysis comprises stripping away or removal of prefixes and suffixes to reduce a word to its simple basic form or root. Reduction of a word typically involves application of various rules implied by morphology. Morphology, a branch of linguistics, studies patterns of word-formation within and across languages. Based on analysis of the patterns, rules may be formulated to model the formation of words in a given language. 
         [0007]    In many languages, words are related to other words by rules. The study of the rules is called morphology. For example, the words “dog”, and “dogs”, are closely related, apparently having the same visible root. People recognize these relations from their knowledge of the rules of word-formation for the English language. This observed “rule” may be applied to other words to show a similar relationship, such as “dish” and “dishes.” The rules then reflect specific patterns of use based on the manner in which words are generated from pieces. 
         [0008]    Familiarity with high-frequency affixes and roots promotes comprehension of numerous words in which they occur, as meaningful chunks. Morphology is typically used to analyze a body of text, called a corpus, to form a lexicon. When there is more than one body of text to be analyzed, the collection is referred to as corpora. A lexicon is an ordered collection of words and associated definitions. A dictionary is an example of a lexicon. Typically, morphology requires the careful analysis of words within a context according to established rules by linguists using prior learned knowledge and lexical resources. The process is time consuming, laborious, and costly. 
         [0009]    Therefore, it would be advantageous to have a method, apparatus, and computer program product for stemming words in a manner that overcomes some or all of the problems discussed above. 
       SUMMARY OF THE INVENTION 
       [0010]    Illustrated embodiments provide a computer implemented method, an apparatus, in the form of a data processing system, and a computer program product for unsupervised stemming schema learning and lexicon acquisition from corpora. In one illustrative embodiment, the computer implemented method obtains a corpus from corpora, analyzes the corpus to deduce a set of possible stemming schema, and reviews and revises the set of possible stemming schema to create a pruned set of stemming schema. The computer implemented method further deduces a lexicon from the corpus using the pruned set of stemming schema. 
         [0011]    In another illustrative embodiment, the data processing system comprises a bus, a memory connected to the bus, a persistent storage connected to the bus, the persistent storage having computer executable program code embodied therein, a communications unit connected to the bus, a display connected to the bus, and a processor unit connected to the bus. The processor unit executes the computer executable program code directing the data processing system to obtain a corpus from corpora, analyzes the corpus to deduce a set of possible stemming schema. The data processing system is then directed to review and revise the set of possible stemming schema to create a pruned set of stemming schema, and deduce a lexicon from the corpus using the pruned set of stemming schema. 
         [0012]    In yet another illustrative embodiment, the computer program product comprises a computer usable recordable medium having computer executable program code tangibly embodied thereon, the computer executable program code comprises, computer executable program code for obtaining a corpus from corpora, computer executable program code for analyzing the corpus to deduce a set of possible stemming schema, and computer executable program code for reviewing and revising the set of possible stemming schema to create a pruned set of stemming schema, and computer executable program code for deducing a lexicon from the corpus using the pruned set of stemming schema. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0013]    The novel features believed characteristic of the invention are set forth in the appended claims. The invention itself, however, as well as a preferred mode of use, further objectives and advantages thereof, will best be understood by reference to the following detailed description of an illustrative embodiment when read in conjunction with the accompanying drawings, wherein: 
           [0014]      FIG. 1  is a pictorial representation of a network of data processing systems in which illustrative embodiments may be implemented; 
           [0015]      FIG. 2  is a block diagram of a data processing system in which illustrative embodiments may be implemented; 
           [0016]      FIG. 3  is a block diagram of a word stemmer in accordance with illustrative embodiments; 
           [0017]      FIG. 4  is a flowchart of an overview of a word stemming process in accordance with illustrative embodiments; and 
           [0018]      FIG. 5  is a flowchart of a lower level view of the word stemming process of  FIG. 4  in accordance with illustrative embodiments. 
       
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 
       [0019]    With reference now to the figures and in particular with reference to  FIGS. 1-2 , exemplary diagrams of data processing environments are provided in which illustrative embodiments may be implemented. It should be appreciated that  FIGS. 1-2  are only exemplary and are not intended to assert or imply any limitation with regard to the environments in which different embodiments may be implemented. Many modifications to the depicted environments may be made. 
         [0020]      FIG. 1  depicts a pictorial representation of a network of data processing systems in which illustrative embodiments may be implemented. Network data processing system  100  is a network of computers in which the illustrative embodiments may be implemented. Network data processing system  100  contains network  102 , which is the medium used to provide communications links between various devices and computers connected together within network data processing system  100 . Network  102  may include connections, such as wire, wireless communication links, or fiber optic cables. 
         [0021]    In the depicted example, server  104  and server  106  connect to network  102  along with storage unit  108 . In addition, clients  110 ,  112 , and  114  connect to network  102 . Clients  110 ,  112 , and  114  may be, for example, personal computers or network computers. In the depicted example, server  104  provides data, such as boot files, operating system images, and applications to clients  110 ,  112 , and  114 . Clients  110 ,  112 , and  114  are clients to server  104  in this example. Network data processing system  100  may include additional servers, clients, and other devices not shown. 
         [0022]    In the depicted example, network data processing system  100  is the Internet with network  102  representing a worldwide collection of networks and gateways that use the Transmission Control Protocol/Internet Protocol (TCP/IP) suite of protocols to communicate with one another. At the heart of the Internet is a backbone of high-speed data communication lines between major nodes or host computers, consisting of thousands of commercial, governmental, educational and other computer systems that route data and messages. Of course, network data processing system  100  also may be implemented as a number of different types of networks, such as for example, an intranet, a local area network (LAN), or a wide area network (WAN).  FIG. 1  is intended as an example, and not as an architectural limitation for the different illustrative embodiments. 
         [0023]    Illustrative embodiments provide a semi-automatic stemming schema approach that can be used as a morphological analyzer that typically does not require a set of exact morphology rules, or it can be used to dynamically build a user-specific lexicon. A set of morphology rules comprises one or more items or rules. For example, morphology rules define usage of word constructs, such as length, compounds composed of more than one word and affixes which are typically prefixes and suffixes. 
         [0024]    Using the process described in the illustrative embodiments typically reduces the time required to update a language lexicon. 
         [0025]    For example, in an illustrative embodiment a user of the described word stemming process on client  110  may access a corpus on server  106  through network  102  to perform the actual stemming of words located in the corpus. The results in the form of a created lexicon may be stored on server  106  or another server, such as server  104  or storage unit  108  for access by another client  112  that performs the function of a linguist to additionally verify and further process the created lexicon. In addition portions or the entire lexicon may be sent to users through network  102 , as required for subsequent uses. 
         [0026]    With reference now to  FIG. 2 , a block diagram of a data processing system is shown in which illustrative embodiments may be implemented. Data processing system  200  is an example of a computer, such as server  104  or client  110  in  FIG. 1 , in which computer usable program code or instructions implementing the processes may be located for the illustrative embodiments. In this illustrative example, data processing system  200  includes communications fabric  202 , which provides communications between processor unit  204 , memory  206 , persistent storage  208 , communications unit  210 , input/output (I/O) unit  212 , and display  214 . 
         [0027]    Processor unit  204  serves to execute instructions for software that may be loaded into memory  206 . Processor unit  204  may be a set of one or more processors or may be a multi-processor core, depending on the particular implementation. Further, processor unit  204  may be implemented using one or more heterogeneous processor systems in which a main processor is present with secondary processors on a single chip. As another illustrative example, processor unit  204  may be a symmetric multi-processor system containing multiple processors of the same type. 
         [0028]    Memory  206 , in these examples, may be, for example, a random access memory or any other suitable volatile or non-volatile storage device. Persistent storage  208  may take various forms depending on the particular implementation. For example, persistent storage  208  may contain one or more components or devices. For example, persistent storage  208  may be a hard drive, a flash memory, a rewritable optical disk, a rewritable magnetic tape, or some combination of the above. The media used by persistent storage  208  also may be removable. For example, a removable hard drive may be used for persistent storage  208   
         [0029]    Communications unit  210 , in these examples, provides for communications with other data processing systems or devices. In these examples, communications unit  210  is a network interface card. Communications unit  210  may provide communications through the use of either or both physical and wireless communications links. Input/output unit  212  allows for input and output of data with other devices that may be connected to data processing system  200 . For example, input/output unit  212  may provide a connection for user input through a keyboard and mouse. Further, input/output unit  212  may send output to a printer. Display  214  provides a mechanism to display information to a user. 
         [0030]    Instructions for the operating system and applications or programs are located on persistent storage  208 . These instructions may be loaded into memory  206  for execution by processor unit  204 . The processes of the different embodiments may be performed by processor unit  204  using computer implemented instructions, which may be located in a memory, such as memory  206 . These instructions are referred to as program code, computer usable program code, computer executable program code, or computer readable program code that may be read and executed by a processor in processor unit  204 . The program code in the different embodiments may be embodied on different physical or tangible computer readable media, such as memory  206  or persistent storage  208 . 
         [0031]    Program code  216  is located in a functional form on computer readable media  218  that is selectively removable and may be loaded onto or transferred to data processing system  200  for execution by processor unit  204 . Program code  216  and computer readable media  218  form computer program product  220 , in these examples. In one example, computer readable media  218  may be in a tangible form, such as, for example, an optical or magnetic disc that is inserted or placed into a drive or other device that is part of persistent storage  208  for transfer onto a storage device, such as a hard drive that is part of persistent storage  208 . In a tangible form, computer readable media  218  also may take the form of a persistent storage, such as a hard drive, a thumb drive, or a flash memory that is connected to data processing system  200 . 
         [0032]    The tangible form of computer readable media  218  is also referred to as computer recordable storage media. In some instances, computer recordable media  218  may not be removable. 
         [0033]    Alternatively, program code  216  may be transferred to data processing system  200  from computer readable media  218  through a communications link to communications unit  210  and/or through a connection to input/output unit  212 . The communications link and/or the connection may be physical or wireless in the illustrative examples. The computer readable media also may take the form of non-tangible media, such as communications links or wireless transmissions containing the program code. The different components illustrated for data processing system  200  are not meant to provide architectural limitations to the manner in which different embodiments may be implemented. The different illustrative embodiments may be implemented in a data processing system including components in addition to or in place of those illustrated for data processing system  200 . Other components shown in  FIG. 2  can be varied from the illustrative examples shown. 
         [0034]    As one example, a storage device in data processing system  200  is any hardware apparatus that may store data. Memory  206 , persistent storage  208 , and computer readable media  218  are examples of storage devices in a tangible form. 
         [0035]    In another example, a bus system may be used to implement communications fabric  202  and may be comprised of one or more buses, such as a system bus or an input/output bus. Of course, the bus system may be implemented using any suitable type of architecture that provides for a transfer of data between different components or devices attached to the bus system. Additionally, a communications unit may include one or more devices used to transmit and receive data, such as a modem or a network adapter. 
         [0036]    Further, a memory may be, for example, memory  206  or a cache such as found in an interface and memory controller hub that may be present in communications fabric  202 . 
         [0037]    Illustrative embodiments provide a semi-automatic or programmatic stemming schema approach that can be used as a morphological analyzer. The approach used in illustrative embodiments typically does not require a set of exact morphology rules, or the described approach can be used to dynamically build a user-specific lexicon. Using the process described in the illustrative embodiments typically reduces the time required to update a language lexicon. 
         [0038]    With reference to  FIG. 3 , a block diagram of a word stemmer in accordance with illustrative embodiments is shown. A word stemmer provides a capability to analyze a word and determine which pieces may be stripped to reduce the word to a root form. The root may also be referred to as a kernel. The pieces removed during analysis are typically the prefix and suffix components, referred to as affixes. Schema stemmer  300  is an example of a word stemmer in accordance with illustrative embodiments. A schema stemmer then is a word stemmer that links the removed prefix and suffix components to a kernel to show the relationships. A portion of schema stemmer  300  is shown comprising a parser  302 , a counter  304 , a mapper  306 , a receiver  308 , a sender  310 , a splitter  312  and a sorter  314 , located within memory  206  of  FIG. 2 . While shown within memory  206 , components of schema stemmer  300  may be located on storage unit  108  of  FIG. 1  until loaded for use. Corpus  316  is shown outside of memory  206 , as corpus  316  may typically be stored on storage unit  108  and portions brought into memory for processing as required in a usual manner of processing data. 
         [0039]    Parser  302  performs text data parsing of the strings of characters comprising corpus  316 . Parser  302  is capable of walking through the characters of text to determine words. Counter  304  is used to maintain a count for frequency determination. Word counts are one common use. Mapper  306  performs a mapping function, such as the mapping of prefix and suffix variants, the affixes associated with a word root, to the word root, the kernel. A mapping indicates a relationship, the schema, between the mapped entities. 
         [0040]    Receiver  308  and sender  310  provide the sending and receiving communication access to bring data, such as words from corpus  316 , and requests in for processing and sending resultant data out of schema stemmer  300 . Splitter  312  provides a capability of chopping words into defined fragments, such as a prefix, a root, and a suffix. The prefix and suffix are commonly referred to as an “affix,” while the root is called a “kernel.” Sorter  314  provides sorting capability to provide an ordered list of words, kernels, or affixes, as required. Sorting may be performed based on the character values or frequency values associated with affixes and kernels. 
         [0041]    With reference to  FIG. 4 , a flowchart of an overview of a word stemming process is shown in accordance with illustrative embodiments. Process  400  is an example of a process using schema stemmer  300  of  FIG. 3 . Process  400  begins (step  402 ) and accesses a corpus, such as corpus  316  of  FIG. 3  (step  404 ). For example, in linguistic studies, a corpus or text corpus is typically a large and structured set of text data, usually stored and accessed electronically. The corpus is typically used in the performance of statistical analysis, and determining occurrences of, or validation of, linguistic rules for a specific language or set of languages comprising the body of text. In another example, a corpus may be used to determine and indicate the lemma, or base form of each word. Analysis and processing of a corpus may typically be used in computational linguistics, speech recognition, and machine translation. 
         [0042]    By analyzing the contents of the corpus, process  400  deduces possible stemming schema to create a set of derived schema (step  406 ). A stemming schema is a unique pairing of a word kernel and associated affix. A schema then represents a transformation from one affix to another. For example, in the words “binary” and “binaries”, the first affix is “y” and the second affix is “ies.” The schema for the word “binar” may then have a transformation of “y” to “ies” according to a rule for plurals. A review and revision of the derived schema is then performed to optimize the remaining schema (step  408 ). In this operation, rare occurrences are eliminated if not already done, as are kernels having only one affix. The reduced set of schema is then more representative of the main body of text processed. 
         [0043]    Deduce a lexicon is performed to exploit the word stemming just performed (step  410 ) with the process terminating thereafter (step  414 ). The lexicon, a list of words associated with lexical information, may be created using the derived and pruned schema in a further analysis of the corpus used as input. In this step, programmatic use of stemming schemas may be used as a morphological analyzer that does not require any set of exact morphology rules on input. Programmatic use of stemming schemas may be used to dynamically build a user-specific lexicon as indicated. 
         [0044]    A linguist, capable of performing analysis and verification, may be used to further refine the result of the generated lexicon in step  408  (step  412 ). In this optional step of a semi-supervised mode, the derived schemas are applied to the corpus in a given Language to programmatically suggest word roots that may then be confirmed or rejected interactively by a linguist. 
         [0045]    With reference to  FIG. 5 , a flowchart of a lower level view of the word stemming process of  FIG. 4  in accordance with illustrative embodiments is shown. Process  500  is a more detailed example of a process using schema stemmer  300  of  FIG. 3  to find stemming schemas from a list of words and associated word frequencies that are deduced from a large corpus being analyzed. Process  500  starts (step  502 ) and obtains a list of words (step  504 ). 
         [0046]    The list of words is typically a corpus that may be part of a set within a corpora or a single corpus, containing words of a language or multiple languages of interest. Frequent concepts are identified by starting with a list of word tokens and their frequency, as in the corpus (step  506 ). Word tokens are created by separating the corpus into individual words separated by punctuation or blanks. Rare words may be removed from the list of words. Frequent concepts are a count of unique occurrences or frequency of words. When processing to identify suffixes rather than prefixes, reversing each word in the list is performed, and the reversed word list is used. 
         [0047]    Finally, the list of words is sorted using binary values of the characters. The frequent affixes are then identified (step  508 ). Each word in the list is chopped into affixes sized from one to “n” characters long, where “n” is the word length. The frequency of each affix is then counted to generate a global mapping between each affix to a respective frequency. As an option, affixes with a frequency determined to be rare due to a low occurrence count may be removed. The result is the creation of a global set of frequent affixes, which is a count of each unique affix identified in the corpus being processed. 
         [0048]    Word splits are then identified (step  510 ). To obtain the word splits, given a minimal kernel size, each word is then chopped into all possible affix+kernel pairs. Kernel size varies from the given minimal size to full word size, including an empty affix. For each split word, starting from the smaller affixes, locate each affix in the global set of affixes from step  508 . If an affix cannot be located in the global set, skip that affix and any larger affixes. If that affix is located, add the kernel+affix to a global “kernel map” that is a container mapping each kernel to a set of that kernel&#39;s affixes. 
         [0049]    Schemas are then created for each kernel, by examination of the set of the kernel&#39;s affixes (step  512 ). Kernels that have only one affix are skipped. For each kernel, all unique pair combinations of affixes in the set are generated. Each of the generated affix pairs is called a “schema.” A schema represents a transformation from one affix to another affix, such as “y” to “ies” as previously shown, including null affixes. A global map from each schema to the schema&#39;s list of kernels is then created. 
         [0050]    The affix count in the concepts is then generated as a global count (step  514 ). For each affix, the occurrences are counted and categorized by kernel size. For example, mapping each affix to a secondary map that gives the affix frequency for each kernel size. For example, the generation may be performed by iterating on the frequent affixes identified in step  508  and locating the affix in the sorted list of words in step  506 . For each word that starts with a given affix, increment the count for the given affix and kernel size. The kernel size is word size minus affix size. Then, add an entry for the null affix. The null affix points to a mapping that counts the frequency of word sizes for all words in the list of step  504 . 
         [0051]    The affix count in the schemas is then generated by generating a global count (step  516 ). This is a similar count to the count obtained in step  514 , performed on the affixes that form the schemas of step  512 . For each affix in the schemas, count the occurrences categorized by kernel size. 
         [0052]    For example, the count may be obtained by iterating on the schemas of step  512  in which each schema maps to a list of kernels. For each of these kernels, and for each of the two affixes in the schema, increment the count for the given affix and kernel size. 
         [0053]    The schema scores are generated to identify the most useful schemas representing the transformations which are used by a large number of kernels (step  518 ). A schema score is created for each schema identified in step  512  and categorized by kernel size. For example, for each schema, get the list of kernels and generate a score for each kernel size and for each of the two affixes in the schema to consider “schema appearances” by counting the number of occurrences of each kernel size, using the mapping information from step  516  and “general appearances” by counting the number of occurrences of each kernel size, using the mapping information from step  514 . 
         [0054]    Both of the counts are cumulative, for example, the count for kernel size  3  sums the counts of kernel size  3  and  4 , up to the max kernel size. For each kernel size, the schema score is the “schema occurrences” divided by the “general occurrences.” Optionally this score is then normalized by multiplying it with the “general frequency”, which is “general occurrences” divided by the number of words in list of step  504 . Normalizing reduces the score of infrequent schemas and “general occurrences” can be factored out of the calculation, leaving just “schema occurrences” divided by the number of words. Each of the two affixes in the schema generates different scores. For each kernel size the minimum of the two scores is used. 
         [0055]    The best schema for each kernel is identified (step  520 ). Identifying the best schema is similar to identifying a representative word for a kernel. For example, for each schema in step  512 , get the corresponding list of kernels and the schema scores from step  518 . Iterate on the kernels in the schema, and for each kernel get the score for this kernel size. The score will be the schema&#39;s score for this kernel. Generate a “best schema” global mapping from kernel to schema. For each new kernel and schema combination, update the “best schema” if the schema&#39;s score exceeds the previous score found for this kernel. 
         [0056]    Schemas are then pruned to keep only high-scoring kernels (step  522 ). For example, iterate over the schemas from step  512  and each kernel of each schema. Keep only those kernels whose best schema from step  520  matches the current schema. If no such kernels are found, delete the schema from the schemas of step  512 . The result of the process of identifying the high scoring schema is the creation of a set of pruned stemming schema. 
         [0057]    Recalculate the scores now that low scoring entries have been removed (step  524 ). For example, reverse-index the container of step  512  creating a mapping from each kernel to a set of the kernel&#39;s associated schemas. Break each word in the list obtained in step  506  into affix and kernel pairs as in step  508 , but skip all pairs in which the kernel is missing from the kernel-to-schema mapping. 
         [0058]    Review the set of schemas for this specific kernel. If one of the affixes in the schema matches the current affix in the affix and kernel pair, update the container of schema scores from step  518  by incrementing the score of the current schema and kernel size by one. 
         [0059]    Generate a set of schemas by score to create a list of the promising schemas (step  526 ). For example, review the list of modified schema scores from step  524 , in which a first list contains average schema scores and a second list contains detailed schema scores. For example to create the average schema score for each schema, average the scores of the various kernel sizes, sort the schemas by this average score, and list the schemas, both affixes and the score. To create the detailed schema score, generate trios of schema, kernel size, and score, and sort by score and list. Each schema may have several scores for different kernel sizes. 
         [0060]    The illustrative embodiments, as shown, provide a semi-automatic or programmatic capability for stemming schemas that can be used as a morphological analyzer. This typically does not require a set of exact morphology rules. The capabilities described can typically be used to dynamically build a user-specific lexicon. In an illustrative embodiment a corpus may be analyzed to identify word kernels and associated schema which may then be used to generate a lexicon related to the corpus. Using the process described in the illustrative embodiments typically reduces the time required to create and update a language lexicon. 
         [0061]    The invention can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment containing both hardware and software elements. In a preferred embodiment, the invention is implemented in software, which includes, but is not limited to, firmware, resident software, microcode, etc. 
         [0062]    Furthermore, the invention can take the form of a computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system. For the purposes of this description, a computer-usable or computer readable medium can be any tangible apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. 
         [0063]    The medium can be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system (or apparatus or device) or a propagation medium. Examples of a computer-readable recordable medium include a semiconductor or solid state memory, magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), a rigid magnetic disk and an optical disk. Current examples of optical disks include compact disk-read only memory (CD-ROM), compact disk-read/write (CD-R/W) and DVD 
         [0064]    A data processing system suitable for storing and/or executing program code will include at least one processor coupled directly or indirectly to memory elements through a system bus. The memory elements can include local memory employed during actual execution of the program code, bulk storage, and cache memories which provide temporary storage of at least some program code in order to reduce the number of times code must be retrieved from bulk storage during execution 
         [0065]    Input/output or I/O devices (including but not limited to keyboards, displays, pointing devices, etc.) can be coupled to the system either directly or through intervening I/O controllers. 
         [0066]    Network adapters may also be coupled to the system to enable the data processing system to become coupled to other data processing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modems, and Ethernet cards are just a few of the currently available types of network adapters. 
         [0067]    The description of the present invention has been presented for purposes of illustration and description, and is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiment was chosen and described in order to best explain the principles of the invention, the practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated.
Illustrated embodiments provide a computer implemented method, an apparatus, and a computer program product for unsupervised stemming schema learning and lexicon acquisition from corpora. In one illustrative embodiment, the computer implemented method obtains a corpus from corpora, analyzes the corpus to deduce a set of possible stemming schema and reviews and revises the set of possible stemming schema, to create a pruned set of stemming schema. The computer implemented method further deduces a lexicon from the corpus using the pruned set of stemming schema.