1. A method of parallel latent Dirichlet allocation (LDA) processing of a dataset in an MPP relational database comprising a master node connected via a network to a plurality of distributed database segments, the dataset comprising a plurality of documents, each document being a mixture of topics where each topic is characterized by a probability distribution over a vocabulary of words, the method comprising: distributing subsets of said documents of said dataset to said plurality of segments, where each one of said documents of a subset is distributed to and stored on one single segment of said relational database; determining in parallel on each segment for each document of the subset on said segment a per-word and a per-document topic count; performing in parallel on each segment LDA analysis using said determined topic counts to assign words to a new topic; calculating in parallel on each segment per-word and per-document probability distributions of topic assignments; iterating said determining, said performing, and said calculating steps on each segment until a predetermined stop condition is satisfied, and storing results locally in memory on each segment, said iterating comprising updating topic assignments at each iteration by providing on each segment two work tables for holding topic assignments that alternate roles as an input table and as an output table, a first work table storing topic assignments from a previous iteration and serving as an input table of topic assignments to a next iteration, and a second work table serving as an output table for updated topic assignments from said next iteration, and alternating roles of said first and second tables for input and output on each said iteration for updating topic assignments; and aggregating results from said plurality of segments to form an inference model.
1. A method of parallel latent Dirichlet allocation (LDA) processing of a dataset in an MPP relational database comprising a master node connected via a network to a plurality of distributed database segments, the dataset comprising a plurality of documents, each document being a mixture of topics where each topic is characterized by a probability distribution over a vocabulary of words, the method comprising: distributing subsets of said documents of said dataset to said plurality of segments, where each one of said documents of a subset is distributed to and stored on one single segment of said relational database; determining in parallel on each segment for each document of the subset on said segment a per-word and a per-document topic count; performing in parallel on each segment LDA analysis using said determined topic counts to assign words to a new topic; calculating in parallel on each segment per-word and per-document probability distributions of topic assignments; iterating said determining, said performing, and said calculating steps on each segment until a predetermined stop condition is satisfied, and storing results locally in memory on each segment, said iterating comprising updating topic assignments at each iteration by providing on each segment two work tables for holding topic assignments that alternate roles as an input table and as an output table, a first work table storing topic assignments from a previous iteration and serving as an input table of topic assignments to a next iteration, and a second work table serving as an output table for updated topic assignments from said next iteration, and alternating roles of said first and second tables for input and output on each said iteration for updating topic assignments; and aggregating results from said plurality of segments to form an inference model. 8. The method of claim 1 further comprising using said inference model to assign new words to a topic.