Reading in millions of lines of AOL query logs for query suggestion/expansion I am working on a query suggestion program, and to faciliate this, I am reading in millions of AOL queries logs. The logs have this format:

AnonID  Query   QueryTime
479 family guy  2006-03-01 16:01:20
479 family guy movie references 2006-03-03 22:37:46
479 top grossing movies of all time 2006-03-03 22:42:42
479 car decals  2006-03-03 23:20:12
479 car window decals   2006-03-03 23:24:05
479 car window sponsor decals   2006-03-03 23:27:17
479 car sponsor decals  2006-03-03 23:28:59
...


Where each "column" is separated by a tab character.

This following code reads in the AOL Query logs into a Guava MultiMap (to allow duplicates) and then puts those into an Apache-Collections PatriciaTrie so that prefixes can be grabbed efficiently for future query suggestion/expansion.

private void readInAolQueries(String directory)
{
    try (Stream&lt;Path&gt; paths = Files.walk(Paths.get(directory)))
    {
        Multimap&lt;String, QueryLog&gt; multimap = TreeMultimap.create();
        paths
                .filter(Files::isRegularFile)
                .filter(path -&gt; path.toString().endsWith(".txt"))
                .filter(path -&gt; path.toString().contains("Clean-Data-"))
                .forEach(path -&gt;
                {
                    String fileName = path.toString();
                    List&lt;QueryLog&gt; queryLogs = new AolQueryLogsProcessor(fileName).getQueryLogs();
                    // Read logs into a multimap to preserve duplicates
                    multimap.putAll(Multimaps.index(queryLogs, QueryLog::getQueryString));
                });
        //Put the multimap into the trie. It now also has duplicates.
        queryTrie.addAll(multimap.asMap());
    }
    catch (IOException e)
    {
        e.printStackTrace();
    }
}


The AOL logs files are in my resources directory (which is passed into the readInAolQueries method), and the files all have the form Clean-Data-XX.txt where XX is a number.

Here is the code for the AolQueryLogsProcessor class from the code above:

public class AolQueryLogsProcessor
{
    private List&lt;QueryLog&gt; queryLogs;

    public AolQueryLogsProcessor(String fileName)
    {
        queryLogs = new ArrayList&lt;&gt;();
        try
        {
            List&lt;String[]&gt; lines = readFile(fileName)
                    .stream()
                    .filter(line -&gt; !line.isEmpty())
                    .map(line -&gt; line.split("\t"))
                    .collect(Collectors.toCollection(ArrayList&lt;String[]&gt;::new));

            //Dates looks like this 2006-03-28 20:39:58
            final DateTimeFormatter formatter = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss");

            //The first line is the header. Skip it.
            for (int i = 1; i &lt; lines.size(); ++i)
            {
                String anonId = lines.get(i)[0];
                String[] query = lines.get(i)[1].split("\\s");
                LocalDateTime timeStamp = LocalDateTime.parse(lines.get(i)[2], formatter);
                QueryLog log = new QueryLog(anonId, query, timeStamp);
                queryLogs.add(log);
            }
        }
        catch (IOException e)
        {
            e.printStackTrace();
        }
    }

    private List&lt;String&gt; readFile(String fileName) throws IOException
    {
        return Files.readAllLines(Paths.get(fileName), Charset.forName("UTF-8"));
    }

    public List&lt;QueryLog&gt; getQueryLogs()
    {
        return Collections.unmodifiableList(queryLogs);
    }
}


I'm using a lot of Java 8 Streams, lambdas, and the like because I am trying to familiarize myself with them.

I'm looking for a better and more efficient way to read in the millions of query logs. On my reasonably fast computer, it takes 45-50 seconds to read in all of the logs.

Is there something I can do to speed up this whole process?
Maybe some of my answers and thoughts from this thread (Scanning through logs (tail -f fashion) parsing and sending to a remote server) can help you.

What I want to point out: It's very important, to measure. If you just post a piece of code and tell, it takes that amount of time and ask, how to increase 'performance', it's usually ghost hunting. If you can reduce the time of a routine by 99% but it only takes 1% of the overall time - you just wasted your time.

My main concern about the app is the use case: You're reading millions of lines into memory into a tree structure for further processing/querying - at least that's my interpretation. Have you thought about using an indexing library, such as Apache Lucene? Usually, people invest more time in better response times, instead of "indexing times" - which is, of course, also important, if you have a real-time or near-real-time requirement. But then I would analyze the software architecture first.

Other than that, I can't identify any major problem with the code. One problem could be the garbage collector, since millions of lines are using a lot of memory. And more memory means more memory managament. But that has to measured too and also analyzed in detail. Playing around with collectors, heap sizes, collector threads and so on, can improve throughput significantly.

Hope this helps,

slowy