queues if we need more control over communication between processeswe can use queue queue data structures are useful for sending messages from one process into one or more other processes any picklable object can be sent into queuebut remember that pickling can be costly operationso keep such objects small to illustrate queueslet' build little search engine for text content that stores all relevant entries in memory this is not the most sensible way to build text-based search enginebut have used this pattern to query numerical data that needed to use cpu-intensive processes to construct chart that was then rendered to the user this particular search engine scans all files in the current directory in parallel process is constructed for each core on the cpu each of these is instructed to load some of the files into memory let' look at the function that does the loading and searchingdef search(pathsquery_qresults_q)lines [for path in pathslines extend( strip(for in path open()query query_q get(while queryresults_q put([ for in lines if query in ]query query_q get(rememberthis function is run in different process (in factit is run in cpucount(different processesfrom the main thread it is passes list of path path objects and two multiprocessing queue objectsone for incoming queries and one to send outgoing results these queues have similar interface to the queue class we discussed in python data structures howeverthey are doing extra work to pickle the data in the queue and pass it into the subprocess over pipe these two queues are set up in the main process and passed through the pipes into the search function inside the child processes the search code is pretty dumbboth in terms of efficiency and of capabilitiesit loops over every line stored in memory and puts the matching ones in list the list is placed on queue and passed back to the main process let' look at the main processwhich sets up these queuesif __name__ ='__main__'from multiprocessing import processqueuecpu_count from path import path cpus cpu_count(