From leblanclarry@example.net Fri Apr 01 01:10:44 2011 Return-Path: <leblanclarry@example.net> Delivered-To: leblanclarry@example.net Received: (qmail 89085 invoked from network); 1 Apr 2011 01:10:44 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Apr 2011 01:10:44 -0000 Received: (qmail 69894 invoked by uid 500); 1 Apr 2011 01:10:43 -0000 Delivered-To: leblanclarry@example.net Received: (qmail 69750 invoked by uid 500); 1 Apr 2011 01:10:43 -0000 Mailing-List: contact leblanclarry@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:leblanclarry@example.net> List-Unsubscribe: <mailto:leblanclarry@example.net> List-Post: <mailto:leblanclarry@example.net> List-Id: <dev.mahout.apache.org> Reply-To: leblanclarry@example.net Delivered-To: mailing list leblanclarry@example.net Received: (qmail 69742 invoked by uid 99); 1 Apr 2011 01:10:43 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 01:10:43 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 01:10:42 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155]) by hel.zones.apache.org (Postfix) with ESMTP id AACFE8CF86 for <leblanclarry@example.net>; Fri, 1 Apr 2011 01:10:05 +0000 (UTC) Date: Fri, 1 Apr 2011 01:10:05 +0000 (UTC) From: "Dmitriy Lyubimov (JIRA)" <leblanclarry@example.net> To: leblanclarry@example.net Message-ID: <leblanclarry@example.net> In-Reply-To: <{{EMAIL+PHONE}}> Subject: [jira] [Commented] (MAHOUT-638) Stochastic svd's is not handling well all cases of sparse vectors MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/MAHOUT-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014338#comment-13014338 ] Dmitriy Lyubimov commented on MAHOUT-638: ----------------------------------------- Let me do one more round on this, please. Please don't close it yet. Thanks. > Stochastic svd's is not handling well all cases of sparse vectors > ------------------------------------------------------------------ > > Key: MAHOUT-638 > URL: https://issues.apache.org/jira/browse/MAHOUT-638 > Project: Mahout > Issue Type: Bug > Components: Math > Affects Versions: 0.5 > Reporter: Dmitriy Lyubimov > Assignee: Dmitriy Lyubimov > Fix For: 0.5 > > Attachments: MAHOUT-638.patch > > > The Mahout patch of the algorithm is not handling all types of sparse input efficiently. BtJob doesn't handle SequentialSparseVector in a way to pick only non-zero elements from initial input and QJob doesn't iterate over RandomAccessSparseVector correctly. With extremely sparse inputs (0.05% non-zero elements) that leads to a terrible inefficiency in the aforementioned jobs (QJob, BtJob). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira From leblanclarry@example.net Fri Apr 01 01:16:48 2011 Return-Path: <leblanclarry@example.net> Delivered-To: leblanclarry@example.net Received: (qmail 94188 invoked from network); 1 Apr 2011 01:16:47 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Apr 2011 01:16:47 -0000 Received: (qmail 76242 invoked by uid 500); 1 Apr 2011 01:16:47 -0000 Delivered-To: leblanclarry@example.net Received: (qmail 76218 invoked by uid 500); 1 Apr 2011 01:16:47 -0000 Mailing-List: contact leblanclarry@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:leblanclarry@example.net> List-Unsubscribe: <mailto:leblanclarry@example.net> List-Post: <mailto:leblanclarry@example.net> List-Id: <dev.mahout.apache.org> Reply-To: leblanclarry@example.net Delivered-To: mailing list leblanclarry@example.net Received: (qmail 76210 invoked by uid 99); 1 Apr 2011 01:16:47 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 01:16:47 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO aegis.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 01:16:42 +0000 Received: from aegis (localhost [83.161.120.155]) by aegis.apache.org (Postfix) with ESMTP id AEA6AC0013 for <leblanclarry@example.net>; Fri, 1 Apr 2011 01:16:20 +0000 (UTC) Date: Fri, 1 Apr 2011 01:16:20 +0000 (UTC) From: Apache Hudson Server <leblanclarry@example.net> To: leblanclarry@example.net Message-ID: <1930072371.11961301620580699.JavaMail.hudson@aegis> Subject: Build failed in Jenkins: Mahout-Quality #709 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org See <https://hudson.apache.org/hudson/job/Mahout-Quality/709/> ------------------------------------------ [...truncated 9136 lines...] [INFO] Using Clover report descriptor: /var/tmp/mvn8754723909663940414resource [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Clover is enabled with initstring '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/target/clover/cloverMerge.db' [WARNING] Clover historical directory [/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/target/clover/history] does not exist, skipping Clover historical report generation ([/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/target/site/clover]) [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Loading coverage database from: '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/target/clover/cloverMerge.db' [INFO] Writing HTML report to '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/target/site/clover' [INFO] Done. Processed 157 packages in 37395ms (238ms per package). [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Clover is enabled with initstring '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/target/clover/cloverMerge.db' [WARNING] Clover historical directory [/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/target/clover/history] does not exist, skipping Clover historical report generation ([/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/target/site/clover/clover.xml]) [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Loading coverage database from: '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/target/clover/cloverMerge.db' [INFO] Writing report to '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/target/site/clover/clover.xml' [INFO] [pmd:pmd {execution: default-cli}] [INFO] ------------------------------------------------------------------------ [INFO] Building Mahout Build Tools [INFO] task-segment: [clover2:clover, pmd:pmd] [INFO] ------------------------------------------------------------------------ [INFO] [clover2:clover {execution: default-cli}] [WARNING] No Clover database found, skipping report generation [INFO] No report being generated for this module. [INFO] Using /default-clover-report descriptor. [INFO] Using Clover report descriptor: /var/tmp/mvn7221761788206398653resource [INFO] [pmd:pmd {execution: default-cli}] [INFO] ------------------------------------------------------------------------ [INFO] Building Mahout Eclipse [INFO] task-segment: [clover2:clover, pmd:pmd] [INFO] ------------------------------------------------------------------------ [INFO] [clover2:clover {execution: default-cli}] [WARNING] No Clover database found, skipping report generation [INFO] No report being generated for this module. [INFO] Using /default-clover-report descriptor. [INFO] Using Clover report descriptor: /var/tmp/mvn4775499155853879426resource [INFO] [pmd:pmd {execution: default-cli}] [INFO] ------------------------------------------------------------------------ [INFO] Building Mahout Math [INFO] task-segment: [clover2:clover, pmd:pmd] [INFO] ------------------------------------------------------------------------ [INFO] [clover2:clover {execution: default-cli}] [INFO] Using /default-clover-report descriptor. [INFO] Using Clover report descriptor: /var/tmp/mvn3114009788525180386resource [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Clover is enabled with initstring '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/math/target/clover/clover.db' [WARNING] Clover historical directory [/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/math/target/clover/history] does not exist, skipping Clover historical report generation ([/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/math/target/site/clover]) [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Loading coverage database from: '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/math/target/clover/clover.db' [INFO] Writing HTML report to '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/math/target/site/clover' [INFO] Done. Processed 17 packages in 2167ms (127ms per package). [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Clover is enabled with initstring '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/math/target/clover/clover.db' [WARNING] Clover historical directory [/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/math/target/clover/history] does not exist, skipping Clover historical report generation ([/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/math/target/site/clover/clover.xml]) [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Loading coverage database from: '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/math/target/clover/clover.db' [INFO] Writing report to '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/math/target/site/clover/clover.xml' [INFO] [pmd:pmd {execution: default-cli}] [INFO] ------------------------------------------------------------------------ [INFO] Building Mahout Core [INFO] task-segment: [clover2:clover, pmd:pmd] [INFO] ------------------------------------------------------------------------ [INFO] [clover2:clover {execution: default-cli}] [INFO] Using /default-clover-report descriptor. [INFO] Using Clover report descriptor: /var/tmp/mvn8720049039021375674resource [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Clover is enabled with initstring '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/core/target/clover/clover.db' [WARNING] Clover historical directory [/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/core/target/clover/history] does not exist, skipping Clover historical report generation ([/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/core/target/site/clover]) [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Loading coverage database from: '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/core/target/clover/clover.db' [INFO] Writing HTML report to '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/core/target/site/clover' [INFO] Done. Processed 106 packages in 13621ms (128ms per package). [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Clover is enabled with initstring '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/core/target/clover/clover.db' [WARNING] Clover historical directory [/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/core/target/clover/history] does not exist, skipping Clover historical report generation ([/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/core/target/site/clover/clover.xml]) [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Loading coverage database from: '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/core/target/clover/clover.db' [INFO] Writing report to '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/core/target/site/clover/clover.xml' [INFO] [pmd:pmd {execution: default-cli}] [INFO] ------------------------------------------------------------------------ [INFO] Building Mahout Taste Webapp [INFO] task-segment: [clover2:clover, pmd:pmd] [INFO] ------------------------------------------------------------------------ [INFO] [clover2:clover {execution: default-cli}] [INFO] Using /default-clover-report descriptor. [INFO] Using Clover report descriptor: /var/tmp/mvn8541901847763950381resource [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Clover is enabled with initstring '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/taste-web/target/clover/clover.db' [WARNING] Clover historical directory [/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/taste-web/target/clover/history] does not exist, skipping Clover historical report generation ([/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/taste-web/target/site/clover]) [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Loading coverage database from: '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/taste-web/target/clover/clover.db' [INFO] Writing HTML report to '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/taste-web/target/site/clover' [INFO] Done. Processed 1 packages in 184ms (184ms per package). [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Clover is enabled with initstring '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/taste-web/target/clover/clover.db' [WARNING] Clover historical directory [/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/taste-web/target/clover/history] does not exist, skipping Clover historical report generation ([/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/taste-web/target/site/clover/clover.xml]) [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Loading coverage database from: '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/taste-web/target/clover/clover.db' [INFO] Writing report to '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/taste-web/target/site/clover/clover.xml' [INFO] [pmd:pmd {execution: default-cli}] [INFO] ------------------------------------------------------------------------ [INFO] Building Mahout Utilities [INFO] task-segment: [clover2:clover, pmd:pmd] [INFO] ------------------------------------------------------------------------ [INFO] [clover2:clover {execution: default-cli}] [INFO] Using /default-clover-report descriptor. [INFO] Using Clover report descriptor: /var/tmp/mvn6665702488537824955resource [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Clover is enabled with initstring '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/utils/target/clover/clover.db' [WARNING] Clover historical directory [/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/utils/target/clover/history] does not exist, skipping Clover historical report generation ([/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/utils/target/site/clover]) [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Loading coverage database from: '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/utils/target/clover/clover.db' [INFO] Writing HTML report to '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/utils/target/site/clover' [INFO] Done. Processed 16 packages in 520ms (32ms per package). [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Clover is enabled with initstring '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/utils/target/clover/clover.db' [WARNING] Clover historical directory [/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/utils/target/clover/history] does not exist, skipping Clover historical report generation ([/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/utils/target/site/clover/clover.xml]) [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Loading coverage database from: '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/utils/target/clover/clover.db' [INFO] Writing report to '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/utils/target/site/clover/clover.xml' [INFO] [pmd:pmd {execution: default-cli}] [INFO] ------------------------------------------------------------------------ [INFO] Building Mahout Examples [INFO] task-segment: [clover2:clover, pmd:pmd] [INFO] ------------------------------------------------------------------------ [INFO] [clover2:clover {execution: default-cli}] [INFO] Using /default-clover-report descriptor. [INFO] Using Clover report descriptor: /var/tmp/mvn828771978514893690resource [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Clover is enabled with initstring '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/examples/target/clover/clover.db' [WARNING] Clover historical directory [/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/examples/target/clover/history] does not exist, skipping Clover historical report generation ([/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/examples/target/site/clover]) [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Loading coverage database from: '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/examples/target/clover/clover.db' [INFO] Writing HTML report to '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/examples/target/site/clover' [INFO] Done. Processed 31 packages in 1406ms (45ms per package). [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Clover is enabled with initstring '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/examples/target/clover/clover.db' [WARNING] Clover historical directory [/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/examples/target/clover/history] does not exist, skipping Clover historical report generation ([/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/examples/target/site/clover/clover.xml]) [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Loading coverage database from: '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/examples/target/clover/clover.db' [INFO] Writing report to '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/examples/target/site/clover/clover.xml' [INFO] [pmd:pmd {execution: default-cli}] [INFO] ------------------------------------------------------------------------ [INFO] Building Mahout Release Package [INFO] task-segment: [clover2:clover, pmd:pmd] [INFO] ------------------------------------------------------------------------ [INFO] [clover2:clover {execution: default-cli}] [WARNING] No Clover database found, skipping report generation [INFO] No report being generated for this module. [INFO] Using /default-clover-report descriptor. [INFO] Using Clover report descriptor: /var/tmp/mvn5669666383597089215resource [INFO] [pmd:pmd {execution: default-cli}] [INFO] [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] ------------------------------------------------------------------------ [INFO] Apache Mahout ......................................... SUCCESS [57.466s] [INFO] Mahout Build Tools .................................... SUCCESS [1.740s] [INFO] Mahout Eclipse ........................................ SUCCESS [1.179s] [INFO] Mahout Math ........................................... SUCCESS [10.209s] [INFO] Mahout Core ........................................... SUCCESS [42.594s] [INFO] Mahout Taste Webapp ................................... SUCCESS [1.513s] [INFO] Mahout Utilities ...................................... SUCCESS [4.537s] [INFO] Mahout Examples ....................................... SUCCESS [7.032s] [INFO] Mahout Release Package ................................ SUCCESS [1.046s] [INFO] ------------------------------------------------------------------------ [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESSFUL [INFO] ------------------------------------------------------------------------ [INFO] Total time: 77 minutes 59 seconds [INFO] Finished at: Fri Apr 01 00:51:49 GMT+00:00 2011 [INFO] Final Memory: 122M/489M [INFO] ------------------------------------------------------------------------ [FINDBUGS] Collecting findbugs analysis files... ERROR: Publisher hudson.plugins.findbugs.FindBugsPublisher aborted due to exception java.lang.ClassCastException: cannot assign instance of hudson.plugins.analysis.core.FilesParser to field hudson.FilePath$FileCallableWrapper.callable of type hudson.FilePath$FileCallable in instance of leblanclarry@example.net$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2004) at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1184) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1916) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1834) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1719) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1305) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348) at hudson.remoting.UserRequest.deserialize(UserRequest.java:178) at hudson.remoting.UserRequest.perform(UserRequest.java:98) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:270) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269) at java.util.concurrent.FutureTask.run(FutureTask.java:123) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:651) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676) at java.lang.Thread.run(Thread.java:595) [PMD] Skipping publisher since build result is FAILURE [TASKS] Skipping publisher since build result is FAILURE Archiving artifacts Recording test results Publishing Javadoc Publishing Clover coverage report... Publishing Clover HTML report... Publishing Clover XML report... Publishing Clover coverage results... From leblanclarry@example.net Fri Apr 01 11:56:49 2011 Return-Path: <leblanclarry@example.net> Delivered-To: leblanclarry@example.net Received: (qmail 80454 invoked from network); 1 Apr 2011 11:56:49 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Apr 2011 11:56:49 -0000 Received: (qmail 78182 invoked by uid 500); 1 Apr 2011 11:56:48 -0000 Delivered-To: leblanclarry@example.net Received: (qmail 78144 invoked by uid 500); 1 Apr 2011 11:56:48 -0000 Mailing-List: contact leblanclarry@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:leblanclarry@example.net> List-Unsubscribe: <mailto:leblanclarry@example.net> List-Post: <mailto:leblanclarry@example.net> List-Id: <dev.mahout.apache.org> Reply-To: leblanclarry@example.net Delivered-To: mailing list leblanclarry@example.net Received: (qmail 78136 invoked by uid 99); 1 Apr 2011 11:56:47 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 11:56:47 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 11:56:44 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155]) by hel.zones.apache.org (Postfix) with ESMTP id D57508D55C for <leblanclarry@example.net>; Fri, 1 Apr 2011 11:56:05 +0000 (UTC) Date: Fri, 1 Apr 2011 11:56:05 +0000 (UTC) From: "Sean Owen (JIRA)" <leblanclarry@example.net> To: leblanclarry@example.net Message-ID: <leblanclarry@example.net> In-Reply-To: <leblanclarry@example.net> Subject: [jira] [Commented] (MAHOUT-646) Cannot run Wikipedia example on Amazon Elastic MapReduce (EMR) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/MAHOUT-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014526#comment-13014526 ] Sean Owen commented on MAHOUT-646: ---------------------------------- Could be... but somehow I doubt it. I note that in MAHOUT-614 we fixed up an apparent problem with this class. The change might have uncovered a different issue (or, er, actually messed it up in a different way). One way forward is to fix it. Another way is to delete it. I say that's an option since WikipediaDatasetCreatorOutputFormat is the one and only class which depends on two classes copied and modified from Hadoop. Going forward we'd want to go back to the main-line version somehow. And if removing use of this custom output format doesn't "hurt", as the issue implies, well, why not just remove it? But the question is... is it really just as well to let this dump to a sequence file or does that defeat the purpose? Robin? > Cannot run Wikipedia example on Amazon Elastic MapReduce (EMR) > -------------------------------------------------------------- > > Key: MAHOUT-646 > URL: https://issues.apache.org/jira/browse/MAHOUT-646 > Project: Mahout > Issue Type: Bug > Components: Classification > Affects Versions: 0.5 > Reporter: Martin Provencher > Priority: Minor > > When I tried to run the Wikipedia example on EMR with all the categories existing in the Wikipedia dump, I got this error : > org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to create file /yatter.tagger/wikipedia/input/_temporary/_attempt__0000_r_000000_0/part-r-00000 for DFSClient_attempt_201103292134_0010_r_000000_0 on client 83.161.120.155 because current leaseholder is trying to recreate file. > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1045) > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:981) > at org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:377) > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:961) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:957) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:955) > at org.apache.hadoop.ipc.Client.call(Client.java:740) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) > at $Proxy1.create(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) > at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) > at $Proxy1.create(Unknown Source) > at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.<init>(DFSClient.java:2709) > at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:491) > at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:195) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:524) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:505) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:412) > at org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.getRecordWriter(TextOutputFormat.java:128) > at org.apache.mahout.classifier.bayes.MultipleTextOutputFormat.getBaseRecordWriter(MultipleTextOutputFormat.java:41) > at org.apache.mahout.classifier.bayes.MultipleOutputFormat$1.write(MultipleOutputFormat.java:81) > at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:517) > at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80) > at org.apache.mahout.classifier.bayes.WikipediaDatasetCreatorReducer.reduce(WikipediaDatasetCreatorReducer.java:35) > at org.apache.mahout.classifier.bayes.WikipediaDatasetCreatorReducer.reduce(WikipediaDatasetCreatorReducer.java:28) > at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176) > at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:575) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:412) > at org.apache.hadoop.mapred.Child.main(Child.java:170) > org.apache.hadoop.ipc.RemoteException: java.io.IOException: failed to create file /yatter.tagger/wikipedia/input/_temporary/_attempt__0000_r_000000_0/part-r-00000 on client 83.161.120.155 either because the filename is invalid or the file exists > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1092) > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:981) > at org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:377) > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:961) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:957) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:955) > at org.apache.hadoop.ipc.Client.call(Client.java:740) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) > at $Proxy1.create(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) > at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) > at $Proxy1.create(Unknown Source) > at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.<init>(DFSClient.java:2709) > at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:491) > at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:195) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:524) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:505) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:412) > at org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.getRecordWriter(TextOutputFormat.java:128) > at org.apache.mahout.classifier.bayes.MultipleTextOutputFormat.getBaseRecordWriter(MultipleTextOutputFormat.java:41) > at org.apache.mahout.classifier.bayes.MultipleOutputFormat$1.write(MultipleOutputFormat.java:81) > at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:517) > at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80) > at org.apache.mahout.classifier.bayes.WikipediaDatasetCreatorReducer.reduce(WikipediaDatasetCreatorReducer.java:35) > at org.apache.mahout.classifier.bayes.WikipediaDatasetCreatorReducer.reduce(WikipediaDatasetCreatorReducer.java:28) > at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176) > at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:575) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:412) > at org.apache.hadoop.mapred.Child.main(Child.java:170) > 4 more : > org.apache.hadoop.ipc.RemoteException: java.io.IOException: failed to create file /yatter.tagger/wikipedia/input/_temporary/_attempt__0000_r_000000_0/part-r-00000 on client 83.161.120.155 either because the filename is invalid or the file exists -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira From leblanclarry@example.net Fri Apr 01 17:01:45 2011 Return-Path: <leblanclarry@example.net> Delivered-To: leblanclarry@example.net Received: (qmail 4788 invoked from network); 1 Apr 2011 17:01:44 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Apr 2011 17:01:44 -0000 Received: (qmail 16217 invoked by uid 500); 1 Apr 2011 17:01:44 -0000 Delivered-To: leblanclarry@example.net Received: (qmail 15841 invoked by uid 500); 1 Apr 2011 17:01:44 -0000 Mailing-List: contact leblanclarry@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:leblanclarry@example.net> List-Unsubscribe: <mailto:leblanclarry@example.net> List-Post: <mailto:leblanclarry@example.net> List-Id: <dev.mahout.apache.org> Reply-To: leblanclarry@example.net Delivered-To: mailing list leblanclarry@example.net Received: (qmail 15832 invoked by uid 99); 1 Apr 2011 17:01:43 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 17:01:43 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of leblanclarry@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-iw0-f170.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 17:01:37 +0000 Received: by iwn3 with SMTP id 3so7838461iwn.1 for <leblanclarry@example.net>; Fri, 01 Apr 2011 10:01:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:date:message-id:subject:from:to :content-type; bh=rZvSrymWftaU1Ilsl19ldZjDD+IrKpBthTXVj1odgFQ=; b=q7pWEHqmje+wouJS3fC12nAVy8Ma/TtmKwvFAIwm0YqvKsRCiiinNtzOucgoau59Kf H9cbPAN/jQW9ml3A0IbK2pzBVnoczke+xGsAhXjb3EuxnWZo4FGjwLZc98F/xz8w11Zl n95rJEBtAFxK5yXn+1vJjmUWma05MmXS/9GlA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=oUhgSTy7COy2x0DqjkPRu0/6lynXRO+dijINK7pwDC7LxxaQ39AgfzGLuRNNyTyzm8 TKgattv/5R1VqDUyPgfATytjgqTmUaUWVnAj7NgVJU4bvNEynPOXd0WkuFeQVSAU4kqR 548H8K7qZdlifCiMSOqhcJeMoXw1en0DiCPRM= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id vp10mr1794581icb.413.1301677277095; Fri, 01 Apr 2011 10:01:17 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Fri, 1 Apr 2011 10:01:16 -0700 (PDT) Date: Fri, 1 Apr 2011 18:01:16 +0100 Message-ID: <leblanclarry@example.net> Subject: Initial draft of April board report From: Sean Owen <leblanclarry@example.net> To: Mahout Dev List <leblanclarry@example.net> Content-Type: multipart/alternative; boundary=bcaec52e5f09b6a5b9049fde5be6 --bcaec52e5f09b6a5b9049fde5be6 Content-Type: text/plain; charset=UTF-8 Initial draft of the April board report is below and in SVN. Anyone care to add more items? discuss the issues here? === Apache Mahout Status Report: April 2011 === ISSUES There are no issues requiring board attention at this time. CURRENT ACTIVITY The project expects to continue with an 0.5 release around May 2011. 93 issues have been resolved for 0.5, with 25 more planned before the release: https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true &jqlQuery=project+%3D+MAHOUT+AND+fixVersion+%3D+%220.5%22 After that, we believe, will be a 1.0 release. From 0.5, the focus will change to making the code base stable and 1.0-ready. COMMITTERS Apache Mahout added Dmitriy Lyubimov and Shannon Quinn as new committers in February 2011. MAHOUT IN ACTION The book "Mahout in Action", published by Manning, has been completed and will be published in May 2011 (delayed slightly from February 2011 date). --bcaec52e5f09b6a5b9049fde5be6-- From leblanclarry@example.net Fri Apr 01 22:49:26 2011 Return-Path: <leblanclarry@example.net> Delivered-To: leblanclarry@example.net Received: (qmail 76864 invoked from network); 1 Apr 2011 22:49:25 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Apr 2011 22:49:25 -0000 Received: (qmail 64463 invoked by uid 500); 1 Apr 2011 22:49:25 -0000 Delivered-To: leblanclarry@example.net Received: (qmail 64424 invoked by uid 500); 1 Apr 2011 22:49:25 -0000 Mailing-List: contact leblanclarry@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:leblanclarry@example.net> List-Unsubscribe: <mailto:leblanclarry@example.net> List-Post: <mailto:leblanclarry@example.net> List-Id: <dev.mahout.apache.org> Reply-To: leblanclarry@example.net Delivered-To: mailing list leblanclarry@example.net Received: (qmail 64416 invoked by uid 99); 1 Apr 2011 22:49:25 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 22:49:25 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of leblanclarry@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-pv0-f170.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 22:49:18 +0000 Received: by pvg16 with SMTP id 16so1313358pvg.1 for <leblanclarry@example.net>; Fri, 01 Apr 2011 15:48:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type:content-transfer-encoding; bh=UAQu9q5Y3Tge/DH/dfEHL7wDzRxEAMut53GYqYhlZNY=; b=bs2YsxnbbcEpSFwSGaN+vz74m0ZjA5lqVWv4kWYare7kZrEVqId78C6fZavZkV6DMG JWpE1BVjpyLCKk0FTopjWieMd2/X4iz/lXiUF2K+f2dm9/438CDRFPUmQ9NnBD3e0z2h 02+QdU73tHwINCPdWdzv+q63cMKweamUqr1gs= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=ULdrZd0Qco/KQQa+LMZt/sa0NF1xElJIQi6SSYJgWpXjEDWv7RTJ4Y3zGMucAR2xnc Qe8yQa9YARWFsVXnMt69fWofpOW3Zt1JEV4ZpTSdEWbAvhgLA/pxrRmXo8RfN+XLySg8 WnyqaeMZlknGrOVS682yEyNzp0BZBl09WGCEE= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id w12mr3422149wff.259.1301698137475; Fri, 01 Apr 2011 15:48:57 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Fri, 1 Apr 2011 15:48:57 -0700 (PDT) In-Reply-To: <leblanclarry@example.net> References: <leblanclarry@example.net> <leblanclarry@example.net> Date: Fri, 1 Apr 2011 15:48:57 -0700 Message-ID: <leblanclarry@example.net> Subject: Re: [jira] [Commented] (MAHOUT-641) DistributedRowMatrix hadoop jobs ignore Configuration set via setConf() From: Lance Norskog <leblanclarry@example.net> To: leblanclarry@example.net Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org If you want this to work smoothly, ban: 'new Job(conf)'. It's a point source of head-scratching errors. On Thu, Mar 31, 2011 at 4:32 PM, Hudson (JIRA) <leblanclarry@example.net> wrote: > > =C2=A0 =C2=A0[ https://issues.apache.org/jira/browse/MAHOUT-641?page=3Dco= m.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedComme= ntId=3D13014281#comment-13014281 ] > > Hudson commented on MAHOUT-641: > ------------------------------- > > Integrated in Mahout-Quality #708 (See [https://hudson.apache.org/hudson/= job/Mahout-Quality/708/]) > > >> DistributedRowMatrix hadoop jobs ignore Configuration set via setConf() >> ----------------------------------------------------------------------- >> >> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Key: MAHOUT-641 >> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 URL: https://iss= ues.apache.org/jira/browse/MAHOUT-641 >> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Project: Mahout >> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Issue Type: Bug >> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Components: Math >> =C2=A0 =C2=A0Affects Versions: 0.4, 0.5 >> =C2=A0 =C2=A0 =C2=A0 =C2=A0 Environment: Mahout 0.4 and 0.5-SNAPSHOT run= with Hadoop 0.20.2 on Mac OS 10.6 and Linux x86_64 2.6.18 >> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Reporter: Jonathan Traupman >> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Labels: configuration, h= adoop >> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Fix For: 0.5 >> >> =C2=A0 =C2=A0 =C2=A0 =C2=A0 Attachments: MAHOUT-641.patch >> >> =C2=A0 Original Estimate: 24h >> =C2=A0Remaining Estimate: 24h >> >> I am using the Distributed Lanczos solver which uses the DistributedRowM= atrix class for it's internal calculation. In our environment, I need to se= t some Configuration properties (specifically hadoop.job.ugi & hadoop.queue= .name), which are when set via the setConf() method on DistributedLanczosSo= lver. These are correctly passed to DistributedRowMatrix via its setConf() = method, but are not passed into the Hadoop JobConfs created by the various = static routines in MatrixMultiplicationJob, TimesSquaredJob, and TransposeJ= ob. > > -- > This message is automatically generated by JIRA. > For more information on JIRA, see: http://www.atlassian.com/software/jira > --=20 Lance Norskog leblanclarry@example.net From leblanclarry@example.net Fri Apr 01 22:55:46 2011 Return-Path: <leblanclarry@example.net> Delivered-To: leblanclarry@example.net Received: (qmail 80294 invoked from network); 1 Apr 2011 22:55:46 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Apr 2011 22:55:46 -0000 Received: (qmail 68707 invoked by uid 500); 1 Apr 2011 22:55:46 -0000 Delivered-To: leblanclarry@example.net Received: (qmail 68674 invoked by uid 500); 1 Apr 2011 22:55:46 -0000 Mailing-List: contact leblanclarry@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:leblanclarry@example.net> List-Unsubscribe: <mailto:leblanclarry@example.net> List-Post: <mailto:leblanclarry@example.net> List-Id: <dev.mahout.apache.org> Reply-To: leblanclarry@example.net Delivered-To: mailing list leblanclarry@example.net Received: (qmail 68666 invoked by uid 99); 1 Apr 2011 22:55:46 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 22:55:46 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 22:55:43 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155]) by hel.zones.apache.org (Postfix) with ESMTP id CC9BD8E7D9 for <leblanclarry@example.net>; Fri, 1 Apr 2011 22:55:05 +0000 (UTC) Date: Fri, 1 Apr 2011 22:55:05 +0000 (UTC) From: "Lance Norskog (JIRA)" <leblanclarry@example.net> To: leblanclarry@example.net Message-ID: <leblanclarry@example.net> In-Reply-To: <leblanclarry@example.net> Subject: [jira] [Commented] (MAHOUT-643) Adding CityBlockSimilarity and DistributedCityBlockDistanceVectorSimilarity MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/MAHOUT-643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014864#comment-13014864 ] Lance Norskog commented on MAHOUT-643: -------------------------------------- Is this Chebyshev or Manhattan distance? Or is it a third formula? If the first, could they all be named consistently? If the second, could there also be a DistanceMeasure? This is the kind of thing that breaks the sanity of new users. > Adding CityBlockSimilarity and DistributedCityBlockDistanceVectorSimilarity > --------------------------------------------------------------------------- > > Key: MAHOUT-643 > URL: https://issues.apache.org/jira/browse/MAHOUT-643 > Project: Mahout > Issue Type: Improvement > Components: Clustering, Math > Affects Versions: 0.5 > Reporter: Daniel McEnnis > Assignee: Sean Owen > Priority: Minor > Labels: distance, patch, similarity > Fix For: 0.5 > > Attachments: MAHOUT-643-2.patch, MAHOUT-643.patch, patch4.txt > > Original Estimate: 1h > Remaining Estimate: 1h > > adding a new distance metric to the 0.5 branch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira From leblanclarry@example.net Fri Apr 01 23:22:31 2011 Return-Path: <leblanclarry@example.net> Delivered-To: leblanclarry@example.net Received: (qmail 59582 invoked from network); 1 Apr 2011 23:22:31 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Apr 2011 23:22:31 -0000 Received: (qmail 87717 invoked by uid 500); 1 Apr 2011 23:22:30 -0000 Delivered-To: leblanclarry@example.net Received: (qmail 87504 invoked by uid 500); 1 Apr 2011 23:22:30 -0000 Mailing-List: contact leblanclarry@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:leblanclarry@example.net> List-Unsubscribe: <mailto:leblanclarry@example.net> List-Post: <mailto:leblanclarry@example.net> List-Id: <dev.mahout.apache.org> Reply-To: leblanclarry@example.net Delivered-To: mailing list leblanclarry@example.net Received: (qmail 87496 invoked by uid 99); 1 Apr 2011 23:22:30 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 23:22:30 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of leblanclarry@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-px0-f171.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 23:22:23 +0000 Received: by pxi7 with SMTP id 7so1257678pxi.30 for <leblanclarry@example.net>; Fri, 01 Apr 2011 16:22:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type:content-transfer-encoding; bh=kU0tb/Zp6prEdOty474lvz/mPg4eOWzRrmgq6faex50=; b=rMKt67A4wINhmoj8GxL2o+z0Z7A8yCfl+Gpze8nklzp0jheXW6i5ajL85Sw8HlTDCG vvWReXk8pj7GiHcyPwsPXjdwKJ2s/PFsZGNIT+1Z11EXPL7zFNHMUFx1/jkC6spVQMei wYtxLAURkBgRJOdThkcXATI5V6ySBaiI8vQug= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=BHSrUULVXcvPyQ66OlqDClglufo1NNyco47fApn4FChezp4sHe6ut7WoraZ2qfaseB KR/PjllFZj/ZgXxTlVsDJayhWaTX9UkS+YjGlgzDxYFjSMr70KcNGER2r0mTzw+89Ebj JFhR7btdLKaxWJdtayhAZcgtKwaVrLt7XtpCA= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id k23mr3420460wfa.316.1301700121305; Fri, 01 Apr 2011 16:22:01 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Fri, 1 Apr 2011 16:22:01 -0700 (PDT) In-Reply-To: <leblanclarry@example.net> References: <leblanclarry@example.net> Date: Fri, 1 Apr 2011 16:22:01 -0700 Message-ID: <leblanclarry@example.net> Subject: Re: Initial draft of April board report From: Lance Norskog <leblanclarry@example.net> To: leblanclarry@example.net Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org Another indicator of traction is the amount of non-committers submitting stuff to the JIRA. On Fri, Apr 1, 2011 at 10:01 AM, Sean Owen <leblanclarry@example.net> wrote: > Initial draft of the April board report is below and in SVN. Anyone care = to > add more items? discuss the issues here? > > > =3D=3D=3D Apache Mahout Status Report: April 2011 =3D=3D=3D > > ISSUES > > There are no issues requiring board attention at this time. > > CURRENT ACTIVITY > > The project expects to continue with an 0.5 release around May 2011. > 93 issues have been resolved for 0.5, with 25 more planned before the > release: > https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=3Dtrue > =C2=A0&jqlQuery=3Dproject+%3D+MAHOUT+AND+fixVersion+%3D+%220.5%22 > > After that, we believe, will be a 1.0 release. From 0.5, the focus will > change to making the code base stable and 1.0-ready. > > COMMITTERS > > Apache Mahout added Dmitriy Lyubimov and Shannon Quinn as new committers > in February 2011. > > MAHOUT IN ACTION > > The book "Mahout in Action", published by Manning, has been completed and > will be published in May 2011 (delayed slightly from February 2011 date). > --=20 Lance Norskog leblanclarry@example.net From leblanclarry@example.net Fri Apr 01 23:32:11 2011 Return-Path: <leblanclarry@example.net> Delivered-To: leblanclarry@example.net Received: (qmail 94138 invoked from network); 1 Apr 2011 23:32:11 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Apr 2011 23:32:11 -0000 Received: (qmail 95257 invoked by uid 500); 1 Apr 2011 23:32:10 -0000 Delivered-To: leblanclarry@example.net Received: (qmail 95217 invoked by uid 500); 1 Apr 2011 23:32:10 -0000 Mailing-List: contact leblanclarry@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:leblanclarry@example.net> List-Unsubscribe: <mailto:leblanclarry@example.net> List-Post: <mailto:leblanclarry@example.net> List-Id: <dev.mahout.apache.org> Reply-To: leblanclarry@example.net Delivered-To: mailing list leblanclarry@example.net Received: (qmail 95209 invoked by uid 99); 1 Apr 2011 23:32:10 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 23:32:10 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO aegis.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 23:32:05 +0000 Received: from aegis (localhost [83.161.120.155]) by aegis.apache.org (Postfix) with ESMTP id D92F7C003F for <leblanclarry@example.net>; Fri, 1 Apr 2011 23:31:43 +0000 (UTC) Date: Fri, 1 Apr 2011 23:31:43 +0000 (UTC) From: Apache Hudson Server <leblanclarry@example.net> To: leblanclarry@example.net Message-ID: <2006379261.13371301700703876.JavaMail.hudson@aegis> In-Reply-To: <1930072371.11961301620580699.JavaMail.hudson@aegis> References: <1930072371.11961301620580699.JavaMail.hudson@aegis> Subject: Build failed in Jenkins: Mahout-Quality #710 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org See <https://hudson.apache.org/hudson/job/Mahout-Quality/710/> ------------------------------------------ [...truncated 9132 lines...] [INFO] Using Clover report descriptor: /var/tmp/mvn1460591759807523687resource [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Clover is enabled with initstring '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/target/clover/cloverMerge.db' [WARNING] Clover historical directory [/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/target/clover/history] does not exist, skipping Clover historical report generation ([/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/target/site/clover]) [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Loading coverage database from: '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/target/clover/cloverMerge.db' [INFO] Writing HTML report to '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/target/site/clover' [INFO] Done. Processed 157 packages in 33588ms (213ms per package). [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Clover is enabled with initstring '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/target/clover/cloverMerge.db' [WARNING] Clover historical directory [/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/target/clover/history] does not exist, skipping Clover historical report generation ([/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/target/site/clover/clover.xml]) [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Loading coverage database from: '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/target/clover/cloverMerge.db' [INFO] Writing report to '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/target/site/clover/clover.xml' [INFO] [pmd:pmd {execution: default-cli}] [INFO] ------------------------------------------------------------------------ [INFO] Building Mahout Build Tools [INFO] task-segment: [clover2:clover, pmd:pmd] [INFO] ------------------------------------------------------------------------ [INFO] [clover2:clover {execution: default-cli}] [WARNING] No Clover database found, skipping report generation [INFO] No report being generated for this module. [INFO] Using /default-clover-report descriptor. [INFO] Using Clover report descriptor: /var/tmp/mvn3291027695084358501resource [INFO] [pmd:pmd {execution: default-cli}] [INFO] ------------------------------------------------------------------------ [INFO] Building Mahout Eclipse [INFO] task-segment: [clover2:clover, pmd:pmd] [INFO] ------------------------------------------------------------------------ [INFO] [clover2:clover {execution: default-cli}] [WARNING] No Clover database found, skipping report generation [INFO] No report being generated for this module. [INFO] Using /default-clover-report descriptor. [INFO] Using Clover report descriptor: /var/tmp/mvn7030681441235862331resource [INFO] [pmd:pmd {execution: default-cli}] [INFO] ------------------------------------------------------------------------ [INFO] Building Mahout Math [INFO] task-segment: [clover2:clover, pmd:pmd] [INFO] ------------------------------------------------------------------------ [INFO] [clover2:clover {execution: default-cli}] [INFO] Using /default-clover-report descriptor. [INFO] Using Clover report descriptor: /var/tmp/mvn5659382849371667608resource [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Clover is enabled with initstring '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/math/target/clover/clover.db' [WARNING] Clover historical directory [/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/math/target/clover/history] does not exist, skipping Clover historical report generation ([/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/math/target/site/clover]) [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Loading coverage database from: '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/math/target/clover/clover.db' [INFO] Writing HTML report to '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/math/target/site/clover' [INFO] Done. Processed 17 packages in 3591ms (211ms per package). [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Clover is enabled with initstring '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/math/target/clover/clover.db' [WARNING] Clover historical directory [/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/math/target/clover/history] does not exist, skipping Clover historical report generation ([/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/math/target/site/clover/clover.xml]) [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Loading coverage database from: '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/math/target/clover/clover.db' [INFO] Writing report to '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/math/target/site/clover/clover.xml' [INFO] [pmd:pmd {execution: default-cli}] [INFO] ------------------------------------------------------------------------ [INFO] Building Mahout Core [INFO] task-segment: [clover2:clover, pmd:pmd] [INFO] ------------------------------------------------------------------------ [INFO] [clover2:clover {execution: default-cli}] [INFO] Using /default-clover-report descriptor. [INFO] Using Clover report descriptor: /var/tmp/mvn8440579012574405159resource [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Clover is enabled with initstring '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/core/target/clover/clover.db' [WARNING] Clover historical directory [/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/core/target/clover/history] does not exist, skipping Clover historical report generation ([/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/core/target/site/clover]) [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Loading coverage database from: '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/core/target/clover/clover.db' [INFO] Writing HTML report to '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/core/target/site/clover' [INFO] Done. Processed 106 packages in 11433ms (107ms per package). [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Clover is enabled with initstring '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/core/target/clover/clover.db' [WARNING] Clover historical directory [/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/core/target/clover/history] does not exist, skipping Clover historical report generation ([/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/core/target/site/clover/clover.xml]) [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Loading coverage database from: '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/core/target/clover/clover.db' [INFO] Writing report to '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/core/target/site/clover/clover.xml' [INFO] [pmd:pmd {execution: default-cli}] [INFO] ------------------------------------------------------------------------ [INFO] Building Mahout Taste Webapp [INFO] task-segment: [clover2:clover, pmd:pmd] [INFO] ------------------------------------------------------------------------ [INFO] [clover2:clover {execution: default-cli}] [INFO] Using /default-clover-report descriptor. [INFO] Using Clover report descriptor: /var/tmp/mvn5674956757549359618resource [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Clover is enabled with initstring '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/taste-web/target/clover/clover.db' [WARNING] Clover historical directory [/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/taste-web/target/clover/history] does not exist, skipping Clover historical report generation ([/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/taste-web/target/site/clover]) [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Loading coverage database from: '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/taste-web/target/clover/clover.db' [INFO] Writing HTML report to '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/taste-web/target/site/clover' [INFO] Done. Processed 1 packages in 159ms (159ms per package). [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Clover is enabled with initstring '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/taste-web/target/clover/clover.db' [WARNING] Clover historical directory [/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/taste-web/target/clover/history] does not exist, skipping Clover historical report generation ([/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/taste-web/target/site/clover/clover.xml]) [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Loading coverage database from: '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/taste-web/target/clover/clover.db' [INFO] Writing report to '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/taste-web/target/site/clover/clover.xml' [INFO] [pmd:pmd {execution: default-cli}] [INFO] ------------------------------------------------------------------------ [INFO] Building Mahout Utilities [INFO] task-segment: [clover2:clover, pmd:pmd] [INFO] ------------------------------------------------------------------------ [INFO] [clover2:clover {execution: default-cli}] [INFO] Using /default-clover-report descriptor. [INFO] Using Clover report descriptor: /var/tmp/mvn6580709384020824148resource [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Clover is enabled with initstring '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/utils/target/clover/clover.db' [WARNING] Clover historical directory [/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/utils/target/clover/history] does not exist, skipping Clover historical report generation ([/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/utils/target/site/clover]) [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Loading coverage database from: '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/utils/target/clover/clover.db' [INFO] Writing HTML report to '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/utils/target/site/clover' [INFO] Done. Processed 16 packages in 535ms (33ms per package). [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Clover is enabled with initstring '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/utils/target/clover/clover.db' [WARNING] Clover historical directory [/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/utils/target/clover/history] does not exist, skipping Clover historical report generation ([/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/utils/target/site/clover/clover.xml]) [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Loading coverage database from: '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/utils/target/clover/clover.db' [INFO] Writing report to '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/utils/target/site/clover/clover.xml' [INFO] [pmd:pmd {execution: default-cli}] [INFO] ------------------------------------------------------------------------ [INFO] Building Mahout Examples [INFO] task-segment: [clover2:clover, pmd:pmd] [INFO] ------------------------------------------------------------------------ [INFO] [clover2:clover {execution: default-cli}] [INFO] Using /default-clover-report descriptor. [INFO] Using Clover report descriptor: /var/tmp/mvn4272620849513588841resource [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Clover is enabled with initstring '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/examples/target/clover/clover.db' [WARNING] Clover historical directory [/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/examples/target/clover/history] does not exist, skipping Clover historical report generation ([/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/examples/target/site/clover]) [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Loading coverage database from: '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/examples/target/clover/clover.db' [INFO] Writing HTML report to '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/examples/target/site/clover' [INFO] Done. Processed 31 packages in 761ms (24ms per package). [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Clover is enabled with initstring '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/examples/target/clover/clover.db' [WARNING] Clover historical directory [/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/examples/target/clover/history] does not exist, skipping Clover historical report generation ([/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/examples/target/site/clover/clover.xml]) [INFO] Clover Version 2.6.3, built on November 20 2009 (build-778) [INFO] Loaded from: /export/home/hudson/.m2/repository/com/cenqua/clover/clover/2.6.3/clover-2.6.3.jar [INFO] Clover: Open Source License registered to Apache. [INFO] Loading coverage database from: '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/examples/target/clover/clover.db' [INFO] Writing report to '/zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Quality/trunk/examples/target/site/clover/clover.xml' [INFO] [pmd:pmd {execution: default-cli}] [INFO] ------------------------------------------------------------------------ [INFO] Building Mahout Release Package [INFO] task-segment: [clover2:clover, pmd:pmd] [INFO] ------------------------------------------------------------------------ [INFO] [clover2:clover {execution: default-cli}] [WARNING] No Clover database found, skipping report generation [INFO] No report being generated for this module. [INFO] Using /default-clover-report descriptor. [INFO] Using Clover report descriptor: /var/tmp/mvn2477224316707682146resource [INFO] [pmd:pmd {execution: default-cli}] [INFO] [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] ------------------------------------------------------------------------ [INFO] Apache Mahout ......................................... SUCCESS [42.557s] [INFO] Mahout Build Tools .................................... SUCCESS [1.752s] [INFO] Mahout Eclipse ........................................ SUCCESS [1.180s] [INFO] Mahout Math ........................................... SUCCESS [12.468s] [INFO] Mahout Core ........................................... SUCCESS [40.377s] [INFO] Mahout Taste Webapp ................................... SUCCESS [1.452s] [INFO] Mahout Utilities ...................................... SUCCESS [4.179s] [INFO] Mahout Examples ....................................... SUCCESS [5.600s] [INFO] Mahout Release Package ................................ SUCCESS [1.048s] [INFO] ------------------------------------------------------------------------ [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESSFUL [INFO] ------------------------------------------------------------------------ [INFO] Total time: 66 minutes 1 second [INFO] Finished at: Fri Apr 01 23:12:54 GMT+00:00 2011 [INFO] Final Memory: 127M/372M [INFO] ------------------------------------------------------------------------ [FINDBUGS] Collecting findbugs analysis files... ERROR: Publisher hudson.plugins.findbugs.FindBugsPublisher aborted due to exception java.lang.ClassCastException: cannot assign instance of hudson.plugins.analysis.core.FilesParser to field hudson.FilePath$FileCallableWrapper.callable of type hudson.FilePath$FileCallable in instance of leblanclarry@example.net$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2004) at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1184) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1916) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1834) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1719) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1305) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348) at hudson.remoting.UserRequest.deserialize(UserRequest.java:178) at hudson.remoting.UserRequest.perform(UserRequest.java:98) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:270) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269) at java.util.concurrent.FutureTask.run(FutureTask.java:123) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:651) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676) at java.lang.Thread.run(Thread.java:595) [PMD] Skipping publisher since build result is FAILURE [TASKS] Skipping publisher since build result is FAILURE Archiving artifacts Recording test results Publishing Javadoc Publishing Clover coverage report... Publishing Clover HTML report... Publishing Clover XML report... Publishing Clover coverage results... From leblanclarry@example.net Fri Apr 01 23:43:46 2011 Return-Path: <leblanclarry@example.net> Delivered-To: leblanclarry@example.net Received: (qmail 24792 invoked from network); 1 Apr 2011 23:43:46 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Apr 2011 23:43:46 -0000 Received: (qmail 99198 invoked by uid 500); 1 Apr 2011 23:43:46 -0000 Delivered-To: leblanclarry@example.net Received: (qmail 99158 invoked by uid 500); 1 Apr 2011 23:43:45 -0000 Mailing-List: contact leblanclarry@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:leblanclarry@example.net> List-Unsubscribe: <mailto:leblanclarry@example.net> List-Post: <mailto:leblanclarry@example.net> List-Id: <dev.mahout.apache.org> Reply-To: leblanclarry@example.net Delivered-To: mailing list leblanclarry@example.net Received: (qmail 99150 invoked by uid 99); 1 Apr 2011 23:43:45 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 23:43:45 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 23:43:43 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155]) by hel.zones.apache.org (Postfix) with ESMTP id C61E18E06E for <leblanclarry@example.net>; Fri, 1 Apr 2011 23:43:05 +0000 (UTC) Date: Fri, 1 Apr 2011 23:43:05 +0000 (UTC) From: "Ted Dunning (JIRA)" <leblanclarry@example.net> To: leblanclarry@example.net Message-ID: <leblanclarry@example.net> In-Reply-To: <leblanclarry@example.net> Subject: [jira] [Commented] (MAHOUT-643) Adding CityBlockSimilarity and DistributedCityBlockDistanceVectorSimilarity MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/MAHOUT-643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014879#comment-13014879 ] Ted Dunning commented on MAHOUT-643: ------------------------------------ Manhattan distance = city block distances = L_1 Chebyshev distance = L_inf > Adding CityBlockSimilarity and DistributedCityBlockDistanceVectorSimilarity > --------------------------------------------------------------------------- > > Key: MAHOUT-643 > URL: https://issues.apache.org/jira/browse/MAHOUT-643 > Project: Mahout > Issue Type: Improvement > Components: Clustering, Math > Affects Versions: 0.5 > Reporter: Daniel McEnnis > Assignee: Sean Owen > Priority: Minor > Labels: distance, patch, similarity > Fix For: 0.5 > > Attachments: MAHOUT-643-2.patch, MAHOUT-643.patch, patch4.txt > > Original Estimate: 1h > Remaining Estimate: 1h > > adding a new distance metric to the 0.5 branch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira From leblanclarry@example.net Sat Apr 02 07:06:50 2011 Return-Path: <leblanclarry@example.net> Delivered-To: leblanclarry@example.net Received: (qmail 76214 invoked from network); 2 Apr 2011 07:06:50 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Apr 2011 07:06:50 -0000 Received: (qmail 3792 invoked by uid 500); 2 Apr 2011 07:06:49 -0000 Delivered-To: leblanclarry@example.net Received: (qmail 3748 invoked by uid 500); 2 Apr 2011 07:06:48 -0000 Mailing-List: contact leblanclarry@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:leblanclarry@example.net> List-Unsubscribe: <mailto:leblanclarry@example.net> List-Post: <mailto:leblanclarry@example.net> List-Id: <dev.mahout.apache.org> Reply-To: leblanclarry@example.net Delivered-To: mailing list leblanclarry@example.net Received: (qmail 3739 invoked by uid 99); 2 Apr 2011 07:06:46 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Apr 2011 07:06:46 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Apr 2011 07:06:44 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155]) by hel.zones.apache.org (Postfix) with ESMTP id 79D428ED80 for <leblanclarry@example.net>; Sat, 2 Apr 2011 07:06:07 +0000 (UTC) Date: Sat, 2 Apr 2011 07:06:07 +0000 (UTC) From: "Sebastian Schelter (JIRA)" <leblanclarry@example.net> To: leblanclarry@example.net Message-ID: <leblanclarry@example.net> Subject: [jira] [Created] (MAHOUT-648) API-changes for optimizing recommender performance in some usecases MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 API-changes for optimizing recommender performance in some usecases ------------------------------------------------------------------- Key: MAHOUT-648 URL: https://issues.apache.org/jira/browse/MAHOUT-648 Project: Mahout Issue Type: New Feature Components: Collaborative Filtering Affects Versions: 0.5 Reporter: Sebastian Schelter Assignee: Sebastian Schelter I'd like to propose a set of small API changes in our recommender code. * add a method *allSimilarItemIDs(long itemID)* to ItemSimilarity, which returns the ids of all similar items * make sure that *GenericItemBasedRecommender.recommend(...)* only makes *a single call to the DataModel* with which it retrieves all preferences for the user to recommend items for * add *a new strategy for finding candidate items* for the most-similar-items and recommendation computation that only calls ItemSimilarity.allSimilarItemIDs(...) and doesn't need to call anything on the DataModel * and an option to GenericItemSimilarity to make it create an in-memory-index to allow *retrieval of all similar items per item in constant time* The purpose of these changes is to make it possible to run a very efficient recommender for usecases, where the major purpose of the recommender is to answer requests for most-similar-items and it you only have to compute "real" recommendations from time to time. A typical scenario where these conditions are met is e-commerce, you have lots of most-similar-items calls as users browse product pages and fill their shopping carts and for the minority of users that log in you have to provide personalized product recommendations. With the proposed changes, you need to precompute the item-similarities and load them into memory, either from a file with FileItemSimilarity or from a database with the new MySQLJDBCInMemoryItemSimilarity and use a GenericItemBasedRecommender with the AllSimilarItemsCandidateItemsStrategy. Requests for most-similar-items can be completely answered from memory (in nearly constant time) without having to touch the DataModel. Answering 100 requests per second on a single machine are no problem using this approach. We can then use *a DataModel that does not need to reside in memory* because its only task is to act as a repository for the users' preferences. When we compute personalized recommendations we need to do exactly one single call to the datastore to retrieve all the preferences for the user we wanna compute recommendations for. This single call should be very fast with our already existing jdbc-backed DataModel's and it should be easy to implement it equally fast in other datastores like Solr for example. One could even start thinking about sharded DataModels with this approach. Another very big advantage of this approach is that *user preferences can now be updated in realtime* as we never need to refresh the datamodel. We only need to refresh the item-similarities from time to time. Memory requirements for the recommender machines would drop drastically as we *only have to store the item-similarities in RAM* whose number should be orders of magnitude smaller than the number of preferences. The API changes in the patch should be fully backwards compatible, so that this new approach is only an additional way to use our recommender code and all currently existing approaches still work as before. Here is an example how such a setup would work using a MySQL database: {noformat} DataSource dataSource = ... DataModel dataModel = new MySQLJDBCDataModel(dataSource); /* load all item-similarities into memory, create an index for fast retrieval of all-similar-item-ids */ ItemSimilarity itemSimilarity = MySQLJDBCInMemoryItemSimilarity(dataSource, true); /* the candidate items for recommendation and most-similar-items are only fetched from our in-memory data structures by this strategy*/ AllSimilarItemsCandidateItemsStrategy allSimilarItemsStrategy = new AllSimilarItemsCandidateItemsStrategy(itemSimilarity); ItemBasedRecommender recommender = new GenericItemBasedRecommender(dataModel, itemSimilarity, allSimilarItemsStrategy, allSimilarItemsStrategy); {noformat} -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira From leblanclarry@example.net Sat Apr 02 07:22:47 2011 Return-Path: <leblanclarry@example.net> Delivered-To: leblanclarry@example.net Received: (qmail 98582 invoked from network); 2 Apr 2011 07:22:46 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Apr 2011 07:22:46 -0000 Received: (qmail 7418 invoked by uid 500); 2 Apr 2011 07:22:45 -0000 Delivered-To: leblanclarry@example.net Received: (qmail 7377 invoked by uid 500); 2 Apr 2011 07:22:45 -0000 Mailing-List: contact leblanclarry@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:leblanclarry@example.net> List-Unsubscribe: <mailto:leblanclarry@example.net> List-Post: <mailto:leblanclarry@example.net> List-Id: <dev.mahout.apache.org> Reply-To: leblanclarry@example.net Delivered-To: mailing list leblanclarry@example.net Received: (qmail 7367 invoked by uid 99); 2 Apr 2011 07:22:44 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Apr 2011 07:22:43 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Apr 2011 07:22:42 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155]) by hel.zones.apache.org (Postfix) with ESMTP id BDAA38F01E for <leblanclarry@example.net>; Sat, 2 Apr 2011 07:22:05 +0000 (UTC) Date: Sat, 2 Apr 2011 07:22:05 +0000 (UTC) From: "Sebastian Schelter (JIRA)" <leblanclarry@example.net> To: leblanclarry@example.net Message-ID: <{{EMAIL+PHONE}}> In-Reply-To: <leblanclarry@example.net> Subject: [jira] [Updated] (MAHOUT-648) API-changes for optimizing recommender performance in some usecases MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/MAHOUT-648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sebastian Schelter updated MAHOUT-648: -------------------------------------- Attachment: MAHOUT-648.patch > API-changes for optimizing recommender performance in some usecases > ------------------------------------------------------------------- > > Key: MAHOUT-648 > URL: https://issues.apache.org/jira/browse/MAHOUT-648 > Project: Mahout > Issue Type: New Feature > Components: Collaborative Filtering > Affects Versions: 0.5 > Reporter: Sebastian Schelter > Assignee: Sebastian Schelter > Attachments: MAHOUT-648.patch > > > I'd like to propose a set of small API changes in our recommender code. > * add a method *allSimilarItemIDs(long itemID)* to ItemSimilarity, which returns the ids of all similar items > * make sure that *GenericItemBasedRecommender.recommend(...)* only makes *a single call to the DataModel* with which it retrieves all preferences for the user to recommend items for > * add *a new strategy for finding candidate items* for the most-similar-items and recommendation computation that only calls ItemSimilarity.allSimilarItemIDs(...) and doesn't need to call anything on the DataModel > * and an option to GenericItemSimilarity to make it create an in-memory-index to allow *retrieval of all similar items per item in constant time* > The purpose of these changes is to make it possible to run a very efficient recommender for usecases, where the major purpose of the recommender is to answer requests for most-similar-items and it you only have to compute "real" recommendations from time to time. A typical scenario where these conditions are met is e-commerce, you have lots of most-similar-items calls as users browse product pages and fill their shopping carts and for the minority of users that log in you have to provide personalized product recommendations. > With the proposed changes, you need to precompute the item-similarities and load them into memory, either from a file with FileItemSimilarity or from a database with the new MySQLJDBCInMemoryItemSimilarity and use a GenericItemBasedRecommender with the AllSimilarItemsCandidateItemsStrategy. Requests for most-similar-items can be completely answered from memory (in nearly constant time) without having to touch the DataModel. Answering 100 requests per second on a single machine are no problem using this approach. > We can then use *a DataModel that does not need to reside in memory* because its only task is to act as a repository for the users' preferences. When we compute personalized recommendations we need to do exactly one single call to the datastore to retrieve all the preferences for the user we wanna compute recommendations for. This single call should be very fast with our already existing jdbc-backed DataModel's and it should be easy to implement it equally fast in other datastores like Solr for example. One could even start thinking about sharded DataModels with this approach. > Another very big advantage of this approach is that *user preferences can now be updated in realtime* as we never need to refresh the datamodel. We only need to refresh the item-similarities from time to time. Memory requirements for the recommender machines would drop drastically as we *only have to store the item-similarities in RAM* whose number should be orders of magnitude smaller than the number of preferences. > The API changes in the patch should be fully backwards compatible, so that this new approach is only an additional way to use our recommender code and all currently existing approaches still work as before. > Here is an example how such a setup would work using a MySQL database: > {noformat} > DataSource dataSource = ... > DataModel dataModel = new MySQLJDBCDataModel(dataSource); > /* load all item-similarities into memory, create an index for fast retrieval of all-similar-item-ids */ > ItemSimilarity itemSimilarity = MySQLJDBCInMemoryItemSimilarity(dataSource, true); > /* the candidate items for recommendation and most-similar-items are only fetched from our in-memory data structures by this strategy*/ > AllSimilarItemsCandidateItemsStrategy allSimilarItemsStrategy = new AllSimilarItemsCandidateItemsStrategy(itemSimilarity); > ItemBasedRecommender recommender = new GenericItemBasedRecommender(dataModel, itemSimilarity, allSimilarItemsStrategy, allSimilarItemsStrategy); > {noformat} -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira From leblanclarry@example.net Sat Apr 02 07:22:50 2011 Return-Path: <leblanclarry@example.net> Delivered-To: leblanclarry@example.net Received: (qmail 98924 invoked from network); 2 Apr 2011 07:22:50 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Apr 2011 07:22:50 -0000 Received: (qmail 7988 invoked by uid 500); 2 Apr 2011 07:22:48 -0000 Delivered-To: leblanclarry@example.net Received: (qmail 7932 invoked by uid 500); 2 Apr 2011 07:22:47 -0000 Mailing-List: contact leblanclarry@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:leblanclarry@example.net> List-Unsubscribe: <mailto:leblanclarry@example.net> List-Post: <mailto:leblanclarry@example.net> List-Id: <dev.mahout.apache.org> Reply-To:
{'source': 'Apache Software Foundation Public Mail Archives'}