From mahout-commits-returnrcantu@example.net Fri May 01 07:32:52 2009 Return-Path: <mahout-commits-returnrcantu@example.net> Delivered-To: rcantu@example.net Received: (qmail 14684 invoked from network); 1 May 2009 07:32:52 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 May 2009 07:32:52 -0000 Received: (qmail 45953 invoked by uid 500); 1 May 2009 07:32:52 -0000 Delivered-To: rcantu@example.net Received: (qmail 45882 invoked by uid 500); 1 May 2009 07:32:52 -0000 Mailing-List: contact rcantu@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rcantu@example.net> List-Unsubscribe: <mailto:rcantu@example.net> List-Post: <mailto:rcantu@example.net> List-Id: <mahout-commits.lucene.apache.org> Reply-To: rcantu@example.net Delivered-To: mailing list rcantu@example.net Received: (qmail 45873 invoked by uid 99); 1 May 2009 07:32:52 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 May 2009 07:32:52 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 May 2009 07:32:49 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 2AFB723888A6; Fri, 1 May 2009 07:32:28 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r770553 - in /lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl: common/ neighborhood/ Date: Fri, 01 May 2009 07:32:27 -0000 To: rcantu@example.net From: rcantu@example.net X-Mailer: svnmailer-1.0.8 Message-Id: <rcantu@example.net> X-Virus-Checked: Checked by ClamAV on apache.org Author: srowen Date: Fri May 1 07:32:27 2009 New Revision: 770553 URL: http://svn.apache.org/viewvc?rev=770553&view=rev Log: Added SamplingIterator and SamplingIterable and fixed up issue with sampling in NearestNUserNeighborhood Added: lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/common/SamplingIterable.java lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/common/SamplingIterator.java Modified: lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/neighborhood/AbstractUserNeighborhood.java lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/neighborhood/NearestNUserNeighborhood.java lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/neighborhood/ThresholdUserNeighborhood.java Added: lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/common/SamplingIterable.java URL: http://svn.apache.org/viewvc/lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/common/SamplingIterable.java?rev=770553&view=auto ============================================================================== --- lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/common/SamplingIterable.java (added) +++ lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/common/SamplingIterable.java Fri May 1 07:32:27 2009 @@ -0,0 +1,45 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.mahout.cf.taste.impl.common; + +import java.util.Iterator; + +/** + * Wraps an {@link Iterable} whose {@link Iterable#iterator()} returns only some subset of the elements + * that it would, as determined by a sampling rate parameter. + */ +public final class SamplingIterable<T> implements Iterable<T> { + + private final Iterable<? extends T> delegate; + private final double samplingRate; + + public SamplingIterable(Iterable<? extends T> delegate, double samplingRate) { + this.delegate = delegate; + this.samplingRate = samplingRate; + } + + @Override + public Iterator<T> iterator() { + return new SamplingIterator<T>(delegate.iterator(), samplingRate); + } + + public static <T> Iterable<T> maybeWrapIterable(Iterable<T> delegate, double samplingRate) { + return samplingRate >= 1.0 ? delegate : new SamplingIterable<T>(delegate, samplingRate); + } + +} \ No newline at end of file Added: lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/common/SamplingIterator.java URL: http://svn.apache.org/viewvc/lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/common/SamplingIterator.java?rev=770553&view=auto ============================================================================== --- lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/common/SamplingIterator.java (added) +++ lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/common/SamplingIterator.java Fri May 1 07:32:27 2009 @@ -0,0 +1,83 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.mahout.cf.taste.impl.common; + +import java.util.Iterator; +import java.util.Random; +import java.util.NoSuchElementException; + +/** + * Wraps an {@link Iterator} and returns only some subset of the elements + * that it would, as determined by a sampling rate parameter. + */ +public final class SamplingIterator<T> implements Iterator<T> { + + private static final Random r = RandomUtils.getRandom(); + + private final Iterator<? extends T> delegate; + private final double samplingRate; + private T next; + private boolean hasNext; + + public SamplingIterator(Iterator<? extends T> delegate, double samplingRate) { + this.delegate = delegate; + this.samplingRate = samplingRate; + this.hasNext = true; + doNext(); + } + + @Override + public boolean hasNext() { + return hasNext; + } + + @Override + public T next() { + if (hasNext) { + T result = next; + doNext(); + return result; + } + throw new NoSuchElementException(); + } + + private void doNext() { + boolean found = false; + while (delegate.hasNext()) { + T delegateNext = delegate.next(); + if (r.nextDouble() < samplingRate) { + next = delegateNext; + found = true; + break; + } + } + if (!found) { + hasNext = false; + next = null; + } + } + + /** + * @throws UnsupportedOperationException + */ + @Override + public void remove() { + throw new UnsupportedOperationException(); + } + +} Modified: lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/neighborhood/AbstractUserNeighborhood.java URL: http://svn.apache.org/viewvc/lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/neighborhood/AbstractUserNeighborhood.java?rev=770553&r1=770552&r2=770553&view=diff ============================================================================== --- lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/neighborhood/AbstractUserNeighborhood.java (original) +++ lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/neighborhood/AbstractUserNeighborhood.java Fri May 1 07:32:27 2009 @@ -20,20 +20,16 @@ import org.apache.mahout.cf.taste.common.Refreshable; import org.apache.mahout.cf.taste.similarity.UserSimilarity; import org.apache.mahout.cf.taste.impl.common.RefreshHelper; -import org.apache.mahout.cf.taste.impl.common.RandomUtils; import org.apache.mahout.cf.taste.model.DataModel; import org.apache.mahout.cf.taste.neighborhood.UserNeighborhood; import java.util.Collection; -import java.util.Random; /** * <p>Contains methods and resources useful to all classes in this package.</p> */ abstract class AbstractUserNeighborhood implements UserNeighborhood { - private static final Random random = RandomUtils.getRandom(); - private final UserSimilarity userSimilarity; private final DataModel dataModel; private final double samplingRate; @@ -64,8 +60,8 @@ return dataModel; } - final boolean sampleForUser() { - return samplingRate >= 1.0 || random.nextDouble() < samplingRate; + final double getSamplingRate() { + return samplingRate; } @Override Modified: lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/neighborhood/NearestNUserNeighborhood.java URL: http://svn.apache.org/viewvc/lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/neighborhood/NearestNUserNeighborhood.java?rev=770553&r1=770552&r2=770553&view=diff ============================================================================== --- lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/neighborhood/NearestNUserNeighborhood.java (original) +++ lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/neighborhood/NearestNUserNeighborhood.java Fri May 1 07:32:27 2009 @@ -22,6 +22,7 @@ import org.apache.mahout.cf.taste.model.DataModel; import org.apache.mahout.cf.taste.model.User; import org.apache.mahout.cf.taste.impl.recommender.TopItems; +import org.apache.mahout.cf.taste.impl.common.SamplingIterable; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -31,7 +32,7 @@ /** * <p>Computes a neighborhood consisting of the nearest n {@link User}s to a given {@link User}. - * "Nearest" is defined by the given {@link org.apache.mahout.cf.taste.similarity.UserSimilarity}.</p> + * "Nearest" is defined by the given {@link UserSimilarity}.</p> */ public final class NearestNUserNeighborhood extends AbstractUserNeighborhood { @@ -97,7 +98,8 @@ TopItems.Estimator<User> estimator = new Estimator(userSimilarityImpl, theUser, minSimilarity); - List<User> neighborhood = TopItems.getTopUsers(n, dataModel.getUsers(), null, estimator); + Iterable<? extends User> users = SamplingIterable.maybeWrapIterable(dataModel.getUsers(), getSamplingRate()); + List<User> neighborhood = TopItems.getTopUsers(n, users, null, estimator); log.trace("UserNeighborhood around user ID '{}' is: {}", userID, neighborhood); Modified: lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/neighborhood/ThresholdUserNeighborhood.java URL: http://svn.apache.org/viewvc/lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/neighborhood/ThresholdUserNeighborhood.java?rev=770553&r1=770552&r2=770553&view=diff ============================================================================== --- lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/neighborhood/ThresholdUserNeighborhood.java (original) +++ lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/neighborhood/ThresholdUserNeighborhood.java Fri May 1 07:32:27 2009 @@ -21,6 +21,7 @@ import org.apache.mahout.cf.taste.similarity.UserSimilarity; import org.apache.mahout.cf.taste.model.DataModel; import org.apache.mahout.cf.taste.model.User; +import org.apache.mahout.cf.taste.impl.common.SamplingIterable; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -33,7 +34,7 @@ /** * <p>Computes a neigbhorhood consisting of all {@link User}s whose similarity to the * given {@link User} meets or exceeds a certain threshold. Similarity is defined by the given - * {@link org.apache.mahout.cf.taste.similarity.UserSimilarity}.</p> + * {@link UserSimilarity}.</p> */ public final class ThresholdUserNeighborhood extends AbstractUserNeighborhood { @@ -83,12 +84,14 @@ DataModel dataModel = getDataModel(); User theUser = dataModel.getUser(userID); List<User> neighborhood = new ArrayList<User>(); - Iterator<? extends User> users = dataModel.getUsers().iterator(); + Iterable<? extends User> usersIterable = + SamplingIterable.maybeWrapIterable(dataModel.getUsers(), getSamplingRate()); + Iterator<? extends User> users = usersIterable.iterator(); UserSimilarity userSimilarityImpl = getUserSimilarity(); while (users.hasNext()) { User user = users.next(); - if (sampleForUser() && !userID.equals(user.getID())) { + if (!userID.equals(user.getID())) { double theSimilarity = userSimilarityImpl.userSimilarity(theUser, user); if (!Double.isNaN(theSimilarity) && theSimilarity >= threshold) { neighborhood.add(user); From mahout-commits-returnrcantu@example.net Fri May 01 16:04:29 2009 Return-Path: <mahout-commits-returnrcantu@example.net> Delivered-To: rcantu@example.net Received: (qmail 30885 invoked from network); 1 May 2009 16:04:28 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 May 2009 16:04:28 -0000 Received: (qmail 84236 invoked by uid 500); 1 May 2009 16:04:28 -0000 Delivered-To: rcantu@example.net Received: (qmail 84166 invoked by uid 500); 1 May 2009 16:04:28 -0000 Mailing-List: contact rcantu@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rcantu@example.net> List-Unsubscribe: <mailto:rcantu@example.net> List-Post: <mailto:rcantu@example.net> List-Id: <mahout-commits.lucene.apache.org> Reply-To: rcantu@example.net Delivered-To: mailing list rcantu@example.net Received: (qmail 84157 invoked by uid 99); 1 May 2009 16:04:28 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 May 2009 16:04:28 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 May 2009 16:04:21 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 0ED5F234C04B for <rcantu@example.net>; Fri, 1 May 2009 09:04:00 -0700 (PDT) Message-ID: <1150454269.1241193840059.JavaMail.www-data@brutus> Date: Fri, 1 May 2009 09:04:00 -0700 (PDT) From: rcantu@example.net To: rcantu@example.net Subject: [CONF] Apache Lucene Mahout: Mahout on Elastic MapReduce (page created) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Mahout on Elastic MapReduce (MAHOUT) created by Stephen Green http://cwiki.apache.org/confluence/display/MAHOUT/Mahout+on+Elastic+MapReduce Content: --------------------------------------------------------------------- This page details the set of steps that was necessary to get an example of k-Means clustering running on Amazon's Elastic MapReduce. The aim here was simply to get something running, but it should provide a good head start if you want to run something else. h1. Getting Started * Get yourself an Elastic MapReduce account. If you're already using EC2, then you can do this from [Amazon's AWS Managment Console|https://console.aws.amazon.com/], which has a tab for running EMR. * Make sure that you're up to date with the Mahout source. h1. Uploading data h1. Running Mahout --------------------------------------------------------------------- CONFLUENCE INFORMATION This message is automatically generated by Confluence Unsubscribe or edit your notifications preferences http://cwiki.apache.org/confluence/users/viewnotifications.action If you think it was sent incorrectly contact one of the administrators http://cwiki.apache.org/confluence/administrators.action If you want more information on Confluence, or have a bug to report see http://www.atlassian.com/software/confluence From mahout-commits-returnrcantu@example.net Fri May 01 17:18:31 2009 Return-Path: <mahout-commits-returnrcantu@example.net> Delivered-To: rcantu@example.net Received: (qmail 62100 invoked from network); 1 May 2009 17:18:30 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 May 2009 17:18:30 -0000 Received: (qmail 32113 invoked by uid 500); 1 May 2009 17:18:30 -0000 Delivered-To: rcantu@example.net Received: (qmail 32080 invoked by uid 500); 1 May 2009 17:18:30 -0000 Mailing-List: contact rcantu@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rcantu@example.net> List-Unsubscribe: <mailto:rcantu@example.net> List-Post: <mailto:rcantu@example.net> List-Id: <mahout-commits.lucene.apache.org> Reply-To: rcantu@example.net Delivered-To: mailing list rcantu@example.net Received: (qmail 32071 invoked by uid 99); 1 May 2009 17:18:30 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 May 2009 17:18:30 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 May 2009 17:18:22 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 54792234C04C for <rcantu@example.net>; Fri, 1 May 2009 10:18:01 -0700 (PDT) Message-ID: <1049203656.1241198281343.JavaMail.www-data@brutus> Date: Fri, 1 May 2009 10:18:01 -0700 (PDT) From: rcantu@example.net To: rcantu@example.net Subject: [CONF] Apache Lucene Mahout: Mahout on Elastic MapReduce (attachment added) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org New files attached to Mahout on Elastic MapReduce (Apache Lucene Mahout) by Stephen Green http://cwiki.apache.org/confluence/display/MAHOUT/Mahout+on+Elastic+MapReduce Attached file(s): --------------------------------------------------------------------- - mahoutjob.tiff (image/tiff, 255 kb) Using the EMR GUI to start a k-means clustering job http://cwiki.apache.org/confluence/download/attachments/116559/mahoutjob.tiff?version=1 --------------------------------------------------------------------- CONFLUENCE INFORMATION This message is automatically generated by Confluence Unsubscribe or edit your notifications preferences http://cwiki.apache.org/confluence/users/viewnotifications.action If you think it was sent incorrectly contact one of the administrators http://cwiki.apache.org/confluence/administrators.action If you want more information on Confluence, or have a bug to report see http://www.atlassian.com/software/confluence From mahout-commits-returnrcantu@example.net Fri May 01 17:20:34 2009 Return-Path: <mahout-commits-returnrcantu@example.net> Delivered-To: rcantu@example.net Received: (qmail 64225 invoked from network); 1 May 2009 17:20:34 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 May 2009 17:20:34 -0000 Received: (qmail 35841 invoked by uid 500); 1 May 2009 17:20:34 -0000 Delivered-To: rcantu@example.net Received: (qmail 35788 invoked by uid 500); 1 May 2009 17:20:34 -0000 Mailing-List: contact rcantu@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rcantu@example.net> List-Unsubscribe: <mailto:rcantu@example.net> List-Post: <mailto:rcantu@example.net> List-Id: <mahout-commits.lucene.apache.org> Reply-To: rcantu@example.net Delivered-To: mailing list rcantu@example.net Received: (qmail 35779 invoked by uid 99); 1 May 2009 17:20:34 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 May 2009 17:20:34 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 May 2009 17:20:25 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 9298F234C04B for <rcantu@example.net>; Fri, 1 May 2009 10:20:04 -0700 (PDT) Message-ID: <60342132.1241198404598.JavaMail.www-data@brutus> Date: Fri, 1 May 2009 10:20:04 -0700 (PDT) From: rcantu@example.net To: rcantu@example.net Subject: [CONF] Apache Lucene Mahout: Mahout on Elastic MapReduce (attachment added) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org New files attached to Mahout on Elastic MapReduce (Apache Lucene Mahout) by Stephen Green http://cwiki.apache.org/confluence/display/MAHOUT/Mahout+on+Elastic+MapReduce Attached file(s): --------------------------------------------------------------------- - emrmh1.png (image/png, 147 kb) Using the EMR GUI to start a k-means clustering job http://cwiki.apache.org/confluence/download/attachments/116559/emrmh1.png?version=1 --------------------------------------------------------------------- CONFLUENCE INFORMATION This message is automatically generated by Confluence Unsubscribe or edit your notifications preferences http://cwiki.apache.org/confluence/users/viewnotifications.action If you think it was sent incorrectly contact one of the administrators http://cwiki.apache.org/confluence/administrators.action If you want more information on Confluence, or have a bug to report see http://www.atlassian.com/software/confluence From mahout-commits-returnrcantu@example.net Fri May 01 17:38:36 2009 Return-Path: <mahout-commits-returnrcantu@example.net> Delivered-To: rcantu@example.net Received: (qmail 81025 invoked from network); 1 May 2009 17:38:36 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 May 2009 17:38:36 -0000 Received: (qmail 68052 invoked by uid 500); 1 May 2009 17:38:36 -0000 Delivered-To: rcantu@example.net Received: (qmail 68012 invoked by uid 500); 1 May 2009 17:38:35 -0000 Mailing-List: contact rcantu@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rcantu@example.net> List-Unsubscribe: <mailto:rcantu@example.net> List-Post: <mailto:rcantu@example.net> List-Id: <mahout-commits.lucene.apache.org> Reply-To: rcantu@example.net Delivered-To: mailing list rcantu@example.net Received: (qmail 68003 invoked by uid 99); 1 May 2009 17:38:35 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 May 2009 17:38:35 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 May 2009 17:38:34 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 5682B23888CA; Fri, 1 May 2009 17:38:14 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r770768 - in /lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/recommender: FarthestNeighborClusterSimilarity.java NearestNeighborClusterSimilarity.java TreeClusteringRecommender.java Date: Fri, 01 May 2009 17:38:14 -0000 To: rcantu@example.net From: rcantu@example.net X-Mailer: svnmailer-1.0.8 Message-Id: <rcantu@example.net> X-Virus-Checked: Checked by ClamAV on apache.org Author: srowen Date: Fri May 1 17:38:13 2009 New Revision: 770768 URL: http://svn.apache.org/viewvc?rev=770768&view=rev Log: (empty) Modified: lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/recommender/FarthestNeighborClusterSimilarity.java lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/recommender/NearestNeighborClusterSimilarity.java lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/recommender/TreeClusteringRecommender.java Modified: lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/recommender/FarthestNeighborClusterSimilarity.java URL: http://svn.apache.org/viewvc/lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/recommender/FarthestNeighborClusterSimilarity.java?rev=770768&r1=770767&r2=770768&view=diff ============================================================================== --- lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/recommender/FarthestNeighborClusterSimilarity.java (original) +++ lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/recommender/FarthestNeighborClusterSimilarity.java Fri May 1 17:38:13 2009 @@ -21,23 +21,20 @@ import org.apache.mahout.cf.taste.common.TasteException; import org.apache.mahout.cf.taste.similarity.UserSimilarity; import org.apache.mahout.cf.taste.impl.common.RefreshHelper; -import org.apache.mahout.cf.taste.impl.common.RandomUtils; +import org.apache.mahout.cf.taste.impl.common.SamplingIterable; import org.apache.mahout.cf.taste.model.User; import java.util.Collection; -import java.util.Random; /** * <p>Defines cluster similarity as the <em>smallest</em> similarity between any two - * {@link org.apache.mahout.cf.taste.model.User}s in the clusters -- that is, it says that clusters are close + * {@link User}s in the clusters -- that is, it says that clusters are close * when <em>all pairs</em> of their members have relatively high similarity.</p> */ public final class FarthestNeighborClusterSimilarity implements ClusterSimilarity { - private static final Random random = RandomUtils.getRandom(); - private final UserSimilarity similarity; - private final double samplingPercentage; + private final double samplingRate; /** * <p>Constructs a {@link FarthestNeighborClusterSimilarity} based on the given {@link UserSimilarity}. @@ -49,19 +46,19 @@ /** * <p>Constructs a {@link FarthestNeighborClusterSimilarity} based on the given {@link UserSimilarity}. - * By setting <code>samplingPercentage</code> to a value less than 1.0, this implementation will only examine + * By setting <code>samplingRate</code> to a value less than 1.0, this implementation will only examine * that fraction of all user-user similarities between two clusters, increasing performance at the expense * of accuracy.</p> */ - public FarthestNeighborClusterSimilarity(UserSimilarity similarity, double samplingPercentage) { + public FarthestNeighborClusterSimilarity(UserSimilarity similarity, double samplingRate) { if (similarity == null) { throw new IllegalArgumentException("similarity is null"); } - if (Double.isNaN(samplingPercentage) || samplingPercentage <= 0.0 || samplingPercentage > 1.0) { - throw new IllegalArgumentException("samplingPercentage is invalid: " + samplingPercentage); + if (Double.isNaN(samplingRate) || samplingRate <= 0.0 || samplingRate > 1.0) { + throw new IllegalArgumentException("samplingRate is invalid: " + samplingRate); } this.similarity = similarity; - this.samplingPercentage = samplingPercentage; + this.samplingRate = samplingRate; } @Override @@ -71,13 +68,12 @@ return Double.NaN; } double leastSimilarity = Double.POSITIVE_INFINITY; - for (User user1 : cluster1) { - if (samplingPercentage >= 1.0 || random.nextDouble() < samplingPercentage) { - for (User user2 : cluster2) { - double theSimilarity = similarity.userSimilarity(user1, user2); - if (theSimilarity < leastSimilarity) { - leastSimilarity = theSimilarity; - } + Iterable<User> someUsers = SamplingIterable.maybeWrapIterable(cluster1, samplingRate); + for (User user1 : someUsers) { + for (User user2 : cluster2) { + double theSimilarity = similarity.userSimilarity(user1, user2); + if (theSimilarity < leastSimilarity) { + leastSimilarity = theSimilarity; } } } Modified: lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/recommender/NearestNeighborClusterSimilarity.java URL: http://svn.apache.org/viewvc/lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/recommender/NearestNeighborClusterSimilarity.java?rev=770768&r1=770767&r2=770768&view=diff ============================================================================== --- lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/recommender/NearestNeighborClusterSimilarity.java (original) +++ lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/recommender/NearestNeighborClusterSimilarity.java Fri May 1 17:38:13 2009 @@ -21,26 +21,23 @@ import org.apache.mahout.cf.taste.common.TasteException; import org.apache.mahout.cf.taste.similarity.UserSimilarity; import org.apache.mahout.cf.taste.impl.common.RefreshHelper; -import org.apache.mahout.cf.taste.impl.common.RandomUtils; +import org.apache.mahout.cf.taste.impl.common.SamplingIterable; import org.apache.mahout.cf.taste.model.User; import java.util.Collection; -import java.util.Random; /** * <p>Defines cluster similarity as the <em>largest</em> similarity between any two - * {@link org.apache.mahout.cf.taste.model.User}s in the clusters -- that is, it says that clusters are close + * {@link User}s in the clusters -- that is, it says that clusters are close * when <em>some pair</em> of their members has high similarity.</p> */ public final class NearestNeighborClusterSimilarity implements ClusterSimilarity { - private static final Random random = RandomUtils.getRandom(); - private final UserSimilarity similarity; - private final double samplingPercentage; + private final double samplingRate; /** - * <p>Constructs a {@link NearestNeighborClusterSimilarity} based on the given {@link org.apache.mahout.cf.taste.similarity.UserSimilarity}. + * <p>Constructs a {@link NearestNeighborClusterSimilarity} based on the given {@link UserSimilarity}. * All user-user similarities are examined.</p> */ public NearestNeighborClusterSimilarity(UserSimilarity similarity) { @@ -48,20 +45,20 @@ } /** - * <p>Constructs a {@link NearestNeighborClusterSimilarity} based on the given {@link org.apache.mahout.cf.taste.similarity.UserSimilarity}. - * By setting <code>samplingPercentage</code> to a value less than 1.0, this implementation will only examine + * <p>Constructs a {@link NearestNeighborClusterSimilarity} based on the given {@link UserSimilarity}. + * By setting <code>samplingRate</code> to a value less than 1.0, this implementation will only examine * that fraction of all user-user similarities between two clusters, increasing performance at the expense * of accuracy.</p> */ - public NearestNeighborClusterSimilarity(UserSimilarity similarity, double samplingPercentage) { + public NearestNeighborClusterSimilarity(UserSimilarity similarity, double samplingRate) { if (similarity == null) { throw new IllegalArgumentException("similarity is null"); } - if (Double.isNaN(samplingPercentage) || samplingPercentage <= 0.0 || samplingPercentage > 1.0) { - throw new IllegalArgumentException("samplingPercentage is invalid: " + samplingPercentage); + if (Double.isNaN(samplingRate) || samplingRate <= 0.0 || samplingRate > 1.0) { + throw new IllegalArgumentException("samplingRate is invalid: " + samplingRate); } this.similarity = similarity; - this.samplingPercentage = samplingPercentage; + this.samplingRate = samplingRate; } @Override @@ -70,14 +67,13 @@ if (cluster1.isEmpty() || cluster2.isEmpty()) { return Double.NaN; } + Iterable<User> someUsers = SamplingIterable.maybeWrapIterable(cluster1, samplingRate); double greatestSimilarity = Double.NEGATIVE_INFINITY; - for (User user1 : cluster1) { - if (samplingPercentage >= 1.0 || random.nextDouble() < samplingPercentage) { - for (User user2 : cluster2) { - double theSimilarity = similarity.userSimilarity(user1, user2); - if (theSimilarity > greatestSimilarity) { - greatestSimilarity = theSimilarity; - } + for (User user1 : someUsers) { + for (User user2 : cluster2) { + double theSimilarity = similarity.userSimilarity(user1, user2); + if (theSimilarity > greatestSimilarity) { + greatestSimilarity = theSimilarity; } } } Modified: lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/recommender/TreeClusteringRecommender.java URL: http://svn.apache.org/viewvc/lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/recommender/TreeClusteringRecommender.java?rev=770768&r1=770767&r2=770768&view=diff ============================================================================== --- lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/recommender/TreeClusteringRecommender.java (original) +++ lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/recommender/TreeClusteringRecommender.java Fri May 1 17:38:13 2009 @@ -60,13 +60,15 @@ */ public final class TreeClusteringRecommender extends AbstractRecommender implements ClusteringRecommender { + private static final Random r = RandomUtils.getRandom(); + private static final Logger log = LoggerFactory.getLogger(TreeClusteringRecommender.class); private final ClusterSimilarity clusterSimilarity; private final int numClusters; private final double clusteringThreshold; private final boolean clusteringByThreshold; - private final double samplingPercentage; + private final double samplingRate; private Map<Object, List<RecommendedItem>> topRecsByUserID; private Collection<Collection<User>> allClusters; private Map<Object, Collection<User>> clustersByUserID; @@ -91,16 +93,16 @@ * @param dataModel {@link DataModel} which provdes {@link User}s * @param clusterSimilarity {@link ClusterSimilarity} used to compute cluster similarity * @param numClusters desired number of clusters to create - * @param samplingPercentage percentage of all cluster-cluster pairs to consider when finding + * @param samplingRate percentage of all cluster-cluster pairs to consider when finding * next-most-similar clusters. Decreasing this value from 1.0 can increase performance at the * cost of accuracy * @throws IllegalArgumentException if arguments are <code>null</code>, or <code>numClusters</code> is - * less than 2, or samplingPercentage is {@link Double#NaN} or nonpositive or greater than 1.0 + * less than 2, or samplingRate is {@link Double#NaN} or nonpositive or greater than 1.0 */ public TreeClusteringRecommender(DataModel dataModel, ClusterSimilarity clusterSimilarity, int numClusters, - double samplingPercentage) { + double samplingRate) { super(dataModel); if (clusterSimilarity == null) { throw new IllegalArgumentException("clusterSimilarity is null"); @@ -108,14 +110,14 @@ if (numClusters < 2) { throw new IllegalArgumentException("numClusters must be at least 2"); } - if (Double.isNaN(samplingPercentage) || samplingPercentage <= 0.0 || samplingPercentage > 1.0) { - throw new IllegalArgumentException("samplingPercentage is invalid: " + samplingPercentage); + if (Double.isNaN(samplingRate) || samplingRate <= 0.0 || samplingRate > 1.0) { + throw new IllegalArgumentException("samplingRate is invalid: " + samplingRate); } this.clusterSimilarity = clusterSimilarity; this.numClusters = numClusters; this.clusteringThreshold = Double.NaN; this.clusteringByThreshold = false; - this.samplingPercentage = samplingPercentage; + this.samplingRate = samplingRate; this.buildClustersLock = new ReentrantLock(); this.refreshHelper = new RefreshHelper(new Callable<Object>() { @Override @@ -147,16 +149,16 @@ * @param clusterSimilarity {@link ClusterSimilarity} used to compute cluster similarity * @param clusteringThreshold clustering similarity threshold; clusters will be aggregated into larger * clusters until the next two nearest clusters' similarity drops below this threshold - * @param samplingPercentage percentage of all cluster-cluster pairs to consider when finding + * @param samplingRate percentage of all cluster-cluster pairs to consider when finding * next-most-similar clusters. Decreasing this value from 1.0 can increase performance at the * cost of accuracy * @throws IllegalArgumentException if arguments are <code>null</code>, or <code>clusteringThreshold</code> is - * {@link Double#NaN}, or samplingPercentage is {@link Double#NaN} or nonpositive or greater than 1.0 + * {@link Double#NaN}, or samplingRate is {@link Double#NaN} or nonpositive or greater than 1.0 */ public TreeClusteringRecommender(DataModel dataModel, ClusterSimilarity clusterSimilarity, double clusteringThreshold, - double samplingPercentage) { + double samplingRate) { super(dataModel); if (clusterSimilarity == null) { throw new IllegalArgumentException("clusterSimilarity is null"); @@ -164,14 +166,14 @@ if (Double.isNaN(clusteringThreshold)) { throw new IllegalArgumentException("clusteringThreshold must not be NaN"); } - if (Double.isNaN(samplingPercentage) || samplingPercentage <= 0.0 || samplingPercentage > 1.0) { - throw new IllegalArgumentException("samplingPercentage is invalid: " + samplingPercentage); + if (Double.isNaN(samplingRate) || samplingRate <= 0.0 || samplingRate > 1.0) { + throw new IllegalArgumentException("samplingRate is invalid: " + samplingRate); } this.clusterSimilarity = clusterSimilarity; this.numClusters = Integer.MIN_VALUE; this.clusteringThreshold = clusteringThreshold; this.clusteringByThreshold = true; - this.samplingPercentage = samplingPercentage; + this.samplingRate = samplingRate; this.buildClustersLock = new ReentrantLock(); this.refreshHelper = new RefreshHelper(new Callable<Object>() { @Override @@ -345,11 +347,10 @@ int size = clusters.size(); Pair<Collection<User>, Collection<User>> nearestPair = null; double bestSimilarity = Double.NEGATIVE_INFINITY; - Random r = RandomUtils.getRandom(); for (int i = 0; i < size; i++) { Collection<User> cluster1 = clusters.get(i); for (int j = i + 1; j < size; j++) { - if (samplingPercentage >= 1.0 || r.nextDouble() < samplingPercentage) { + if (samplingRate >= 1.0 || r.nextDouble() < samplingRate) { Collection<User> cluster2 = clusters.get(j); double similarity = clusterSimilarity.getSimilarity(cluster1, cluster2); if (!Double.isNaN(similarity) && similarity > bestSimilarity) { From mahout-commits-returnrcantu@example.net Fri May 01 17:48:30 2009 Return-Path: <mahout-commits-returnrcantu@example.net> Delivered-To: rcantu@example.net Received: (qmail 87709 invoked from network); 1 May 2009 17:48:30 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 May 2009 17:48:30 -0000 Received: (qmail 91826 invoked by uid 500); 1 May 2009 17:48:29 -0000 Delivered-To: rcantu@example.net Received: (qmail 91762 invoked by uid 500); 1 May 2009 17:48:29 -0000 Mailing-List: contact rcantu@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rcantu@example.net> List-Unsubscribe: <mailto:rcantu@example.net> List-Post: <mailto:rcantu@example.net> List-Id: <mahout-commits.lucene.apache.org> Reply-To: rcantu@example.net Delivered-To: mailing list rcantu@example.net Received: (qmail 91753 invoked by uid 99); 1 May 2009 17:48:29 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 May 2009 17:48:29 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 May 2009 17:48:21 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 142F1234C051 for <rcantu@example.net>; Fri, 1 May 2009 10:48:00 -0700 (PDT) Message-ID: <675353200.1241200080081.JavaMail.www-data@brutus> Date: Fri, 1 May 2009 10:48:00 -0700 (PDT) From: rcantu@example.net To: rcantu@example.net Subject: [CONF] Apache Lucene Mahout: Mahout on Elastic MapReduce (attachment added) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org New files attached to Mahout on Elastic MapReduce (Apache Lucene Mahout) by Stephen Green http://cwiki.apache.org/confluence/display/MAHOUT/Mahout+on+Elastic+MapReduce Attached file(s): --------------------------------------------------------------------- - mahout.png (image/png, 89 kb) Using the EMR GUI to start a k-means clustering job http://cwiki.apache.org/confluence/download/attachments/116559/mahout.png?version=1 --------------------------------------------------------------------- CONFLUENCE INFORMATION This message is automatically generated by Confluence Unsubscribe or edit your notifications preferences http://cwiki.apache.org/confluence/users/viewnotifications.action If you think it was sent incorrectly contact one of the administrators http://cwiki.apache.org/confluence/administrators.action If you want more information on Confluence, or have a bug to report see http://www.atlassian.com/software/confluence From mahout-commits-returnrcantu@example.net Fri May 01 18:39:33 2009 Return-Path: <mahout-commits-returnrcantu@example.net> Delivered-To: rcantu@example.net Received: (qmail 27154 invoked from network); 1 May 2009 18:39:33 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 May 2009 18:39:33 -0000 Received: (qmail 67902 invoked by uid 500); 1 May 2009 18:39:33 -0000 Delivered-To: rcantu@example.net Received: (qmail 67828 invoked by uid 500); 1 May 2009 18:39:33 -0000 Mailing-List: contact rcantu@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rcantu@example.net> List-Unsubscribe: <mailto:rcantu@example.net> List-Post: <mailto:rcantu@example.net> List-Id: <mahout-commits.lucene.apache.org> Reply-To: rcantu@example.net Delivered-To: mailing list rcantu@example.net Received: (qmail 67819 invoked by uid 99); 1 May 2009 18:39:33 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 May 2009 18:39:33 +0000 X-ASF-Spam-Status: No, hits=-1998.5 required=10.0 tests=ALL_TRUSTED,WEIRD_PORT X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 May 2009 18:39:22 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 391A3234C04B for <rcantu@example.net>; Fri, 1 May 2009 11:39:00 -0700 (PDT) Message-ID: <753386566.1241203140232.JavaMail.www-data@brutus> Date: Fri, 1 May 2009 11:39:00 -0700 (PDT) From: rcantu@example.net To: rcantu@example.net Subject: [CONF] Apache Lucene Mahout: Mahout on Elastic MapReduce (page edited) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Mahout on Elastic MapReduce (MAHOUT) edited by Stephen Green Page: http://cwiki.apache.org/confluence/display/MAHOUT/Mahout+on+Elastic+MapReduce Changes: http://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=116559&originalVersion=5&revisedVersion=6 Content: --------------------------------------------------------------------- This page details the set of steps that was necessary to get an example of k-Means clustering running on Amazon's Elastic MapReduce (EMR). The aim here was simply to get something running, but it should provide a good head start if you want to run something else. I started out on the [QuickStart] page and went from there. Along the way, I encountered some problems and posted to the Amazon EMR forums to get some help. The [resulting thread|http://developer.amazonwebservices.com/connect/thread.jspa?threadID=30945&tstart=15] might have some useful information if you're having trouble. h1. Getting Started * Get yourself an EMR account. If you're already using EC2, then you can do this from [Amazon's AWS Managment Console|https://console.aws.amazon.com/], which has a tab for running EMR. * Get the [ElasticFox|https://addons.mozilla.org/en-US/firefox/addon/11626] and [S3Fox|https://addons.mozilla.org/en-US/firefox/search?q=s3fox&cat=all] Firefox extensions. These will make it easy to monitor running EMR instances, upload code and data, and download results. * Download the Ruby command line client for EMR. You can do things from the GUI, but when you're in the midst of trying to get something running, the CLI client will make life a lot easier. * Have a look at [Common Problems Running Job Flows|http://developer.amazonwebservices.com/connect/thread.jspa?messageID=124694&#124694] and [Developing and Debugging Job Flows|http://developer.amazonwebservices.com/connect/message.jspa?messageID=124695#124695] in the EMR rcantu@example.net. They were tremendously useful. * Make sure that you're up to date with the Mahout source. The fix for [Issue 118|http://issues.apache.org/jira/browse/MAHOUT-118] is required to get things running when you're sending output to an S3 bucket. * Build the Mahout core and examples. Note that the Hadoop that's running on EMR is a modified version of Hadoop 0.18.3 that includes some back-ported stuff from later versions of Hadoop. The EMR GUI in the AWS Management Console provides a number of examples of using EMR, and you might want to try running one of these to get started. One big gotcha that I discovered is that the S3N file system for Hadoop has a couple of weird cases that boil down to the following advice: if you're naming a directory in an s3n URI, make sure that it ends in a slash and you should not try to use a top-level S3 bucket name as the place where your Mahout output will be going, you should always include a subdirectory. h1. Uploading Code and Data I decided that I would use separate S3 buckets for the Mahout code, the input for the clustering (I used the synthetic control data, you can find it easily from the [QuickStart] page), and the output of the clustering. I used S3Fox to make two buckets: {{mahout-code}} and {{mahout-input}} and then uploaded {{mahout-examples-0.1.job}} to the {{mahout-code}} bucket. I copied the {{synthetic_control.data}} file to the {{mahout-input}} bucket. You don't actually need to make an output bucket, as the Mahout examples will create one if the one you specify doesn't exist. h1. Running k-means Clustering EMR offers two modes for running MapReduce jobs. The first is a "streaming" mode where you provide the source for single-step mapper and reducer functions (you can use languages other than Java for this). The second mode is called "Custom Jar" and it gives you full control over the job steps that will run. This is the mode that we need to use to run Mahout. In order to run in Custom Jar mode, you need to look at the example that you want to run (in my case, {{org.apache.mahout.clustering.syntheticcontrol.kmeans.Job}}) and figure out the arguments that you need to provide to the job. Essentially, you need to know the command line that you would give to bin/hadoop in order to run the job, including whatever parameters the job needs to run. Certainly, you could provide a jar for EMR that has a {{Main-Class}} attribute and make sure that the default {{main}} for that class will do the right thing, but that won't work with the default Mahout examples, because they expect the input and output to be in HDFS and not S3. h2. Using the GUI The EMR GUI is an easy way to start up a Custom Jar run, but it doesn't have the full functionality of the CLI. Basically, you tell the GUI where in S3 the jar file is using a Hadoop s3n URI like {{s3n://mahout-code/mahout-examples-0.1.job}}. The GUI will check and make sure that the given file exists, which is a nice sanity check. You can then provide the arguments for the job just as you would on the command line. The arguments for the k-means job that I wanted to run were as follows: {noformat} org.apache.mahout.clustering.syntheticcontrol.kmeans.Job s3n://mahout-input/synthetic_control.data s3n://mahout-output/kmeans/ org.apache.mahout.utils.EuclideanDistanceMeasure 80 55 0.5 10 {noformat} You can see what the configuration screen looks like here: !mahout.png|thumbnail! The main failing with the GUI mode is that you can only specify a single job to run, and you can't run another job in the same set of instances. Recall that on AWS you pay for partial hours at the hourly rate, so if your job fails in the first 10 seconds, you pay for the full hour and if you try again, you're going to paying for another hour. Because of this, I strongly suggest using the CLI once you're minmally familiar with EMR. h2. Using the CLI If you're in development mode, and trying things out, EMR allows you to set up a set of instances and leave them running. Once you've done this, you can add job steps to the set of instances as you like. This solves the "10 second failure" problem that I described above and lets you get full value for your EMR dollar. Amazon has pretty good [documentation for the CLI|http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/index.html?CHAP_RunningaJob.html], which you'll need to read to figure out how to do things like set up your AWS credentials for the EMR CLI. You can start up a job flow that will keep running using an invocation like the following: {noformat} ./elastic-mapreduce --create --alive \ --log-uri s3n://mahout-logs/ --key_pair aura \ --num-instances 4 --name kmeans {noformat} Note that I've named the job flow {{kmeans}}, I've told it to use 4 small instances, and I'll have the logs stored in an S3 bucket at the end of the run. This call returns the name of the job flow, and you'll need that for subsequent calls to add steps to the job flow. Let's list our job flows: {noformat} [stgreen@dhcp-ubur02-74-153 14:16:15 emr]$ ./elastic-mapreduce --list j-3JB4UF7CQQ025 WAITING ec2-174-129-90-97.compute-1.amazonaws.com kmeans {noformat} Everything's started up, and it's waiting for us to add a step to the job. When I started the job flow, I specified a key pair that I created earlier so that I can log into the master while the job flow is running: {noformat} [stgreen@dhcp-ubur02-74-153 14:14:01 emr]$ ssh rcantu@example.net Linux domU-12-31-39-00-84-13 83.161.120.155-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 -------------------------------------------------------------------------------- Welcome to Amazon Elastic MapReduce running Hadoop 0.18.3 and Debian/Lenny. Hadoop is installed in /home/hadoop. Log files are in /mnt/var/log/hadoop. Check /mnt/var/log/hadoop/steps for diagnosing step failures. The Hadoop UI can be accessed via the command: lynx http://localhost:9100/ -------------------------------------------------------------------------------- Last login: Fri May 1 18:12:13 2009 from 83.161.120.155 hadoop@domU-12-31-39-00-84-13:~$ {noformat} Let's add a step to run the same job that I ran via the GUI: {noformat} ./elastic-mapreduce -j j-3JB4UF7CQQ025 \ --jar s3n://mahout-code/mahout-examples-0.1.job \ --main-class org.apache.mahout.clustering.syntheticcontrol.kmeans.Job \ --arg s3n://mahout-input/synthetic_control.data \ --arg s3n://mahout-output/kmeans/ \ --arg org.apache.mahout.utils.EuclideanDistanceMeasure \ --arg 80 --arg 55 --arg 0.5 --arg 10 {noformat} When you do this, the job flow goes into the {{RUNNING}} state for a while and then returns to {{WAITING}} once the step has finished. You can use the CLI or the GUI to monitor the step while it runs. Once you've finished with your job flow, you can shut it down the following way: {noformat} ./elastic-mapreduce -j j-3JB4UF7CQQ025 --terminate {noformat} and go look in your S3 buckets to find your output and logs (if things didn't go very well!) --------------------------------------------------------------------- CONFLUENCE INFORMATION This message is automatically generated by Confluence Unsubscribe or edit your notifications preferences http://cwiki.apache.org/confluence/users/viewnotifications.action If you think it was sent incorrectly contact one of the administrators http://cwiki.apache.org/confluence/administrators.action If you want more information on Confluence, or have a bug to report see http://www.atlassian.com/software/confluence From mahout-commits-returnrcantu@example.net Mon May 04 22:26:14 2009 Return-Path: <mahout-commits-returnrcantu@example.net> Delivered-To: rcantu@example.net Received: (qmail 5030 invoked from network); 4 May 2009 22:26:14 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 May 2009 22:26:14 -0000 Received: (qmail 81063 invoked by uid 500); 4 May 2009 22:26:14 -0000 Delivered-To: rcantu@example.net Received: (qmail 81003 invoked by uid 500); 4 May 2009 22:26:14 -0000 Mailing-List: contact rcantu@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rcantu@example.net> List-Unsubscribe: <mailto:rcantu@example.net> List-Post: <mailto:rcantu@example.net> List-Id: <mahout-commits.lucene.apache.org> Reply-To: rcantu@example.net Delivered-To: mailing list rcantu@example.net Received: (qmail 80994 invoked by uid 99); 4 May 2009 22:26:13 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 May 2009 22:26:13 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 May 2009 22:26:12 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 0E1D123888CA; Mon, 4 May 2009 22:25:51 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r771475 - /lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/similarity/BooleanTanimotoCoefficientSimilarity.java Date: Mon, 04 May 2009 22:25:50 -0000 To: rcantu@example.net From: rcantu@example.net X-Mailer: svnmailer-1.0.8 Message-Id: <rcantu@example.net> X-Virus-Checked: Checked by ClamAV on apache.org Author: otis Date: Mon May 4 22:25:50 2009 New Revision: 771475 URL: http://svn.apache.org/viewvc?rev=771475&view=rev Log: - Removed null checks which should be caught by the instanceof checks at the beginning of the method Modified: lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/similarity/BooleanTanimotoCoefficientSimilarity.java Modified: lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/similarity/BooleanTanimotoCoefficientSimilarity.java URL: http://svn.apache.org/viewvc/lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/similarity/BooleanTanimotoCoefficientSimilarity.java?rev=771475&r1=771474&r2=771475&view=diff ============================================================================== --- lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/similarity/BooleanTanimotoCoefficientSimilarity.java (original) +++ lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/similarity/BooleanTanimotoCoefficientSimilarity.java Mon May 4 22:25:50 2009 @@ -54,10 +54,6 @@ @Override public double userSimilarity(User user1, User user2) { - - if (user1 == null || user2 == null) { - throw new IllegalArgumentException("user1 or user2 is null"); - } if (!(user1 instanceof BooleanPrefUser && user2 instanceof BooleanPrefUser)) { throw new IllegalArgumentException(); } @@ -86,4 +82,4 @@ return "BooleanTanimotoCoefficientSimilarity[dataModel:" + dataModel + ']'; } -} \ No newline at end of file +} From mahout-commits-returnrcantu@example.net Tue May 05 11:48:34 2009 Return-Path: <mahout-commits-returnrcantu@example.net> Delivered-To: rcantu@example.net Received: (qmail 92610 invoked from network); 5 May 2009 11:48:33 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 5 May 2009 11:48:33 -0000 Received: (qmail 7420 invoked by uid 500); 5 May 2009 11:48:33 -0000 Delivered-To: rcantu@example.net Received: (qmail 7348 invoked by uid 500); 5 May 2009 11:48:33 -0000 Mailing-List: contact rcantu@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rcantu@example.net> List-Unsubscribe: <mailto:rcantu@example.net> List-Post: <mailto:rcantu@example.net> List-Id: <mahout-commits.lucene.apache.org> Reply-To: rcantu@example.net Delivered-To: mailing list rcantu@example.net Received: (qmail 7339 invoked by uid 99); 5 May 2009 11:48:33 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 May 2009 11:48:33 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 May 2009 11:48:31 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id CAC932388979; Tue, 5 May 2009 11:43:09 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r771673 - /lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/recommender/BooleanUserGenericUserBasedRecommender.java Date: Tue, 05 May 2009 11:43:08 -0000 To: rcantu@example.net From: rcantu@example.net X-Mailer: svnmailer-1.0.8 Message-Id: <rcantu@example.net> X-Virus-Checked: Checked by ClamAV on apache.org Author: srowen Date: Tue May 5 11:43:05 2009 New Revision: 771673 URL: http://svn.apache.org/viewvc?rev=771673&view=rev Log: (empty) Modified: lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/recommender/BooleanUserGenericUserBasedRecommender.java Modified: lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/recommender/BooleanUserGenericUserBasedRecommender.java URL: http://svn.apache.org/viewvc/lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/recommender/BooleanUserGenericUserBasedRecommender.java?rev=771673&r1=771672&r2=771673&view=diff ============================================================================== --- lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/recommender/BooleanUserGenericUserBasedRecommender.java (original) +++ lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/impl/recommender/BooleanUserGenericUserBasedRecommender.java Tue May 5 11:43:05 2009 @@ -176,7 +176,6 @@ if (theNeighborhood.isEmpty()) { return Double.NaN; } - double preference = 0.0; double totalSimilarity = 0.0; for (User user : theNeighborhood) { if (!user.equals(theUser)) { @@ -185,13 +184,12 @@ if (pref != null) { double theSimilarity = similarity.userSimilarity(theUser, user) + 1.0; if (!Double.isNaN(theSimilarity)) { - preference += theSimilarity * pref.getValue(); totalSimilarity += theSimilarity; } } } } - return totalSimilarity == 0.0 ? Double.NaN : preference / totalSimilarity; + return totalSimilarity == 0.0 ? Double.NaN : totalSimilarity; } private static Set<Object> getAllOtherItems(Iterable<User> theNeighborhood, User theUser) { From mahout-commits-returnrcantu@example.net Tue May 05 11:55:08 2009 Return-Path: <mahout-commits-returnrcantu@example.net> Delivered-To: rcantu@example.net Received: (qmail 94621 invoked from network); 5 May 2009 11:55:08 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 5 May 2009 11:55:08 -0000 Received: (qmail 16070 invoked by uid 500); 5 May 2009 11:55:08 -0000 Delivered-To: rcantu@example.net Received: (qmail 15996 invoked by uid 500); 5 May 2009 11:55:07 -0000 Mailing-List: contact rcantu@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rcantu@example.net> List-Unsubscribe: <mailto:rcantu@example.net> List-Post: <mailto:rcantu@example.net> List-Id: <mahout-commits.lucene.apache.org> Reply-To: rcantu@example.net Delivered-To: mailing list rcantu@example.net Received: (qmail 15987 invoked by uid 99); 5 May 2009 11:55:07 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 May 2009 11:55:07 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 May 2009 11:55:05 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 14879238893B; Tue, 5 May 2009 11:49:44 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r771675 - in /lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/hadoop: RecommenderJob.java RecommenderMapper.java Date: Tue, 05 May 2009 11:49:43 -0000 To: rcantu@example.net From: rcantu@example.net X-Mailer: svnmailer-1.0.8 Message-Id: <rcantu@example.net> X-Virus-Checked: Checked by ClamAV on apache.org Author: srowen Date: Tue May 5 11:49:41 2009 New Revision: 771675 URL: http://svn.apache.org/viewvc?rev=771675&view=rev Log: (empty) Modified: lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/hadoop/RecommenderJob.java lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/hadoop/RecommenderMapper.java Modified: lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/hadoop/RecommenderJob.java URL: http://svn.apache.org/viewvc/lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/hadoop/RecommenderJob.java?rev=771675&r1=771674&r2=771675&view=diff ============================================================================== --- lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/hadoop/RecommenderJob.java (original) +++ lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/hadoop/RecommenderJob.java Tue May 5 11:49:41 2009 @@ -37,14 +37,14 @@ * <p>Command line arguments are:</p> * <ol> * <li>Fully-qualified class name of {@link Recommender} to use to make recommendations. - * Note that it must have a no-arg constructor.</li> + * Note that it must have a constructor which takes a {@link org.apache.mahout.cf.taste.model.DataModel} + * argument.</li> * <li>Number of recommendations to compute per user</li> * <li>Location of a text file containing user IDs for which recommendations should be computed, * one per line</li> * <li>Location of a data model file containing preference data, suitable for use with * {@link org.apache.mahout.cf.taste.impl.model.file.FileDataModel}</li> * <li>Output path where reducer output should go</li> - * <li>Number of mapper tasks to use</li> * </ol> * * <p>Example:</p> Modified: lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/hadoop/RecommenderMapper.java URL: http://svn.apache.org/viewvc/lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/hadoop/RecommenderMapper.java?rev=771675&r1=771674&r2=771675&view=diff ============================================================================== --- lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/hadoop/RecommenderMapper.java (original) +++ lucene/mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/hadoop/RecommenderMapper.java Tue May 5 11:49:41 2009 @@ -45,14 +45,17 @@ * * <p>Note that there is no corresponding {@link org.apache.hadoop.mapred.Reducer}; this * implementation can only partially take advantage of the mapreduce paradigm and only - * really leverages it for easy parallelization.</p> + * really leverages it for easy parallelization. Therefore, use the + * {@link org.apache.hadoop.mapred.lib.IdentityReducer} when running this on Hadoop.</p> + * + * @see RecommenderJob */ public final class RecommenderMapper extends MapReduceBase implements Mapper<LongWritable, Text, Text, RecommendedItemsWritable> { static final String RECOMMENDER_CLASS_NAME = "recommenderClassName"; - static final String RECOMMENDATIONS_PER_USER = "recommendadtionsPerUser"; + static final String RECOMMENDATIONS_PER_USER = "recommendationsPerUser"; static final String DATA_MODEL_FILE = "dataModelFile"; private Recommender recommender; @@ -92,8 +95,10 @@ throw new RuntimeException(ioe); } try { - Class<? extends Recommender> recommenderClass = Class.forName(recommenderClassName).asSubclass(Recommender.class); - Constructor<? extends Recommender> constructor = recommenderClass.getConstructor(DataModel.class); + Class<? extends Recommender> recommenderClass = + Class.forName(recommenderClassName).asSubclass(Recommender.class); + Constructor<? extends Recommender> constructor = + recommenderClass.getConstructor(DataModel.class); recommender = constructor.newInstance(fileDataModel); } catch (NoSuchMethodException nsme) { throw new RuntimeException(nsme); From mahout-commits-returnrcantu@example.net Wed May 06 12:18:33 2009 Return-Path: <mahout-commits-returnrcantu@example.net> Delivered-To: rcantu@example.net Received: (qmail 68403 invoked from network); 6 May 2009 12:18:33 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 6 May 2009 12:18:33 -0000 Received: (qmail 71012 invoked by uid 500); 6 May 2009 12:18:33 -0000 Delivered-To: rcantu@example.net Received: (qmail 70960 invoked by uid 500); 6 May 2009 12:18:32 -0000 Mailing-List: contact rcantu@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rcantu@example.net> List-Unsubscribe: <mailto:rcantu@example.net> List-Post: <mailto:rcantu@example.net> List-Id: <mahout-commits.lucene.apache.org> Reply-To: rcantu@example.net Delivered-To: mailing list rcantu@example.net Received: (qmail 70946 invoked by uid 99); 6 May 2009 12:18:31 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 May 2009 12:18:31 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 May 2009 12:18:21 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id BBEA2234C041 for <rcantu@example.net>; Wed, 6 May 2009 05:18:00 -0700 (PDT) Message-ID: <1375886189.1241612280768.JavaMail.www-data@brutus> Date: Wed, 6 May 2009 05:18:00 -0700 (PDT) From: rcantu@example.net To: rcantu@example.net Subject: [CONF] Apache Lucene Mahout: MahoutIntegration (page created) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org MahoutIntegration (MAHOUT) created by shliuzw http://cwiki.apache.org/confluence/display/MAHOUT/MahoutIntegration Content: --------------------------------------------------------------------- --------------------------------------------------------------------- CONFLUENCE INFORMATION This message is automatically generated by Confluence Unsubscribe or edit your notifications preferences http://cwiki.apache.org/confluence/users/viewnotifications.action If you think it was sent incorrectly contact one of the administrators http://cwiki.apache.org/confluence/administrators.action If you want more information on Confluence, or have a bug to report see http://www.atlassian.com/software/confluence From mahout-commits-returnrcantu@example.net Thu May 07 18:42:36 2009 Return-Path: <mahout-commits-returnrcantu@example.net> Delivered-To: rcantu@example.net Received: (qmail 2927 invoked from network); 7 May 2009 18:42:36 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 7 May 2009 18:42:36 -0000 Received: (qmail 67831 invoked by uid 500); 7 May 2009 18:42:36 -0000 Delivered-To: rcantu@example.net Received: (qmail 67771 invoked by uid 500); 7 May 2009 18:42:35 -0000 Mailing-List: contact rcantu@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rcantu@example.net> List-Unsubscribe: <mailto:rcantu@example.net> List-Post: <mailto:rcantu@example.net> List-Id: <mahout-commits.lucene.apache.org> Reply-To: rcantu@example.net Delivered-To: mailing list rcantu@example.net Received: (qmail 67748 invoked by uid 99); 7 May 2009 18:42:34 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 May 2009 18:42:34 +0000 X-ASF-Spam-Status: No, hits=-1998.5 required=10.0 tests=ALL_TRUSTED,WEIRD_PORT X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 May 2009 18:42:21 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 5CA42234C044 for <rcantu@example.net>; Thu, 7 May 2009 11:42:00 -0700 (PDT) Message-ID: <1840930411.1241721720378.JavaMail.www-data@brutus> Date: Thu, 7 May 2009 11:42:00 -0700 (PDT) From: rcantu@example.net To: rcantu@example.net Subject: [CONF] Apache Lucene Mahout: Mahout on Elastic MapReduce (page edited) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Mahout on Elastic MapReduce (MAHOUT) edited by Stephen Green Page: http://cwiki.apache.org/confluence/display/MAHOUT/Mahout+on+Elastic+MapReduce Changes: http://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=116559&originalVersion=6&revisedVersion=7 Content: --------------------------------------------------------------------- This page details the set of steps that was necessary to get an example of k-Means clustering running on Amazon's Elastic MapReduce (EMR). The aim here was simply to get something running, but it should provide a good head start if you want to run something else. I started out on the [QuickStart] page and went from there. Along the way, I encountered some problems and posted to the Amazon EMR forums to get some help. The [resulting thread|http://developer.amazonwebservices.com/connect/thread.jspa?threadID=30945&tstart=15] might have some useful information if you're having trouble. h1. Getting Started * Get yourself an EMR account. If you're already using EC2, then you can do this from [Amazon's AWS Managment Console|https://console.aws.amazon.com/], which has a tab for running EMR. * Get the [ElasticFox|https://addons.mozilla.org/en-US/firefox/addon/11626] and [S3Fox|https://addons.mozilla.org/en-US/firefox/search?q=s3fox&cat=all] Firefox extensions. These will make it easy to monitor running EMR instances, upload code and data, and download results. * Download the [Ruby command line client for EMR|http://developer.amazonwebservices.com/connect/entry.jspa?externalID=2264&categoryID=262]. You can do things from the GUI, but when you're in the midst of trying to get something running, the CLI client will make life a lot easier. * Have a look at [Common Problems Running Job Flows|http://developer.amazonwebservices.com/connect/thread.jspa?messageID=124694&#124694] and [Developing and Debugging Job Flows|http://developer.amazonwebservices.com/connect/message.jspa?messageID=124695#124695] in the EMR rcantu@example.net. They were tremendously useful. * Make sure that you're up to date with the Mahout source. The fix for [Issue 118|http://issues.apache.org/jira/browse/MAHOUT-118] is required to get things running when you're sending output to an S3 bucket. * Build the Mahout core and examples. Note that the Hadoop that's running on EMR is a modified version of Hadoop 0.18.3 that includes some back-ported stuff from later versions of Hadoop. The EMR GUI in the AWS Management Console provides a number of examples of using EMR, and you might want to try running one of these to get started. One big gotcha that I discovered is that the S3N file system for Hadoop has a couple of weird cases that boil down to the following advice: if you're naming a directory in an s3n URI, make sure that it ends in a slash and you should not try to use a top-level S3 bucket name as the place where your Mahout output will be going, you should always include a subdirectory. h1. Uploading Code and Data I decided that I would use separate S3 buckets for the Mahout code, the input for the clustering (I used the synthetic control data, you can find it easily from the [QuickStart] page), and the output of the clustering. I used S3Fox to make two buckets: {{mahout-code}} and {{mahout-input}} and then uploaded {{mahout-examples-0.1.job}} to the {{mahout-code}} bucket. I copied the {{synthetic_control.data}} file to the {{mahout-input}} bucket. You don't actually need to make an output bucket, as the Mahout examples will create one if the one you specify doesn't exist. h1. Running k-means Clustering EMR offers two modes for running MapReduce jobs. The first is a "streaming" mode where you provide the source for single-step mapper and reducer functions (you can use languages other than Java for this). The second mode is called "Custom Jar" and it gives you full control over the job steps that will run. This is the mode that we need to use to run Mahout. In order to run in Custom Jar mode, you need to look at the example that you want to run (in my case, {{org.apache.mahout.clustering.syntheticcontrol.kmeans.Job}}) and figure out the arguments that you need to provide to the job. Essentially, you need to know the command line that you would give to bin/hadoop in order to run the job, including whatever parameters the job needs to run. Certainly, you could provide a jar for EMR that has a {{Main-Class}} attribute and make sure that the default {{main}} for that class will do the right thing, but that won't work with the default Mahout examples, because they expect the input and output to be in HDFS and not S3. h2. Using the GUI The EMR GUI is an easy way to start up a Custom Jar run, but it doesn't have the full functionality of the CLI. Basically, you tell the GUI where in S3 the jar file is using a Hadoop s3n URI like {{s3n://mahout-code/mahout-examples-0.1.job}}. The GUI will check and make sure that the given file exists, which is a nice sanity check. You can then provide the arguments for the job just as you would on the command line. The arguments for the k-means job that I wanted to run were as follows: {noformat} org.apache.mahout.clustering.syntheticcontrol.kmeans.Job s3n://mahout-input/synthetic_control.data s3n://mahout-output/kmeans/ org.apache.mahout.utils.EuclideanDistanceMeasure 80 55 0.5 10 {noformat} You can see what the configuration screen looks like here: !mahout.png|thumbnail! The main failing with the GUI mode is that you can only specify a single job to run, and you can't run another job in the same set of instances. Recall that on AWS you pay for partial hours at the hourly rate, so if your job fails in the first 10 seconds, you pay for the full hour and if you try again, you're going to paying for another hour. Because of this, I strongly suggest using the CLI once you're minmally familiar with EMR. h2. Using the CLI If you're in development mode, and trying things out, EMR allows you to set up a set of instances and leave them running. Once you've done this, you can add job steps to the set of instances as you like. This solves the "10 second failure" problem that I described above and lets you get full value for your EMR dollar. Amazon has pretty good [documentation for the CLI|http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/index.html?CHAP_RunningaJob.html], which you'll need to read to figure out how to do things like set up your AWS credentials for the EMR CLI. You can start up a job flow that will keep running using an invocation like
{'source': 'Apache Software Foundation Public Mail Archives'}