From mahout-usershane48@example.com Wed Oct 01 11:19:38 2008 Return-Path: <mahout-usershane48@example.com> Delivered-To: shane48@example.com Received: (qmail 92194 invoked from network); 1 Oct 2008 11:19:37 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Oct 2008 11:19:37 -0000 Received: (qmail 62647 invoked by uid 500); 1 Oct 2008 11:19:35 -0000 Delivered-To: shane48@example.com Received: (qmail 62613 invoked by uid 500); 1 Oct 2008 11:19:35 -0000 Mailing-List: contact shane48@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:shane48@example.com> List-Unsubscribe: <mailto:shane48@example.com> List-Post: <mailto:shane48@example.com> List-Id: <mahout-user.lucene.apache.org> Reply-To: shane48@example.com Delivered-To: mailing list shane48@example.com Received: (qmail 62602 invoked by uid 99); 1 Oct 2008 11:19:34 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Oct 2008 04:19:34 -0700 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of shane48@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-gx0-f13.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Oct 2008 11:18:34 +0000 Received: by gxk6 with SMTP id 6so5840gxk.5 for <shane48@example.com>; Wed, 01 Oct 2008 04:18:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=icTxP2j1MHCVehWa/OGrOlzQ1l69qjPwBc7p6SMv0Js=; b=u0qIKoOsIKfaEaPPYvD8jjJY7tEJwMJNKE/wkMt/coBlYBytPAKdFcAOMIvp8akcV3 MGEN6JqA0vS6QYYm83UlFqP4/sOcQYR93xAauxbh13YGpd9KnqkyPOHz1zLDzvBlAK5/ 1PpaJKnsvijj7Qs0rxCK6+ZpEQqUFgodwwuQ8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=M3CBqaBYtG2gWJF0xSrbGlrWCYiJnHyQYfhng7DkXm4Wesa0QYTPqTP1qSgeTEd2bM vmkVF7bCivjrHhdtSGSYjfrHRxrSRZBP+oHYQffzR3pncvUD0KSIyF7/XhfgfpXfhoqA Ssg+hzQ9LaMdrrEEw7r8B8Ilp9OlqCb/Sahpg= Received: by 83.161.120.155 with SMTP id g18mr11871040ybd.137.1222859888730; Wed, 01 Oct 2008 04:18:08 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Wed, 1 Oct 2008 04:18:08 -0700 (PDT) Message-ID: <shane48@example.com> Date: Wed, 1 Oct 2008 12:18:08 +0100 From: "Sean Owen" <shane48@example.com> To: shane48@example.com Subject: Re: Recommending when working with binary data sets In-Reply-To: <shane48@example.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <shane48@example.com> X-Virus-Checked: Checked by ClamAV on apache.org On Tue, Sep 30, 2008 at 7:55 PM, Otis Gospodnetic <shane48@example.com> wrote: > I think their algo then translates to: > > for item i1 in {1, 2, 3 ...100} { > for item i2 in {1, 10, 20} { > computeSimilarity(i1, i2) > } > } Yes though you would not consider recommending items 1, 10, or 20 since the user already has expressed a preference for them. > sim (1, 1) = 1.0 (... and you therefore wouldn't have to compute this) > sim (10, 1) = same as sim (1,10) (... sim(x,x) isn't recorded in the similarity matrix, and sim(x,y) is only recorded for x < y to avoid this redundancy, because we assume it's symmetric. You know all this, just commenting.) > So, then we have a list of most similarity items for each item: > item 1: 20, 10 > item 10: 20, 1 > item 20: 10, 1 Yes those are most similar but of course they are, since those are the very items the user has already expressed a preference for! > Is this interpretation correct? If it is, I don't quite yet see the benefit of computing similarity between items for a single person. Of course they will be similar, unless we want to create groups of items for a single person with a diverse set of interests for some reason. Not quite, what you are doing is computing an estimated preference for everything the user hasn't already computed a preference for. And then of course you recommend those with the highest estimated preference. You do this by computing a weighted average. For every other item in the universe (2, 3, 4, ...) compute its similarity to all the user's preferred items (1, 10, 20), and sum those similarities times the user's preferences for those items. User-based recommenders work the same way but kinda turned on its side -- you compute similiar *users* then... > Wouldn't one need to build item-item similarity matrix for all Joe's items (1, 10, 20) and all new items (101-200)? Yes > If so, what's the point of figuring item-item similarity for all items Joe already consumed? There isn't. :) Well... Let me digress a bit to perhaps explain why item-based recommenders have some interesting power. Often, your items are such that you have some a priori, external idea of how similar they are -- beyond just a notion based on user preferences. For example you might say that two mystery books are similar and a mystery and a sci-fi book are not. This could form the basis of an item-item similarity metric. This is good because: 1) it's more, external info being injected into the system, which could improve recommendations, and 2) often it is quite cheap to compute this notion, to define the complete item-item similarity matrix cheaply. It's not cheap to compute the whole thing based on, say, Pearson correlations between items based on preferences! And interestingly this reasoning doesn't apply so much to user-based recommenders. You usually don't have an a priori notion of user-user similarity, and, these notions are likely to change as you learn more about *users* whereas we expect *item* similarity to be relatively stable the more we learn. That's why item-based recommenders are not just the reverse of user-based recommenders, but a bit interesting in their own right. It's not 100% symmetric. Not sure if that was useful.... From mahout-usershane48@example.com Wed Oct 01 14:51:10 2008 Return-Path: <mahout-usershane48@example.com> Delivered-To: shane48@example.com Received: (qmail 50371 invoked from network); 1 Oct 2008 14:51:10 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Oct 2008 14:51:10 -0000 Received: (qmail 96788 invoked by uid 500); 1 Oct 2008 14:50:52 -0000 Delivered-To: shane48@example.com Received: (qmail 96764 invoked by uid 500); 1 Oct 2008 14:50:52 -0000 Mailing-List: contact shane48@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:shane48@example.com> List-Unsubscribe: <mailto:shane48@example.com> List-Post: <mailto:shane48@example.com> List-Id: <mahout-user.lucene.apache.org> Reply-To: shane48@example.com Delivered-To: mailing list shane48@example.com Received: (qmail 96746 invoked by uid 99); 1 Oct 2008 14:50:52 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Oct 2008 07:50:52 -0700 X-ASF-Spam-Status: No, hits=2.0 required=10.0 tests=RCVD_IN_BL_SPAMCOP_NET,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO web50304.mail.re2.yahoo.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with SMTP; Wed, 01 Oct 2008 14:49:48 +0000 Received: (qmail 6417 invoked by uid 60001); 1 Oct 2008 14:49:21 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type:Message-ID; b=6gTk0wgjIlQQtC0REF7ElVy/5Zy+4axL7fOoQtr9+HpOop3ESlENSV9dhFWQAFzAX0GXQSqkEXCzgDwXFFQtYB+c2uxIgD8yjfv8RhUbZ4Y437CvJ9Ss6ruYOI+fUERE19H0+Q6jC2uM52sUh+sFSrJiWtsc3UUPYC32Qavo4Es=; Received: from [83.161.120.155] by web50304.mail.re2.yahoo.com via HTTP; Wed, 01 Oct 2008 07:49:21 PDT X-Mailer: YahooMailRC/1096.40 YahooMailWebService/83.161.120.155 Date: Wed, 1 Oct 2008 07:49:21 -0700 (PDT) From: Otis Gospodnetic <shane48@example.com> Subject: Re: Recommending when working with binary data sets To: shane48@example.com MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Message-ID: <{{EMAIL+PHONE}}> X-Virus-Checked: Checked by ClamAV on apache.org Hi Sean, ----- Original Message ---- > From: Sean Owen <shane48@example.com> > To: shane48@example.com > Sent: Wednesday, October 1, 2008 7:18:08 AM > Subject: Re: Recommending when working with binary data sets > > On Tue, Sep 30, 2008 at 7:55 PM, Otis Gospodnetic > wrote: > > I think their algo then translates to: > > > > for item i1 in {1, 2, 3 ...100} { > > for item i2 in {1, 10, 20} { > > computeSimilarity(i1, i2) > > } > > } > > Yes though you would not consider recommending items 1, 10, or 20 > since the user already has expressed a preference for them. Right. > > sim (1, 1) = 1.0 > > (... and you therefore wouldn't have to compute this) > > > sim (10, 1) = same as sim (1,10) > > (... sim(x,x) isn't recorded in the similarity matrix, and sim(x,y) is > only recorded for x < y to avoid this redundancy, because we assume > it's symmetric. You know all this, just commenting.) Right. > > So, then we have a list of most similarity items for each item: > > item 1: 20, 10 > > item 10: 20, 1 > > item 20: 10, 1 > > Yes those are most similar but of course they are, since those are the > very items the user has already expressed a preference for! Right, right, but isn't that what http://dsonline.computer.org/portal/site/dsonline/menuitem.9ed3d9924aeb0dcd82ccc6716bbe36ec/index.jsp?&pName=dso_level1&path=dsonline/2003_Archives/0301/d&file=wp1lind.xml&xsl=article.xsl describes? If I understand the article correctly, they go through several possible approaches to building REs - user-item CF -- too expensive, user clustering --- low quality, search-based -- low quality and then present their novel item-to-item CF as something that doesn't have performance/scale/quality problems. I think they said it works as my example above, but it doesn't make sense to me - by doing the above all I'm doing is looking at similarity between items that Joe already consumed, and I'm really after *new* things to recommend. > > Is this interpretation correct? If it is, I don't quite yet see the benefit > of computing similarity between items for a single person. Of course they will > be similar, unless we want to create groups of items for a single person with a > diverse set of interests for some reason. > > Not quite, what you are doing is computing an estimated preference for > everything the user hasn't already computed a preference for. And then > of course you recommend those with the highest estimated preference. Right, that's page 23 in PCI. I thought Amazonians were talking about something different. It'ss tarting to sound like they are talking about the same thing. In my case I'm interested in working with binary preferences, though, so if I understand things correctly weighted average doesn't really make sense in that context. Is this correct? > You do this by computing a weighted average. For every other item in > the universe (2, 3, 4, ...) compute its similarity to all the user's > preferred items (1, 10, 20), and sum those similarities times the > user's preferences for those items. This sounds exactly like what's described on pages 22-25 in Programming Collective Intelligence. My understanding was that what's in PCI is "an old known approach" whereas what is in that article about Amazon is a "new, different, and patented approach". Are they both actually talking about the same algo? i.e. Amazon article is the same as what's on p22-25 in PCI? > User-based recommenders work the same way but kinda turned on its side > -- you compute similiar *users* then... > > > > Wouldn't one need to build item-item similarity matrix for all Joe's items (1, > 10, 20) and all new items (101-200)? > > Yes > > > If so, what's the point of figuring item-item similarity for all items Joe > already consumed? > > There isn't. :) OK, so the summary is that you still have to compute item1-item2 similarity matrix where item1 comes from the set of items consumed by Joe and item2 comes from the set of items I want to consider for recommendation (i.e. it can be only a subset of all items, e.g. only new items since the last time Joe was recommended some items 3 days ago). If my assumption that computing weighted average doesn't make sense in the context of binary preferences, then the above is really it - it all boils down to computing item1-item2 matrix based on item-item similarity. While doing that we try to make that item1-item2 as small as possible and try to compute it ahead of time, so that at the "give Joe N recommendations" time we can simply take topN from that1-item2 matrix. Is this correct or am I missing some pieces of the puzzle? > Well... > > Let me digress a bit to perhaps explain why item-based recommenders > have some interesting power. > > Often, your items are such that you have some a priori, external idea > of how similar they are -- beyond just a notion based on user > preferences. For example you might say that two mystery books are > similar and a mystery and a sci-fi book are not. This could form the > basis of an item-item similarity metric. This is good because: > > 1) it's more, external info being injected into the system, which > could improve recommendations, and > 2) often it is quite cheap to compute this notion, to define the > complete item-item similarity matrix cheaply. It's not cheap to > compute the whole thing based on, say, Pearson correlations between > items based on preferences! > > And interestingly this reasoning doesn't apply so much to user-based > recommenders. You usually don't have an a priori notion of user-user > similarity, and, these notions are likely to change as you learn more > about *users* whereas we expect *item* similarity to be relatively > stable the more we learn. Si si :) > That's why item-based recommenders are not just the reverse of > user-based recommenders, but a bit interesting in their own right. > It's not 100% symmetric. Otis From mahout-usershane48@example.com Wed Oct 01 16:36:08 2008 Return-Path: <mahout-usershane48@example.com> Delivered-To: shane48@example.com Received: (qmail 71767 invoked from network); 1 Oct 2008 16:36:08 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Oct 2008 16:36:08 -0000 Received: (qmail 92845 invoked by uid 500); 1 Oct 2008 16:36:06 -0000 Delivered-To: shane48@example.com Received: (qmail 92821 invoked by uid 500); 1 Oct 2008 16:36:05 -0000 Mailing-List: contact shane48@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:shane48@example.com> List-Unsubscribe: <mailto:shane48@example.com> List-Post: <mailto:shane48@example.com> List-Id: <mahout-user.lucene.apache.org> Reply-To: shane48@example.com Delivered-To: mailing list shane48@example.com Received: (qmail 92810 invoked by uid 99); 1 Oct 2008 16:36:05 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Oct 2008 09:36:05 -0700 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of shane48@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO wr-out-0506.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Oct 2008 16:35:05 +0000 Received: by wr-out-0506.google.com with SMTP id c30so256774wra.21 for <shane48@example.com>; Wed, 01 Oct 2008 09:35:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=+42R4SA7ZNLI7So0EB4boDdLRt9XDY0bHSw08yzzUnY=; b=AgOn/GFy0Fnk2AMO1LtdGehfuX+fB1ZuDIyk6OQ6uaArD802ldIwy4jvkAl9Rqs5ok 6vxPVQtAkqX7E1wlOzP5jKQZf12J7/YyhzQMVWV9mfBGK0cMJRZF8vbEtAbSAeg9YKDS skkUuQWDVk2qAUC0ezhg80LankBV0AlxuuQZc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=eKR978lS+8UFiP5ewDZUdmYJccbs1TggQ56kP0jYGcVIcNjTLjEZzY2AFBkcvHsl8d pYs1g/dygb0Pj+k3cDunLmGWzAaHc8eLbOFAhdh6cAdbhVPTYuj5G3keqUPz1oVOSkbN wSlatRI9j4RcBEda4ITEawJPSsnK2lJjgvQw8= Received: by 83.161.120.155 with SMTP id o1mr12342805ybm.87.1222878921934; Wed, 01 Oct 2008 09:35:21 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Wed, 1 Oct 2008 09:35:21 -0700 (PDT) Message-ID: <shane48@example.com> Date: Wed, 1 Oct 2008 17:35:21 +0100 From: "Sean Owen" <shane48@example.com> To: shane48@example.com Subject: Re: Recommending when working with binary data sets In-Reply-To: <{{EMAIL+PHONE}}> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <{{EMAIL+PHONE}}> X-Virus-Checked: Checked by ClamAV on apache.org On Wed, Oct 1, 2008 at 3:49 PM, Otis Gospodnetic <shane48@example.com> wrote: > If I understand the article correctly, they go through several possible approaches to building REs - user-item CF -- too expensive, user clustering --- low quality, search-based -- low quality and then present their novel item-to-item CF as something that doesn't have performance/scale/quality problems. I think they said it works as my example above, but it doesn't make sense to me - by doing the above all I'm doing is looking at similarity between items that Joe already consumed, and I'm really after *new* things to recommend. I have not seen this before. I think I just read somewhere that the item-based system I outlined (as does Collective Intelligence) is "what Amazon does" and that could be wrong. They first say, basically, compute only as much of the item-item similarity matrix as makes sense -- if nobody ever bought X and Y together, don't bother computing it. OK. As far as my stuff goes -- similarities are not all precomputed in the interest of time but are cached as much as memory permits. Taking it to an extreme and having a system that can cache it all -- yeah, that's going to help speed! It says... "Given a similar-items table, the algorithm finds items similar to each of the user's purchases and ratings, aggregates those items, and then recommends the most popular or correlated items." It's not clear what exactly this means, whether the top recommendations are merely items that are most similar to *something* you bought/rated, or most similar *on average to everything you bought/rated*, or is a weighted average like we were discussing. I kinda think it has to be the latter for two reasons: 1) If I hated something, I don't want to recommend things like it! so rating seems to matter 2) Taking into account rating does not appreciably slow anything down >> > Is this interpretation correct? If it is, I don't quite yet see the benefit >> of computing similarity between items for a single person. Of course they will >> be similar, unless we want to create groups of items for a single person with a >> diverse set of interests for some reason. Oh, no I think they are saying they compute similarity between all pairs of items in the catalog -- well, all pairs that actually co-occur in some user's purchase/rating history. > In my case I'm interested in working with binary preferences, though, so if I understand things correctly weighted average doesn't really make sense in that context. Is this correct? Yeah, er, it still works to an extent but not really meaningfully. It seems that other approaches are more appropriate and faster in this special case. This, I haven't dug into much. CI covers it reasonably well. You or I could sit down and adapt the common approaches to this special case in any number of clever ways, I'm sure. > This sounds exactly like what's described on pages 22-25 in Programming Collective Intelligence. My understanding was that what's in PCI is "an old known approach" whereas what is in that article about Amazon is a "new, different, and patented approach". Are they both actually talking about the same algo? i.e. Amazon article is the same as what's on p22-25 in PCI? This I don't know. This was my impression, that they're basically doing plain ol' item-based recommendation -- with a heap of tweaks and heuristics and so on internally, no doubt, that they aren't going to publish. For example, one area where one can add a lot of proprietary value is translating user actions (purchases, ratings, views) into the "right" inferred preference value. > OK, so the summary is that you still have to compute item1-item2 similarity matrix where item1 comes from the set of items consumed by Joe and item2 comes from the set of items I want to consider for recommendation (i.e. it can be only a subset of all items, e.g. only new items since the last time Joe was recommended some items 3 days ago). > > If my assumption that computing weighted average doesn't make sense in the context of binary preferences, then the above is really it - it all boils down to computing item1-item2 matrix based on item-item similarity. > While doing that we try to make that item1-item2 as small as possible and try to compute it ahead of time, so that at the "give Joe N recommendations" time we can simply take topN from that1-item2 matrix. > Is this correct or am I missing some pieces of the puzzle? Yeah that is my understanding. They build as much of the complete item-item matrix as makes sense ahead of time. The only issue with that is, well, you can't just go update it every time a pref changes. So either you don't immediately respond to new ratings, or you have to do so in some clever other way. For my stuff, everything has an nearly immediate effect -- updates take effect immediately but due to caching may not percolate up the recommender system and have an effect for a bit. This is a sort of middle ground between recomputing everything all the time, and never. (And it is still a problematic way to approach this.) From mahout-usershane48@example.com Thu Oct 02 13:26:22 2008 Return-Path: <mahout-usershane48@example.com> Delivered-To: shane48@example.com Received: (qmail 67538 invoked from network); 2 Oct 2008 13:26:22 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Oct 2008 13:26:22 -0000 Received: (qmail 75668 invoked by uid 500); 2 Oct 2008 13:26:20 -0000 Delivered-To: shane48@example.com Received: (qmail 75641 invoked by uid 500); 2 Oct 2008 13:26:19 -0000 Mailing-List: contact shane48@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:shane48@example.com> List-Unsubscribe: <mailto:shane48@example.com> List-Post: <mailto:shane48@example.com> List-Id: <mahout-user.lucene.apache.org> Reply-To: shane48@example.com Delivered-To: mailing list shane48@example.com Received: (qmail 75619 invoked by uid 99); 2 Oct 2008 13:26:19 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Oct 2008 06:26:19 -0700 X-ASF-Spam-Status: No, hits=1.2 required=10.0 tests=SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO spunkymail-a10.g.dreamhost.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Oct 2008 13:25:15 +0000 Received: from [83.161.120.155] (adsl-074-229-189-244.sip.rmo.bellsouth.net [83.161.120.155]) by spunkymail-a10.g.dreamhost.com (Postfix) with ESMTP id 2ED701615F9; Thu, 2 Oct 2008 06:25:45 -0700 (PDT) Message-Id: <shane48@example.com> From: Grant Ingersoll <shane48@example.com> To: shane48@example.com, shane48@example.com Content-Type: text/plain; charset=WINDOWS-1252; format=flowed; delsp=yes Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Apple Message framework v929.2) Subject: Fwd: CFP open for ApacheCon Europe 2009 Date: Thu, 2 Oct 2008 09:25:09 -0400 References: <shane48@example.com> X-Mailer: Apple Mail (2.929.2) X-Virus-Checked: Checked by ClamAV on apache.org FYI. Begin forwarded message: > From: Noirin Shirley <shane48@example.com> > Date: October 2, 2008 4:20:28 AM EDT > To: shane48@example.com, shane48@example.com, shane48@example.com > Subject: CFP open for ApacheCon Europe 2009 > Reply-To: shane48@example.com > > If you only have thirty seconds: > > The Call for Papers for ApacheCon Europe 2009, to be held in =20 > Amsterdam, from 23rd to 27th March, is now open! Submit your =20 > proposals at http://eu.apachecon.com/c/aceu2009/cfp/ before 24th =20 > October. > > Remember that early bird prices for ApacheCon US 2008, to be held in =20= > New Orleans, from 3rd to 7th November, will go up this Friday, at =20 > midnight Eastern time! > > Sponsorship opportunities for ApacheCon US 2008 and ApacheCon EU =20 > 2009 are still available. If you or your company are interested in =20 > becoming a sponsor, please contact Delia Frees at =20 > shane48@example.com for details. > > ******* > > If you want all the details: > > ApacheCon Europe 2009 - Leading the Wave of Open Source > Amsterdam, The Netherlands > 23rd to 27th March, 2009 > > Call for Papers Opens for ApacheCon Europe 2009 > > The Apache Software Foundation (ASF) invites submissions to its =20 > official conference, ApacheCon Europe 2009. To be held 23rd to 27th =20= > March, 2009 at the M=F6venpick Hotel Amsterdam City Centre, ApacheCon =20= > serves as a forum for showcasing the ASF's latest developments, =20 > including its projects, membership, and community. ApacheCon offers =20= > unparalleled educational opportunities, with dedicated =20 > presentations, hands-on trainings, and sessions that address core =20 > technology, development, business/marketing, and licensing issues in =20= > Open Source. > > ApacheCon's wide range of activities are designed to promote the =20 > exchange of ideas amongst ASF Members, innovators, developers, =20 > vendors, and users interested in the future of Open Source =20 > technology. The conference program includes competitively selected =20 > presentations, trainings/workshops, and a small number of invited =20 > speakers. All sessions undergo a peer review process by the =20 > ApacheCon Conference Planning team. The following information =20 > provides presentation category descriptions, and information about =20 > how to submit your > proposal. > > Conference Themes and Topics > > APACHECON 2009 - LEADING THE WAVE OF OPEN SOURCE > > Building on the success of the last two years, we are excited to =20 > return to Amsterdam in 2009. We'll be continuing to offer our very =20 > popular two-day trainings, including certifications of completion =20 > for those who fulfill all the requirements of these trainings. > > The ASF comprises some of the most active and recognized developers =20= > in the Open Source community. By bringing together the pioneers, =20 > developers, and users of flagship Open Source technologies, =20 > ApacheCon provides an influential platform for dialogue, between the =20= > speaker and the audience, between project contributors and the =20 > community at large, traversing a wide range of ideas, expertise, and =20= > personalities. > > ApacheCon welcomes submissions from like-minded delegates across =20 > many fields, geographic locations, and areas of development. The =20 > breadth and loosely-structured nature of the Apache community lends =20= > itself to conference content that is also somewhat loosely-=20 > structured. Common themes of interest address groundbreaking =20 > technologies and emerging trends, successful practices (from =20 > development to deployment), and lessons learned (tips, tools, and =20 > tricks). In addition to technical content, ApacheCon invites =20 > Business Track submissions that address Open Source business, =20 > marketing, and legal/licensing issues. > > Topics appropriate for submission to this conference are manifold, =20 > and may include but are not restricted to: > > - Apache HTTP server topics such as installation, configuration, and =20= > migration > - ASF-wide projects such as Lucene, SpamAssassin, Jackrabbit, and =20 > Maven > - Scripting languages and dynamic content such as Java, Perl, =20 > Python, Ruby, XSL, and PHP > - Security and e-commerce > - Performance tuning, load balancing and high availability > - New technologies and broader initiatives such as Web Services and =20= > Web 2.0 > - ASF-Incubated projects such as Sling, UIMA, and Shindig > > > Submission Guidelines > Submissions must include > - Title > - Speaker name, with affiliation and email address > - Speaker bio (100 words or less) > - Short description (50 words or less) > - Full description including abstract and objectives (200 words or > less) > - Expertise level (beginner to advanced) > - Format and duration (trainings vs. general presentation; half-, =20 > full- or two-day workshop, etc.) > - Intended audience and maximum number of participants (trainings =20 > only) > - Background knowledge expected of the participants (trainings only) > > > Types of Presentations > > - Trainings/Workshops > - General Sessions > - Case Studies/Industry Profiles > - Invited Keynotes/Panels/Speakers > - Corporate Showcases & Demonstrations > > BoF sessions and Fast Feather Track talks will be selected separately > > Pre Conference Trainings/Workshops > Held on the first and second day of the conference =96 2008-03-23 and =20= > 2008-03-24, Trainings require a registration fee beyond the regular =20= > conference fee. Proposals may be submitted for half-day (3 hours), =20 > full-day (6 hours), or two-day (12 hours) training sessions, aimed =20 > at providing in-depth, hands-on development experience or related =20 > continuing education. Training submissions are welcome at beginner, =20= > intermediate, and expert levels. > > General Sessions > General Sessions include presentations on practical development =20 > applications, insight into high-interest projects, best practices =20 > and key advances, overcoming implementation challenges, and industry =20= > innovations. Especially welcome are submissions that extend =20 > participants' understanding the role of ASF projects and their =20 > influence on the Open Source shane48@example.com. General Sessions =20 > are scheduled for 50 minutes and are accessible to all conference =20 > delegates. > > Case Study/Industry Profile > Practitioners are invited to submit presentations that focus on how =20= > implementing particular ASF technologies led to improved products/=20 > solutions, service offerings, changes in work practices, among other =20= > successes. Proposals that highlight overcoming interesting =20 > challenges in application design and developing innovative =20 > frameworks using multiple ASF projects are particularly encouraged. =20= > NOTE: Marketing-oriented submissions aimed at promoting specific =20 > organizations or products will not be accepted. > > Invited Keynotes/Panels/Speakers > Each conference the ApacheCon Planning team invites select =20 > presenters dealing with engaging, dialectical, and challenging =20 > subjects to present in keynote and/or panel formats. Topics include =20= > cutting-edge technology development, industry leadership, hot or =20 > emerging trends, opinions on controversial issues, insight on =20 > technology paradigms, and contrasting viewpoints in complementary =20 > professional areas. Those interested in suggesting a candidate for =20 > an invited speaker opportunity should submit a brief proposal with =20 > the speaker's name, affiliation, background/bio, overview of topics =20= > of interest, and contact information. > > Important Dates > Please submit your completed proposal, at = http://eu.apachecon.com/c/aceu2009/cfp/=20 > before Friday, 24th October, 2008 Midnight GMT. > > About ApacheCon 2009 > ApacheCon is co-produced by the Apache Software Foundation and Stone =20= > Circle Productions. The ApacheCon Planning team comprises ASF =20 > Members from all over the world working on a wholly-volunteer basis. =20= > For more information, visit > http://eu.apachecon.com/c/aceu2009/ > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: shane48@example.com > For additional commands, e-mail: shane48@example.com > -------------------------- Grant Ingersoll Lucene Helpful Hints: http://wiki.apache.org/lucene-java/BasicsOfPerformance http://wiki.apache.org/lucene-java/LuceneFAQ From mahout-usershane48@example.com Mon Oct 06 09:27:23 2008 Return-Path: <mahout-usershane48@example.com> Delivered-To: shane48@example.com Received: (qmail 47859 invoked from network); 6 Oct 2008 09:27:22 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 6 Oct 2008 09:27:22 -0000 Received: (qmail 48701 invoked by uid 500); 6 Oct 2008 09:27:21 -0000 Delivered-To: shane48@example.com Received: (qmail 48496 invoked by uid 500); 6 Oct 2008 09:27:20 -0000 Mailing-List: contact shane48@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:shane48@example.com> List-Unsubscribe: <mailto:shane48@example.com> List-Post: <mailto:shane48@example.com> List-Id: <mahout-user.lucene.apache.org> Reply-To: shane48@example.com Delivered-To: mailing list shane48@example.com Received: (qmail 48485 invoked by uid 99); 6 Oct 2008 09:27:20 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Oct 2008 02:27:20 -0700 X-ASF-Spam-Status: No, hits=-0.9 required=10.0 tests=DNS_FROM_SECURITYSAGE,RCVD_IN_DNSWL_LOW,SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of shane48@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO ciao.gmane.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Oct 2008 09:26:17 +0000 Received: from list by ciao.gmane.org with local (Exim 4.43) id 1KmmMe-0006IR-7e for shane48@example.com; Mon, 06 Oct 2008 09:26:52 +0000 Received: from 93-137.63-81.stat.fixnetdata.ch ([83.161.120.155]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for <shane48@example.com>; Mon, 06 Oct 2008 09:26:52 +0000 Received: from werner.mueller by 93-137.63-81.stat.fixnetdata.ch with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for <shane48@example.com>; Mon, 06 Oct 2008 09:26:52 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: shane48@example.com From: werner mueller <shane48@example.com> Subject: Re: Some questions for starters Date: Mon, 06 Oct 2008 11:26:44 +0200 Lines: 124 Message-ID: <shane48@example.com> References: <shane48@example.com> <shane48@example.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Complaints-To: shane48@example.com X-Gmane-NNTP-Posting-Host: 93-137.63-81.stat.fixnetdata.ch User-Agent: Thunderbird 83.161.120.155 (Windows/20080914) In-Reply-To: <shane48@example.com> Sender: news <shane48@example.com> X-Virus-Checked: Checked by ClamAV on apache.org Hallo Thanks for the reply! I'll try to be more specific (which is a bit diffucult atm). The contract example you gave comes quite close. For example: if a contract from this month is not in a 5% range of what is was in the last month, give an alarm. similar things could be done on phone numbers or on numbers of connections. the thing i know no one has an idea of: is it 5% or 25%? these numbers are only clear after at least a year. and i dont assume anyone will tweak the system continousily. on the other hand: from december to january a jump of 25% may be absolutly normal (some companies 'stop' working in december as the whole staff must go on vacation). so beside the change of cost also a timely context should be reflected in the limit warnings. looking at a single contract seems a bad thing. So I could look at all contracts of the same company, try to figure out some average change in costs. If 80% of the contracts have an increase the alarm limit could be higher. But here again: why 80% (its a random number right now)? This would get easier once a full 12 month data set is stored. until then... try and error. but thats hard to sell. (espacially the time required for tweaking is hard to specify). beside the price of the contract there is information like: number of connections, time of the single connections (people work between 8-17h), number of products used (sms, telephony on foreign countries, data options, etc.) right now i dont know how to figure out the numbers to declare behaviour as 'unusual' based on limits that differ from company to company (a plumber may have different limits that a consulting company). of course there is the option: let the user choose the limits. this has two drawbacks: - where should the user know the limits from? and - some user have to look at thousands of contracts. so i would prefer the system to work on its own (as much as possible). i don't know whether there are some 'weighting' algorithms out there that do similar things. so any hint may help a lot. The whole alarming feature will be part of the web-application. so whatever we need to do on the cluster must be done via JDBC (from do-it-yourself to call a stored procedure). i dont know if we are allowed to run java applications inside the DB directly. I know we execute some tools to import data that run on the db hosts but in a different JVM. But actually the system requiremts question i have goes more into: do i need a computing cloud and MapReduce or are algorithms to learn from data independant of such things? Thanks :) kind regards werner Grant Ingersoll schrieb: > > On Sep 28, 2008, at 6:26 PM, werner mueller wrote: > >> Hallo's >> >> finally i find some time to ask boring questions :) >> >> I some sort of stumbled across the mahout project at apachecon08 in >> amsterdam. But i havent found the time for looking into it deeply. >> >> I would like to ask for some hints / links / directions for a >> 'predictions' feature. i read through the mahout wiki and found some >> interesting links. but since i com more from the applications part and i >> am not that much into databases i need some help getting started. >> >> we develop a reporting application for a telcommunication company. >> mainly we store data in an oracle cluster. it consists of a star-schema. >> the application mainly offers to create reports on two data sources: >> costs and traffic. the data amount is about 1-2 terabytes. >> >> the idea came up to implement some 'alarming' features. so customers >> could set up some limits for contracts, phone numbers etc to get >> notified once the limits are reached or the data 'behaves strange' (too >> strong increases for a period, other ideas to come...). > > Can you give an example? It sounds like you simply want the user to say > "if contracts > X, then alarm", but I gather not, since you are asking > here. Or are you looking for the user > to not be involved in setting the thresholds, but instead to learn from > past examples where there was a problem? For instance, you have > failures from before, but you don't particularly know why it failed > (i.e. what features caused the problem). > >> >> >> i would like to ask if there is something of use in mahout or whether >> you would recommend to keep such features 'simple' on a statistical >> basis and not use learning techniques at all? > > Well, simple is usually better, if it solves your problem. > >> >> >> on the other hand the more boring questions: do i need a hadoop cluster >> for your implementations or could i run them on oracle based clusters as >> well? > > I don't know enough about Oracle clusters to render an opinion. If your > asking if Mahout will run inside the Oracle JVM, I'm guessing that would > be a stretch at this point, but I don't have anything to base that on. > > -Grant > > > From mahout-usershane48@example.com Tue Oct 07 12:23:05 2008 Return-Path: <mahout-usershane48@example.com> Delivered-To: shane48@example.com Received: (qmail 40298 invoked from network); 7 Oct 2008 12:23:05 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 7 Oct 2008 12:23:05 -0000 Received: (qmail 34366 invoked by uid 500); 7 Oct 2008 12:23:04 -0000 Delivered-To: shane48@example.com Received: (qmail 34339 invoked by uid 500); 7 Oct 2008 12:23:03 -0000 Mailing-List: contact shane48@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:shane48@example.com> List-Unsubscribe: <mailto:shane48@example.com> List-Post: <mailto:shane48@example.com> List-Id: <mahout-user.lucene.apache.org> Reply-To: shane48@example.com Delivered-To: mailing list shane48@example.com Received: (qmail 34328 invoked by uid 99); 7 Oct 2008 12:23:03 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Oct 2008 05:23:03 -0700 X-ASF-Spam-Status: No, hits=2.1 required=10.0 tests=DNS_FROM_SECURITYSAGE,RCVD_IN_BL_SPAMCOP_NET,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO h1349259.stratoserver.net) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Oct 2008 12:21:57 +0000 Received: from 83.161.120.155.net.de.o2.com ([83.161.120.155] helo=MotokoKusanagi.isabel-drost.de) by h1349259.stratoserver.net with esmtpsa (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.63) (envelope-from <shane48@example.com>) id 1KnBX1-0005fV-Mu for shane48@example.com; Tue, 07 Oct 2008 12:19:21 +0000 From: Isabel Drost <shane48@example.com> Reply-To: shane48@example.com To: shane48@example.com Subject: Re: Some questions for starters Date: Tue, 7 Oct 2008 14:19:08 +0200 User-Agent: KMail/1.9.9 References: <shane48@example.com> <shane48@example.com> <shane48@example.com> In-Reply-To: <shane48@example.com> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart2040636.OTG8Ag3ePP"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Message-Id: <shane48@example.com> X-Virus-Checked: Checked by ClamAV on apache.org --nextPart2040636.OTG8Ag3ePP Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline On Monday 06 October 2008, werner mueller wrote: > of course there is the option: let the user choose the limits. this has > two drawbacks: > - where should the user know the limits from? and > - some user have to look at thousands of contracts. > so i would prefer the system to work on its own (as much as possible). To me your problem looks something like the following: You have a number of clients. For each client you store the same kind of da= ta=20 sets, but the exact values differ depending on the client. Your task is - given previous datasets - classify new incoming data as norm= al=20 or unusual. You already have identified several features that might help you classify t= he=20 incoming data. The only thing you are missing is for each client a good=20 combination of the features. I see two possibilities of solving your problem: There are algorithms for instance in the intrusion detection community that= =20 deal with the problem of discovering unusual data in a stream of normal dat= a.=20 You might find the algorithms you are looking for there. Maybe someone more= =20 familiar with this area than myself can answer your questions on this list. The second possibility would be to manually label previous datasets as "ok"= =20 and "strange", train a classifier on it and apply it to new incoming data.= =20 Only problem here: You need labeled data for each client, you need to retra= in=20 each time the data changes. Isabel =2D-=20 Vail's Second Axiom: The amount of work to be done increases in proportion = to=20 the amount of work already completed. |\ _,,,---,,_ Web: <http://www.isabel-drost.de> /,`.-'`' -. ;-;;,_ |,4- ) )-,_..;\ ( `'-' '---''(_/--' `-'\_) (fL) IM: <xmpp://MaineC.@spaceboyz.net> --nextPart2040636.OTG8Ag3ePP Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) iEYEABECAAYFAkjrU7wACgkQPJpCBAwIhbSyPgCdHc7HplEdaLq3Ix2XoWO9st8M dGoAni3FoB1pxil26pkZ8hwTHV4qrqzd =XMPC -----END PGP SIGNATURE----- --nextPart2040636.OTG8Ag3ePP-- From mahout-usershane48@example.com Tue Oct 07 20:49:58 2008 Return-Path: <mahout-usershane48@example.com> Delivered-To: shane48@example.com Received: (qmail 68987 invoked from network); 7 Oct 2008 20:49:58 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 7 Oct 2008 20:49:58 -0000 Received: (qmail 77718 invoked by uid 500); 7 Oct 2008 20:49:56 -0000 Delivered-To: shane48@example.com Received: (qmail 77691 invoked by uid 500); 7 Oct 2008 20:49:56 -0000 Mailing-List: contact shane48@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:shane48@example.com> List-Unsubscribe: <mailto:shane48@example.com> List-Post: <mailto:shane48@example.com> List-Id: <mahout-user.lucene.apache.org> Reply-To: shane48@example.com Delivered-To: mailing list shane48@example.com Received: (qmail 77680 invoked by uid 99); 7 Oct 2008 20:49:56 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Oct 2008 13:49:56 -0700 X-ASF-Spam-Status: No, hits=2.1 required=10.0 tests=DNS_FROM_SECURITYSAGE,HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of shane48@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO wa-out-1112.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Oct 2008 20:48:53 +0000 Received: by wa-out-1112.google.com with SMTP id n4so2053885wag.5 for <shane48@example.com>; Tue, 07 Oct 2008 13:49:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:in-reply-to:mime-version:content-type:references; bh=q/kCGDeFFE7RI0heCfv7+S1SNBbnYFY+1rPkh1Ayu9k=; b=ekyHqgieGNEp3t+uVISbn5VQX+N6j0/1ySNG5aAY+yZFbGht/zKWgyeHfxnFV3HV0b bCKw8QWvyF7r00oY5eLuQrDT2DOAfSEY7QODocPY195SQ+DusvQV5TC6IFkKw00ykCt/ Bx9zRCUebiAHLpiHatbExVA0ZrfAfZCfvo3NI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version :content-type:references; b=mVmuKYEgJAuG69woAV6b+lzZH9/BOuhaLHmtnd0zZQDh3ZUhV2/osoQjszXVYxgqdo prZsrfHEbCgNUgafm8H6ndSJqETJ60nAvv23jFZMjSLDTfaQS7iwU01J32KoIFcVowF7 jtM2CMZA++LvIehA00ihVSekd94CyrjYIrRGQ= Received: by 83.161.120.155 with SMTP id 3mr8051391way.22.1223412568180; Tue, 07 Oct 2008 13:49:28 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Tue, 7 Oct 2008 13:49:28 -0700 (PDT) Message-ID: <shane48@example.com> Date: Tue, 7 Oct 2008 13:49:28 -0700 From: "Chuck Lam" <shane48@example.com> To: shane48@example.com, shane48@example.com Subject: Re: Some questions for starters In-Reply-To: <shane48@example.com> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_58644_13945683.1223412568172" References: <shane48@example.com> <shane48@example.com> <shane48@example.com> <shane48@example.com> X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_58644_13945683.1223412568172 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Werner, The devil is in the details, but it sounds like your problem falls under the general class of "anomaly detection," of which intrusion detection (as mentioned by Isabel) is one application. The basic kind of anomaly detection usually starts out with "hypothesis testing" from statistics. Any intro book to statistics would have a chapter on that. In particular, Student's t-test and chi-square testing are often used. The textbook example of their application would be something like deciding if the medical condition of one group of people (who, say, smokes) is significantly different (i.e. worse) than another group. Both Student's t-test and chi-square testing work on comparing two *groups*, with one group generally consider "normal." If you want to detect a *single* item/contract as being anomalous or not, usually you build a statistical model over the "normal" group and see the probability of the single item occurring under this model. Building a statistical model can be as simple as fitting a Gaussian or can be super complicated. Again it depends on the details of your problem and what assumptions you can make. If you have labeled training data, then it becomes a classification problem. I haven't thought through whether/which/how these algorithms can be implemented in a Hadoop/MapReduce framework, but let's figure out which one is more appropriate for your problem first... On Tue, Oct 7, 2008 at 5:19 AM, Isabel Drost <shane48@example.com>wrote: > On Monday 06 October 2008, werner mueller wrote: > > of course there is the option: let the user choose the limits. this has > > two drawbacks: > > - where should the user know the limits from? and > > - some user have to look at thousands of contracts. > > so i would prefer the system to work on its own (as much as possible). > > To me your problem looks something like the following: > > You have a number of clients. For each client you store the same kind of > data > sets, but the exact values differ depending on the client. > > Your task is - given previous datasets - classify new incoming data as > normal > or unusual. > > You already have identified several features that might help you classify > the > incoming data. The only thing you are missing is for each client a good > combination of the features. > > I see two possibilities of solving your problem: > > There are algorithms for instance in the intrusion detection community that > deal with the problem of discovering unusual data in a stream of normal > data. > You might find the algorithms you are looking for there. Maybe someone more > familiar with this area than myself can answer your questions on this list. > > The second possibility would be to manually label previous datasets as "ok" > and "strange", train a classifier on it and apply it to new incoming data. > Only problem here: You need labeled data for each client, you need to > retrain > each time the data changes. > > Isabel > > -- > Vail's Second Axiom: The amount of work to be done increases in > proportion to > the amount of work already completed. > |\ _,,,---,,_ Web: <http://www.isabel-drost.de> > /,`.-'`' -. ;-;;,_ > |,4- ) )-,_..;\ ( `'-' > '---''(_/--' `-'\_) (fL) IM: <xmpp://MaineC.@spaceboyz.net> > ------=_Part_58644_13945683.1223412568172-- From mahout-usershane48@example.com Thu Oct 09 12:34:57 2008 Return-Path: <mahout-usershane48@example.com> Delivered-To: shane48@example.com Received: (qmail 27906 invoked from network); 9 Oct 2008 12:34:57 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 9 Oct 2008 12:34:57 -0000 Received: (qmail 3120 invoked by uid 500); 9 Oct 2008 12:34:55 -0000 Delivered-To: shane48@example.com Received: (qmail 3094 invoked by uid 500); 9 Oct 2008 12:34:55 -0000 Mailing-List: contact shane48@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:shane48@example.com> List-Unsubscribe: <mailto:shane48@example.com> List-Post: <mailto:shane48@example.com> List-Id: <mahout-user.lucene.apache.org> Reply-To: shane48@example.com Delivered-To: mailing list shane48@example.com Received: (qmail 3083 invoked by uid 99); 9 Oct 2008 12:34:55 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Oct 2008 05:34:55 -0700 X-ASF-Spam-Status: No, hits=0.1 required=10.0 tests=DNS_FROM_SECURITYSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of shane48@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO yw-out-2324.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Oct 2008 12:33:52 +0000 Received: by yw-out-2324.google.com with SMTP id 3so699270ywj.5 for <shane48@example.com>; Thu, 09 Oct 2008 05:34:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:mime-version:content-type:content-transfer-encoding :content-disposition; bh=2iml7KdQ5R7HeSWZ6L9LXKd6sziVbbx8hWc8YqRSTag=; b=d6JIlgDCZLay6luD1/sG1HitvLbiqmiH69ubC4to/REyDmchr4d0QHQ8cWh7jCwlqx m7TQ+QxKRqm9zBzvJFW8cxWW2PbZzYwoROltWVASOyyeVM3UsVWY+V+J+J7vKSf4GZlZ g/0fGJLn+edywEPf8SUGhtk8cenGmcEZtp2+o= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type :content-transfer-encoding:content-disposition; b=vCv/9eKc2P5K58o++ou7VfErKNXlzaE2ADYHR0fMJ3RhYFpRm1D5hbwUc0RJ15FVOg kiWuWWbE52t205jWIDhhrTegwJhDlyX66P5joWoEW1RqBywZBkyOujnkGN1Ov0Ead0Al KMJ6sQjtYnrSEHQ4wR5WdI5MyzscHnYRd7/dQ= Received: by 83.161.120.155 with SMTP id s19mr113221ybg.101.1223555658927; Thu, 09 Oct 2008 05:34:18 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Thu, 9 Oct 2008 05:34:18 -0700 (PDT) Message-ID: <shane48@example.com> Date: Thu, 9 Oct 2008 13:34:18 +0100 From: "Sean Owen" <shane48@example.com> To: shane48@example.com Subject: What are the business cases for collaborative filtering? MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Virus-Checked: Checked by ClamAV on apache.org I mentioned this to Otis but thought I'd expand it -- As some of you may know I am actually an MBA student. I have worked on the collaborative filtering portion of Mahout (formerly called Taste) for several years. I have always been curious to know what, exactly, enterprises need out of collaborative filtering. What product would they like to see? Now I have the opportunity to use this as the basis of a business school project. So I am wondering if anyone can comment on any of these questions, without naming names if it's confidential -- - What do enterprises imagine they can get out of collaborative filtering? - At what stage is the interest -- barely know about it, ready to use it, other? - What do people use to solve these needs today, custom code, other products? - What's the biggest need enterprises have from collaborative filtering products or services? - What's the one product or feature an enterprise would love to see in the market, in terms of CF? Thanks for any thoughts, folks! Sean From mahout-usershane48@example.com Fri Oct 10 04:25:49 2008 Return-Path: <mahout-usershane48@example.com> Delivered-To: shane48@example.com Received: (qmail 71784 invoked from network); 10 Oct 2008 04:25:49 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 10 Oct 2008 04:25:49 -0000 Received: (qmail 55731 invoked by uid 500); 10 Oct 2008 04:25:48 -0000 Delivered-To: shane48@example.com Received: (qmail 55700 invoked by uid 500); 10 Oct 2008 04:25:48 -0000 Mailing-List: contact shane48@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:shane48@example.com> List-Unsubscribe: <mailto:shane48@example.com> List-Post: <mailto:shane48@example.com> List-Id: <mahout-user.lucene.apache.org> Reply-To: shane48@example.com Delivered-To: mailing list shane48@example.com Received: (qmail 55689 invoked by uid 99); 10 Oct 2008 04:25:48 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Oct 2008 21:25:48 -0700 X-ASF-Spam-Status: No, hits=1.3 required=10.0 tests=DNS_FROM_SECURITYSAGE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO ag-out-0708.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Oct 2008 04:24:41 +0000 Received: by ag-out-0708.google.com with SMTP id 23so542949agd.11 for <shane48@example.com>; Thu, 09 Oct 2008 21:25:07 -0700 (PDT) Received: by 83.161.120.155 with SMTP id p1mr1452403ybi.61.1223612706687; Thu, 09 Oct 2008 21:25:06 -0700 (PDT) Received: from C0A80168.tipt.aol.com (h-64-236-138-3.aoltw.net [83.161.120.155]) by mx.google.com with ESMTPS id 5sm995082ywl.4.283.161.120.155.25.04 (version=TLSv1/SSLv3 cipher=RC4-MD5); Thu, 09 Oct 2008 21:25:05 -0700 (PDT) Message-ID: <shane48@example.com> Date: Fri, 10 Oct 2008 15:25:02 +1100 From: Ian Holsman <shane48@example.com> User-Agent: Thunderbird 83.161.120.155 (Macintosh/20080914) MIME-Version: 1.0 To: shane48@example.com Subject: Re: What are the business cases for collaborative filtering? References: <shane48@example.com> In-Reply-To: <shane48@example.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Sean Owen wrote: > I mentioned this to Otis but thought I'd expand it -- > > As some of you may know I am actually an MBA student. I have worked on > the collaborative filtering portion of Mahout (formerly called Taste) > for several years. I have always been curious to know what, exactly, > enterprises need out of collaborative filtering. What product would > they like to see? > Hi Sean. as a ex-MBA student (I got it last year) and a guy who managed to get this on his companies radar I might be able to help ;-) > Now I have the opportunity to use this as the basis of a business > school project. So I am wondering if anyone can comment on any of > these questions, without naming names if it's confidential -- > > - What do enterprises imagine they can get out of collaborative filtering? > Firstly we are a 'consumer web site', different businesses will have different value drivers. We don't sell things , but rely on advertising as our primary revenue driver. (for Advertising the 2 things that influence that is how many people click through/see the ad, and how much we can charge for it). we also have other KPIs that we believe are lead indicators. (that is we increases these and the money will follow). They are engagement (time spent on the page), page views and return visits. we see CF
{'source': 'Apache Software Foundation Public Mail Archives'}