From garciaoscar@example.com Tue Mar 01 04:03:02 2011 Return-Path: <garciaoscar@example.com> Delivered-To: garciaoscar@example.com Received: (qmail 18502 invoked from network); 1 Mar 2011 04:03:02 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Mar 2011 04:03:02 -0000 Received: (qmail 28064 invoked by uid 500); 1 Mar 2011 04:03:02 -0000 Delivered-To: garciaoscar@example.com Received: (qmail 27945 invoked by uid 500); 1 Mar 2011 04:02:59 -0000 Mailing-List: contact garciaoscar@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:garciaoscar@example.com> List-Unsubscribe: <mailto:garciaoscar@example.com> List-Post: <mailto:garciaoscar@example.com> List-Id: <issues.hbase.apache.org> Delivered-To: mailing list garciaoscar@example.com Received: (qmail 27931 invoked by uid 99); 1 Mar 2011 04:02:57 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 04:02:57 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 04:02:57 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155]) by hel.zones.apache.org (Postfix) with ESMTP id EA45C42E64 for <garciaoscar@example.com>; Tue, 1 Mar 2011 04:02:36 +0000 (UTC) Date: Tue, 1 Mar 2011 04:02:36 +0000 (UTC) From: "stack (JIRA)" <garciaoscar@example.com> To: garciaoscar@example.com Message-ID: <garciaoscar@example.com> In-Reply-To: <garciaoscar@example.com> Subject: [jira] Commented: (HBASE-3529) Add search to HBase MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HBASE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000693#comment-13000693 ] stack commented on HBASE-3529: ------------------------------ @Jason Which offset are you talking off? The storefile in hbase keeps offsets in a file index. When we ask to read from a position in the hfile, dfsclient does a quick calc to figure which block and then relatively, the offset into the target block. Are you talking of something more fine grained or something else? > Add search to HBase > ------------------- > > Key: HBASE-3529 > URL: https://issues.apache.org/jira/browse/HBASE-3529 > Project: HBase > Issue Type: Improvement > Affects Versions: 0.90.0 > Reporter: Jason Rutherglen > > Using the Apache Lucene library we can add freetext search to HBase. The advantages of this are: > * HBase is highly scalable and distributed > * HBase is realtime > * Lucene is a fast inverted index and will soon be realtime (see LUCENE-2312) > * Lucene offers many types of queries not currently available in HBase (eg, AND, OR, NOT, phrase, etc) > * It's easier to build scalable realtime systems on top of already architecturally sound, scalable realtime data system, eg, HBase. > * Scaling realtime search will be as simple as scaling HBase. > Phase 1 - Indexing: > * Integrate Lucene into HBase such that an index mirrors a given region. This means cascading add, update, and deletes between a Lucene index and an HBase region (and vice versa). > * Define meta-data to mark a region as indexed, and use a Solr schema to allow the user to define the fields and analyzers. > * Integrate with the HLog to ensure that index recovery can occur properly (eg, on region server failure) > * Mirror region splits with indexes (use Lucene's IndexSplitter?) > * When a region is written to HDFS, also write the corresponding Lucene index to HDFS. > * A row key will be the ID of a given Lucene document. The Lucene docstore will explicitly not be used because the document/row data is stored in HBase. We will need to solve what the best data structure for efficiently mapping a docid -> row key is. It could be a docstore, field cache, column stride fields, or some other mechanism. > * Write unit tests for the above > Phase 2 - Queries: > * Enable distributed Lucene queries > * Regions that have Lucene indexes are inherently available and may be searched on, meaning there's no need for a separate search related system in Zookeeper. > * Integrate search with HBase's RPC mechanism -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira From garciaoscar@example.com Tue Mar 01 04:22:03 2011 Return-Path: <garciaoscar@example.com> Delivered-To: garciaoscar@example.com Received: (qmail 75090 invoked from network); 1 Mar 2011 04:22:02 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Mar 2011 04:22:02 -0000 Received: (qmail 40716 invoked by uid 500); 1 Mar 2011 04:22:01 -0000 Delivered-To: garciaoscar@example.com Received: (qmail 40599 invoked by uid 500); 1 Mar 2011 04:21:59 -0000 Mailing-List: contact garciaoscar@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:garciaoscar@example.com> List-Unsubscribe: <mailto:garciaoscar@example.com> List-Post: <mailto:garciaoscar@example.com> List-Id: <issues.hbase.apache.org> Delivered-To: mailing list garciaoscar@example.com Received: (qmail 40591 invoked by uid 99); 1 Mar 2011 04:21:58 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 04:21:58 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 04:21:57 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155]) by hel.zones.apache.org (Postfix) with ESMTP id 4DF1D43291 for <garciaoscar@example.com>; Tue, 1 Mar 2011 04:21:37 +0000 (UTC) Date: Tue, 1 Mar 2011 04:21:37 +0000 (UTC) From: "Jason Rutherglen (JIRA)" <garciaoscar@example.com> To: garciaoscar@example.com Message-ID: <garciaoscar@example.com> In-Reply-To: <garciaoscar@example.com> Subject: [jira] Commented: (HBASE-3529) Add search to HBase MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HBASE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000697#comment-13000697 ] Jason Rutherglen commented on HBASE-3529: ----------------------------------------- Sorry, I thought through the file access a little more. I think we can use the block local reader as is, because Lucene reads the postings sequentially, we don't really need random file access (eg, the offset issue more or less goes away), we simply need to allow seek'ing forward, and most postings will live inside of a single (64 - 128MB block). The issue with this system is we may need to maintain an FSInputStream per thread per file because we probably don't want to open a new FSInputStream per query given the overhead or creating and destroying them? Will this cause issues with the max file descriptors? > Add search to HBase > ------------------- > > Key: HBASE-3529 > URL: https://issues.apache.org/jira/browse/HBASE-3529 > Project: HBase > Issue Type: Improvement > Affects Versions: 0.90.0 > Reporter: Jason Rutherglen > > Using the Apache Lucene library we can add freetext search to HBase. The advantages of this are: > * HBase is highly scalable and distributed > * HBase is realtime > * Lucene is a fast inverted index and will soon be realtime (see LUCENE-2312) > * Lucene offers many types of queries not currently available in HBase (eg, AND, OR, NOT, phrase, etc) > * It's easier to build scalable realtime systems on top of already architecturally sound, scalable realtime data system, eg, HBase. > * Scaling realtime search will be as simple as scaling HBase. > Phase 1 - Indexing: > * Integrate Lucene into HBase such that an index mirrors a given region. This means cascading add, update, and deletes between a Lucene index and an HBase region (and vice versa). > * Define meta-data to mark a region as indexed, and use a Solr schema to allow the user to define the fields and analyzers. > * Integrate with the HLog to ensure that index recovery can occur properly (eg, on region server failure) > * Mirror region splits with indexes (use Lucene's IndexSplitter?) > * When a region is written to HDFS, also write the corresponding Lucene index to HDFS. > * A row key will be the ID of a given Lucene document. The Lucene docstore will explicitly not be used because the document/row data is stored in HBase. We will need to solve what the best data structure for efficiently mapping a docid -> row key is. It could be a docstore, field cache, column stride fields, or some other mechanism. > * Write unit tests for the above > Phase 2 - Queries: > * Enable distributed Lucene queries > * Regions that have Lucene indexes are inherently available and may be searched on, meaning there's no need for a separate search related system in Zookeeper. > * Integrate search with HBase's RPC mechanism -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira From garciaoscar@example.com Tue Mar 01 04:26:00 2011 Return-Path: <garciaoscar@example.com> Delivered-To: garciaoscar@example.com Received: (qmail 92538 invoked from network); 1 Mar 2011 04:26:00 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Mar 2011 04:26:00 -0000 Received: (qmail 44423 invoked by uid 500); 1 Mar 2011 04:26:00 -0000 Delivered-To: garciaoscar@example.com Received: (qmail 44321 invoked by uid 500); 1 Mar 2011 04:25:58 -0000 Mailing-List: contact garciaoscar@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:garciaoscar@example.com> List-Unsubscribe: <mailto:garciaoscar@example.com> List-Post: <mailto:garciaoscar@example.com> List-Id: <issues.hbase.apache.org> Delivered-To: mailing list garciaoscar@example.com Received: (qmail 44313 invoked by uid 99); 1 Mar 2011 04:25:57 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 04:25:57 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 04:25:56 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155]) by hel.zones.apache.org (Postfix) with ESMTP id E44EE43498 for <garciaoscar@example.com>; Tue, 1 Mar 2011 04:25:36 +0000 (UTC) Date: Tue, 1 Mar 2011 04:25:36 +0000 (UTC) From: "stack (JIRA)" <garciaoscar@example.com> To: garciaoscar@example.com Message-ID: <garciaoscar@example.com> In-Reply-To: <garciaoscar@example.com> Subject: [jira] Commented: (HBASE-3529) Add search to HBase MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HBASE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000706#comment-13000706 ] stack commented on HBASE-3529: ------------------------------ @Jason Currently HBase keeps all files open all the time (Yeah, users have to up their ulimit if they have more than a smidgeon of data in hbase--requirement #4 or #5). > Add search to HBase > ------------------- > > Key: HBASE-3529 > URL: https://issues.apache.org/jira/browse/HBASE-3529 > Project: HBase > Issue Type: Improvement > Affects Versions: 0.90.0 > Reporter: Jason Rutherglen > > Using the Apache Lucene library we can add freetext search to HBase. The advantages of this are: > * HBase is highly scalable and distributed > * HBase is realtime > * Lucene is a fast inverted index and will soon be realtime (see LUCENE-2312) > * Lucene offers many types of queries not currently available in HBase (eg, AND, OR, NOT, phrase, etc) > * It's easier to build scalable realtime systems on top of already architecturally sound, scalable realtime data system, eg, HBase. > * Scaling realtime search will be as simple as scaling HBase. > Phase 1 - Indexing: > * Integrate Lucene into HBase such that an index mirrors a given region. This means cascading add, update, and deletes between a Lucene index and an HBase region (and vice versa). > * Define meta-data to mark a region as indexed, and use a Solr schema to allow the user to define the fields and analyzers. > * Integrate with the HLog to ensure that index recovery can occur properly (eg, on region server failure) > * Mirror region splits with indexes (use Lucene's IndexSplitter?) > * When a region is written to HDFS, also write the corresponding Lucene index to HDFS. > * A row key will be the ID of a given Lucene document. The Lucene docstore will explicitly not be used because the document/row data is stored in HBase. We will need to solve what the best data structure for efficiently mapping a docid -> row key is. It could be a docstore, field cache, column stride fields, or some other mechanism. > * Write unit tests for the above > Phase 2 - Queries: > * Enable distributed Lucene queries > * Regions that have Lucene indexes are inherently available and may be searched on, meaning there's no need for a separate search related system in Zookeeper. > * Integrate search with HBase's RPC mechanism -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira From garciaoscar@example.com Tue Mar 01 05:10:03 2011 Return-Path: <garciaoscar@example.com> Delivered-To: garciaoscar@example.com Received: (qmail 42550 invoked from network); 1 Mar 2011 05:10:03 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Mar 2011 05:10:03 -0000 Received: (qmail 70973 invoked by uid 500); 1 Mar 2011 05:10:03 -0000 Delivered-To: garciaoscar@example.com Received: (qmail 70855 invoked by uid 500); 1 Mar 2011 05:10:01 -0000 Mailing-List: contact garciaoscar@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:garciaoscar@example.com> List-Unsubscribe: <mailto:garciaoscar@example.com> List-Post: <mailto:garciaoscar@example.com> List-Id: <issues.hbase.apache.org> Delivered-To: mailing list garciaoscar@example.com Received: (qmail 70815 invoked by uid 99); 1 Mar 2011 05:10:00 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 05:10:00 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 05:09:57 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155]) by hel.zones.apache.org (Postfix) with ESMTP id E18C843E17 for <garciaoscar@example.com>; Tue, 1 Mar 2011 05:09:36 +0000 (UTC) Date: Tue, 1 Mar 2011 05:09:36 +0000 (UTC) From: "Jason Rutherglen (JIRA)" <garciaoscar@example.com> To: garciaoscar@example.com Message-ID: <garciaoscar@example.com> In-Reply-To: <garciaoscar@example.com> Subject: [jira] Commented: (HBASE-3529) Add search to HBase MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/HBASE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000721#comment-13000721 ] Jason Rutherglen commented on HBASE-3529: ----------------------------------------- Ah, going back to storing the "row, qualifier and timestamp" in a Lucene document/docstore, is that does require totally random reads. I wonder if there's some efficient way to store row pointers in RAM (compression?) or a Hadooop data structure that can be used? I think that storing this information in the Lucene field cache is going to cause OOMs. It'd be great if we could simply store a long that points to the exact row and column family we'd like to reference, as that could easily be stored in RAM, and would possibly enable faster lookup? > Add search to HBase > ------------------- > > Key: HBASE-3529 > URL: https://issues.apache.org/jira/browse/HBASE-3529 > Project: HBase > Issue Type: Improvement > Affects Versions: 0.90.0 > Reporter: Jason Rutherglen > > Using the Apache Lucene library we can add freetext search to HBase. The advantages of this are: > * HBase is highly scalable and distributed > * HBase is realtime > * Lucene is a fast inverted index and will soon be realtime (see LUCENE-2312) > * Lucene offers many types of queries not currently available in HBase (eg, AND, OR, NOT, phrase, etc) > * It's easier to build scalable realtime systems on top of already architecturally sound, scalable realtime data system, eg, HBase. > * Scaling realtime search will be as simple as scaling HBase. > Phase 1 - Indexing: > * Integrate Lucene into HBase such that an index mirrors a given region. This means cascading add, update, and deletes between a Lucene index and an HBase region (and vice versa). > * Define meta-data to mark a region as indexed, and use a Solr schema to allow the user to define the fields and analyzers. > * Integrate with the HLog to ensure that index recovery can occur properly (eg, on region server failure) > * Mirror region splits with indexes (use Lucene's IndexSplitter?) > * When a region is written to HDFS, also write the corresponding Lucene index to HDFS. > * A row key will be the ID of a given Lucene document. The Lucene docstore will explicitly not be used because the document/row data is stored in HBase. We will need to solve what the best data structure for efficiently mapping a docid -> row key is. It could be a docstore, field cache, column stride fields, or some other mechanism. > * Write unit tests for the above > Phase 2 - Queries: > * Enable distributed Lucene queries > * Regions that have Lucene indexes are inherently available and may be searched on, meaning there's no need for a separate search related system in Zookeeper. > * Integrate search with HBase's RPC mechanism -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira From garciaoscar@example.com Tue Mar 01 05:40:05 2011 Return-Path: <garciaoscar@example.com> Delivered-To: garciaoscar@example.com Received: (qmail 46987 invoked from network); 1 Mar 2011 05:40:04 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Mar 2011 05:40:04 -0000 Received: (qmail 89959 invoked by uid 500); 1 Mar 2011 05:40:04 -0000 Delivered-To: garciaoscar@example.com Received: (qmail 89857 invoked by uid 500); 1 Mar 2011 05:40:01 -0000 Mailing-List: contact garciaoscar@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:garciaoscar@example.com> List-Unsubscribe: <mailto:garciaoscar@example.com> List-Post: <mailto:garciaoscar@example.com> List-Id: <issues.hbase.apache.org> Delivered-To: mailing list garciaoscar@example.com Received: (qmail 89847 invoked by uid 99); 1 Mar 2011 05:40:00 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 05:40:00 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 05:39:57 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155]) by hel.zones.apache.org (Postfix) with ESMTP id E581143452 for <garciaoscar@example.com>; Tue, 1 Mar 2011 05:39:36 +0000 (UTC) Date: Tue, 1 Mar 2011 05:39:36 +0000 (UTC) From: "stack (JIRA)" <garciaoscar@example.com> To: garciaoscar@example.com Message-ID: <garciaoscar@example.com> In-Reply-To: <garciaoscar@example.com> Subject: [jira] Commented: (HBASE-3529) Add search to HBase MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/HBASE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000730#comment-13000730 ] stack commented on HBASE-3529: ------------------------------ Are you thinking you could exploit hbase scan somehow? If so, how you think it would work? Whats a lucene docid? A long? Or a double? You could toBytes that and that'd be the hbase row (HBase rows are byte arrays). The column family could be one byte -- that'd give you 256 maximum column family names. Qualifier probably has to be lucene document field name. You could try and keep these short. Timestamp is a long. So thats two longs (docid + ts), one byte for cf, and say, 8 characters for field name.. thats about 25 bytes or so per lucene doc. Will that cause you to run out of mem? > Add search to HBase > ------------------- > > Key: HBASE-3529 > URL: https://issues.apache.org/jira/browse/HBASE-3529 > Project: HBase > Issue Type: Improvement > Affects Versions: 0.90.0 > Reporter: Jason Rutherglen > > Using the Apache Lucene library we can add freetext search to HBase. The advantages of this are: > * HBase is highly scalable and distributed > * HBase is realtime > * Lucene is a fast inverted index and will soon be realtime (see LUCENE-2312) > * Lucene offers many types of queries not currently available in HBase (eg, AND, OR, NOT, phrase, etc) > * It's easier to build scalable realtime systems on top of already architecturally sound, scalable realtime data system, eg, HBase. > * Scaling realtime search will be as simple as scaling HBase. > Phase 1 - Indexing: > * Integrate Lucene into HBase such that an index mirrors a given region. This means cascading add, update, and deletes between a Lucene index and an HBase region (and vice versa). > * Define meta-data to mark a region as indexed, and use a Solr schema to allow the user to define the fields and analyzers. > * Integrate with the HLog to ensure that index recovery can occur properly (eg, on region server failure) > * Mirror region splits with indexes (use Lucene's IndexSplitter?) > * When a region is written to HDFS, also write the corresponding Lucene index to HDFS. > * A row key will be the ID of a given Lucene document. The Lucene docstore will explicitly not be used because the document/row data is stored in HBase. We will need to solve what the best data structure for efficiently mapping a docid -> row key is. It could be a docstore, field cache, column stride fields, or some other mechanism. > * Write unit tests for the above > Phase 2 - Queries: > * Enable distributed Lucene queries > * Regions that have Lucene indexes are inherently available and may be searched on, meaning there's no need for a separate search related system in Zookeeper. > * Integrate search with HBase's RPC mechanism -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira From garciaoscar@example.com Tue Mar 01 07:06:05 2011 Return-Path: <garciaoscar@example.com> Delivered-To: garciaoscar@example.com Received: (qmail 39803 invoked from network); 1 Mar 2011 07:06:05 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Mar 2011 07:06:05 -0000 Received: (qmail 52714 invoked by uid 500); 1 Mar 2011 07:06:05 -0000 Delivered-To: garciaoscar@example.com Received: (qmail 52589 invoked by uid 500); 1 Mar 2011 07:06:02 -0000 Mailing-List: contact garciaoscar@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:garciaoscar@example.com> List-Unsubscribe: <mailto:garciaoscar@example.com> List-Post: <mailto:garciaoscar@example.com> List-Id: <issues.hbase.apache.org> Delivered-To: mailing list garciaoscar@example.com Received: (qmail 52580 invoked by uid 99); 1 Mar 2011 07:06:00 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 07:06:00 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 07:05:58 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155]) by hel.zones.apache.org (Postfix) with ESMTP id CF0BA43C14 for <garciaoscar@example.com>; Tue, 1 Mar 2011 07:05:37 +0000 (UTC) Date: Tue, 1 Mar 2011 07:05:37 +0000 (UTC) From: "stack (JIRA)" <garciaoscar@example.com> To: garciaoscar@example.com Message-ID: <garciaoscar@example.com> In-Reply-To: <garciaoscar@example.com> Subject: [jira] Updated: (HBASE-3574) Move ServerLoad off the heartbeat; prereq of hbase-1502 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/HBASE-3574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-3574: ------------------------- Attachment: 3574.txt This patch serializes HServerLoad into the regionserver ephemeral node instead of passing the load over the heartbeat. It starts a Chore inside regionserver to update the znode every 30 seconds. I'm not going to finish this patch. To complete, I'd have to make investment in translating znode content into HServerInfo, a class deprecated by patch over in hbase-1502. Instead, I'm going to close this issue and roll it into hbase-1502. > Move ServerLoad off the heartbeat; prereq of hbase-1502 > ------------------------------------------------------- > > Key: HBASE-3574 > URL: https://issues.apache.org/jira/browse/HBASE-3574 > Project: HBase > Issue Type: Task > Reporter: stack > Attachments: 3574.txt > > > The master ui and shell report on cluster load by reading the serverload delivered by each RS on heartbeat. HBASE-1502 is removing heartbeat. Need to do serverload otherwise. I'm thinking of writing it instead into the RS ephemeral node and having the master UI, etc. read it from zk instead. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira From garciaoscar@example.com Tue Mar 01 07:08:02 2011 Return-Path: <garciaoscar@example.com> Delivered-To: garciaoscar@example.com Received: (qmail 40026 invoked from network); 1 Mar 2011 07:08:02 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Mar 2011 07:08:02 -0000 Received: (qmail 53092 invoked by uid 500); 1 Mar 2011 07:08:02 -0000 Delivered-To: garciaoscar@example.com Received: (qmail 52973 invoked by uid 500); 1 Mar 2011 07:07:59 -0000 Mailing-List: contact garciaoscar@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:garciaoscar@example.com> List-Unsubscribe: <mailto:garciaoscar@example.com> List-Post: <mailto:garciaoscar@example.com> List-Id: <issues.hbase.apache.org> Delivered-To: mailing list garciaoscar@example.com Received: (qmail 52961 invoked by uid 99); 1 Mar 2011 07:07:58 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 07:07:58 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 07:07:56 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155]) by hel.zones.apache.org (Postfix) with ESMTP id E145143D01 for <garciaoscar@example.com>; Tue, 1 Mar 2011 07:07:36 +0000 (UTC) Date: Tue, 1 Mar 2011 07:07:36 +0000 (UTC) From: "stack (JIRA)" <garciaoscar@example.com> To: garciaoscar@example.com Message-ID: <garciaoscar@example.com> In-Reply-To: <garciaoscar@example.com> Subject: [jira] Resolved: (HBASE-3574) Move ServerLoad off the heartbeat; prereq of hbase-1502 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HBASE-3574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-3574. -------------------------- Resolution: Won't Fix Resolving as won't fix here, but will do over in hbase-1502. > Move ServerLoad off the heartbeat; prereq of hbase-1502 > ------------------------------------------------------- > > Key: HBASE-3574 > URL: https://issues.apache.org/jira/browse/HBASE-3574 > Project: HBase > Issue Type: Task > Reporter: stack > Attachments: 3574.txt > > > The master ui and shell report on cluster load by reading the serverload delivered by each RS on heartbeat. HBASE-1502 is removing heartbeat. Need to do serverload otherwise. I'm thinking of writing it instead into the RS ephemeral node and having the master UI, etc. read it from zk instead. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira From garciaoscar@example.com Tue Mar 01 13:16:02 2011 Return-Path: <garciaoscar@example.com> Delivered-To: garciaoscar@example.com Received: (qmail 3357 invoked from network); 1 Mar 2011 13:16:02 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Mar 2011 13:16:02 -0000 Received: (qmail 26014 invoked by uid 500); 1 Mar 2011 13:16:02 -0000 Delivered-To: garciaoscar@example.com Received: (qmail 25903 invoked by uid 500); 1 Mar 2011 13:15:59 -0000 Mailing-List: contact garciaoscar@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:garciaoscar@example.com> List-Unsubscribe: <mailto:garciaoscar@example.com> List-Post: <mailto:garciaoscar@example.com> List-Id: <issues.hbase.apache.org> Delivered-To: mailing list garciaoscar@example.com Received: (qmail 25877 invoked by uid 99); 1 Mar 2011 13:15:58 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 13:15:58 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 13:15:57 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155]) by hel.zones.apache.org (Postfix) with ESMTP id 4A9424311D for <garciaoscar@example.com>; Tue, 1 Mar 2011 13:15:37 +0000 (UTC) Date: Tue, 1 Mar 2011 13:15:37 +0000 (UTC) From: "Jason Rutherglen (JIRA)" <garciaoscar@example.com> To: garciaoscar@example.com Message-ID: <garciaoscar@example.com> In-Reply-To: <garciaoscar@example.com> Subject: [jira] Commented: (HBASE-3529) Add search to HBase MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HBASE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000866#comment-13000866 ] Jason Rutherglen commented on HBASE-3529: ----------------------------------------- @Stack Thanks for the analysis. I forgot to mention that each subquery would also require it's own FSInputStream, which would be too many file descriptors. The heap required for 25 bytes * 2 mil docs is 50MB, eg, that's too much? I think we can go ahead with the positional read which'd only require an FSInputStream per file, to be shared by all readers of that file (using FileChannel.read(ByteBuffer dst, long position) underneath. Given the number of blocks per Lucene file will be < 10 and the blocks are of a fixed size, we can divide the (offset / blocksize) to efficiently obtain the block index? I think it'll be efficient to translate a file offset into a local block file, eg, I'm not sure why LocatedBlocks.findBlock uses a binary search because I'm not familiar enough with HDFS. Then we'd just need to cache the LocatedBlock(s), instead of looking them up from the DataNode on each small read byte[1024] call. In summary: * DFSClient.DFSInputStream.getBlockRange looks fast enough for many calls per second * locatedBlocks.findBlock uses a binary search for some reason, that'll be a bottleneck, why can't we divide the number the offset by the number of blocks. Oh ok, that's because block sizes are variable. I guess if the number of blocks is small the binary search will always be fast? Or we can detect if the blocks are of the same size and divide to get the correct block? * DFSClient.DFSInputStream.fetchBlockByteRange is a hotspot because it calls chooseDataNode, whose return value [DNAddrPair] can be cached inside of LocatedBlock? * Later in fetchBlockByteRange we call DFSClient.createClientDatanodeProtocolProxy() and make a local RPC call, getBlockPathInfo. I think the results of this [BlockPathInfo] can be cached into LocatedBlock as well? * Then instead of instantiating a new BlockReader object, we can call FileChannel.read(ByteBuffer b, long pos) directly? * With this solution in place we can safely store documents in the docstore without any worries, and in addition use the system that most efficient in Lucene today, all the while using the fewest file descriptors possible. > Add search to HBase > ------------------- > > Key: HBASE-3529 > URL: https://issues.apache.org/jira/browse/HBASE-3529 > Project: HBase > Issue Type: Improvement > Affects Versions: 0.90.0 > Reporter: Jason Rutherglen > > Using the Apache Lucene library we can add freetext search to HBase. The advantages of this are: > * HBase is highly scalable and distributed > * HBase is realtime > * Lucene is a fast inverted index and will soon be realtime (see LUCENE-2312) > * Lucene offers many types of queries not currently available in HBase (eg, AND, OR, NOT, phrase, etc) > * It's easier to build scalable realtime systems on top of already architecturally sound, scalable realtime data system, eg, HBase. > * Scaling realtime search will be as simple as scaling HBase. > Phase 1 - Indexing: > * Integrate Lucene into HBase such that an index mirrors a given region. This means cascading add, update, and deletes between a Lucene index and an HBase region (and vice versa). > * Define meta-data to mark a region as indexed, and use a Solr schema to allow the user to define the fields and analyzers. > * Integrate with the HLog to ensure that index recovery can occur properly (eg, on region server failure) > * Mirror region splits with indexes (use Lucene's IndexSplitter?) > * When a region is written to HDFS, also write the corresponding Lucene index to HDFS. > * A row key will be the ID of a given Lucene document. The Lucene docstore will explicitly not be used because the document/row data is stored in HBase. We will need to solve what the best data structure for efficiently mapping a docid -> row key is. It could be a docstore, field cache, column stride fields, or some other mechanism. > * Write unit tests for the above > Phase 2 - Queries: > * Enable distributed Lucene queries > * Regions that have Lucene indexes are inherently available and may be searched on, meaning there's no need for a separate search related system in Zookeeper. > * Integrate search with HBase's RPC mechanism -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira From garciaoscar@example.com Tue Mar 01 18:21:01 2011 Return-Path: <garciaoscar@example.com> Delivered-To: garciaoscar@example.com Received: (qmail 21815 invoked from network); 1 Mar 2011 18:21:01 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Mar 2011 18:21:01 -0000 Received: (qmail 2701 invoked by uid 500); 1 Mar 2011 18:21:01 -0000 Delivered-To: garciaoscar@example.com Received: (qmail 2570 invoked by uid 500); 1 Mar 2011 18:20:58 -0000 Mailing-List: contact garciaoscar@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:garciaoscar@example.com> List-Unsubscribe: <mailto:garciaoscar@example.com> List-Post: <mailto:garciaoscar@example.com> List-Id: <issues.hbase.apache.org> Delivered-To: mailing list garciaoscar@example.com Received: (qmail 2550 invoked by uid 99); 1 Mar 2011 18:20:57 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 18:20:57 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 18:20:57 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155]) by hel.zones.apache.org (Postfix) with ESMTP id E987D4A95F for <garciaoscar@example.com>; Tue, 1 Mar 2011 18:20:36 +0000 (UTC) Date: Tue, 1 Mar 2011 18:20:36 +0000 (UTC) From: "Prakash Khemani (JIRA)" <garciaoscar@example.com> To: garciaoscar@example.com Message-ID: <garciaoscar@example.com> Subject: [jira] Created: (HBASE-3585) isLegalFamilyName() can throw ArrayOutOfBoundException MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 isLegalFamilyName() can throw ArrayOutOfBoundException ------------------------------------------------------ Key: HBASE-3585 URL: https://issues.apache.org/jira/browse/HBASE-3585 Project: HBase Issue Type: Bug Components: client Affects Versions: 0.90.1 Reporter: Prakash Khemani Priority: Minor org.apache.hadoop.hbase.HColumnDescriptor.isLegalFamilyName(byte[]) accesses byte[0] w/o first checking the array length. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira From garciaoscar@example.com Tue Mar 01 19:12:02 2011 Return-Path: <garciaoscar@example.com> Delivered-To: garciaoscar@example.com Received: (qmail 3552 invoked from network); 1 Mar 2011 19:12:01 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Mar 2011 19:12:01 -0000 Received: (qmail 81903 invoked by uid 500); 1 Mar 2011 19:12:01 -0000 Delivered-To: garciaoscar@example.com Received: (qmail 81766 invoked by uid 500); 1 Mar 2011 19:11:59 -0000 Mailing-List: contact garciaoscar@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:garciaoscar@example.com> List-Unsubscribe: <mailto:garciaoscar@example.com> List-Post: <mailto:garciaoscar@example.com> List-Id: <issues.hbase.apache.org> Delivered-To: mailing list garciaoscar@example.com Received: (qmail 81747 invoked by uid 99); 1 Mar 2011 19:11:58 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 19:11:58 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 19:11:57 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155]) by hel.zones.apache.org (Postfix) with ESMTP id 71C7D4ABE0 for <garciaoscar@example.com>; Tue, 1 Mar 2011 19:11:37 +0000 (UTC) Date: Tue, 1 Mar 2011 19:11:37 +0000 (UTC) From: "Jean-Daniel Cryans (JIRA)" <garciaoscar@example.com> To: garciaoscar@example.com Message-ID: <garciaoscar@example.com> Subject: [jira] Created: (HBASE-3586) Randomize the selection of regions to balance MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 Randomize the selection of regions to balance --------------------------------------------- Key: HBASE-3586 URL: https://issues.apache.org/jira/browse/HBASE-3586 Project: HBase Issue Type: Improvement Affects Versions: 0.90.1 Reporter: Jean-Daniel Cryans Priority: Critical Fix For: 0.90.2 Currently LoadBalancer goes through the list of regions per RS and grabs the few first ones to balance. This is not bad, but that list is often sorted naturally since the a RS that boots will open the regions in a sequential and sorted order (since it comes from .META.) which means that we're balancing regions starting in an almost sorted fashion. We discovered that because one of our internal users created a new table starting with letter "p" which has now grown to 100 regions in the last few hours and they are all served by 1 region server. Looking at the master's log, the balancer has moved as many regions from that region server but they are all from the same table that starts with letter "a" (and the regions that were moved all come one after the other). The part of the code that should be modified is: {code} for (HRegionInfo hri: regions) { // Don't rebalance meta regions. if (hri.isMetaRegion()) continue; regionsToMove.add(new RegionPlan(hri, serverInfo, null)); numTaken++; if (numTaken >= numToOffload) break; } {code} -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira From garciaoscar@example.com Tue Mar 01 19:35:03 2011 Return-Path: <garciaoscar@example.com> Delivered-To: garciaoscar@example.com Received: (qmail 87515 invoked from network); 1 Mar 2011 19:35:03 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Mar 2011 19:35:03 -0000 Received: (qmail 15926 invoked by uid 500); 1 Mar 2011 19:35:03 -0000 Delivered-To: garciaoscar@example.com Received: (qmail 15674 invoked by uid 500); 1 Mar 2011 19:35:01 -0000 Mailing-List: contact garciaoscar@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:garciaoscar@example.com> List-Unsubscribe: <mailto:garciaoscar@example.com> List-Post: <mailto:garciaoscar@example.com> List-Id: <issues.hbase.apache.org> Delivered-To: mailing list garciaoscar@example.com Received: (qmail 15653 invoked by uid 99); 1 Mar 2011 19:35:01 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 19:35:01 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 19:34:58 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155]) by hel.zones.apache.org (Postfix) with ESMTP id 0EEFF4A541 for <garciaoscar@example.com>; Tue, 1 Mar 2011 19:34:37 +0000 (UTC) Date: Tue, 1 Mar 2011 19:34:37 +0000 (UTC) From: "Ted Yu (JIRA)" <garciaoscar@example.com> To: garciaoscar@example.com Message-ID: <garciaoscar@example.com> In-Reply-To: <garciaoscar@example.com> Subject: [jira] Commented: (HBASE-3586) Randomize the selection of regions to balance MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/HBASE-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13001047#comment-13001047 ] Ted Yu commented on HBASE-3586: ------------------------------- See my comment at 29/Jan/11 00:53 in HBASE-3373 > Randomize the selection of regions to balance > --------------------------------------------- > > Key: HBASE-3586 > URL: https://issues.apache.org/jira/browse/HBASE-3586 > Project: HBase > Issue Type: Improvement > Affects Versions: 0.90.1 > Reporter: Jean-Daniel Cryans > Priority: Critical > Fix For: 0.90.2 > > > Currently LoadBalancer goes through the list of regions per RS and grabs the few first ones to balance. This is not bad, but that list is often sorted naturally since the a RS that boots will open the regions in a sequential and sorted order (since it comes from .META.) which means that we're balancing regions starting in an almost sorted fashion. > We discovered that because one of our internal users created a new table starting with letter "p" which has now grown to 100 regions in the last few hours and they are all served by 1 region server. Looking at the master's log, the balancer has moved as many regions from that region server but they are all from the same table that starts with letter "a" (and the regions that were moved all come one after the other). > The part of the code that should be modified is: > {code} > for (HRegionInfo hri: regions) { > // Don't rebalance meta regions. > if (hri.isMetaRegion()) continue; > regionsToMove.add(new RegionPlan(hri, serverInfo, null)); > numTaken++; > if (numTaken >= numToOffload) break; > } > {code} -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira From garciaoscar@example.com Tue Mar 01 19:58:58 2011 Return-Path: <garciaoscar@example.com> Delivered-To: garciaoscar@example.com Received: (qmail 42699 invoked from network); 1 Mar 2011 19:58:58 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Mar 2011 19:58:58 -0000 Received: (qmail 80207 invoked by uid 500); 1 Mar 2011 19:58:58 -0000 Delivered-To: garciaoscar@example.com Received: (qmail 80137 invoked by uid 500); 1 Mar 2011 19:58:57 -0000 Mailing-List: contact garciaoscar@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:garciaoscar@example.com> List-Unsubscribe: <mailto:garciaoscar@example.com> List-Post: <mailto:garciaoscar@example.com> List-Id: <issues.hbase.apache.org> Delivered-To: mailing list garciaoscar@example.com Received: (qmail 80129 invoked by uid 99); 1 Mar 2011 19:58:57 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 19:58:57 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 19:58:57 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155]) by hel.zones.apache.org (Postfix) with ESMTP id 241BB4A0CD for <garciaoscar@example.com>; Tue, 1 Mar 2011 19:58:37 +0000 (UTC) Date: Tue, 1 Mar 2011 19:58:37 +0000 (UTC) From: "Jean-Daniel Cryans (JIRA)" <garciaoscar@example.com> To: garciaoscar@example.com Message-ID: <garciaoscar@example.com> In-Reply-To: <garciaoscar@example.com> Subject: [jira] Commented: (HBASE-3586) Randomize the selection of regions to balance MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HBASE-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13001075#comment-13001075 ] Jean-Daniel Cryans commented on HBASE-3586: ------------------------------------------- You're talking of sorting regionsToMove by creation time, in this jira I'm talking about filling this structure by choosing regions in a slightly different way. Unless that's what you meant too, that we should choose regions from those lists according to their creation time? > Randomize the selection of regions to balance > --------------------------------------------- > > Key: HBASE-3586 > URL: https://issues.apache.org/jira/browse/HBASE-3586 > Project: HBase > Issue Type: Improvement > Affects Versions: 0.90.1 > Reporter: Jean-Daniel Cryans > Priority: Critical > Fix For: 0.90.2 > > > Currently LoadBalancer goes through the list of regions per RS and grabs the few first ones to balance. This is not bad, but that list is often sorted naturally since the a RS that boots will open the regions in a sequential and sorted order (since it comes from .META.) which means that we're balancing regions starting in an almost sorted fashion. > We discovered that because one of our internal users created a new table starting with letter "p" which has now grown to 100 regions in the last few hours and they are all served by 1 region server. Looking at the master's log, the balancer has moved as many regions from that region server but they are all from the same table that starts with letter "a" (and the regions that were moved all come one after the other). > The part of the code that should be modified is: > {code} > for (HRegionInfo hri: regions) { > // Don't rebalance meta regions. > if (hri.isMetaRegion()) continue; > regionsToMove.add(new RegionPlan(hri, serverInfo, null)); > numTaken++; > if (numTaken >= numToOffload) break; > } > {code} -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira From garciaoscar@example.com Tue Mar 01 19:59:01 2011 Return-Path: <garciaoscar@example.com> Delivered-To: garciaoscar@example.com Received: (qmail 42774 invoked from network); 1 Mar 2011 19:59:00 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Mar 2011 19:59:00 -0000 Received: (qmail 80739 invoked by uid 500); 1 Mar 2011 19:59:00 -0000 Delivered-To: garciaoscar@example.com Received: (qmail 80486 invoked by uid 500); 1 Mar 2011 19:59:00 -0000 Mailing-List: contact garciaoscar@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:garciaoscar@example.com> List-Unsubscribe: <mailto:garciaoscar@example.com> List-Post: <mailto:garciaoscar@example.com> List-Id: <issues.hbase.apache.org> Delivered-To: mailing list garciaoscar@example.com Received: (qmail 80478 invoked by uid 99); 1 Mar 2011 19:59:00 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 19:59:00 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 19:58:58 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155]) by hel.zones.apache.org (Postfix) with ESMTP id 755B54A0D1 for <garciaoscar@example.com>; Tue, 1 Mar 2011 19:58:37 +0000 (UTC) Date: Tue, 1 Mar 2011 19:58:37 +0000 (UTC) From: "stack (JIRA)" <garciaoscar@example.com> To: garciaoscar@example.com Message-ID: <garciaoscar@example.com> Subject: [jira] Commented: (HBASE-2166) Inconsistent/wrong treatment of timestamp parameters in the old Thrift API MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/HBASE-2166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13001073#comment-13001073 ] stack commented on HBASE-2166: ------------------------------ This just came up again on the list, Usman Waheed. > Inconsistent/wrong treatment of timestamp parameters in the old Thrift API > -------------------------------------------------------------------------- > > Key: HBASE-2166 > URL: https://issues.apache.org/jira/browse/HBASE-2166 > Project: HBase > Issue Type: Bug > Components: thrift > Affects Versions: 0.20.2 > Reporter: Lars Francke > Assignee: Lars Francke > > The old Thrift API treats timestamps wrong/inconsistent. > getRowTs should return a specific version or null if that doesn't exist. Currently it does seem to treat the timestamp as exclusive so the row is not found. Same goes for getVerTs but there it might only be a documentation problem. > I'll go through the old API and will try to find any such problems and correct them in a way that is consistent with the Java API where possible. > Peter Falk reported this on the mailing list. Thanks! -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira From garciaoscar@example.com Tue Mar 01 20:13:00 2011 Return-Path: <garciaoscar@example.com> Delivered-To: garciaoscar@example.com Received: (qmail 98635 invoked from network); 1 Mar 2011 20:13:00 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Mar 2011 20:13:00 -0000 Received: (qmail 12066 invoked by uid 500); 1 Mar 2011 20:13:00 -0000 Delivered-To: garciaoscar@example.com Received: (qmail 12024 invoked by uid 500); 1 Mar 2011 20:13:00 -0000 Mailing-List: contact garciaoscar@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:garciaoscar@example.com> List-Unsubscribe: <mailto:garciaoscar@example.com> List-Post: <mailto:garciaoscar@example.com> List-Id: <issues.hbase.apache.org> Delivered-To: mailing list garciaoscar@example.com Received: (qmail 12016 invoked by uid 99); 1 Mar 2011 20:12:59 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 20:12:59 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 20:12:57 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155]) by hel.zones.apache.org (Postfix) with ESMTP id D585B4A70B for <garciaoscar@example.com>; Tue, 1 Mar 2011 20:12:36 +0000 (UTC) Date: Tue, 1 Mar 2011 20:12:36 +0000 (UTC) From: "Ted Yu (JIRA)" <garciaoscar@example.com> To: garciaoscar@example.com Message-ID: <garciaoscar@example.com> In-Reply-To: <garciaoscar@example.com> Subject: [jira] Commented: (HBASE-3586) Randomize the selection of regions to balance MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/HBASE-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13001087#comment-13001087 ] Ted Yu commented on HBASE-3586: ------------------------------- That's what I meant. LoadBalancer should select/move young regions. > Randomize the selection of regions to balance > --------------------------------------------- > > Key: HBASE-3586 > URL: https://issues.apache.org/jira/browse/HBASE-3586 > Project: HBase > Issue Type: Improvement > Affects Versions: 0.90.1 > Reporter: Jean-Daniel Cryans > Priority: Critical > Fix For: 0.90.2 > > > Currently LoadBalancer goes through the list of regions per RS and grabs the few first ones to balance. This is not bad, but that list is often sorted naturally since the a RS that boots will open the regions in a sequential and sorted order (since it comes from .META.) which means that we're balancing regions starting in an almost sorted fashion. > We discovered that because one of our internal users created a new table starting with letter "p" which has now grown to 100 regions in the last few hours and they are all served by 1 region server. Looking at the master's log, the balancer has moved as many regions from that region server but they are all from the same table that starts with letter "a" (and the regions that were moved all come one after the other). > The part of the code that should be modified is: > {code} > for (HRegionInfo hri: regions) { > // Don't rebalance meta regions. > if (hri.isMetaRegion()) continue; > regionsToMove.add(new RegionPlan(hri, serverInfo, null)); > numTaken++; > if (numTaken >= numToOffload) break; > } > {code} -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira From garciaoscar@example.com Tue Mar 01 20:22:58 2011 Return-Path: <garciaoscar@example.com> Delivered-To: garciaoscar@example.com Received: (qmail 26100 invoked from network); 1 Mar 2011 20:22:58 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Mar 2011 20:22:58 -0000 Received: (qmail 28652 invoked by uid 500); 1 Mar 2011 20:22:58 -0000 Delivered-To: garciaoscar@example.com Received: (qmail 28588 invoked by uid 500); 1 Mar 2011 20:22:57 -0000 Mailing-List: contact garciaoscar@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:garciaoscar@example.com> List-Unsubscribe: <mailto:garciaoscar@example.com> List-Post: <mailto:garciaoscar@example.com> List-Id: <issues.hbase.apache.org> Delivered-To: mailing list garciaoscar@example.com Received: (qmail 28580 invoked by uid 99); 1 Mar 2011 20:22:57 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 20:22:57 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 20:22:56 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155]) by hel.zones.apache.org (Postfix) with ESMTP id D6DD44AB25 for <garciaoscar@example.com>; Tue, 1 Mar 2011 20:22:36 +0000 (UTC) Date: Tue, 1 Mar 2011 20:22:36 +0000 (UTC) From: "Jean-Daniel Cryans (JIRA)" <garciaoscar@example.com> To: garciaoscar@example.com Message-ID: <garciaoscar@example.com> In-Reply-To: <garciaoscar@example.com> Subject: [jira] Commented: (HBASE-3586) Randomize the selection of regions to balance MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HBASE-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13001093#comment-13001093 ] Jean-Daniel Cryans commented on HBASE-3586: ------------------------------------------- Ah ok that wasn't clear, well let's try it out! > Randomize the selection of regions to balance > --------------------------------------------- > > Key: HBASE-3586 > URL: https://issues.apache.org/jira/browse/HBASE-3586 > Project: HBase > Issue Type: Improvement > Affects Versions: 0.90.1 > Reporter: Jean-Daniel Cryans > Priority: Critical > Fix For: 0.90.2 > > > Currently LoadBalancer goes through the list of regions per RS and grabs the few first ones to balance. This is not bad, but that list is often sorted naturally since the a RS that boots will open the regions in a sequential and sorted order (since it comes from .META.) which means that we're balancing regions starting in an almost sorted fashion. > We discovered that because one of our internal users created a new table starting with letter "p" which has now grown to 100 regions in the last few hours and they are all served by 1 region server. Looking at the master's log, the balancer has moved as many regions from that region server but they are all from the same table that starts with letter "a" (and the regions that were moved all come one after the other). > The part of the code that should be modified is: > {code} > for (HRegionInfo hri: regions) { > // Don't rebalance meta regions. > if (hri.isMetaRegion()) continue; > regionsToMove.add(new RegionPlan(hri, serverInfo, null)); > numTaken++; > if (numTaken >= numToOffload) break; > } > {code} -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira From garciaoscar@example.com Tue Mar 01 20:25:01 2011 Return-Path: <garciaoscar@example.com> Delivered-To: garciaoscar@example.com Received: (qmail 35977 invoked from network); 1 Mar 2011 20:25:01 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Mar 2011 20:25:01 -0000 Received: (qmail 33934 invoked by uid 500); 1 Mar 2011 20:25:01 -0000 Delivered-To: garciaoscar@example.com Received: (qmail 33867 invoked by uid 500); 1 Mar 2011 20:25:00 -0000 Mailing-List: contact garciaoscar@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:garciaoscar@example.com> List-Unsubscribe: <mailto:garciaoscar@example.com> List-Post: <mailto:garciaoscar@example.com> List-Id: <issues.hbase.apache.org> Delivered-To: mailing list garciaoscar@example.com Received: (qmail 33859 invoked by uid 99); 1 Mar 2011 20:25:00 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 20:25:00 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 20:24:58 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155]) by hel.zones.apache.org (Postfix) with ESMTP id 1BD384AC15 for <garciaoscar@example.com>; Tue, 1 Mar 2011 20:24:37 +0000 (UTC) Date: Tue, 1 Mar 2011 20:24:37 +0000 (UTC) From: "Andrew Purtell (JIRA)" <garciaoscar@example.com> To: garciaoscar@example.com Message-ID: <{{EMAIL+PHONE}}> In-Reply-To: <garciaoscar@example.com> Subject: [jira] Commented: (HBASE-3582) HMaster and HRegionServer should be able to login from Kerberos keytab when running on security-enabled Hadoop MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/HBASE-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13001094#comment-13001094 ] Andrew Purtell commented on HBASE-3582: --------------------------------------- +1 I say commit it Gary. > HMaster and HRegionServer should be able to login from Kerberos keytab when running on security-enabled Hadoop > -------------------------------------------------------------------------------------------------------------- > > Key: HBASE-3582 > URL: https://issues.apache.org/jira/browse/HBASE-3582 > Project: HBase > Issue Type: Bug > Components: security > Affects Versions: 0.90.0, 0.90.1 > Reporter: Gary Helmling > > Currently HBase can run on top of Hadoop 0.20 with security APIs, but only using the "simple" authentication method. There is currently no configuration or hooks for the HBase process to obtain Kerberos credentials so it can authenticate against secure Hadoop. > We should extend the current {{org.apache.hadoop.hbase.security.User}} hooks to allow obtaining credentials from a keytab file when security is enabled. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira From garciaoscar@example.com Tue Mar 01 20:26:58 2011 Return-Path: <garciaoscar@example.com> Delivered-To: garciaoscar@example.com Received: (qmail 43970 invoked from network); 1 Mar 2011 20:26:58 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Mar 2011 20:26:58 -0000 Received: (qmail 40385 invoked by uid 500); 1 Mar 2011 20:26:58 -0000 Delivered-To: garciaoscar@example.com
{'source': 'Apache Software Foundation Public Mail Archives'}