From hbase-useralyssa14@example.net Wed Apr 01 00:48:35 2009 Return-Path: <hbase-useralyssa14@example.net> Delivered-To: alyssa14@example.net Received: (qmail 46947 invoked from network); 1 Apr 2009 00:48:35 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Apr 2009 00:48:35 -0000 Received: (qmail 97397 invoked by uid 500); 1 Apr 2009 00:48:34 -0000 Delivered-To: alyssa14@example.net Received: (qmail 97367 invoked by uid 500); 1 Apr 2009 00:48:34 -0000 Mailing-List: contact alyssa14@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:alyssa14@example.net> List-Unsubscribe: <mailto:alyssa14@example.net> List-Post: <mailto:alyssa14@example.net> List-Id: <hbase-user.hadoop.apache.org> Reply-To: alyssa14@example.net Delivered-To: mailing list alyssa14@example.net Received: (qmail 97357 invoked by uid 99); 1 Apr 2009 00:48:34 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 00:48:34 +0000 X-ASF-Spam-Status: No, hits=-4.0 required=10.0 tests=RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO emailgw01.pnl.gov) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 00:48:25 +0000 X-IronPort-AV: E=Sophos;i="4.39,304,1235980800"; d="scan'208";a="94626848" Received: from emailbh03.pnl.gov ([83.161.120.155]) by emailgw01.pnl.gov with ESMTP; 31 Mar 2009 17:47:53 -0700 Received: from EMAIL02.pnl.gov ([83.161.120.155]) by emailbh03.pnl.gov with Microsoft SMTPSVC(581)243-3561); Tue, 31 Mar 2009 17:47:53 -0700 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Subject: Novice Hbase user needs help with data upload - gets a RetriesExhaustedException, followed by NoServerForRegionException Date: Tue, 31 Mar 2009 17:47:52 -0700 Message-ID: <alyssa14@example.net> In-Reply-To: <alyssa14@example.net> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Novice Hbase user needs help with data upload - gets a RetriesExhaustedException, followed by NoServerForRegionException thread-index: AcmyURBEFse3MBBZRnWcULw8YYBv5QAD3lVg References: <alyssa14@example.net> <alyssa14@example.net> <alyssa14@example.net> <alyssa14@example.net> <alyssa14@example.net> From: "Taylor, Ronald C" <alyssa14@example.net> To: <alyssa14@example.net> Cc: "Taylor, Ronald C" <alyssa14@example.net> X-OriginalArrivalTime: 01 Apr 2009 00:47:53.0101 (UTC) FILETIME=[7D552BD0:01C9B263] X-Virus-Checked: Checked by ClamAV on apache.org =20 Hello folks, This is my first msg to the list - I just joined today, and I am a novice Hadoop/HBase programmer. I have a question: I have written a Java program to create an HBase table and then enter a number of rows into the table. The only way I have found so far to do this is to enter each row one-by-one, creating a new BatchUpdate updateObj for each row, doing about ten updateObj.put()'s to add the column data, and then doing a tableObj.commit(updateObj). There's probably a more efficient way (happy to hear, if so!), but this is what I'm starting with. When I do this on input that creates 3000 rows, the program works fine. When I try this on input that would create 300,000 rows (still relatively small for an HBase table, I would think), the program terminates around row 160,000 or so, generating first an RetriesExhaustedException, followed by NoServerForRegionException. The HBase server crashes, and I have to restart it. The Hadoop server appears to remain OK and does not need restarting. Can anybody give me any guidance? I presume that I might need to adjust some setting for larger input in the HBase and/or Hadoop config files. At present, I am using default settings. I have installed Hadoop 0.19.0 and HBase 0.19.0 in the "pseudo" cluster mode on a single machine, my Red Hat Linux desktop, which has 3 Gb RAM.=20 Any help / suggestions would be much appreciated. Cheers,=20 Ron Taylor ___________________________________________ Ronald Taylor, Ph.D. Computational Biology & Bioinformatics Group Pacific Northwest National Laboratory 902 Battelle Boulevard P.O. Box 999, MSIN K7-90 Richland, WA 99352 USA Office: (581)243-3561 Email: alyssa14@example.net www.pnl.gov From hbase-useralyssa14@example.net Wed Apr 01 01:32:49 2009 Return-Path: <hbase-useralyssa14@example.net> Delivered-To: alyssa14@example.net Received: (qmail 68196 invoked from network); 1 Apr 2009 01:32:46 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Apr 2009 01:32:46 -0000 Received: (qmail 39747 invoked by uid 500); 1 Apr 2009 01:32:45 -0000 Delivered-To: alyssa14@example.net Received: (qmail 39717 invoked by uid 500); 1 Apr 2009 01:32:45 -0000 Mailing-List: contact alyssa14@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:alyssa14@example.net> List-Unsubscribe: <mailto:alyssa14@example.net> List-Post: <mailto:alyssa14@example.net> List-Id: <hbase-user.hadoop.apache.org> Reply-To: alyssa14@example.net Delivered-To: mailing list alyssa14@example.net Received: (qmail 39707 invoked by uid 99); 1 Apr 2009 01:32:45 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 01:32:45 +0000 X-ASF-Spam-Status: No, hits=-1.0 required=10.0 tests=RCVD_IN_DNSWL_LOW,SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of alyssa14@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO ciao.gmane.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 01:32:35 +0000 Received: from list by ciao.gmane.org with local (Exim 4.43) id 1LopJL-0006lu-Bf for alyssa14@example.net; Wed, 01 Apr 2009 01:32:11 +0000 Received: from adsl-75-0-68-60.dsl.ltrkar.sbcglobal.net ([83.161.120.155]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for <alyssa14@example.net>; Wed, 01 Apr 2009 01:32:11 +0000 Received: from sales by adsl-75-0-68-60.dsl.ltrkar.sbcglobal.net with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for <alyssa14@example.net>; Wed, 01 Apr 2009 01:32:11 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: alyssa14@example.net From: "Billy Pearson" <alyssa14@example.net> Subject: Re: Curious about using HBase Date: Tue, 31 Mar 2009 20:31:56 -0500 Lines: 104 Message-ID: <alyssa14@example.net> References: <alyssa14@example.net> Mime-Version: 1.0 Content-Type: text/plain; format=flowed; charset="utf-8"; reply-type=original Content-Transfer-Encoding: 7bit X-Complaints-To: alyssa14@example.net X-Gmane-NNTP-Posting-Host: adsl-75-0-68-60.dsl.ltrkar.sbcglobal.net In-Reply-To: <alyssa14@example.net> X-MSMail-Priority: Normal X-Newsreader: Microsoft Windows Mail 6.0.6001.18000 X-MimeOLE: Produced By Microsoft MimeOLE V6.0.6001.18049 Sender: news <alyssa14@example.net> X-Virus-Checked: Checked by ClamAV on apache.org I agree with Jonathan Gray about the two tables to have a random access to both what achievement's a player has got and ALL players that have x achievement's you will need two tables Things you should look in to changing about your setup to be more efficient would be to only use a player table and cache the other stuff and use MR jobs if you do not need random access to it Example Do you reality need access to this query? What players have a given achievement? This data can be mined form the player table with a MR job for updating stats or emailing or what every you use it for internally. If not provided to players but you need it for something internal and can wait a few hours for it then a MR job can provide this data. This is the query you will need to make the achievement table and will require you to update both the players and achievement table on changes to keep everything in sync If you reality have to have this query I would still thank about going with a cached version of it that gets updated every X hours/days depending on resources. Who are the first 25 people to have a given achievement? This could be cached somehow on local disk sense it will never change once 25 have made the achievement. What are all the possible achievements? This should be easy to cache outside hbase also sense you said the new achievement will not be common. Just have to update it if you add new achievement's What achievements does a given player NOT have? Using a cache from above you could pull a players achievement's from the players table and remove the ones he has and use what's left as Jonathan Gray pointed to With the above suggestions they should help to scale with less resources. Billy Pearson "Matthew Runo" <alyssa14@example.net> wrote in message news:alyssa14@example.net... > Hello folks! > > I'm running a site (http://www.realmranker.com) that works on medium sized > data sets from the World of Warcraft armory. I'm currently doing > everything > in MySQL on a "Large" EC2 instance, but hitting some budgetary limits in > how > much more data I can put there. > > First, here is my current MySQL architecture: > > player - table that holds the player data, currently about 500,000 rows > (expecting 2,000,000+ rows long term) > achievement - table that holds the root achievement data, currently a few > 1000 rows, not expecting all that much growth > player_achievement - table that maps a player to all the achievement's > they've completed. Currentl about 60,000,000 rows, expecting 250,000,000+ > long term > > Here is the layout of player_achievment right now: > CREATE TABLE `player_achievement` ( > `playerAchievementId` int(11) unsigned NOT NULL AUTO_INCREMENT, > `playerId` int(11) unsigned NOT NULL, > `achievementId` int(11) unsigned NOT NULL, > `guildId` int(11) unsigned DEFAULT NULL, > `dateCompleted` varchar(100) NOT NULL, > PRIMARY KEY (`playerAchievementId`), > UNIQUE KEY `playerId` (`playerId`,`achievementId`), > KEY `playerId_2` (`playerId`), > KEY `achievementId` (`achievementId`), > KEY `guildId` (`guildId`), > KEY `dateCompleted` (`dateCompleted`) > ) > > I'd like to move this table into HBase, since it's simply getting too > large > for the amount of server resources I can afford. I assume that HBase would > preform faster than MySQL on my (small by HBase standards) dataset. > > My question is simply what is the best way to do this? I'm assuming > something like this: > > player_achievement table > achievementId row > players family > player1, player2, player3.....playerN columns > date completed would be the data under each column > > I need to do the following types of queries: > - What players have a given achievement? > - What achievements does a give player have? > - What achievements does a given player NOT have? > - What are all the possible achievements? > - Who are the first 25 people to have a given achievement? > > I'm sorry for the really high level of this email - I'm still getting up > to > speed on HBase, and not quite sure it's exactly the answer to my prayers > that I think it might be.. > > --Matthew Runo > From hbase-useralyssa14@example.net Wed Apr 01 02:25:59 2009 Return-Path: <hbase-useralyssa14@example.net> Delivered-To: alyssa14@example.net Received: (qmail 97887 invoked from network); 1 Apr 2009 02:25:59 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Apr 2009 02:25:59 -0000 Received: (qmail 75034 invoked by uid 500); 1 Apr 2009 02:25:58 -0000 Delivered-To: alyssa14@example.net Received: (qmail 74986 invoked by uid 500); 1 Apr 2009 02:25:58 -0000 Mailing-List: contact alyssa14@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:alyssa14@example.net> List-Unsubscribe: <mailto:alyssa14@example.net> List-Post: <mailto:alyssa14@example.net> List-Id: <hbase-user.hadoop.apache.org> Reply-To: alyssa14@example.net Delivered-To: mailing list alyssa14@example.net Received: (qmail 74976 invoked by uid 99); 1 Apr 2009 02:25:58 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 02:25:58 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of alyssa14@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO wa-out-1112.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 02:25:52 +0000 Received: by wa-out-1112.google.com with SMTP id j32so1751411waf.29 for <alyssa14@example.net>; Tue, 31 Mar 2009 19:25:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=5TaI2UlhN7zz72RGpG8S2l2NfiETkS4u/hicF8hpYFw=; b=GypnhB8Vp8Vh1v6aDfuhU+yDksGljNclEa0rRSs9TGwYDXEI4RaerbpoArmdoaYJRr RyXhYOjqdyrGHGvICh00N7Cks8tE2UF/jpDTnQStb2P3/YFJMjtzyxu7qbM6KdzPvY/X 3RHhuQ0gsyvhrR81Z7yHiOq0hy/hzKhjFGM6Y= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=WRxZesqEocYE48/E0rcDoCbIHNNuxS6vUPgADDYm199MIHaXHJQvH6c/BWarcq7aD/ RpKANdVavnUOf33yoqy2rbDJBh2yW5dVdy+cDtNPvosILF9BfatrX4EtPLqA+tyqbP7e YipGz8508LCayCezN4j5JAAWeFyxgRhsXfMls= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id q15mr4825724waf.199.1238552732106; Tue, 31 Mar 2009 19:25:32 -0700 (PDT) Date: Tue, 31 Mar 2009 19:25:32 -0700 Message-ID: <alyssa14@example.net> Subject: financial time series database From: Bradford Cross <alyssa14@example.net> To: alyssa14@example.net Content-Type: multipart/alternative; boundary=00163646ccd6a202640466750711 X-Virus-Checked: Checked by ClamAV on apache.org --00163646ccd6a202640466750711 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Greetings, I am prototyping a financial time series database on top of HBase and trying to head my head around what a good design would look like. As I understand it, I have rows, column families, columns and cells. Since the only think that Hbase really "indexes" is row keys, it seems natural in a way to represent the rowkeys as the date/time. As a simple example: Bar data: { "2009/1/17" : { "open":"100", "high":"102", "low":"99", "close":"101" "volume":"1000256" } } Quote data: { "2009/1/17:11:23:04" : { "bid":"100.01", "ask":"100.02", "bidsize":"10000", "asksize":"100200" } } But there are many other issues to think about. In financial time series data we have small amounts of data within each "observation" and we can have lots of observations. We can have millions of observations per time series (f.ex. all historical trade and quote date for a particular stock since 1993)across hundreds of thousands of individual instruments (f.ex. across all stocks that have traded since 1993.) The write patterns fit HBase nicely, because it is a write once and append pattern. This is followed by loads of offline processes for simulating trading models and such. These query patterns look like "all quotes for all stocks between the dates of 1/1/996 and 12/31/2008." So the querying is typically across a date range, and we can further filter the query by instrument types. So I am not sure what makes sense for efficiency because I do not understand HBase well enough yet. What kinds of mixes of rows, column families, and columns should I be thinking about? Does my simplistic approach make any sense? That would mean each row is a key-value pair where the key is is the date/time and the value is the "observation." I suppose this leads to a "table per time series" model. Does that make sense or is there overhead to having lots of tables? --00163646ccd6a202640466750711-- From hbase-useralyssa14@example.net Wed Apr 01 03:03:27 2009 Return-Path: <hbase-useralyssa14@example.net> Delivered-To: alyssa14@example.net Received: (qmail 4069 invoked from network); 1 Apr 2009 03:03:27 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Apr 2009 03:03:27 -0000 Received: (qmail 792 invoked by uid 500); 1 Apr 2009 03:03:25 -0000 Delivered-To: alyssa14@example.net Received: (qmail 723 invoked by uid 500); 1 Apr 2009 03:03:25 -0000 Mailing-List: contact alyssa14@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:alyssa14@example.net> List-Unsubscribe: <mailto:alyssa14@example.net> List-Post: <mailto:alyssa14@example.net> List-Id: <hbase-user.hadoop.apache.org> Reply-To: alyssa14@example.net Delivered-To: mailing list alyssa14@example.net Received: (qmail 713 invoked by uid 99); 1 Apr 2009 03:03:25 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 03:03:25 +0000 X-ASF-Spam-Status: No, hits=1.2 required=10.0 tests=SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO rv-out-0506.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 03:03:16 +0000 Received: by rv-out-0506.google.com with SMTP id g37so1951390rvb.5 for <alyssa14@example.net>; Tue, 31 Mar 2009 20:02:55 -0700 (PDT) MIME-Version: 1.0 Sender: alyssa14@example.net Received: by 83.161.120.155 with SMTP id f20mr3783922rvl.94.1238554975479; Tue, 31 Mar 2009 20:02:55 -0700 (PDT) In-Reply-To: <alyssa14@example.net> References: <alyssa14@example.net> <alyssa14@example.net> <alyssa14@example.net> Date: Wed, 1 Apr 2009 12:02:55 +0900 X-Google-Sender-Auth: feed00d7d69c4488 Message-ID: <alyssa14@example.net> Subject: Re: Schema to store graph From: "Edward J. Yoon" <alyssa14@example.net> To: alyssa14@example.net Cc: alyssa14@example.net, alyssa14@example.net Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org One thing is Hbase 0.19 doesn't work with over 5,000 qualifier of one column so I couldn't test/benchmark for large scale. On Tue, Mar 31, 2009 at 6:04 PM, Amandeep Khurana <alyssa14@example.net> wrote: > Response below > > > Amandeep Khurana > Computer Science Graduate Student > University of California, Santa Cruz > > > On Tue, Mar 31, 2009 at 1:58 AM, Edward J. Yoon <alyssa14@example.net>wrote: > >> Hama store the sparse graph using Hbase as an sparse adjacency matrix. >> One of reason is to perform matrix decomposition for large sparse >> graphs. Anyway, I guess If you store the graph like that, you'll only >> need update the row 'v/w' to add v to w's/w to v's list of neighbors. > > > I didnt quite understand the last line here. > > I did think of a sparse matrix as well but not sure which is a better > approach. Thats why I posted here... > > Share about your experiences with Hama... > >> >> >> Just FYI, You also may want to see -- >> http://blog.udanax.org/2009/02/breadth-first-search-mapreduce.html >> >> If you have any advice for us, Pls let us know. >> >> On Tue, Mar 31, 2009 at 5:09 PM, Amandeep Khurana <alyssa14@example.net> >> wrote: >> > What would be a good schema in HBase to store information pertaining to a >> > many to many graph? I was thinking of having the node id as the row key, >> the >> > type of relation as the column family, the relation name for the column >> > identifier and the actual cell containing the key of the node that is >> being >> > connected with. >> > >> > >> > Amandeep Khurana >> > Computer Science Graduate Student >> > University of California, Santa Cruz >> > >> >> >> >> -- >> Best Regards, Edward J. Yoon >> alyssa14@example.net >> http://blog.udanax.org >> > -- Best Regards, Edward J. Yoon alyssa14@example.net http://blog.udanax.org From hbase-useralyssa14@example.net Wed Apr 01 03:09:19 2009 Return-Path: <hbase-useralyssa14@example.net> Delivered-To: alyssa14@example.net Received: (qmail 5109 invoked from network); 1 Apr 2009 03:09:18 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Apr 2009 03:09:18 -0000 Received: (qmail 8841 invoked by uid 500); 1 Apr 2009 03:09:18 -0000 Delivered-To: alyssa14@example.net Received: (qmail 8786 invoked by uid 500); 1 Apr 2009 03:09:17 -0000 Mailing-List: contact alyssa14@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:alyssa14@example.net> List-Unsubscribe: <mailto:alyssa14@example.net> List-Post: <mailto:alyssa14@example.net> List-Id: <hbase-user.hadoop.apache.org> Reply-To: alyssa14@example.net Delivered-To: mailing list alyssa14@example.net Received: (qmail 8776 invoked by uid 99); 1 Apr 2009 03:09:17 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 03:09:17 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of alyssa14@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO qw-out-2122.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 03:09:09 +0000 Received: by qw-out-2122.google.com with SMTP id 5so2045630qwi.35 for <alyssa14@example.net>; Tue, 31 Mar 2009 20:08:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=cAQ9pwzH/svGcYYJFtCXfcVb85pwYXW4S+OFKZrUn5s=; b=PyYx3P4ZT7x50565fSqZYKm85X1BZ/HUkd8aF9ckBtrBN2Wb1G/wM4YHzySnbpUtAP GUMdNyrfE536LCw7okngfyHlh8lWwYuIp3uqLf89KnEmfxomjn9EuKiL2Vo65018jdrz YwdmxDlCugL9U5haldBcGlDVEGk3Ioejc9AZA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=d6Z78UuVvJPF4FthhaCIZfiDebuE/PU3OiPKLaOvAJ7KD7jtdIwvYvsL/enfaG2rXu OLBvNYQoedAkkBqCtHfmp8MwYnjnDemeJcAvZakkG5jQUdhhPzXCSd4tebxIHFU+UQyG HJn80zm/9te90XwpAeg9N5fL6WusIFrZ7+d0Y= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id m2mr2981950qcn.21.1238555326273; Tue, 31 Mar 2009 20:08:46 -0700 (PDT) In-Reply-To: <alyssa14@example.net> References: <alyssa14@example.net> Date: Wed, 1 Apr 2009 11:08:46 +0800 Message-ID: <alyssa14@example.net> Subject: Re: financial time series database From: zsongbo <alyssa14@example.net> To: alyssa14@example.net Content-Type: multipart/alternative; boundary=0016368324ea41d8e2046675a28d X-Virus-Checked: Checked by ClamAV on apache.org --0016368324ea41d8e2046675a28d Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit If the rowkey is date/time and the data is original sequential by date/time, when load/insert data into table, only one region (the one node) is active to receive new data. The load performance will be pool. On Wed, Apr 1, 2009 at 10:25 AM, Bradford Cross <alyssa14@example.net>wrote: > Greetings, > > I am prototyping a financial time series database on top of HBase and > trying > to head my head around what a good design would look like. > > As I understand it, I have rows, column families, columns and cells. > > Since the only think that Hbase really "indexes" is row keys, it seems > natural in a way to represent the rowkeys as the date/time. > > As a simple example: > > Bar data: > > { > "2009/1/17" : { > "open":"100", > "high":"102", > "low":"99", > "close":"101" > "volume":"1000256" > } > } > > > Quote data: > > { > "2009/1/17:11:23:04" : { > "bid":"100.01", > "ask":"100.02", > "bidsize":"10000", > "asksize":"100200" > } > } > > But there are many other issues to think about. > > In financial time series data we have small amounts of data within each > "observation" and we can have lots of observations. We can have millions > of > observations per time series (f.ex. all historical trade and quote date for > a particular stock since 1993)across hundreds of thousands of individual > instruments (f.ex. across all stocks that have traded since 1993.) > > The write patterns fit HBase nicely, because it is a write once and append > pattern. This is followed by loads of offline processes for simulating > trading models and such. These query patterns look like "all quotes for > all > stocks between the dates of 1/1/996 and 12/31/2008." So the querying is > typically across a date range, and we can further filter the query by > instrument types. > > So I am not sure what makes sense for efficiency because I do not > understand > HBase well enough yet. > > What kinds of mixes of rows, column families, and columns should I be > thinking about? > > Does my simplistic approach make any sense? That would mean each row is a > key-value pair where the key is is the date/time and the value is the > "observation." I suppose this leads to a "table per time series" model. > Does that make sense or is there overhead to having lots of tables? > --0016368324ea41d8e2046675a28d-- From hbase-useralyssa14@example.net Wed Apr 01 03:09:37 2009 Return-Path: <hbase-useralyssa14@example.net> Delivered-To: alyssa14@example.net Received: (qmail 5171 invoked from network); 1 Apr 2009 03:09:37 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Apr 2009 03:09:37 -0000 Received: (qmail 9261 invoked by uid 500); 1 Apr 2009 03:09:36 -0000 Delivered-To: alyssa14@example.net Received: (qmail 9235 invoked by uid 500); 1 Apr 2009 03:09:36 -0000 Mailing-List: contact alyssa14@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:alyssa14@example.net> List-Unsubscribe: <mailto:alyssa14@example.net> List-Post: <mailto:alyssa14@example.net> List-Id: <hbase-user.hadoop.apache.org> Reply-To: alyssa14@example.net Delivered-To: mailing list alyssa14@example.net Received: (qmail 9225 invoked by uid 99); 1 Apr 2009 03:09:36 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 03:09:36 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of alyssa14@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-qy0-f134.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 03:09:29 +0000 Received: by qyk40 with SMTP id 40so7195844qyk.5 for <alyssa14@example.net>; Tue, 31 Mar 2009 20:09:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=RVEV/Uf1+WAlLB0pwO0DoemkWVIINalocs2gu3M5PYw=; b=xNPwhyu/Ic9oDF4obUlBFrtOMk/DLUnSVoShp4pqKUaBF60cC7+HE62JkCNrbgcqP5 IW5M/HEKYZGkSNFo5LXZDrX9qSONjGJbQulbcabJAdAtLTYNJoJL+/XRB1jsMAjACyYC gLOsMeN4gFM+WR1tI5a8OreA/rew54iR+s8k8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=HTiErI7SWH0ESeNEDP4F8IhITRmtS/bBtSMNE6JCSJcaFFKZ+wZ2mQwcw7P3pc1oJ7 kIxesPHe3BEW5Y4s+yDWF3+WupbTlzqS6rItNdUUPs8z/nWSwjV9zGgwvv3zewAGVM8B DP1Kxlo5NS3Rgl5CLrHuE3h15xEOogUcu3xNg= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id m4mr3032237qcl.55.1238555348876; Tue, 31 Mar 2009 20:09:08 -0700 (PDT) In-Reply-To: <alyssa14@example.net> References: <alyssa14@example.net> <alyssa14@example.net> Date: Wed, 1 Apr 2009 11:09:08 +0800 Message-ID: <alyssa14@example.net> Subject: Re: financial time series database From: zsongbo <alyssa14@example.net> To: alyssa14@example.net Content-Type: multipart/alternative; boundary=001636aa2c4c9abfd6046675a364 X-Virus-Checked: Checked by ClamAV on apache.org --001636aa2c4c9abfd6046675a364 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit If the rowkey is date/time and the data is original sequential by date/time, when load/insert data into table, only one region (the one node) is active to receive new data. The load performance will be bad. On Wed, Apr 1, 2009 at 11:08 AM, zsongbo <alyssa14@example.net> wrote: > If the rowkey is date/time and the data is original sequential by > date/time, when load/insert data into table, only one region (the one > node) is active to receive new data. The load performance will be pool. > > > On Wed, Apr 1, 2009 at 10:25 AM, Bradford Cross < > alyssa14@example.net> wrote: > >> Greetings, >> >> I am prototyping a financial time series database on top of HBase and >> trying >> to head my head around what a good design would look like. >> >> As I understand it, I have rows, column families, columns and cells. >> >> Since the only think that Hbase really "indexes" is row keys, it seems >> natural in a way to represent the rowkeys as the date/time. >> >> As a simple example: >> >> Bar data: >> >> { >> "2009/1/17" : { >> "open":"100", >> "high":"102", >> "low":"99", >> "close":"101" >> "volume":"1000256" >> } >> } >> >> >> Quote data: >> >> { >> "2009/1/17:11:23:04" : { >> "bid":"100.01", >> "ask":"100.02", >> "bidsize":"10000", >> "asksize":"100200" >> } >> } >> >> But there are many other issues to think about. >> >> In financial time series data we have small amounts of data within each >> "observation" and we can have lots of observations. We can have millions >> of >> observations per time series (f.ex. all historical trade and quote date >> for >> a particular stock since 1993)across hundreds of thousands of individual >> instruments (f.ex. across all stocks that have traded since 1993.) >> >> The write patterns fit HBase nicely, because it is a write once and append >> pattern. This is followed by loads of offline processes for simulating >> trading models and such. These query patterns look like "all quotes for >> all >> stocks between the dates of 1/1/996 and 12/31/2008." So the querying is >> typically across a date range, and we can further filter the query by >> instrument types. >> >> So I am not sure what makes sense for efficiency because I do not >> understand >> HBase well enough yet. >> >> What kinds of mixes of rows, column families, and columns should I be >> thinking about? >> >> Does my simplistic approach make any sense? That would mean each row is a >> key-value pair where the key is is the date/time and the value is the >> "observation." I suppose this leads to a "table per time series" model. >> Does that make sense or is there overhead to having lots of tables? >> > > --001636aa2c4c9abfd6046675a364-- From hbase-useralyssa14@example.net Wed Apr 01 04:54:41 2009 Return-Path: <hbase-useralyssa14@example.net> Delivered-To: alyssa14@example.net Received: (qmail 18664 invoked from network); 1 Apr 2009 04:54:41 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Apr 2009 04:54:41 -0000 Received: (qmail 77420 invoked by uid 500); 1 Apr 2009 04:54:40 -0000 Delivered-To: alyssa14@example.net Received: (qmail 77361 invoked by uid 500); 1 Apr 2009 04:54:40 -0000 Mailing-List: contact alyssa14@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:alyssa14@example.net> List-Unsubscribe: <mailto:alyssa14@example.net> List-Post: <mailto:alyssa14@example.net> List-Id: <hbase-user.hadoop.apache.org> Reply-To: alyssa14@example.net Delivered-To: mailing list alyssa14@example.net Received: (qmail 77351 invoked by uid 99); 1 Apr 2009 04:54:40 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 04:54:40 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of alyssa14@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO qw-out-2122.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 04:54:33 +0000 Received: by qw-out-2122.google.com with SMTP id 5so2069614qwi.35 for <alyssa14@example.net>; Tue, 31 Mar 2009 21:54:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=VYDDFW2G4BwhL2tqzKe27E0tFP+dzqIKS5urHwoAxOU=; b=YIgSeahK2UjrWsqHGhfOyIS1EsqMg4esJpguKVlVD95YQ4svTW8kXZ/LALyz10i1lc WhGR1w+muQxoLayB4PLGW9Ggba2syRj43jaRB33hDjVScmkYZEgUQqt94eAaH6YcRO1z wXUGgt/0jrC2KolTlJ6lY/jwMXzJG+WMDe3LU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=TiBkDAtVfbrSMofEgfwora5l7p4Y34lyjScVqJJkBFQjCKc0NhUho0fgBy7hbEq1KP J0HhlkKFbGs2V1/WFTeuFOATdmb8qxQEPZ0bE3PyxY4YFT4V0PHco12SoPlIJBiz6Nuq rxBkX68L3B7sayJRj1nRzO/yOpa9ZhMqFvxyM= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id r2mr2944449qcj.29.1238561652532; Tue, 31 Mar 2009 21:54:12 -0700 (PDT) In-Reply-To: <alyssa14@example.net> References: <alyssa14@example.net> <alyssa14@example.net> <alyssa14@example.net> <alyssa14@example.net> <alyssa14@example.net> <alyssa14@example.net> <alyssa14@example.net> Date: Wed, 1 Apr 2009 12:54:12 +0800 Message-ID: <alyssa14@example.net> Subject: Re: HDFS unbalance issue. (HBase over HDFS) From: zsongbo <alyssa14@example.net> To: alyssa14@example.net Content-Type: multipart/alternative; boundary=0016364ee12a54e7890466771b9e X-Virus-Checked: Checked by ClamAV on apache.org --0016364ee12a54e7890466771b9e Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Thanks stack. (i am schubert) Yes, I have found the fix in 0.20. And I just made a temporary fix based on branch-0.19 and it work fine. On Mon, Mar 30, 2009 at 7:16 PM, stack <alyssa14@example.net> wrote: > Thanks for doing the digging Schubert. I agree, its an ugly issue. > Another > gentleman reported he'd tripped over the same thing in private mail. I'd > suggest you file an issue against HADOOP HDFS and add the below (I"ve > opened > HBASE-1296 so we can track it in our project). > > Good stuff, > St.Ack > > On Fri, Mar 27, 2009 at 9:50 AM, schubert zhang <alyssa14@example.net> wrote: > > > Sorry "But I found the namenode is fair to process the invalidating for > > each > > datanode." > > should be: > > > > "But I found the namenode is unfair to process the invalidating for each > > datanode." > > > > > > On Fri, Mar 27, 2009 at 3:49 PM, schubert zhang <alyssa14@example.net> > wrote: > > > > > Thanks Samuel, > > > Your information is very correct. > > > I have also read code about garbage collection of invalidating blocks. > > > > > > But I found the namenode is fair to process the invalidating for each > > > datanode. > > > In my cluster, there are 5 datanode. The storage IDs are: > > > > > > node1: DS- 978762906-83.161.120.155-50010-1237686434530 > > > node2: DS- 489086185-83.161.120.155-50010-1237686416330 > > > node3: DS-(581)243-3561.161.120.155-50010-1237686426395 > > > node4: DS-1024388083-83.161.120.155-50010-1237686404482 > > > node5: DS-2136798339-83.161.120.155-50010-1237686444430 > > > I know the storage ID is generated > > > by > org.apache.hadoop.hdfs.server.datanode.DataNode.setNewStorageID(...). > > > > > > In org.apache.hadoop.hdfs.server.namenode.FSNamesystem > > > > > > // Keeps a Collection for every named machine containing > > > // blocks that have recently been invalidated and are thought to live > > > // on the machine in question. > > > // Mapping: StorageID -> ArrayList<Block> > > > // > > > private Map<String, Collection<Block>> recentInvalidateSets = > > > new TreeMap<String, Collection<Block>>(); > > > > > > In > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.ReplicationMonitor > > > This thread run in interval: replicationRecheckInterval=3000 > > milliseconds. > > > > > > Into computeDatanodeWork() > > > nodesToProcess = 2. > > > > > > then into computeInvalidateWork(nodesToProcess) > > > the for cycle will only exec 2 cycles. > > > > > > for each cycle, go into invalidateWorkForOneNode() > > > it will always get the first node to invalidate blocks on this node. > > > String firstNodeId = recentInvalidateSets.keySet().iterator().next(); > > > > > > TreeMap is a stored map, so, the ketSet is: > > > [1024388083-83.161.120.155-50010-1237686404482, > > > (581)243-3561.161.120.155-50010-1237686426395, > > > 2136798339-83.161.120.155-50010-1237686444430, > > > 489086185-83.161.120.155-50010-1237686416330, > > > 978762906-83.161.120.155-50010-1237686434530] > > > > > > So, the sequence of node list in recentInvalidateSets is: > > > [node4, node3, node5, node2, node1] > > > > > > So, every time in invalidateWorkForOneNode(), it will always process > > node4 > > > then node3, then node2 and then node1. > > > > > > My application is a HBase write-heavy application. > > > So, there is many blocks need invalidate in each datanode. So when each > > > 3000 milliseconds, at most, there is only two datanode is processed. > > Since > > > the node1 is the last one in the TreeMap, it have no change to be > garbage > > > collected. > > > > > > I think HDFS namenode should fix this issue. > > > > > > Schubert > > > > > > On Thu, Mar 26, 2009 at 2:57 PM, Samuel Guo <alyssa14@example.net> > wrote: > > > > > >> After a file is deleted, HDFS does not immediately reclaim the > available > > >> physical storage. It does so only lazily during garbage collection. > When > > a > > >> file is deleted by the application, the master remove the file's > > metadata > > >> from *FSNamesystem* and logs the deletion immediately. And the file's > > >> deleted blocks information will be collected in each > > DataNodeDescriptor's > > >> *invalidateBlocks* set in Namenode. During the heartbeats between NN > and > > >> DN, > > >> NN will scan the specified DN's DataNodeDescriptor's invalidateBlocks > > set, > > >> find the blocks to be deleted in DN and send a *DNA_INVALIDATE* > > >> BlockCommand > > >> to DN. And the *BlockScanner* thread running on DN will scan, find and > > >> delete these blocks after DN receives the *DNA_INVALIDATE* > BlockCommand. > > >> > > >> You can search *DNA_INVALIDATE* in DataNode.java and NameNode.java > > files, > > >> and find the logic of the garbage collection. Hope it will be helpful. > > >> > > >> On Thu, Mar 26, 2009 at 11:07 AM, schubert zhang <alyssa14@example.net> > > >> wrote: > > >> > > >> > Tanks Andrew and Billy. > > >> > I think the subject of this mail thread is not appropriate, it may > not > > >> be a > > >> > balance issue. > > >> > The problem seems the block deleting scheduler in HDFS. > > >> > > > >> > Last night(timezone:+8), I slow down my application, and this > morning, > > I > > >> > found almost all garbage blocks are deleted. > > >> > Here is the current blocks number of each datanode: > > >> > node1: 10651 > > >> > node2: 10477 > > >> > node3: 12185 > > >> > node4: 11607 > > >> > node5: 14000 > > >> > > > >> > It seems fine. > > >> > But I want to study the code of HDFS and make clear the policy of > > >> deleting > > >> > blocks on datanodes. If anyone in the hadoop community can give me > > some > > >> > advices? > > >> > > > >> > Schubert > > >> > > > >> > On Thu, Mar 26, 2009 at 7:55 AM, Andrew Purtell < > alyssa14@example.net> > > >> > wrote: > > >> > > > >> > > > >> > > > > >> > > > From: schubert zhang <alyssa14@example.net> > > >> > > > From another point of view, I think HBase cannot control to > > >> > > > delete blocks on which node, it would just delete files, and > > >> > > > HDFS delete blocks where the blocks locating. > > >> > > > > >> > > Yes, that is exactly correct. > > >> > > > > >> > > Best regards, > > >> > > > > >> > > - Andy > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > > > > > > --0016364ee12a54e7890466771b9e-- From hbase-useralyssa14@example.net Wed Apr 01 07:41:39 2009 Return-Path: <hbase-useralyssa14@example.net> Delivered-To: alyssa14@example.net Received: (qmail 29124 invoked from network); 1 Apr 2009 07:41:38 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Apr 2009 07:41:38 -0000 Received: (qmail 30689 invoked by uid 500); 1 Apr 2009 07:41:38 -0000 Delivered-To: alyssa14@example.net Received: (qmail 30633 invoked by uid 500); 1 Apr 2009 07:41:38 -0000 Mailing-List: contact alyssa14@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:alyssa14@example.net> List-Unsubscribe: <mailto:alyssa14@example.net> List-Post: <mailto:alyssa14@example.net> List-Id: <hbase-user.hadoop.apache.org> Reply-To: alyssa14@example.net Delivered-To: mailing list alyssa14@example.net Received: (qmail 30623 invoked by uid 99); 1 Apr 2009 07:41:38 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 07:41:37 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of alyssa14@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO yw-out-2324.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 07:41:29 +0000 Received: by yw-out-2324.google.com with SMTP id 2so1806174ywt.29 for <alyssa14@example.net>; Wed, 01 Apr 2009 00:41:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:in-reply-to :references:date:x-google-sender-auth:message-id:subject:from:to:cc :content-type; bh=JCeFz3s7HeKNxJBlgbjEzMyoZtiuCpNx3NZNz9GPwUo=; b=IphAJgU94X+f3+R6QjqcyvBURauU5i0AXvCs5o6AvGjJuxvAwVrFKbMJCL3KhRrkEO X8m+midG2jP01Ifr/dQRZ3FGgiFPl+0gcz7rREkRfIguHMMV9ZWa82UWlTkBMF5u+Yqh LGAPrCYLxPj5gjp6lCeBmqCx/AlEZut5K4P1s= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; b=GCWLFCytT8ncXpNHgdyJS1EahYKixrFiEz6QosR3vwd+9NlEcdvbmuSmLkH16Zr6Gk tlb2yLkb/oNSGj0ntk9m+3O7jMddW3aOQifhRMG//hMJ/TjReoRV+6GEUmWw6mmJ+n0s oRBHnWy5gq6b02YWU7K6+QXOusSsj2K7cNhpE= MIME-Version: 1.0 Sender: alyssa14@example.net Received: by 83.161.120.155 with SMTP id 13mr2756437ang.10.1238571668098; Wed, 01 Apr 2009 00:41:08 -0700 (PDT) In-Reply-To: <alyssa14@example.net> References: <alyssa14@example.net> <alyssa14@example.net> <alyssa14@example.net> <alyssa14@example.net> Date: Wed, 1 Apr 2009 09:41:08 +0200 X-Google-Sender-Auth: 0ed35845842dd25b Message-ID: <alyssa14@example.net> Subject: Re: Schema to store graph From: stack <alyssa14@example.net> To: alyssa14@example.net Cc: alyssa14@example.net, alyssa14@example.net Content-Type: multipart/alternative; boundary=0016e644dd444e4f47046679708d X-Virus-Checked: Checked by ClamAV on apache.org --0016e644dd444e4f47046679708d Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Edward is referring to https://issues.apache.org/jira/browse/HBASE-867. We need to fix it for 0.20.0 hbase release. St.Ack On Wed, Apr 1, 2009 at 5:02 AM, Edward J. Yoon <alyssa14@example.net>wrote: > One thing is Hbase 0.19 doesn't work with over 5,000 qualifier of one > column so I couldn't test/benchmark for large scale. > > On Tue, Mar 31, 2009 at 6:04 PM, Amandeep Khurana <alyssa14@example.net> > wrote: > > Response below > > > > > > Amandeep Khurana > > Computer Science Graduate Student > > University of California, Santa Cruz > > > > > > On Tue, Mar 31, 2009 at 1:58 AM, Edward J. Yoon <alyssa14@example.net > >wrote: > > > >> Hama store the sparse graph using Hbase as an sparse adjacency matrix. > >> One of reason is to perform matrix decomposition for large sparse > >> graphs. Anyway, I guess If you store the graph like that, you'll only > >> need update the row 'v/w' to add v to w's/w to v's list of neighbors. > > > > > > I didnt quite understand the last line here. > > > > I did think of a sparse matrix as well but not sure which is a better > > approach. Thats why I posted here... > > > > Share about your experiences with Hama... > > > >> > >> > >> Just FYI, You also may want to see -- > >> http://blog.udanax.org/2009/02/breadth-first-search-mapreduce.html > >> > >> If you have any advice for us, Pls let us know. > >> > >> On Tue, Mar 31, 2009 at 5:09 PM, Amandeep Khurana <alyssa14@example.net> > >> wrote: > >> > What would be a good schema in HBase to store information pertaining > to a > >> > many to many graph? I was thinking of having the node id as the row > key, > >> the > >> > type of relation as the column family, the relation name for the > column > >> > identifier and the actual cell containing the key of the node that is > >> being > >> > connected with. > >> > > >> > > >> > Amandeep Khurana > >> > Computer Science Graduate Student > >> > University of California, Santa Cruz > >> > > >> > >> > >> > >> -- > >> Best Regards, Edward J. Yoon > >> alyssa14@example.net > >> http://blog.udanax.org > >> > > > > > > -- > Best Regards, Edward J. Yoon > alyssa14@example.net > http://blog.udanax.org > --0016e644dd444e4f47046679708d-- From hbase-useralyssa14@example.net Wed Apr 01 07:45:03 2009 Return-Path: <hbase-useralyssa14@example.net> Delivered-To: alyssa14@example.net Received: (qmail 30638 invoked from network); 1 Apr 2009 07:45:03 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Apr 2009 07:45:03 -0000 Received: (qmail 32637 invoked by uid 500); 1 Apr 2009 07:45:02 -0000 Delivered-To: alyssa14@example.net Received: (qmail 32579 invoked by uid 500); 1 Apr 2009 07:45:02 -0000 Mailing-List: contact alyssa14@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:alyssa14@example.net> List-Unsubscribe: <mailto:alyssa14@example.net> List-Post: <mailto:alyssa14@example.net> List-Id: <hbase-user.hadoop.apache.org> Reply-To: alyssa14@example.net Delivered-To: mailing list alyssa14@example.net Received: (qmail 32569 invoked by uid 99); 1 Apr 2009 07:45:02 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 07:45:02 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of alyssa14@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO yx-out-2324.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 07:44:55 +0000 Received: by yx-out-2324.google.com with SMTP id 3so1805125yxj.29 for <alyssa14@example.net>; Wed, 01 Apr 2009 00:44:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:in-reply-to :references:date:x-google-sender-auth:message-id:subject:from:to :content-type; bh=o/50rdYTkFPAkQKZq1N1UUNY0lIAZbd76ns9r2QTdJs=; b=m4GO3JrAxE6EfWhhNO4AdzaceH2cFLXjDHBKAUobPWVujYoSicdmsRsepzUKo5pQH4 B7JZoJrQJetw6BezqunR+NPw48QQNhTFbFGrMyOrw9xMVpwGtK9RSHE+FQHFRcguxAfn 6G6KjlJMSgKgJbAUnsSq0P2lMXMwPYDEPmJBA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type; b=eUhOxmrOK4umrbWQHJYpEuDQDl7HW8hJp3ovBbjF4tkdyhzIrK8Un08Oa9fIGBb8rs 3Iq40B7dwP5UhH7ZyjJA2eaYqW3U96fXkKODGDefzvIr9lcBsVF1u/ho+ulzkqnGtjYQ PH0G6ZT37xGi5rkvZjS/ILGo0KGkj5lbfl25w= MIME-Version: 1.0 Sender: alyssa14@example.net Received: by 83.161.120.155 with SMTP id t16mr2106506ane.29.1238571874972; Wed, 01 Apr 2009 00:44:34 -0700 (PDT) In-Reply-To: <alyssa14@example.net> References: <alyssa14@example.net> <alyssa14@example.net> <alyssa14@example.net> <alyssa14@example.net> <alyssa14@example.net> <alyssa14@example.net> <alyssa14@example.net> <alyssa14@example.net> Date: Wed, 1 Apr 2009 09:44:34 +0200 X-Google-Sender-Auth: 2166d1ff20ed6ee7 Message-ID: <alyssa14@example.net> Subject: Re: HDFS unbalance issue. (HBase over HDFS) From: stack <alyssa14@example.net> To: alyssa14@example.net Content-Type: multipart/alternative; boundary=0016368e2089a2f6340466797cb5 X-Virus-Checked: Checked by ClamAV on apache.org --0016368e2089a2f6340466797cb5 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit What was the fix? Thanks, St.Ack On Wed, Apr 1, 2009 at 6:54 AM, zsongbo <alyssa14@example.net> wrote: > Thanks stack. (i am schubert) > Yes, I have found the fix in 0.20. > And I just made a temporary fix based on branch-0.19 and it work fine. > > On Mon, Mar 30, 2009 at 7:16 PM, stack <alyssa14@example.net> wrote: > > > Thanks for doing the digging Schubert. I agree, its an ugly issue. > > Another > > gentleman reported he'd tripped over the same thing in private mail. I'd > > suggest you file an issue against HADOOP HDFS and add the below (I"ve > > opened > > HBASE-1296 so we can track it in our project). > > > > Good stuff, > > St.Ack > > > > On Fri, Mar 27, 2009 at 9:50 AM, schubert zhang <alyssa14@example.net> > wrote: > > > > > Sorry "But I found the namenode is fair to process the invalidating for > > > each > > > datanode." > > > should be: > > > > > > "But I found the namenode is unfair to process the invalidating for > each > > > datanode." > > > > > > > > > On Fri, Mar 27, 2009 at 3:49 PM, schubert zhang <alyssa14@example.net> > > wrote: > > > > > > > Thanks Samuel, > > > > Your information is very correct. > > > > I have also read code about garbage collection of invalidating > blocks. > > > > > > > > But I found the namenode is fair to process the invalidating for each > > > > datanode. > > > > In my cluster, there are 5 datanode. The storage IDs are: > > > > > > > > node1: DS- 978762906-83.161.120.155-50010-1237686434530 > > > > node2: DS- 489086185-83.161.120.155-50010-1237686416330 > > > > node3: DS-(581)243-3561.161.120.155-50010-1237686426395 > > > > node4: DS-1024388083-83.161.120.155-50010-1237686404482 > > > > node5: DS-2136798339-83.161.120.155-50010-1237686444430 > > > > I know the storage ID is generated > > > > by > > org.apache.hadoop.hdfs.server.datanode.DataNode.setNewStorageID(...). > > > > > > > > In org.apache.hadoop.hdfs.server.namenode.FSNamesystem > > > > > > > > // Keeps a Collection for every named machine containing > > > > // blocks that have recently been invalidated and are thought to > live > > > > // on the machine in question. > > > > // Mapping: StorageID -> ArrayList<Block> > > > > // > > > > private Map<String, Collection<Block>> recentInvalidateSets = > > > > new TreeMap<String, Collection<Block>>(); > > > > > > > > In > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.ReplicationMonitor > > > > This thread run in interval: replicationRecheckInterval=3000 > > > milliseconds. > > > > > > > > Into computeDatanodeWork() > > > > nodesToProcess = 2. > > > > > > > > then into computeInvalidateWork(nodesToProcess) > > > > the for cycle will only exec 2 cycles. > > > > > > > > for each cycle, go into invalidateWorkForOneNode() > > > > it will always get the first node to invalidate blocks on this node. > > > > String firstNodeId = > recentInvalidateSets.keySet().iterator().next(); > > > > > > > > TreeMap is a stored map, so, the ketSet is: > > > > [1024388083-83.161.120.155-50010-1237686404482, > > > > (581)243-3561.161.120.155-50010-1237686426395, > > > > 2136798339-83.161.120.155-50010-1237686444430, > > > > 489086185-83.161.120.155-50010-1237686416330, > > > > 978762906-83.161.120.155-50010-1237686434530] > > > > > > > > So, the sequence of node list in recentInvalidateSets is: > > > > [node4, node3, node5, node2, node1] > > > > > > > > So, every time in invalidateWorkForOneNode(), it will always process > > > node4 > > > > then node3, then node2 and then node1. > > > > > > > > My application is a HBase write-heavy application. > > > > So, there is many blocks need invalidate in each datanode. So when > each > > > > 3000 milliseconds, at most, there is only two datanode is processed. > > > Since > > > > the node1 is the last one in the TreeMap, it have no change to be > > garbage > > > > collected. > > > > > > > > I think HDFS namenode should fix this issue. > > > > > > > > Schubert > > > > > > > > On Thu, Mar 26, 2009 at 2:57 PM, Samuel Guo <alyssa14@example.net> > > wrote: > > > > > > > >> After a file is deleted, HDFS does not immediately reclaim the > > available > > > >> physical storage. It does so only lazily during garbage collection. > > When > > > a > > > >> file is deleted by the application, the master remove the file's > > > metadata > > > >> from *FSNamesystem* and logs the deletion immediately. And the > file's > > > >> deleted blocks information will be collected in each > > > DataNodeDescriptor's > > > >> *invalidateBlocks* set in Namenode. During the heartbeats between NN > > and > > > >> DN, > > > >> NN will scan the specified DN's DataNodeDescriptor's > invalidateBlocks > > > set, > > > >> find the blocks to be deleted in DN and send a *DNA_INVALIDATE* > > > >> BlockCommand > > > >> to DN. And the *BlockScanner* thread running on DN will scan, find > and > > > >> delete these blocks after DN receives the *DNA_INVALIDATE* > > BlockCommand. > > > >> > > > >> You can search *DNA_INVALIDATE* in DataNode.java and NameNode.java > > > files, > > > >> and find the logic of the garbage collection. Hope it will be > helpful. > > > >> > > > >> On Thu, Mar 26, 2009 at 11:07 AM, schubert zhang <alyssa14@example.net > > > > > >> wrote: > > > >> > > > >> > Tanks Andrew and Billy. > > > >> > I think the subject of this mail thread is not appropriate, it may > > not > > > >> be a > > > >> > balance issue. > > > >> > The problem seems the block deleting scheduler in HDFS. > > > >> > > > > >> > Last night(timezone:+8), I slow down my application, and this > > morning, > > > I > > > >> > found almost all garbage blocks are deleted. > > > >> > Here is the current blocks number of each datanode: > > > >> > node1: 10651 > > > >> > node2: 10477 > > > >> > node3: 12185 > > > >> > node4: 11607 > > > >> > node5: 14000 > > > >> > > > > >> > It seems fine. > > > >> > But I want to study the code of HDFS and make clear the policy of > > > >> deleting > > > >> > blocks on datanodes. If anyone in the hadoop community can give > me > > > some > > > >> > advices? > > > >> > > > > >> > Schubert > > > >> > > > > >> > On Thu, Mar 26, 2009 at 7:55 AM, Andrew Purtell < > > alyssa14@example.net> > > > >> > wrote: > > > >> > > > > >> > > > > >> > > > > > >> > > > From: schubert zhang <alyssa14@example.net> > > > >> > > > From another point of view, I think HBase cannot control to > > > >> > > > delete blocks on which node, it would just delete files, and > > > >> > > > HDFS delete blocks where the blocks locating. > > > >> > > > > > >> > > Yes, that is exactly correct. > > > >> > > > > > >> > > Best regards, > > > >> > > > > > >> > > - Andy > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > >> > > > > > > > > > > > > > > --0016368e2089a2f6340466797cb5-- From hbase-useralyssa14@example.net Wed Apr 01 07:48:27 2009 Return-Path: <hbase-useralyssa14@example.net> Delivered-To: alyssa14@example.net Received: (qmail 32165 invoked from network); 1 Apr 2009 07:48:26 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Apr 2009 07:48:26 -0000 Received: (qmail 37784 invoked by uid 500); 1 Apr 2009 07:48:26 -0000 Delivered-To: alyssa14@example.net Received: (qmail 37719 invoked by uid 500); 1 Apr 2009 07:48:26 -0000 Mailing-List: contact alyssa14@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:alyssa14@example.net> List-Unsubscribe: <mailto:alyssa14@example.net> List-Post: <mailto:alyssa14@example.net> List-Id: <hbase-user.hadoop.apache.org> Reply-To: alyssa14@example.net Delivered-To: mailing list alyssa14@example.net Received: (qmail 37704 invoked by uid 99); 1 Apr 2009 07:48:26 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 07:48:26 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of alyssa14@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO rv-out-0506.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 07:48:15 +0000 Received: by rv-out-0506.google.com with SMTP id k40so2984662rvb.29 for <alyssa14@example.net>; Wed, 01 Apr 2009 00:47:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :received:message-id:subject:from:to:cc:content-type; bh=LJHpXjX82fyLsftQA+KtShzTowMuC1gd7biBOUTCn1A=; b=oLWdfN/lOimVZEmqjrksApqGf7X0R9Vb1GyRLOCPsZTcjPuZMZxB+JDSfGZD5Plbmy rx0S859+71mvBq+WjpNVOzktdH+LpErLO3o6WH7KFY0d78TvKWoVDSVpjX6wlbCG44JB t5SngnTdrgucmdGzw2+8kVACk8rQ0bUMF0sFs= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=tqokAzTfN/eIx2YXekTLauc79BXZDZrVl5RAmzn/+wtjNf6XXXwNnmgByqze0Syc5f XKPl+PwwBpkfxC5GbYpgVw4V72DIqmXORC/LmFLO3KIhvh4KX3zfORtsDLV1mERJ+iBX 7R5pC9mSxqxRfroAqAQ1mGMXN8kQlOfI4+t/Q= MIME-Version: 1.0 In-Reply-To: <alyssa14@example.net> References: <alyssa14@example.net> <alyssa14@example.net> <alyssa14@example.net> <alyssa14@example.net> Date: Wed, 1 Apr 2009 00:47:39 -0700 Received: by 83.161.120.155 with SMTP id a6mr3916255rvk.36.1238572074321; Wed, 01 Apr 2009 00:47:54 -0700 (PDT) Message-ID: <alyssa14@example.net> Subject: Re: Schema to store graph From: Amandeep Khurana <alyssa14@example.net> To: alyssa14@example.net Cc: alyssa14@example.net, alyssa14@example.net Content-Type: multipart/alternative; boundary=000e0cd24f2484c5e9046679889d X-Virus-Checked: Checked by ClamAV on apache.org --000e0cd24f2484c5e9046679889d Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Right. Edward, I didnt understand what you were trying to say with this: Anyway, I guess If you store the graph like that, you'll only need update the row 'v/w' to add
{'source': 'Apache Software Foundation Public Mail Archives'}