From common-commits-returnshane48@example.com Wed Sep 01 18:56:30 2010 Return-Path: <common-commits-returnshane48@example.com> Delivered-To: shane48@example.com Received: (qmail 99369 invoked from network); 1 Sep 2010 18:56:29 -0000 Received: from unknown (HELO mail.apache.org) (83.161.120.155) by 83.161.120.155 with SMTP; 1 Sep 2010 18:56:29 -0000 Received: (qmail 67208 invoked by uid 500); 1 Sep 2010 18:56:29 -0000 Delivered-To: shane48@example.com Received: (qmail 67182 invoked by uid 500); 1 Sep 2010 18:56:28 -0000 Mailing-List: contact shane48@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:shane48@example.com> List-Unsubscribe: <mailto:shane48@example.com> List-Post: <mailto:shane48@example.com> List-Id: <common-commits.hadoop.apache.org> Reply-To: shane48@example.com Delivered-To: mailing list shane48@example.com Received: (qmail 67175 invoked by uid 500); 1 Sep 2010 18:56:28 -0000 Delivered-To: shane48@example.com Received: (qmail 67172 invoked by uid 99); 1 Sep 2010 18:56:28 -0000 Received: from Unknown (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Sep 2010 18:56:28 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eos.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Sep 2010 18:56:11 +0000 Received: from eosnew.apache.org (localhost [83.161.120.155]) by eos.apache.org (Postfix) with ESMTP id 49173BA1; Wed, 1 Sep 2010 18:55:49 +0000 (UTC) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Apache Wiki <shane48@example.com> To: Apache Wiki <shane48@example.com> Date: Wed, 01 Sep 2010 18:55:49 -0000 Message-ID: <shane48@example.com> Subject: =?utf-8?q?=5BHadoop_Wiki=5D_Trivial_Update_of_=22Hbase/MasterRewrite=22_b?= =?utf-8?q?y_stack?= X-Virus-Checked: Checked by ClamAV on apache.org Dear Wiki user, You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for ch= ange notification. The "Hbase/MasterRewrite" page has been changed by stack. http://wiki.apache.org/hadoop/Hbase/MasterRewrite?action=3Ddiff&rev1=3D16&r= ev2=3D17 -------------------------------------------------- =3D Design Notes for Master Rewrite =3D + = + '''MOST OF THE BELOW HAS BEEN DONE BY [[https://issues.apache.org/jira/br= owse/HBASE-2692|HBASE-2692]]. OUTSTANDING IS MOVING SCHEMA UP TO ZK. -- St= .Ack 20100901''' + = Initial Master Rewrite design came of conversations had at the hbase hack= athon held at StumbleUpon, August 5-7, 2009 ([[https://issues.apache.org/ji= ra/secure/attachment/12418561/HBase+Hackathon+Notes+-+Sunday.pdf|Jon Gray k= ept notes]]). The umbrella issue for the master rewrite is [[https://issue= s.apache.org/jira/browse/HBASE-1816|HBASE-1816]]. Timeline is hbase 0.21.0. = =3D=3D Table of Contents =3D=3D @@ -53, +56 @@ = <<Anchor(moveall)>> =3D=3D=3D Move all state, state transitions, and schema to go via zookeep= er =3D=3D=3D + = + '''STATE TRANSITIONS VIA ZK IS DONE: See [[https://issues.apache.org/jira= /browse/HBASE-2692|HBASE-2692]]''' Currently state transitions are done inside master shuffling between Maps= triggered by messages carried on the back of regionserver heartbeats. Mov= e all to zookeeper. = (Patrick Hunt and Mahadev have been helping with the below via [[http://w= iki.apache.org/hadoop/ZooKeeper/HBaseUseCases|HBase Zookeeper Use Cases]]) From common-commits-returnshane48@example.com Wed Sep 01 19:55:56 2010 Return-Path: <common-commits-returnshane48@example.com> Delivered-To: shane48@example.com Received: (qmail 16050 invoked from network); 1 Sep 2010 19:55:56 -0000 Received: from unknown (HELO mail.apache.org) (83.161.120.155) by 83.161.120.155 with SMTP; 1 Sep 2010 19:55:56 -0000 Received: (qmail 27635 invoked by uid 500); 1 Sep 2010 19:55:54 -0000 Delivered-To: shane48@example.com Received: (qmail 27171 invoked by uid 500); 1 Sep 2010 19:55:53 -0000 Mailing-List: contact shane48@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:shane48@example.com> List-Unsubscribe: <mailto:shane48@example.com> List-Post: <mailto:shane48@example.com> List-Id: <common-commits.hadoop.apache.org> Reply-To: shane48@example.com Delivered-To: mailing list shane48@example.com Received: (qmail 27064 invoked by uid 500); 1 Sep 2010 19:55:53 -0000 Delivered-To: shane48@example.com Received: (qmail 27035 invoked by uid 99); 1 Sep 2010 19:55:53 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Sep 2010 19:55:53 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eos.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Sep 2010 19:55:49 +0000 Received: from eosnew.apache.org (localhost [83.161.120.155]) by eos.apache.org (Postfix) with ESMTP id 55A04BAA; Wed, 1 Sep 2010 19:55:28 +0000 (UTC) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Apache Wiki <shane48@example.com> To: Apache Wiki <shane48@example.com> Date: Wed, 01 Sep 2010 19:55:28 -0000 Message-ID: <shane48@example.com> Subject: =?utf-8?q?=5BHadoop_Wiki=5D_Update_of_=22FAQ=22_by_Eugene_Koontz?= Dear Wiki user, You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for ch= ange notification. The "FAQ" page has been changed by Eugene Koontz. The comment on this change is: HADOOP-6369 : fix broken link. http://wiki.apache.org/hadoop/FAQ?action=3Ddiff&rev1=3D71&rev2=3D72 -------------------------------------------------- =3D Hadoop FAQ =3D + <<BR>> <<Anchor(1)>> '''1. [[#A1|What is Hadoop?]]''' - = - <<BR>> - <<Anchor(1)>> - '''1. [[#1|What is Hadoop?]]''' = [[http://hadoop.apache.org/core/|Hadoop]] is a distributed computing plat= form written in Java. It incorporates features similar to those of the [[h= ttp://en.wikipedia.org/wiki/Google_File_System|Google File System]] and of = [[http://en.wikipedia.org/wiki/MapReduce|MapReduce]]. For some details, se= e HadoopMapReduce. = - <<BR>> - <<Anchor(2)>> - '''2. [[#2|What platform does Hadoop run on?]]''' + <<BR>> <<Anchor(2)>> '''2. [[#A2|What platform does Hadoop run on?]]''' = - 1. Java 1.6.x or higher, preferably from Sun -see HadoopJavaVersions + 1. Java 1.6.x or higher, preferably from Sun -see HadoopJavaVersions - 2. Linux and Windows are the supported operating systems, but BSD, Mac = OS/X, and OpenSolaris are known to work. (Windows requires the installation= of [[http://www.cygwin.com/|Cygwin]]). + 1. Linux and Windows are the supported operating systems, but BSD, Mac O= S/X, and OpenSolaris are known to work. (Windows requires the installation = of [[http://www.cygwin.com/|Cygwin]]). = - = - <<Anchor(2.1)>> - ''2.1 [[#2.1|Building / Testing Hadoop on Windows]]'' + <<Anchor(2.1)>> ''2.1 [[#A2.1|Building / Testing Hadoop on Windows]]'' = The Hadoop build on Windows can be run from inside a Windows (not cygwin)= command prompt window. = @@ -28, +21 @@ set JAVA_HOME=3Dc:\jdk83.161.120.155 set PATH=3D%PATH%;%ANT_HOME%\bin }}} - = then open a command prompt window, cd to your workspace directory (in my = case it is c:\workspace\hadoop) and run ant. Since I am interested in runni= ng the contrib test cases I do the following: = {{{ ant -l build.log -Dtest.output=3Dyes test-contrib }}} - = other targets work similarly. I just wanted to document this because I sp= ent some time trying to figure out why the ant build would not run from a c= ygwin command prompt window. If you are building/testing on Windows, and ha= ven't figured it out yet, this should get you started. = - = - <<BR>> - <<Anchor(3)>> - '''3. [[#3|How well does Hadoop scale?]]''' + <<BR>> <<Anchor(3)>> '''3. [[#A3|How well does Hadoop scale?]]''' = Hadoop has been demonstrated on clusters of up to 2000 nodes. Sort perfo= rmance on 900 nodes is good (sorting 9TB of data on 900 nodes takes around = 1.8 hours) and [[attachment:sort900-20080115.png|improving]] using these no= n-default configuration values: = - * `dfs.block.size =3D 134217728` + * `dfs.block.size =3D 134217728` - * `dfs.namenode.handler.count =3D 40` + * `dfs.namenode.handler.count =3D 40` - * `mapred.reduce.parallel.copies =3D 20` + * `mapred.reduce.parallel.copies =3D 20` - * `mapred.child.java.opts =3D -Xmx512m` + * `mapred.child.java.opts =3D -Xmx512m` - * `fs.inmemory.size.mb =3D 200` + * `fs.inmemory.size.mb =3D 200` - * `io.sort.factor =3D 100` + * `io.sort.factor =3D 100` - * `io.sort.mb =3D 200` + * `io.sort.mb =3D 200` - * `io.file.buffer.size =3D 131072` + * `io.file.buffer.size =3D 131072` = - Sort performances on 1400 nodes and 2000 nodes are pretty good too - sort= ing 14TB of data on a 1400-node cluster takes 2.2 hours; sorting 20TB on a = 2000-node cluster takes 2.5 hours. The updates to the above configuration b= eing: = + Sort performances on 1400 nodes and 2000 nodes are pretty good too - sort= ing 14TB of data on a 1400-node cluster takes 2.2 hours; sorting 20TB on a = 2000-node cluster takes 2.5 hours. The updates to the above configuration b= eing: + = - * `mapred.job.tracker.handler.count =3D 60` + * `mapred.job.tracker.handler.count =3D 60` - * `mapred.reduce.parallel.copies =3D 50` + * `mapred.reduce.parallel.copies =3D 50` - * `tasktracker.http.threads =3D 50` + * `tasktracker.http.threads =3D 50` - * `mapred.child.java.opts =3D -Xmx1024m` + * `mapred.child.java.opts =3D -Xmx1024m` = - <<BR>> - <<Anchor(4)>> - '''4. [[#4|Do I have to write my application in Java?]]''' + <<BR>> <<Anchor(4)>> '''4. [[#A4|Do I have to write my application in Jav= a?]]''' = - No. There are several ways to incorporate non-Java code. = + No. There are several ways to incorporate non-Java code. + = - * HadoopStreaming permits any shell command to be used as a map or redu= ce function. = + * HadoopStreaming permits any shell command to be used as a map or reduc= e function. - * [[http://svn.apache.org/viewvc/hadoop/core/trunk/src/c%2B%2B/libhdfs|= libhdfs]], a JNI-based C API for talking to hdfs (only). + * [[http://svn.apache.org/viewvc/hadoop/core/trunk/src/c++/libhdfs|libhd= fs]], a JNI-based C API for talking to hdfs (only). - * [[http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/ma= pred/pipes/package-summary.html|Hadoop Pipes]], a [[http://www.swig.org/|SW= IG]]-compatible C++ API (non-JNI) to write map-reduce jobs. + * [[http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/map= red/pipes/package-summary.html|Hadoop Pipes]], a [[http://www.swig.org/|SWI= G]]-compatible C++ API (non-JNI) to write map-reduce jobs. = - = - <<BR>> - <<Anchor(5)>> - '''5. [[#5|How can I help to make Hadoop better?]]''' + <<BR>> <<Anchor(5)>> '''5. [[#A5|How can I help to make Hadoop better?]]'= '' = If you have trouble figuring how to use Hadoop, then, once you've figured= something out (perhaps with the help of the [[http://hadoop.apache.org/cor= e/mailing_lists.html|mailing lists]]), pass that knowledge on to others by = adding something to this wiki. = If you find something that you wish were done better, and know how to fix= it, read HowToContribute, and contribute a patch. = - = - <<BR>> - <<Anchor(6)>> - '''6. [[#6|HDFS. If I add new data-nodes to the cluster will HDFS move th= e blocks to the newly added nodes in order to balance disk space utilizatio= n between the nodes?]]''' + <<BR>> <<Anchor(6)>> '''6. [[#A6|HDFS. If I add new data-nodes to the clu= ster will HDFS move the blocks to the newly added nodes in order to balance= disk space utilization between the nodes?]]''' = No, HDFS will not move blocks to new nodes automatically. However, newly = created files will likely have their blocks placed on the new nodes. = There are several ways to rebalance the cluster manually. + = 1. Select a subset of files that take up a good percentage of your disk = space; copy them to new locations in HDFS; remove the old copies of the fil= es; rename the new copies to their original names. - 2. A simpler way, with no interruption of service, is to turn up the rep= lication of files, wait for transfers to stabilize, and then turn the repli= cation back down. + 1. A simpler way, with no interruption of service, is to turn up the rep= lication of files, wait for transfers to stabilize, and then turn the repli= cation back down. - 3. Yet another way to re-balance blocks is to turn off the data-node, wh= ich is full, wait until its blocks are replicated, and then bring it back a= gain. The over-replicated blocks will be randomly removed from different no= des, so you really get them rebalanced not just removed from the current no= de. + 1. Yet another way to re-balance blocks is to turn off the data-node, wh= ich is full, wait until its blocks are replicated, and then bring it back a= gain. The over-replicated blocks will be randomly removed from different no= des, so you really get them rebalanced not just removed from the current no= de. - 4. Finally, you can use the bin/start-balancer.sh command to run a balan= cing process to move blocks around the cluster automatically. See = + 1. Finally, you can use the bin/start-balancer.sh command to run a balan= cing process to move blocks around the cluster automatically. See * [[http://hadoop.apache.org/core/docs/current/hdfs_user_guide.html#Reb= alancer|HDFS User Guide: Rebalancer]]; * [[http://developer.yahoo.com/hadoop/tutorial/module2.html#rebalancing= |HDFS Tutorial: Rebalancing]]; * [[http://hadoop.apache.org/core/docs/current/commands_manual.html#bal= ancer|HDFS Commands Guide: balancer]]. = - <<BR>> - <<Anchor(7)>> - '''7. [[#7|HDFS. What is the purpose of the secondary name-node?]]''' + <<BR>> <<Anchor(7)>> '''7. [[#A7|HDFS. What is the purpose of the seconda= ry name-node?]]''' = + The term "secondary name-node" is somewhat misleading. It is not a name-n= ode in the sense that data-nodes cannot connect to the secondary name-node,= and in no event it can replace the primary name-node in case of its failur= e. - The term "secondary name-node" is somewhat misleading. - It is not a name-node in the sense that data-nodes cannot connect to the = secondary name-node, - and in no event it can replace the primary name-node in case of its failu= re. = + The only purpose of the secondary name-node is to perform periodic checkp= oints. The secondary name-node periodically downloads current name-node ima= ge and edits log files, joins them into new image and uploads the new image= back to the (primary and the only) name-node. See [[http://hadoop.apache.o= rg/core/docs/current/hdfs_user_guide.html#Secondary+Namenode|User Guide]]. - The only purpose of the secondary name-node is to perform periodic checkp= oints. - The secondary name-node periodically downloads current name-node image an= d edits log files, - joins them into new image and uploads the new image back to the (primary = and the only) name-node. - See [[http://hadoop.apache.org/core/docs/current/hdfs_user_guide.html#Sec= ondary+Namenode|User Guide]]. = + So if the name-node fails and you can restart it on the same physical nod= e then there is no need to shutdown data-nodes, just the name-node need to= be restarted. If you cannot use the old node anymore you will need to copy= the latest image somewhere else. The latest image can be found either on t= he node that used to be the primary before failure if available; or on the = secondary name-node. The latter will be the latest checkpoint without subse= quent edits logs, that is the most recent name space modifications may be = missing there. You will also need to restart the whole cluster in this case. - So if the name-node fails and you can restart it on the same physical nod= e then there is no need = - to shutdown data-nodes, just the name-node need to be restarted. - If you cannot use the old node anymore you will need to copy the latest i= mage somewhere else. - The latest image can be found either on the node that used to be the prim= ary before failure if available; - or on the secondary name-node. The latter will be the latest checkpoint w= ithout subsequent edits logs, = - that is the most recent name space modifications may be missing there. - You will also need to restart the whole cluster in this case. = - = - <<BR>> - <<Anchor(8)>> - '''8. [[#8|MR. What is the Distributed Cache used for?]]''' + <<BR>> <<Anchor(8)>> '''8. [[#A8|MR. What is the Distributed Cache used f= or?]]''' = The distributed cache is used to distribute large read-only files that ar= e needed by map/reduce jobs to the cluster. The framework will copy the nec= essary files from a url (either hdfs: or http:) on to the slave node before= any tasks for the job are executed on that node. The files are only copied= once per job and so should not be modified by the application. = - = - <<BR>> - <<Anchor(9)>> - '''9. [[#9|MR. Can I write create/write-to hdfs files directly from my ma= p/reduce tasks?]]''' + <<BR>> <<Anchor(9)>> '''9. [[#A9|MR. Can I write create/write-to hdfs fil= es directly from my map/reduce tasks?]]''' = Yes. (Clearly, you want this since you need to create/write-to files othe= r than the output-file written out by [[http://hadoop.apache.org/core/docs/= current/api/org/apache/hadoop/mapred/OutputCollector.html|OutputCollector]]= .) = @@ -148, +112 @@ = The entire discussion holds true for maps of jobs with reducer=3DNONE (i.= e. 0 reduces) since output of the map, in that case, goes directly to hdfs. = - = - <<BR>> - <<Anchor(10)>> - '''10. [[#10|MR. How do I get each of my maps to work on one complete inp= ut-file and not allow the framework to split-up my files?]]''' + <<BR>> <<Anchor(10)>> '''10. [[#A10|MR. How do I get each of my maps to w= ork on one complete input-file and not allow the framework to split-up my f= iles?]]''' = Essentially a job's input is represented by the [[http://hadoop.apache.or= g/core/docs/current/api/org/apache/hadoop/mapred/InputFormat.html|InputForm= at]](interface)/[[http://hadoop.apache.org/core/docs/current/api/org/apache= /hadoop/mapred/FileInputFormat.html|FileInputFormat]](base class). = For this purpose one would need a 'non-splittable' [[http://hadoop.apache= .org/core/docs/current/api/org/apache/hadoop/mapred/FileInputFormat.html|Fi= leInputFormat]] i.e. an input-format which essentially tells the map-reduce= framework that it cannot be split-up and processed. To do this you need yo= ur particular input-format to return '''false''' for the [[http://hadoop.ap= ache.org/core/docs/current/api/org/apache/hadoop/mapred/FileInputFormat.htm= l#isSplitable(org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path)= |isSplittable]] call. - = + = - E.g. '''org.apache.hadoop.mapred.Sort``Validator.Record``Stats``Checker.N= on``Splitable``Sequence``File``Input``Format''' in [[http://svn.apache.org/= viewvc/hadoop/core/trunk/src/test/org/apache/hadoop/mapred/SortValidator.ja= va|src/test/org/apache/hadoop/mapred/SortValidator.java]] + E.g. '''org.apache.hadoop.mapred.SortValidator.RecordStatsChecker.NonSpli= tableSequenceFileInputFormat''' in [[http://svn.apache.org/viewvc/hadoop/co= re/trunk/src/test/org/apache/hadoop/mapred/SortValidator.java|src/test/org/= apache/hadoop/mapred/SortValidator.java]] = In addition to implementing the InputFormat interface and having isSplita= ble(...) returning false, it is also necessary to implement the RecordReade= r interface for returning the whole content of the input file. (default is = LineRecordReader, which splits the file into separate lines) = The other, quick-fix option, is to set [[http://hadoop.apache.org/core/do= cs/current/hadoop-default.html#mapred.min.split.size|mapred.min.split.size]= ] to large enough value. = - = - <<BR>> - <<Anchor(11)>> - '''11. [[#11|Why I do see broken images in jobdetails.jsp page?]]''' + <<BR>> <<Anchor(11)>> '''11. [[#A11|Why I do see broken images in jobdeta= ils.jsp page?]]''' = - In hadoop-0.15, Map / Reduce task completion graphics are added. The grap= hs are produced as SVG(Scalable Vector Graphics) images, which are basicall= y xml files, embedded in html content. The graphics are tested successfully= in Firefox 2 on Ubuntu and MAC OS. However for other browsers, one should = install an additional plugin to the browser to see the SVG images. Adobe's = SVG Viewer can be found at [[http://www.adobe.com/svg/viewer/install/]]. = + In hadoop-0.15, Map / Reduce task completion graphics are added. The grap= hs are produced as SVG(Scalable Vector Graphics) images, which are basicall= y xml files, embedded in html content. The graphics are tested successfully= in Firefox 2 on Ubuntu and MAC OS. However for other browsers, one should = install an additional plugin to the browser to see the SVG images. Adobe's = SVG Viewer can be found at http://www.adobe.com/svg/viewer/install/. = - = - = - <<BR>> - <<Anchor(12)>> - '''12. [[#12|HDFS. Does the name-node stay in safe mode till all under-re= plicated files are fully replicated?]]''' + <<BR>> <<Anchor(12)>> '''12. [[#A12|HDFS. Does the name-node stay in safe= mode till all under-replicated files are fully replicated?]]''' = + No. During safe mode replication of blocks is prohibited. The name-node = awaits when all or majority of data-nodes report their blocks. - No. During safe mode replication of blocks is prohibited. = - The name-node awaits when all or majority of data-nodes report their bloc= ks. = + Depending on how safe mode parameters are configured the name-node will s= tay in safe mode until a specific percentage of blocks of the system is ''= minimally'' replicated [[http://hadoop.apache.org/core/docs/current/hadoop= -default.html#dfs.replication.min|dfs.replication.min]]. If the safe mode t= hreshold [[http://hadoop.apache.org/core/docs/current/hadoop-default.html#= dfs.safemode.threshold.pct|dfs.safemode.threshold.pct]] is set to 1 then a= ll blocks of all files should be minimally replicated. - Depending on how safe mode parameters are configured the name-node will s= tay in safe mode = - until a specific percentage of blocks of the system is ''minimally'' repl= icated = - [[http://hadoop.apache.org/core/docs/current/hadoop-default.html#dfs.repl= ication.min|dfs.replication.min]]. - If the safe mode threshold = - [[http://hadoop.apache.org/core/docs/current/hadoop-default.html#dfs.safe= mode.threshold.pct|dfs.safemode.threshold.pct]] = - is set to 1 then all blocks of all = - files should be minimally replicated. = + Minimal replication does not mean full replication. Some replicas may be = missing and in order to replicate them the name-node needs to leave safe m= ode. - Minimal replication does not mean full replication. Some replicas may be = missing and in = - order to replicate them the name-node needs to leave safe mode. = + Learn more about safe mode [[http://hadoop.apache.org/hdfs/docs/current/h= dfs_user_guide.html#Safemode|in the HDFS Users' Guide]]. - Learn more about safe mode - [[http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/dfs/Na= meNode.html#setSafeMode(org.apache.hadoop.dfs.FSConstants.SafeModeAction)|h= ere]]. = - = - <<BR>> - <<Anchor(13)>> - '''13. [[#13|MR. I see a maximum of 2 maps/reduces spawned concurrently o= n each TaskTracker, how do I increase that?]]''' + <<BR>> <<Anchor(13)>> '''13. [[#A13|MR. I see a maximum of 2 maps/reduces= spawned concurrently on each TaskTracker, how do I increase that?]]''' = Use the configuration knob: [[http://hadoop.apache.org/core/docs/current/= hadoop-default.html#mapred.tasktracker.map.tasks.maximum|mapred.tasktracker= .map.tasks.maximum]] and [[http://hadoop.apache.org/core/docs/current/hadoo= p-default.html#mapred.tasktracker.reduce.tasks.maximum|mapred.tasktracker.r= educe.tasks.maximum]] to control the number of maps/reduces spawned simulta= neously on a !TaskTracker. By default, it is set to ''2'', hence one sees a= maximum of 2 maps and 2 reduces at a given instance on a !TaskTracker. = You can set those on a per-tasktracker basis to accurately reflect your h= ardware (i.e. set those to higher nos. on a beefier tasktracker etc.). = - <<BR>> - <<Anchor(14)>> - '''14. [[#14|MR. Submitting map/reduce jobs as a different user doesn't w= ork.]]''' + <<BR>> <<Anchor(14)>> '''14. [[#A14|MR. Submitting map/reduce jobs as a d= ifferent user doesn't work.]]''' = + The problem is that you haven't configured your map/reduce system direc= tory to a fixed value. The default works for single node systems, but not f= or "real" clusters. I like to use: - The problem is that you haven't configured your map/reduce system = - directory to a fixed value. The default works for single node systems, bu= t not for = - "real" clusters. I like to use: = {{{ <property> @@ -218, +156 @@ </description> </property> }}} + Note that this directory is in your default file system and must be acc= essible from both the client and server machines and is typically in HDFS. = - Note that this directory is in your default file system and must be = - accessible from both the client and server machines and is typically = - in HDFS. - = - = - <<BR>> - <<Anchor(15)>> - '''15. [[#15|HDFS. How do I set up a hadoop node to use multiple volumes?= ]]''' + <<BR>> <<Anchor(15)>> '''15. [[#A15|HDFS. How do I set up a hadoop node t= o use multiple volumes?]]''' = + ''Data-nodes'' can store blocks in multiple directories typically allocat= ed on different local disk drives. In order to setup multiple directories o= ne needs to specify a comma separated list of pathnames as a value of the c= onfiguration parameter [[http://hadoop.apache.org/core/docs/current/hadoop= -default.html#dfs.data.dir|dfs.data.dir]]. Data-nodes will attempt to place= equal amount of data in each of the directories. - ''Data-nodes'' can store blocks in multiple directories typically allocat= ed on different local disk drives. - In order to setup multiple directories one needs to specify a comma separ= ated list of pathnames as a value of - the configuration parameter = - [[http://hadoop.apache.org/core/docs/current/hadoop-default.html#dfs.data= .dir|dfs.data.dir]]. - Data-nodes will attempt to place equal amount of data in each of the dire= ctories. = + The ''name-node'' also supports multiple directories, which in the case s= tore the name space image and the edits log. The directories are specified = via the [[http://hadoop.apache.org/core/docs/current/hadoop-default.html#d= fs.name.dir|dfs.name.dir]] configuration parameter. The name-node directori= es are used for the name space data replication so that the image and the = log could be restored from the remaining volumes if one of them fails. - The ''name-node'' also supports multiple directories, which in the case s= tore the name space image and the edits log. - The directories are specified via the = - [[http://hadoop.apache.org/core/docs/current/hadoop-default.html#dfs.name= .dir|dfs.name.dir]] - configuration parameter. - The name-node directories are used for the name space data replication so= that the image and the = - log could be restored from the remaining volumes if one of them fails. = = - = - <<BR>> - <<Anchor(16)>> - '''16. [[#16|HDFS. What happens if one Hadoop client renames a file or a = directory containing this file while another client is still writing into i= t?]]''' + <<BR>> <<Anchor(16)>> '''16. [[#A16|HDFS. What happens if one Hadoop clie= nt renames a file or a directory containing this file while another client = is still writing into it?]]''' = + Starting with release hadoop-0.15, a file will appear in the name space a= s soon as it is created. If a writer is writing to a file and another clie= nt renames either the file itself or any of its path components, then the = original writer will get an IOException either when it finishes writing to = the current block or when it closes the file. - Starting with release hadoop-0.15, a file will appear in the name space a= s soon as it is created. = - If a writer is writing to a file and another client renames either the fi= le itself or any of its path = - components, then the original writer will get an IOException either when = it finishes writing to the current = - block or when it closes the file. = - = - <<BR>> - <<Anchor(17)>> - '''17. [[#17|HDFS. I want to make a large cluster smaller by taking out a= bunch of nodes simultaneously. How can this be done?]]''' + <<BR>> <<Anchor(17)>> '''17. [[#A17|HDFS. I want to make a large cluster = smaller by taking out a bunch of nodes simultaneously. How can this be done= ?]]''' = + On a large cluster removing one or two data-nodes will not lead to any da= ta loss, because name-node will replicate their blocks as long as it will = detect that the nodes are dead. With a large number of nodes getting remove= d or dying the probability of losing data is higher. - On a large cluster removing one or two data-nodes will not lead to any da= ta loss, because = - name-node will replicate their blocks as long as it will detect that the = nodes are dead. - With a large number of nodes getting removed or dying the probability of = losing data is higher. = + Hadoop offers the ''decommission'' feature to retire a set of existing da= ta-nodes. The nodes to be retired should be included into the ''exclude fil= e'', and the exclude file name should be specified as a configuration para= meter [[http://hadoop.apache.org/core/docs/current/hadoop-default.html#dfs.= hosts.exclude|dfs.hosts.exclude]]. This file should have been specified dur= ing namenode startup. It could be a zero length file. You must use the full= hostname, ip or ip:port format in this file. Then the shell command + = - Hadoop offers the ''decommission'' feature to retire a set of existing da= ta-nodes. - The nodes to be retired should be included into the ''exclude file'', and= the exclude file name should = - be specified as a configuration parameter - [[http://hadoop.apache.org/core/docs/current/hadoop-default.html#dfs.host= s.exclude|dfs.hosts.exclude]]. This file should have been specified during = namenode startup. It could be a zero length file. You must use the full hos= tname, ip or ip:port format in this file. = - Then the shell command {{{ bin/hadoop dfsadmin -refreshNodes }}} - should be called, which forces the name-node to re-read the exclude file + should be called, which forces the name-node to re-read the exclude file = and start the decommission process. - and start the decommission process. = + Decommission does not happen momentarily since it requires replication of= potentially a large number of blocks and we do not want the cluster to be= overwhelmed with just this one job. The decommission progress can be monit= ored on the name-node Web UI. Until all blocks are replicated the node wil= l be in "Decommission In Progress" state. When decommission is done the sta= te will change to "Decommissioned". The nodes can be removed whenever deco= mmission is finished. - Decommission does not happen momentarily since it requires replication of= potentially a large number of blocks = - and we do not want the cluster to be overwhelmed with just this one job. - The decommission progress can be monitored on the name-node Web UI. = - Until all blocks are replicated the node will be in "Decommission In Prog= ress" state. - When decommission is done the state will change to "Decommissioned". = - The nodes can be removed whenever decommission is finished. = - The decommission process can be terminated at any time by editing the con= figuration or the exclude files = + The decommission process can be terminated at any time by editing the con= figuration or the exclude files and repeating the {{{-refreshNodes}}} comm= and. - and repeating the {{{-refreshNodes}}} command. = - <<BR>> - <<Anchor(18)>> - '''18. [[#18|What kind of hardware scales best for Hadoop?]]''' + <<BR>> <<Anchor(18)>> '''18. [[#A18|What kind of hardware scales best for= Hadoop?]]''' = The short answer is dual processor/dual core machines with 4-8GB of RAM u= sing ECC memory. Machines should be moderately high-end commodity machines = to be most cost-effective and typically cost 1/2 - 2/3 the cost of normal p= roduction application servers but are not desktop-class machines. This cost= tends to be $2-5K. For a more detailed discussion, see MachineScaling page. = - <<BR>> - <<Anchor(19)>> - '''19. [[#19|Wildcard characters doesn't work correctly in FsShell.]]''' + <<BR>> <<Anchor(19)>> '''19. [[#A19|Wildcard characters doesn't work corr= ectly in FsShell.]]''' = + When you issue a command in !FsShell, you may want to apply that command = to more than one file. !FsShell provides a wildcard character to help you d= o so. The * (asterisk) character can be used to take the place of any set = of characters. For example, if you would like to list all the files in your= account which begin with the letter '''x''', you could use the ls command = with the * wildcard: - When you issue a command in !FsShell, you may want to apply that command = to more than one file. !FsShell provides a wildcard character to help you d= o so. = - The * (asterisk) character can be used to take the place of any set of ch= aracters. For example, if you would like to list all the files in your acco= unt which begin with the - letter '''x''', you could use the ls command with the * wildcard: = = {{{ bin/hadoop dfs -ls x* }}} - = Sometimes, the native OS wildcard support causes unexpected results. To a= void this problem, Enclose the expression in '''Single''' or '''Double''' q= uotes and it should work correctly. = {{{ bin/hadoop dfs -ls 'in*' }}} - = - <<BR>> - <<Anchor(20)>> - '''20. [[#20|How does GridGain compare to Hadoop?]]''' + <<BR>> <<Anchor(20)>> '''20. [[#A20|How does GridGain compare to Hadoop?]= ]''' = !GridGain does not support data intensive jobs. For more details, see Had= oopVsGridGain. = - <<BR>> - <<Anchor(21)>> - '''21. [[#21|HDFS. Can I have multiple files in HDFS use different block = sizes?]]''' + <<BR>> <<Anchor(21)>> '''21. [[#A21|HDFS. Can I have multiple files in HD= FS use different block sizes?]]''' = - Yes. HDFS provides api to specify block size when you create a file. <<BR= >> - See - [[http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/fs/Fil= eSystem.html#create(org.apache.hadoop.fs.Path,%20boolean,%20int,%20short,%2= 0long)|FileSystem.create(Path, overwrite, bufferSize, replication, blockSiz= e, progress)]] + Yes. HDFS provides api to specify block size when you create a file. <<BR= >> See [[http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/f= s/FileSystem.html#create(org.apache.hadoop.fs.Path,%20boolean,%20int,%20sho= rt,%20long)|FileSystem.create(Path, overwrite, bufferSize, replication, blo= ckSize, progress)]] = - <<BR>> - <<Anchor(22)>> - '''22. [[#22|Does HDFS make block boundaries between records?]]''' + <<BR>> <<Anchor(22)>> '''22. [[#A22|Does HDFS make block boundaries betwe= en records?]]''' = No, HDFS does not provide record-oriented API and therefore is not aware = of records and boundaries between them. = - <<BR>> - <<Anchor(23)>> - '''23. [[#23|How do Map/Reduce InputSplit's handle record boundaries corr= ectly?]]''' + <<BR>> <<Anchor(23)>> '''23. [[#A23|How do Map/Reduce InputSplit's handle= record boundaries correctly?]]''' = It is the responsibility of the InputSplit's RecordReader to start and en= d at a record boundary. For SequenceFile's every 2k bytes has a 20 bytes ''= 'sync''' mark between the records. These sync marks allow the RecordReader = to seek to the start of the InputSplit, which contains a file, offset and l= ength and find the first sync mark after the start of the split. The Record= Reader continues processing records until it reaches the first sync mark af= ter the end of the split. The first split of each file naturally starts imm= ediately and not after the first sync mark. In this way, it is guaranteed t= hat each record will be processed by exactly one mapper. = Text files are handled similarly, using newlines instead of sync marks. = - <<BR>> - <<Anchor(24)>> - '''24. [[#24|HDFS. What happens when two clients try to write into the sa= me HDFS file?]]''' + <<BR>> <<Anchor(24)>> '''24. [[#A24|HDFS. What happens when two clients t= ry to write into the same HDFS file?]]''' = + HDFS supports exclusive writes only. <<BR>> When the first client contact= s the name-node to open the file for writing, the name-node grants a lease = to the client to create this file. When the second client tries to open th= e same file for writing, the name-node will see that the lease for the fil= e is already granted to another client, and will reject the open request fo= r the second client. - HDFS supports exclusive writes only. <<BR>> - When the first client contacts the name-node to open the file for writing, - the name-node grants a lease to the client to create this file. = - When the second client tries to open the same file for writing, the name-= node = - will see that the lease for the file is already granted to another client, - and will reject the open request for the second client. = - <<BR>> - <<Anchor(25)>> - '''25. [[#25|I have a new node I want to add to a running Hadoop cluster;= how do I start services on just one node?]]''' + <<BR>> <<Anchor(25)>> '''25. [[#A25|I have a new node I want to add to a = running Hadoop cluster; how do I start services on just one node?]]''' = - This also applies to the case where a machine has crashed and rebooted, e= tc, and you need to get it to rejoin the cluster. You do not need to shutdo= wn and/or restart the entire cluster in this case. = + This also applies to the case where a machine has crashed and rebooted, e= tc, and you need to get it to rejoin the cluster. You do not need to shutdo= wn and/or restart the entire cluster in this case. = First, add the new node's DNS name to the conf/slaves file on the master = node. = @@ -359, +234 @@ $ bin/hadoop-daemon.sh start datanode $ bin/hadoop-daemon.sh start tasktracker }}} - = - <<BR>> - <<Anchor(26)>> - '''26. [[#26|Is there an easy way to see the status and health of my clus= ter?]]''' + <<BR>> <<Anchor(26)>> '''26. [[#A26|Is there an easy way to see the statu= s and health of my cluster?]]''' = There are web-based interfaces to both the JobTracker (MapReduce master) = and NameNode (HDFS master) which display status pages about the state of th= e entire system. By default, these are located at http://job.tracker.addr:5= 0030/ and http://name.node.addr:50070/. = - The JobTracker status page will display the state of all nodes, as well a= s the job queue and status about all currently running jobs and tasks. The = NameNode status page will display the state of all nodes and the amount of = free space, and provides the ability to browse the DFS via the web. = + The JobTracker status page will display the state of all nodes, as well a= s the job queue and status about all currently running jobs and tasks. The = NameNode status page will display the state of all nodes and the amount of = free space, and provides the ability to browse the DFS via the web. = You can also see some basic HDFS cluster health data by running: + = {{{ $ bin/hadoop dfsadmin -report }}} - = - = - <<BR>> - <<Anchor(27)>> - '''27. [[#27|How do I change final output file name with the desired name= rather than in partitions like part-00000, part-00001 ?]]''' + <<BR>> <<Anchor(27)>> '''27. [[#A27|How do I change final output file nam= e with the desired name rather than in partitions like part-00000, part-000= 01 ?]]''' = You can subclass the [[http://svn.apache.org/viewvc/hadoop/core/trunk/src= /mapred/org/apache/hadoop/mapred/OutputFormat.java?view=3Dmarkup|OutputForm= at.java]] class and write your own. You can look at the code of [[http://sv= n.apache.org/viewvc/hadoop/core/trunk/src/mapred/org/apache/hadoop/mapred/T= extOutputFormat.java?view=3Dmarkup|TextOutputFormat]] [[http://svn.apache.o= rg/viewvc/hadoop/core/trunk/src/mapred/org/apache/hadoop/mapred/MultipleOut= putFormat.java?view=3Dmarkup|MultipleOutputFormat.java]] etc. for reference= . It might be the case that you only need to do minor changes to any of the= existing Output Format classes. To do that you can just subclass that clas= s and override the methods you need to change. = - <<BR>> - <<Anchor(28)>> - '''28. [[#28|How much network bandwidth might I need between racks in a m= edium size (40-80 node) Hadoop cluster?]]''' + <<BR>> <<Anchor(28)>> '''28. [[#A28|How much network bandwidth might I ne= ed between racks in a medium size (40-80 node) Hadoop cluster?]]''' = The true answer depends on the types of jobs you're running. As a back of= the envelope calculation one might figure something like this: = + 60 nodes total on 2 racks =3D 30 nodes per rack Each node might process a= bout 100MB/sec of data In the case of a sort job where the intermediate dat= a is the same size as the input data, that means each node needs to shuffle= 100MB/sec of data In aggregate, each rack is then producing about 3GB/sec = of data However, given even reducer spread across the racks, each rack will= need to send 1.5GB/sec to reducers running on the other rack. Since the co= nnection is full duplex, that means you need 1.5GB/sec of bisection bandwid= th for this theoretical job. So that's 12Gbps. - 60 nodes total on 2 racks =3D 30 nodes per rack - Each node might process about 100MB/sec of data - In the case of a sort job where the intermediate data is the same size as= the input data, that means each node needs to shuffle 100MB/sec of data - In aggregate, each rack is then producing about 3GB/sec of data - However, given even reducer spread across the racks, each rack will need = to send 1.5GB/sec to reducers running on the other rack. - Since the connection is full duplex, that means you need 1.5GB/sec of bis= ection bandwidth for this theoretical job. So that's 12Gbps. = However, the above calculations are probably somewhat of an upper bound. = A large number of jobs have significant data reduction during the map phase= , either by some kind of filtering/selection going on in the Mapper itself,= or by good usage of Combiners. Additionally, intermediate data compression= can cut the intermediate data transfer by a significant factor. Lastly, al= though your disks can probably provide 100MB sustained throughput, it's rar= e to see a MR job which can sustain disk speed IO through the entire pipeli= ne. So, I'd say my estimate is at least a factor of 2 too high. = So, the simple answer is that 4-6Gbps is most likely just fine for most p= ractical jobs. If you want to be extra safe, many inexpensive switches can = operate in a "stacked" configuration where the bandwidth between them is es= sentially backplane speed. That should scale you to 96 nodes with plenty of= headroom. Many inexpensive gigabit switches also have one or two 10GigE po= rts which can be used effectively to connect to each other or to a 10GE cor= e. = - '''29.[[#29|How to limit Data node's disk usage?]]''' + '''29.[[#A29|How to limit Data node's disk usage?]]''' = Use dfs.datanode.du.reserved configuration value in $HADOOP_HOME/conf/hdf= s-site.xml for limiting disk usage. = @@ -409, +271 @@ <description>Reserved space in bytes per volume. Always leave this mu= ch space free for non dfs use. </description> </property> - }}} = + }}} - = - <<Anchor(30)>> - ''2.1 [[#30|When writing a New InputFormat, what is the format for the ar= ray of string returned by InputSplit\#getLocations()?]]'' + <<Anchor(30)>> ''2.1 [[#A30|When writing a New InputFormat, what is the f= ormat for the array of string returned by InputSplit\#getLocations()?]]'' = It appears that DatanodeID.getHost() is the standard place to retrieve th= is name, and the machineName variable, populated in DataNode.java\#startDat= aNode, is where the name is first set. The first method attempted is to get= "slave.host.name" from the configuration; if that is not available, DNS.ge= tDefaultHost is used instead. =20 From common-commits-returnshane48@example.com Thu Sep 02 00:37:11 2010 Return-Path: <common-commits-returnshane48@example.com> Delivered-To: shane48@example.com Received: (qmail 96472 invoked from network); 2 Sep 2010 00:37:11 -0000 Received: from unknown (HELO mail.apache.org) (83.161.120.155) by 83.161.120.155 with SMTP; 2 Sep 2010 00:37:11 -0000 Received: (qmail 77942 invoked by uid 500); 2 Sep 2010 00:37:11 -0000 Delivered-To: shane48@example.com Received: (qmail 77866 invoked by uid 500); 2 Sep 2010 00:37:10 -0000 Mailing-List: contact shane48@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:shane48@example.com> List-Unsubscribe: <mailto:shane48@example.com> List-Post: <mailto:shane48@example.com> List-Id: <common-commits.hadoop.apache.org> Reply-To: shane48@example.com Delivered-To: mailing list shane48@example.com Received: (qmail 77836 invoked by uid 99); 2 Sep 2010 00:37:10 -0000 Received: from Unknown (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Sep 2010 00:37:10 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Sep 2010 00:36:50 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id EE5302388980; Thu, 2 Sep 2010 00:35:30 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r991780 - in /hadoop/common/trunk: ./ src/java/org/apache/hadoop/ipc/ src/test/core/org/apache/hadoop/ipc/ Date: Thu, 02 Sep 2010 00:35:30 -0000 To: shane48@example.com From: shane48@example.com X-Mailer: svnmailer-1.0.8 Message-Id: <shane48@example.com> X-Virus-Checked: Checked by ClamAV on apache.org Author: hairong Date: Thu Sep 2 00:35:30 2010 New Revision: 991780 URL: http://svn.apache.org/viewvc?rev=991780&view=rev Log: HADOOP-6907. Rpc client doesn't use the per-connection conf to figure out server's Kerberos principal. Contributed by Kan Zhang. Modified: hadoop/common/trunk/CHANGES.txt hadoop/common/trunk/src/java/org/apache/hadoop/ipc/Client.java hadoop/common/trunk/src/java/org/apache/hadoop/ipc/WritableRpcEngine.java hadoop/common/trunk/src/test/core/org/apache/hadoop/ipc/TestIPC.java hadoop/common/trunk/src/test/core/org/apache/hadoop/ipc/TestSaslRPC.java Modified: hadoop/common/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/common/trunk/CHANGES.txt?rev=991780&r1=991779&r2=991780&view=diff ============================================================================== --- hadoop/common/trunk/CHANGES.txt (original) +++ hadoop/common/trunk/CHANGES.txt Thu Sep 2 00:35:30 2010 @@ -226,6 +226,9 @@ Trunk (unreleased changes) HADOOP-6913. Circular initialization between UserGroupInformation and KerberosName (Kan Zhang via boryas) + HADOOP-6907. Rpc client doesn't use the per-connection conf to figure + out server's Kerberos principal (Kan Zhang via hairong) + Release 0.21.0 - Unreleased INCOMPATIBLE CHANGES Modified: hadoop/common/trunk/src/java/org/apache/hadoop/ipc/Client.java URL: http://svn.apache.org/viewvc/hadoop/common/trunk/src/java/org/apache/hadoop/ipc/Client.java?rev=991780&r1=991779&r2=991780&view=diff ============================================================================== --- hadoop/common/trunk/src/java/org/apache/hadoop/ipc/Client.java (original) +++ hadoop/common/trunk/src/java/org/apache/hadoop/ipc/Client.java Thu Sep 2 00:35:30 2010 @@ -37,6 +37,7 @@ import java.security.PrivilegedException import java.util.Hashtable; import java.util.Iterator; import java.util.Random; +import java.util.Set; import java.util.Map.Entry; import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicLong; @@ -45,6 +46,8 @@ import javax.net.SocketFactory; import org.apache.commons.logging.*; +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.classification.InterfaceStability; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.io.IOUtils; import org.apache.hadoop.io.Text; @@ -80,12 +83,6 @@ public class Client { private int counter; // counter for call ids private AtomicBoolean running = new AtomicBoolean(true); // if client runs final private Configuration conf; - final private int maxIdleTime; //connections will be culled if it was idle for - //maxIdleTime msecs - final private int maxRetries; //the max. no. of retries for socket connections - private boolean tcpNoDelay; // if T then disable Nagle's Algorithm - private int pingInterval; // how often sends ping to the server in msecs - final private boolean doPing; //do we need to send ping message private SocketFactory socketFactory; // how to create sockets private int refCount = 1; @@ -220,6 +217,12 @@ public class Client { private DataInputStream in; private DataOutputStream out; private int rpcTimeout; + private int maxIdleTime; //connections will be culled if it was idle for + //maxIdleTime msecs + private int maxRetries; //the max. no. of retries for socket connections + private boolean tcpNoDelay; // if T then disable Nagle's Algorithm + private boolean doPing; //do we need to send ping message + private int pingInterval; // how often sends ping to the server in msecs // currently active calls private Hashtable<Integer, Call> calls = new Hashtable<Integer, Call>(); @@ -235,6 +238,15 @@ public class Client { remoteId.getAddress().getHostName()); } this.rpcTimeout = remoteId.getRpcTimeout(); + this.maxIdleTime = remoteId.getMaxIdleTime(); + this.maxRetries = remoteId.getMaxRetries(); + this.tcpNoDelay = remoteId.getTcpNoDelay(); + this.doPing = remoteId.getDoPing(); + this.pingInterval = remoteId.getPingInterval(); + if (LOG.isDebugEnabled()) { + LOG.debug("The ping interval is" + this.pingInterval + "ms."); + } + UserGroupInformation ticket = remoteId.getTicket(); Class<?> protocol = remoteId.getProtocol(); this.useSasl = UserGroupInformation.isSecurityEnabled(); @@ -256,15 +268,9 @@ public class Client { } KerberosInfo krbInfo = protocol.getAnnotation(KerberosInfo.class); if (krbInfo != null) { - String serverKey = krbInfo.serverPrincipal(); - if (serverKey == null) { - throw new IOException( - "Can't obtain server Kerberos config key from KerberosInfo"); - } - serverPrincipal = SecurityUtil.getServerPrincipal( - conf.get(serverKey), server.getAddress().getCanonicalHostName()); + serverPrincipal = remoteId.getServerPrincipal(); if (LOG.isDebugEnabled()) { - LOG.debug("RPC Server Kerberos principal name for protocol=" + LOG.debug("RPC Server's Kerberos principal name for protocol=" + protocol.getCanonicalName() + " is " + serverPrincipal); } } @@ -882,15 +888,6 @@ public class Client { public Client(Class<? extends Writable> valueClass, Configuration conf, SocketFactory factory) { this.valueClass = valueClass; - this.maxIdleTime = - conf.getInt("ipc.client.connection.maxidletime", 10000); //10s - this.maxRetries = conf.getInt("ipc.client.connect.max.retries", 10); - this.tcpNoDelay = conf.getBoolean("ipc.client.tcpnodelay", false); - this.doPing = conf.getBoolean("ipc.client.ping", true); - this.pingInterval = getPingInterval(conf); - if (LOG.isDebugEnabled()) { - LOG.debug("The ping interval is" + this.pingInterval + "ms."); - } this.conf = conf; this.socketFactory = factory; } @@ -942,7 +939,7 @@ public class Client { /** Make a call, passing <code>param</code>, to the IPC server running at * <code>address</code>, returning the value. Throws exceptions if there are * network problems or if the remote code threw an exception. - * @deprecated Use {@link #call(Writable, InetSocketAddress, Class, UserGroupInformation, int)} instead + * @deprecated Use {@link #call(Writable, ConnectionId)} instead */ @Deprecated public Writable call(Writable param, InetSocketAddress address) @@ -955,27 +952,60 @@ public class Client { * the value. * Throws exceptions if there are network problems or if the remote code * threw an exception. - * @deprecated Use {@link #call(Writable, InetSocketAddress, Class, UserGroupInformation, int)} instead + * @deprecated Use {@link #call(Writable, ConnectionId)} instead */ @Deprecated public Writable call(Writable param, InetSocketAddress addr, UserGroupInformation ticket) throws InterruptedException, IOException { - return call(param, addr, null, ticket, 0); + ConnectionId remoteId = ConnectionId.getConnectionId(addr, null, ticket, 0, + conf); + return call(param, remoteId); } /** Make a call, passing <code>param</code>, to the IPC server running at * <code>address</code> which is servicing the <code>protocol</code> protocol, - * with the <code>ticket</code> credentials, returning the value. + * with the <code>ticket</code> credentials and <code>rpcTimeout</code> as + * timeout, returning the value. * Throws exceptions if there are network problems or if the remote code - * threw an exception. */ + * threw an exception. + * @deprecated Use {@link #call(Writable, ConnectionId)} instead + */ + @Deprecated public Writable call(Writable param, InetSocketAddress addr, Class<?> protocol, UserGroupInformation ticket, int rpcTimeout) throws InterruptedException, IOException { + ConnectionId remoteId = ConnectionId.getConnectionId(addr, protocol, + ticket, rpcTimeout, conf); + return call(param, remoteId); + } + + /** + * Make a call, passing <code>param</code>, to the IPC server running at + * <code>address</code> which is servicing the <code>protocol</code> protocol, + * with the <code>ticket</code> credentials, <code>rpcTimeout</code> as + * timeout and <code>conf</code> as conf for this connection, returning the + * value. Throws exceptions if there are network problems or if the remote + * code threw an exception. + */ + public Writable call(Writable param, InetSocketAddress addr, + Class<?> protocol, UserGroupInformation ticket, + int rpcTimeout, Configuration conf) + throws InterruptedException, IOException { + ConnectionId remoteId = ConnectionId.getConnectionId(addr, protocol, + ticket, rpcTimeout, conf); + return call(param, remoteId); + } + + /** Make a call, passing <code>param</code>, to the IPC server defined by + * <code>remoteId</code>, returning the value. + * Throws exceptions if there are network problems or if the remote code + * threw an exception. */ + public Writable call(Writable param, ConnectionId remoteId) + throws InterruptedException, IOException { Call call = new Call(param); - Connection connection = getConnection( - addr, protocol, ticket, rpcTimeout, call); + Connection connection = getConnection(remoteId, call); connection.sendParam(call); // send the parameter boolean interrupted = false; synchronized (call) { @@ -998,7 +1028,7 @@ public class Client { call.error.fillInStackTrace(); throw call.error; } else { // local exception - throw wrapException(addr, call.error); + throw wrapException(remoteId.getAddress(), call.error); } } else { return call.value; @@ -1038,25 +1068,34 @@ public class Client { } /** - * Makes a set of calls in parallel. Each parameter is sent to the - * corresponding address. When all values are available, or have timed out - * or errored, the collected results are returned in an array. The array - * contains nulls for calls that timed out or errored. - * @deprecated Use {@link #call(Writable[], InetSocketAddress[], Class, UserGroupInformation)} instead + * @deprecated Use {@link #call(Writable[], InetSocketAddress[], + * Class, UserGroupInformation, Configuration)} instead */ @Deprecated public Writable[] call(Writable[] params, InetSocketAddress[] addresses) throws IOException, InterruptedException { - return call(params, addresses, null, null); + return call(params, addresses, null, null, conf); + } + + /** + * @deprecated Use {@link #call(Writable[], InetSocketAddress[], + * Class, UserGroupInformation, Configuration)} instead + */ + @Deprecated + public Writable[] call(Writable[] params, InetSocketAddress[] addresses, + Class<?> protocol, UserGroupInformation ticket) + throws IOException, InterruptedException { + return call(params, addresses, protocol, ticket, conf); } + /** Makes a set of calls in parallel. Each parameter is sent to the * corresponding address. When all values are available, or have timed out * or errored, the collected results are returned in an array. The array * contains nulls for calls that timed out or errored. */ - public Writable[] call(Writable[] params, InetSocketAddress[] addresses, - Class<?> protocol, UserGroupInformation ticket) - throws IOException, InterruptedException { + public Writable[] call(Writable[] params, InetSocketAddress[] addresses, + Class<?> protocol, UserGroupInformation ticket, Configuration conf) + throws IOException, InterruptedException { if (addresses.length == 0) return new Writable[0]; ParallelResults results = new ParallelResults(params.length); @@ -1064,8 +1103,9 @@ public class Client { for (int i = 0; i < params.length; i++) { ParallelCall call = new ParallelCall(params[i], results, i); try { - Connection connection = - getConnection(addresses[i], protocol, ticket, 0, call); + ConnectionId remoteId = ConnectionId.getConnectionId(addresses[i], + protocol, ticket, 0, conf); + Connection connection = getConnection(remoteId, call); connection.sendParam(call); // send each parameter } catch (IOException e) { // log errors @@ -1084,12 +1124,18 @@ public class Client { } } + // for unit testing only + @InterfaceAudience.Private + @InterfaceStability.Unstable + Set<ConnectionId> getConnectionIds() { + synchronized (connections) { + return connections.keySet(); + } + } + /** Get a connection from the pool, or create a new one and add it to the - * pool. Connections to a given host/port
{'source': 'Apache Software Foundation Public Mail Archives'}