From hadoop-commits-returnbkim@example.net Mon May 01 17:50:53 2006 Return-Path: <hadoop-commits-returnbkim@example.net> Delivered-To: bkim@example.net Received: (qmail 35721 invoked from network); 1 May 2006 17:50:53 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 May 2006 17:50:53 -0000 Received: (qmail 12419 invoked by uid 500); 1 May 2006 17:50:53 -0000 Delivered-To: bkim@example.net Received: (qmail 12389 invoked by uid 500); 1 May 2006 17:50:52 -0000 Mailing-List: contact bkim@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:bkim@example.net> List-Unsubscribe: <mailto:bkim@example.net> List-Post: <mailto:bkim@example.net> List-Id: <hadoop-commits.lucene.apache.org> Reply-To: bkim@example.net Delivered-To: mailing list bkim@example.net Received: (qmail 12370 invoked by uid 99); 1 May 2006 17:50:52 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 May 2006 10:50:52 -0700 X-ASF-Spam-Status: No, hits=-9.4 required=10.0 tests=ALL_TRUSTED,NO_REAL_NAME X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO minotaur.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with SMTP; Mon, 01 May 2006 10:50:51 -0700 Received: (qmail 35622 invoked by uid 65534); 1 May 2006 17:50:31 -0000 Message-ID: <bkim@example.net> Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r398650 - in /lucene/hadoop/trunk: CHANGES.txt src/webapps/mapred/jobdetails.jsp src/webapps/mapred/jobtracker.jsp Date: Mon, 01 May 2006 17:50:30 -0000 To: bkim@example.net From: bkim@example.net X-Mailer: svnmailer-1.0.8 X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N Author: cutting Date: Mon May 1 10:50:28 2006 New Revision: 398650 URL: http://svn.apache.org/viewcvs?rev=398650&view=rev Log: HADOOP-177. Page through tasks in web ui. Modified: lucene/hadoop/trunk/CHANGES.txt lucene/hadoop/trunk/src/webapps/mapred/jobdetails.jsp lucene/hadoop/trunk/src/webapps/mapred/jobtracker.jsp Modified: lucene/hadoop/trunk/CHANGES.txt URL: http://svn.apache.org/viewcvs/lucene/hadoop/trunk/CHANGES.txt?rev=398650&r1=398649&r2=398650&view=diff ============================================================================== --- lucene/hadoop/trunk/CHANGES.txt (original) +++ lucene/hadoop/trunk/CHANGES.txt Mon May 1 10:50:28 2006 @@ -128,6 +128,10 @@ to contact the job tracker for five attempts, not just one as before. (omalley via cutting) +35. HADOOP-177. Change MapReduce web interface to page through tasks. + Previously, when jobs had more than a few thousand tasks they + could crash web browsers. (Mahadev Konar via cutting) + Release 0.1.1 - 2006-04-08 Modified: lucene/hadoop/trunk/src/webapps/mapred/jobdetails.jsp URL: http://svn.apache.org/viewcvs/lucene/hadoop/trunk/src/webapps/mapred/jobdetails.jsp?rev=398650&r1=398649&r2=398650&view=diff ============================================================================== --- lucene/hadoop/trunk/src/webapps/mapred/jobdetails.jsp (original) +++ lucene/hadoop/trunk/src/webapps/mapred/jobdetails.jsp Mon May 1 10:50:28 2006 @@ -5,16 +5,30 @@ import="java.io.*" import="java.util.*" import="org.apache.hadoop.mapred.*" + import="java.lang.Integer" %> + <% String jobid = request.getParameter("jobid"); + String type = request.getParameter("type"); + String pagenum = request.getParameter("pagenum"); + int pnum = Integer.parseInt(pagenum); + int next_page = pnum+1; + int numperpage = 2000; JobTracker tracker = JobTracker.getTracker(); JobInProgress job = (JobInProgress) tracker.getJob(jobid); JobProfile profile = (job != null) ? (job.getProfile()) : null; JobStatus status = (job != null) ? (job.getStatus()) : null; - - TaskReport[] mapTaskReports = (job != null) ? tracker.getMapTaskReports(jobid) : null; - TaskReport[] reduceTaskReports = (job != null) ? tracker.getReduceTaskReports(jobid) : null; + TaskReport[] reports = null; + int start_index = (pnum - 1) * numperpage; + int end_index = start_index + numperpage; + int report_len = 0; + if ("map".equals(type)){ + reports = (job != null) ? tracker.getMapTaskReports(jobid) : null; + } + else{ + reports = (job != null) ? tracker.getReduceTaskReports(jobid) : null; + } %> <html> @@ -26,70 +40,63 @@ No job found<br> <% } else { - %> + %> <h1>Job '<%=jobid%>'</h1> <b>Job File:</b> <%=profile.getJobFile()%><br> <b>The job started at:</b> <%= new Date(job.getStartTime())%><br> -<% +<% if (status.getRunState() == JobStatus.RUNNING) { out.print("The job is still running.<br>\n"); } else if (status.getRunState() == JobStatus.SUCCEEDED) { - out.print("<b>The job completed at:</b> " + new Date(job.getFinishTime()) + "<br>\n"); - } else if (status.getRunState() == JobStatus.FAILED) { - out.print("<b>The job failed at:</b> " + new Date(job.getFinishTime()) + "<br>\n"); - } -%> -<hr> - -<h2>Map Tasks</h2> - <center> - <table border=2 cellpadding="5" cellspacing="2"> - <tr><td align="center">Task</td><td>Complete</td><td>Status</td><td>Errors</td></tr> + out.print("<b>The job completed at:</b> " + new Date(job.getFinishTime()) + - <% - - for (int i = 0; i < mapTaskReports.length; i++) { - TaskReport report = mapTaskReports[i]; +"<br>\n"); + } else if (status.getRunState() == JobStatus.FAILED) { + out.print("<b>The job failed at:</b> " + new Date(job.getFinishTime()) + - out.print("<tr><td><a href=\"taskdetails.jsp?jobid=" + jobid + "&taskid=" + report.getTaskId() + "\">" + report.getTaskId() + "</a></td>"); - out.print("<td>" + report.getProgress() + "</td>"); - out.print("<td>" + report.getState() + "</td>"); - String[] diagnostics = report.getDiagnostics(); - for (int j = 0; j < diagnostics.length ; j++) { - out.print("<td><pre>" + diagnostics[j] + "</pre></td>"); - } - out.print("</tr>\n"); +"<br>\n"); + } + report_len = reports.length; + + if (report_len <= start_index) { + out.print("<b>No such tasks</b>"); + }else{ + out.print("<hr>"); + out.print("<h2>Tasks</h2>"); + out.print("<center>"); + out.print("<table border=2 cellpadding=\"5\" cellspacing=\"2\">"); + out.print("<tr><td align=\"center\">Task</td><td>Complete</td><td>Status</td><td>Errors</td></tr>"); + if (end_index > report_len){ + end_index = report_len; } - %> - </table> - </center> -<hr> - - -<h2>Reduce Tasks</h2> - <center> - <table border=2 cellpadding="5" cellspacing="2"> - <tr><td align="center">Task</td><td>Complete</td><td>State</td><td>Errors</td></tr> - - <% - for (int i = 0; i < reduceTaskReports.length; i++) { - TaskReport report = reduceTaskReports[i]; - - out.print("<tr><td><a href=\"taskdetails.jsp?jobid=" + jobid + "&taskid=" + report.getTaskId() + "\">" + report.getTaskId() + "</a></td>"); - out.print("<td>" + report.getProgress() + "</td>"); - out.print("<td>" + report.getState() + "</td>"); - - String[] diagnostics = report.getDiagnostics(); - for (int j = 0; j < diagnostics.length ; j++) { - out.print("<td><pre>" + diagnostics[j] + "</pre></td>"); - } - out.print("</tr>\n"); + for (int i = start_index ; i < end_index; i++) { + TaskReport report = reports[i]; + out.print("<tr><td><a href=\"taskdetails.jsp?jobid=" + jobid + +"&taskid=" + report.getTaskId() + "\">" + report.getTaskId() + "</a></td>"); + out.print("<td>" + report.getProgress() + "</td>"); + out.print("<td>" + report.getState() + "</td>"); + String[] diagnostics = report.getDiagnostics(); + for (int j = 0; j < diagnostics.length ; j++) { + out.print("<td><pre>" + diagnostics[j] + "</pre></td>"); + } + out.print("</tr>\n"); } + out.print("</table>"); + out.print("</center>"); + } + %> - </table> - </center> - <% + + <% + if (end_index < report_len) { + out.print("<div style=\"text-align:right\">" + "<a href=\"/jobdetails.jsp?jobid="+ jobid + "&type=" + type +"&pagenum=" + next_page + + "\">" + "Next" + "</a></div>"); + } + if (start_index != 0) { + out.print("<div style=\"text-align:right\">" + "<a href=\"/jobdetails.jsp?jobid="+ jobid + "&type=" + type +"&pagenum=" + (pnum -1) + + "\">" + "Prev" + "</a></div>"); + } } %> Modified: lucene/hadoop/trunk/src/webapps/mapred/jobtracker.jsp URL: http://svn.apache.org/viewcvs/lucene/hadoop/trunk/src/webapps/mapred/jobtracker.jsp?rev=398650&r1=398649&r2=398650&view=diff ============================================================================== --- lucene/hadoop/trunk/src/webapps/mapred/jobtracker.jsp (original) +++ lucene/hadoop/trunk/src/webapps/mapred/jobtracker.jsp Mon May 1 10:50:28 2006 @@ -82,10 +82,14 @@ int completedReduces = job.finishedReduces(); String name = profile.getJobName(); - out.print("<tr><td><a href=\"jobdetails.jsp?jobid=" + jobid + "\">" + - jobid + "</a></td><td>"+ profile.getUser() + "</td><td>" + - ("".equals(name) ? "&nbsp;" : name) + "</td><td>" + - percentFormat.format(completedRatio) + "%</td><td>" + + out.print( "<tr><td>" + jobid + + "<br>" + "<a href=\"jobdetails.jsp?jobid=" + jobid + "&type=" + "map" + "&pagenum=" + 1 + "\">" + + "Maps" + "</a>" + "<br>" + + "<a href=\"jobdetails.jsp?jobid=" + jobid + "&type=" + "reduce" + "&pagenum=" + 1 + "\">" + + "Reduces" + "</a></td>" + + "<td>" + profile.getUser() + "</td>" + + "<td>" + ("".equals(name) ? "&nbsp;" : name) + "</td>" + + "<td>" + percentFormat.format(completedRatio) + "%</td><td>" + desiredMaps + "</td><td>" + completedMaps + "</td><td>" + desiredReduces + "</td><td> " + completedReduces + "</td></tr>\n"); From hadoop-commits-returnbkim@example.net Mon May 01 19:46:06 2006 Return-Path: <hadoop-commits-returnbkim@example.net> Delivered-To: bkim@example.net Received: (qmail 675 invoked from network); 1 May 2006 19:45:59 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 May 2006 19:45:58 -0000 Received: (qmail 6257 invoked by uid 500); 1 May 2006 19:45:54 -0000 Delivered-To: bkim@example.net Received: (qmail 6237 invoked by uid 500); 1 May 2006 19:45:53 -0000 Mailing-List: contact bkim@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:bkim@example.net> List-Unsubscribe: <mailto:bkim@example.net> List-Post: <mailto:bkim@example.net> List-Id: <hadoop-commits.lucene.apache.org> Reply-To: bkim@example.net Delivered-To: mailing list bkim@example.net Received: (qmail 6223 invoked by uid 99); 1 May 2006 19:45:53 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 May 2006 12:45:53 -0700 X-ASF-Spam-Status: No, hits=-8.6 required=10.0 tests=ALL_TRUSTED,INFO_TLD,NO_REAL_NAME X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO minotaur.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with SMTP; Mon, 01 May 2006 12:45:52 -0700 Received: (qmail 369 invoked by uid 65534); 1 May 2006 19:45:31 -0000 Message-ID: <bkim@example.net> Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r398668 - in /lucene/hadoop/trunk: CHANGES.txt src/java/org/apache/hadoop/dfs/DataNode.java src/java/org/apache/hadoop/dfs/DatanodeProtocol.java src/java/org/apache/hadoop/dfs/NameNode.java Date: Mon, 01 May 2006 19:45:26 -0000 To: bkim@example.net From: bkim@example.net X-Mailer: svnmailer-1.0.8 X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N Author: cutting Date: Mon May 1 12:45:24 2006 New Revision: 398668 URL: http://svn.apache.org/viewcvs?rev=398668&view=rev Log: HADOOP-178. Piggyback DFS blockwork requests on heartbeat responses, reducing traffic. Also move blockwork delay on startup from datanode to namenode, fixing a problems when the namenode alone restarts. Contributed by Hairong Kuang. Modified: lucene/hadoop/trunk/CHANGES.txt lucene/hadoop/trunk/src/java/org/apache/hadoop/dfs/DataNode.java lucene/hadoop/trunk/src/java/org/apache/hadoop/dfs/DatanodeProtocol.java lucene/hadoop/trunk/src/java/org/apache/hadoop/dfs/NameNode.java Modified: lucene/hadoop/trunk/CHANGES.txt URL: http://svn.apache.org/viewcvs/lucene/hadoop/trunk/CHANGES.txt?rev=398668&r1=398667&r2=398668&view=diff ============================================================================== --- lucene/hadoop/trunk/CHANGES.txt (original) +++ lucene/hadoop/trunk/CHANGES.txt Mon May 1 12:45:24 2006 @@ -132,6 +132,12 @@ Previously, when jobs had more than a few thousand tasks they could crash web browsers. (Mahadev Konar via cutting) +36. HADOOP-178. In DFS, piggyback blockwork requests from datanodes + on heartbeat responses from namenode. This reduces the volume of + RPC traffic. Also move startup delay in blockwork from datanode + to namenode. This fixes a problem where restarting the namenode + triggered a lot of uneeded replication. (Hairong Kuang via cutting) + Release 0.1.1 - 2006-04-08 Modified: lucene/hadoop/trunk/src/java/org/apache/hadoop/dfs/DataNode.java URL: http://svn.apache.org/viewcvs/lucene/hadoop/trunk/src/java/org/apache/hadoop/dfs/DataNode.java?rev=398668&r1=398667&r2=398668&view=diff ============================================================================== --- lucene/hadoop/trunk/src/java/org/apache/hadoop/dfs/DataNode.java (original) +++ lucene/hadoop/trunk/src/java/org/apache/hadoop/dfs/DataNode.java Mon May 1 12:45:24 2006 @@ -89,7 +89,6 @@ int xmitsInProgress = 0; Daemon dataXceiveServer = null; long blockReportInterval; - private long datanodeStartupPeriod; /** * Create the DataNode given a configuration and a dataDir. @@ -128,8 +127,6 @@ conf.getLong("dfs.blockreport.intervalMsec", BLOCKREPORT_INTERVAL); this.blockReportInterval = blockReportIntervalBasis - new Random().nextInt((int)(blockReportIntervalBasis/10)); - this.datanodeStartupPeriod = - conf.getLong("dfs.datanode.startupMsec", DATANODE_STARTUP_PERIOD); } /** @@ -159,7 +156,6 @@ */ public void offerService() throws Exception { long lastHeartbeat = 0, lastBlockReport = 0; - long sendStart = System.currentTimeMillis(); LOG.info("using BLOCKREPORT_INTERVAL of " + blockReportInterval + "msec"); // @@ -171,100 +167,91 @@ // // Every so often, send heartbeat or block-report // - synchronized (receivedBlockList) { - if (now - lastHeartbeat > HEARTBEAT_INTERVAL) { + if (now - lastHeartbeat > HEARTBEAT_INTERVAL) { + // + // All heartbeat messages include following info: + // -- Datanode name + // -- data transfer port + // -- Total capacity + // -- Bytes remaining + // + BlockCommand cmd = namenode.sendHeartbeat(localName, + data.getCapacity(), data.getRemaining(), xmitsInProgress); + //LOG.info("Just sent heartbeat, with name " + localName); + lastHeartbeat = now; + + if (cmd != null && cmd.transferBlocks()) { + // + // Send a copy of a block to another datanode + // + Block blocks[] = cmd.getBlocks(); + DatanodeInfo xferTargets[][] = cmd.getTargets(); + + for (int i = 0; i < blocks.length; i++) { + if (!data.isValidBlock(blocks[i])) { + String errStr = "Can't send invalid block " + blocks[i]; + LOG.info(errStr); + namenode.errorReport(localName, errStr); + break; + } else { + if (xferTargets[i].length > 0) { + LOG.info("Starting thread to transfer block " + blocks[i] + " to " + xferTargets[i]); + new Daemon(new DataTransfer(xferTargets[i], blocks[i])).start(); + } + } + } + } else if (cmd != null && cmd.invalidateBlocks()) { // - // All heartbeat messages include following info: - // -- Datanode name - // -- data transfer port - // -- Total capacity - // -- Bytes remaining - // - namenode.sendHeartbeat(localName, data.getCapacity(), data.getRemaining()); - //LOG.info("Just sent heartbeat, with name " + localName); - lastHeartbeat = now; - } - if (now - lastBlockReport > blockReportInterval) { - // - // Send latest blockinfo report if timer has expired. - // Get back a list of local block(s) that are obsolete - // and can be safely GC'ed. - // - Block toDelete[] = namenode.blockReport(localName, data.getBlockReport()); - data.invalidate(toDelete); - lastBlockReport = now; - continue; - } - if (receivedBlockList.size() > 0) { + // Some local block(s) are obsolete and can be + // safely garbage-collected. + // + data.invalidate(cmd.getBlocks()); + } + } + + // send block report + if (now - lastBlockReport > blockReportInterval) { + // + // Send latest blockinfo report if timer has expired. + // Get back a list of local block(s) that are obsolete + // and can be safely GC'ed. + // + Block toDelete[] = namenode.blockReport(localName, data.getBlockReport()); + data.invalidate(toDelete); + lastBlockReport = now; + continue; + } + + // check if there are newly received blocks + Block [] blockArray=null; + synchronized( receivedBlockList ) { + if (receivedBlockList.size() > 0) { // // Send newly-received blockids to namenode // - Block blockArray[] = (Block[]) receivedBlockList.toArray(new Block[receivedBlockList.size()]); + blockArray = (Block[]) receivedBlockList.toArray(new Block[receivedBlockList.size()]); receivedBlockList.removeAllElements(); - namenode.blockReceived(localName, blockArray); - } - - // - // Only perform block operations (transfer, delete) after - // a startup quiet period. The assumption is that all the - // datanodes will be started together, but the namenode may - // have been started some time before. (This is esp. true in - // the case of network interruptions.) So, wait for some time - // to pass from the time of connection to the first block-transfer. - // Otherwise we transfer a lot of blocks unnecessarily. - // - if (now - sendStart > datanodeStartupPeriod) { - // - // Check to see if there are any block-instructions from the - // namenode that this datanode should perform. - // - BlockCommand cmd = namenode.getBlockwork(localName, xmitsInProgress); - if (cmd != null && cmd.transferBlocks()) { - // - // Send a copy of a block to another datanode - // - Block blocks[] = cmd.getBlocks(); - DatanodeInfo xferTargets[][] = cmd.getTargets(); - - for (int i = 0; i < blocks.length; i++) { - if (!data.isValidBlock(blocks[i])) { - String errStr = "Can't send invalid block " + blocks[i]; - LOG.info(errStr); - namenode.errorReport(localName, errStr); - break; - } else { - if (xferTargets[i].length > 0) { - LOG.info("Starting thread to transfer block " + blocks[i] - + " to " + xferTargets[i][0].getName() - + (xferTargets[i].length > 1 ? " and " - + (xferTargets[i].length-1) + " more destination(s)" : "" )); - new Daemon(new DataTransfer(xferTargets[i], blocks[i])).start(); - } - } - } - } else if (cmd != null && cmd.invalidateBlocks()) { - // - // Some local block(s) are obsolete and can be - // safely garbage-collected. - // - data.invalidate(cmd.getBlocks()); - } } - - // - // There is no work to do; sleep until hearbeat timer elapses, - // or work arrives, and then iterate again. - // - long waitTime = HEARTBEAT_INTERVAL - (now - lastHeartbeat); + } + if( blockArray != null ) { + namenode.blockReceived(localName, blockArray); + } + + // + // There is no work to do; sleep until hearbeat timer elapses, + // or work arrives, and then iterate again. + // + long waitTime = HEARTBEAT_INTERVAL - (System.currentTimeMillis() - lastHeartbeat); + synchronized( receivedBlockList ) { if (waitTime > 0 && receivedBlockList.size() == 0) { try { receivedBlockList.wait(waitTime); } catch (InterruptedException ie) { } } - } - } - } + } // synchronized + } // while (shouldRun) + } // offerService /** * Server used for receiving/sending a block of data. Modified: lucene/hadoop/trunk/src/java/org/apache/hadoop/dfs/DatanodeProtocol.java URL: http://svn.apache.org/viewcvs/lucene/hadoop/trunk/src/java/org/apache/hadoop/dfs/DatanodeProtocol.java?rev=398668&r1=398667&r2=398668&view=diff ============================================================================== --- lucene/hadoop/trunk/src/java/org/apache/hadoop/dfs/DatanodeProtocol.java (original) +++ lucene/hadoop/trunk/src/java/org/apache/hadoop/dfs/DatanodeProtocol.java Mon May 1 12:45:24 2006 @@ -30,9 +30,14 @@ interface DatanodeProtocol { /** * sendHeartbeat() tells the NameNode that the DataNode is still - * alive and well. Includes some status info, too. + * alive and well. Includes some status info, too. + * It also gives the NameNode a chance to return a "BlockCommand" object. + * A BlockCommand tells the DataNode to invalidate local block(s), + * or to copy them to other DataNodes, etc. */ - public void sendHeartbeat(String sender, long capacity, long remaining) throws IOException; + public BlockCommand sendHeartbeat(String sender, + long capacity, long remaining, + int xmitsInProgress) throws IOException; /** * blockReport() tells the NameNode about all the locally-stored blocks. @@ -56,13 +61,4 @@ * awry. Useful for debugging. */ public void errorReport(String sender, String msg) throws IOException; - - /** - * The DataNode periodically calls getBlockwork(). It includes a - * small amount of status information, but mainly gives the NameNode - * a chance to return a "BlockCommand" object. A BlockCommand tells - * the DataNode to invalidate local block(s), or to copy them to other - * DataNodes, etc. - */ - public BlockCommand getBlockwork(String sender, int xmitsInProgress) throws IOException; } Modified: lucene/hadoop/trunk/src/java/org/apache/hadoop/dfs/NameNode.java URL: http://svn.apache.org/viewcvs/lucene/hadoop/trunk/src/java/org/apache/hadoop/dfs/NameNode.java?rev=398668&r1=398667&r2=398668&view=diff ============================================================================== --- lucene/hadoop/trunk/src/java/org/apache/hadoop/dfs/NameNode.java (original) +++ lucene/hadoop/trunk/src/java/org/apache/hadoop/dfs/NameNode.java Mon May 1 12:45:24 2006 @@ -61,7 +61,9 @@ private FSNamesystem namesystem; private Server server; private int handlerCount = 2; - + private long datanodeStartupPeriod; + private volatile long firstBlockReportTime; + /** only used for testing purposes */ private boolean stopRequested = false; @@ -83,10 +85,12 @@ /** * Create a NameNode at the specified location and start it. */ - public NameNode(File dir, int port, Configuration conf) throws IOException { + public NameNode(File dir, int port, Configuration conf) throws IOException { this.namesystem = new FSNamesystem(dir, conf); this.handlerCount = conf.getInt("dfs.namenode.handler.count", 10); this.server = RPC.getServer(this, port, handlerCount, false, conf); + this.datanodeStartupPeriod = + conf.getLong("dfs.datanode.startupMsec", DATANODE_STARTUP_PERIOD); this.server.start(); } @@ -353,34 +357,31 @@ // DatanodeProtocol //////////////////////////////////////////////////////////////// /** - */ - public void sendHeartbeat(String sender, long capacity, long remaining) { - namesystem.gotHeartbeat(new UTF8(sender), capacity, remaining); - } - - public Block[] blockReport(String sender, Block blocks[]) { - LOG.info("Block report from "+sender+": "+blocks.length+" blocks."); - return namesystem.processReport(blocks, new UTF8(sender)); - } - - public void blockReceived(String sender, Block blocks[]) { - for (int i = 0; i < blocks.length; i++) { - namesystem.blockReceived(blocks[i], new UTF8(sender)); - } - } - - /** - */ - public void errorReport(String sender, String msg) { - // Log error message from datanode - //LOG.info("Report from " + sender + ": " + msg); - } - - /** + * Data node notify the name node that it is alive * Return a block-oriented command for the datanode to execute. * This will be either a transfer or a delete operation. */ - public BlockCommand getBlockwork(String sender, int xmitsInProgress) { + public BlockCommand sendHeartbeat(String sender, long capacity, long remaining, + int xmitsInProgress) { + namesystem.gotHeartbeat(new UTF8(sender), capacity, remaining); + + // + // Only ask datanodes to perform block operations (transfer, delete) + // after a startup quiet period. The assumption is that all the + // datanodes will be started together, but the namenode may + // have been started some time before. (This is esp. true in + // the case of network interruptions.) So, wait for some time + // to pass from the time of connection to the first block-transfer. + // Otherwise we transfer a lot of blocks unnecessarily. + // + // Hairong: Ideally in addition we also look at the history. For example, + // we should wait until at least 98% of datanodes are connected to the server + // + if( firstBlockReportTime==0 || + System.currentTimeMillis()-firstBlockReportTime < datanodeStartupPeriod) { + return null; + } + // // Ask to perform pending transfers, if any // @@ -400,6 +401,26 @@ return new BlockCommand(blocks); } return null; + } + + public Block[] blockReport(String sender, Block blocks[]) { + if( firstBlockReportTime==0) + firstBlockReportTime=System.currentTimeMillis(); + + return namesystem.processReport(blocks, new UTF8(sender)); + } + + public void blockReceived(String sender, Block blocks[]) { + for (int i = 0; i < blocks.length; i++) { + namesystem.blockReceived(blocks[i], new UTF8(sender)); + } + } + + /** + */ + public void errorReport(String sender, String msg) { + // Log error message from datanode + //LOG.info("Report from " + sender + ": " + msg); } /** From hadoop-commits-returnbkim@example.net Mon May 01 20:01:25 2006 Return-Path: <hadoop-commits-returnbkim@example.net> Delivered-To: bkim@example.net Received: (qmail 10210 invoked from network); 1 May 2006 20:01:25 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 May 2006 20:01:25 -0000 Received: (qmail 31974 invoked by uid 500); 1 May 2006 20:01:25 -0000 Delivered-To: bkim@example.net Received: (qmail 31958 invoked by uid 500); 1 May 2006 20:01:25 -0000 Mailing-List: contact bkim@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:bkim@example.net> List-Unsubscribe: <mailto:bkim@example.net> List-Post: <mailto:bkim@example.net> List-Id: <hadoop-commits.lucene.apache.org> Reply-To: bkim@example.net Delivered-To: mailing list bkim@example.net Received: (qmail 31948 invoked by uid 99); 1 May 2006 20:01:25 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 May 2006 13:01:25 -0700 X-ASF-Spam-Status: No, hits=-9.4 required=10.0 tests=ALL_TRUSTED,NO_REAL_NAME X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO minotaur.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with SMTP; Mon, 01 May 2006 13:01:24 -0700 Received: (qmail 9821 invoked by uid 65534); 1 May 2006 20:00:59 -0000 Message-ID: <bkim@example.net> Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r398674 - in /lucene/hadoop/trunk: CHANGES.txt src/java/org/apache/hadoop/dfs/FSDirectory.java Date: Mon, 01 May 2006 20:00:58 -0000 To: bkim@example.net From: bkim@example.net X-Mailer: svnmailer-1.0.8 X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N Author: cutting Date: Mon May 1 13:00:57 2006 New Revision: 398674 URL: http://svn.apache.org/viewcvs?rev=398674&view=rev Log: HADOOP-183. If the namendode is restarted with different minimum or maximum replication counts, existing files' replication counts are now automatically adjusted to be within the newly configured bounds. Contributed by Hairong Kuang. Modified: lucene/hadoop/trunk/CHANGES.txt lucene/hadoop/trunk/src/java/org/apache/hadoop/dfs/FSDirectory.java Modified: lucene/hadoop/trunk/CHANGES.txt URL: http://svn.apache.org/viewcvs/lucene/hadoop/trunk/CHANGES.txt?rev=398674&r1=398673&r2=398674&view=diff ============================================================================== --- lucene/hadoop/trunk/CHANGES.txt (original) +++ lucene/hadoop/trunk/CHANGES.txt Mon May 1 13:00:57 2006 @@ -138,6 +138,10 @@ to namenode. This fixes a problem where restarting the namenode triggered a lot of uneeded replication. (Hairong Kuang via cutting) +37. HADOOP-183. If the DFS namenode is restarted with different + minimum and/or maximum replication counts, existing files' + replication counts are now automatically adjusted to be within the + newly configured bounds. (Hairong Kuang via cutting) Release 0.1.1 - 2006-04-08 Modified: lucene/hadoop/trunk/src/java/org/apache/hadoop/dfs/FSDirectory.java URL: http://svn.apache.org/viewcvs/lucene/hadoop/trunk/src/java/org/apache/hadoop/dfs/FSDirectory.java?rev=398674&r1=398673&r2=398674&view=diff ============================================================================== --- lucene/hadoop/trunk/src/java/org/apache/hadoop/dfs/FSDirectory.java (original) +++ lucene/hadoop/trunk/src/java/org/apache/hadoop/dfs/FSDirectory.java Mon May 1 13:00:57 2006 @@ -403,8 +403,10 @@ UTF8 name = new UTF8(); name.readFields(in); // version 0 does not support per file replication - if( !(imgVersion >= 0) ) + if( !(imgVersion >= 0) ) { replication = in.readShort(); // other versions do + replication = adjustReplication( replication, conf ); + } int numBlocks = in.readInt(); Block blocks[] = null; if (numBlocks > 0) { @@ -484,6 +486,7 @@ name = (UTF8) writables[0]; replication = Short.parseShort( ((UTF8)writables[1]).toString()); + replication = adjustReplication( replication, conf ); } // get blocks aw = new ArrayWritable(Block.class); @@ -501,8 +504,11 @@ UTF8 repl = new UTF8(); src.readFields(in); repl.readFields(in); + replication=adjustReplication( + fromLogReplication(repl), + conf); unprotectedSetReplication(src.toString(), - fromLogReplication(repl), + replication, null); break; } @@ -541,6 +547,17 @@ return numEdits; } + private static short adjustReplication( short replication, Configuration conf) { + short minReplication = (short)conf.getInt("dfs.replication.min", 1); + if( replication<minReplication ) { + replication = minReplication; + } + short maxReplication = (short)conf.getInt("dfs.replication.max", 512); + if( replication>maxReplication ) { + replication = maxReplication; + } + return replication; + } /** * Save the contents of the FS image */ From hadoop-commits-returnbkim@example.net Tue May 02 18:16:19 2006 Return-Path: <hadoop-commits-returnbkim@example.net> Delivered-To: bkim@example.net Received: (qmail 857 invoked from network); 2 May 2006 18:16:19 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 May 2006 18:16:19 -0000 Received: (qmail 62609 invoked by uid 500); 2 May 2006 18:16:19 -0000 Delivered-To: bkim@example.net Received: (qmail 62584 invoked by uid 500); 2 May 2006 18:16:18 -0000 Mailing-List: contact bkim@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:bkim@example.net> List-Unsubscribe: <mailto:bkim@example.net> List-Post: <mailto:bkim@example.net> List-Id: <hadoop-commits.lucene.apache.org> Reply-To: bkim@example.net Delivered-To: mailing list bkim@example.net Received: (qmail 62564 invoked by uid 99); 2 May 2006 18:16:17 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 May 2006 11:16:17 -0700 X-ASF-Spam-Status: No, hits=-8.6 required=10.0 tests=ALL_TRUSTED,INFO_TLD,NO_REAL_NAME X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO minotaur.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with SMTP; Tue, 02 May 2006 11:16:17 -0700 Received: (qmail 99879 invoked by uid 65534); 2 May 2006 18:15:56 -0000 Message-ID: <bkim@example.net> Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r398994 - in /lucene/hadoop/trunk: CHANGES.txt src/java/org/apache/hadoop/mapred/TaskTracker.java Date: Tue, 02 May 2006 18:15:56 -0000 To: bkim@example.net From: bkim@example.net X-Mailer: svnmailer-1.0.8 X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N Author: cutting Date: Tue May 2 11:15:53 2006 New Revision: 398994 URL: http://svn.apache.org/viewcvs?rev=398994&view=rev Log: HADOOP-186. Better error handling in TaskTracker's top-level loop. Also improve calculation of time to send next heartbeat. Contributed by Owen O'Malley. Modified: lucene/hadoop/trunk/CHANGES.txt lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/TaskTracker.java Modified: lucene/hadoop/trunk/CHANGES.txt URL: http://svn.apache.org/viewcvs/lucene/hadoop/trunk/CHANGES.txt?rev=398994&r1=398993&r2=398994&view=diff ============================================================================== --- lucene/hadoop/trunk/CHANGES.txt (original) +++ lucene/hadoop/trunk/CHANGES.txt Tue May 2 11:15:53 2006 @@ -143,6 +143,10 @@ replication counts are now automatically adjusted to be within the newly configured bounds. (Hairong Kuang via cutting) +38. HADOOP-186. Better error handling in TaskTracker's top-level + loop. Also improve calculation of time to send next heartbeat. + (omalley via cutting) + Release 0.1.1 - 2006-04-08 1. Added CHANGES.txt, logging all significant changes to Hadoop. (cutting) Modified: lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/TaskTracker.java URL: http://svn.apache.org/viewcvs/lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/TaskTracker.java?rev=398994&r1=398993&r2=398994&view=diff ============================================================================== --- lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/TaskTracker.java (original) +++ lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/TaskTracker.java Tue May 2 11:15:53 2006 @@ -18,7 +18,6 @@ import org.apache.hadoop.fs.*; import org.apache.hadoop.io.*; import org.apache.hadoop.ipc.*; -import org.apache.hadoop.conf.*; import org.apache.hadoop.util.*; import java.io.*; @@ -215,6 +214,7 @@ */ int offerService() throws Exception { long lastHeartbeat = 0; + this.fs = FileSystem.getNamed(jobClient.getFilesystemName(), this.fConf); while (running) { long now = System.currentTimeMillis(); @@ -227,15 +227,16 @@ } continue; } + lastHeartbeat = now; // // Emit standard hearbeat message to check in with JobTracker // Vector taskReports = new Vector(); synchronized (this) { - for (Iterator it = runningTasks.keySet().iterator(); it.hasNext(); ) { - String taskid = (String) it.next(); - TaskInProgress tip = (TaskInProgress) runningTasks.get(taskid); + for (Iterator it = runningTasks.values().iterator(); + it.hasNext(); ) { + TaskInProgress tip = (TaskInProgress) it.next(); TaskStatus status = tip.createStatus(); taskReports.add(status); if (status.getRunState() != TaskStatus.RUNNING) { @@ -252,9 +253,6 @@ // // Xmit the heartbeat // - if (justStarted) { - this.fs = FileSystem.getNamed(jobClient.getFilesystemName(), this.fConf); - } TaskTrackerStatus status = new TaskTrackerStatus(taskTrackerName, localHostname, @@ -269,11 +267,16 @@ // // Check if we should create a new Task // - if (mapTotal < maxCurrentTasks || reduceTotal < maxCurrentTasks) { - Task t = jobClient.pollForNewTask(taskTrackerName); - if (t != null) { - startNewTask(t); - } + try { + if (mapTotal < maxCurrentTasks || reduceTotal < maxCurrentTasks) { + Task t = jobClient.pollForNewTask(taskTrackerName); + if (t != null) { + startNewTask(t); + } + } + } catch (IOException ie) { + LOG.info("Problem launching task: " + + StringUtils.stringifyException(ie)); } // @@ -292,7 +295,12 @@ " seconds. Killing."; LOG.info(tip.getTask().getTaskId() + ": " + msg); tip.reportDiagnosticInfo(msg); - tip.killAndCleanup(true); + try { + tip.killAndCleanup(true); + } catch (IOException ie) { + LOG.info("Problem cleaning task up: " + + StringUtils.stringifyException(ie)); + } } } } @@ -307,16 +315,25 @@ // // Check for any Tasks whose job may have ended // + try { String[] toCloseIds = jobClient.pollForTaskWithClosedJob(taskTrackerName); if (toCloseIds != null) { synchronized (this) { for (int i = 0; i < toCloseIds.length; i++) { TaskInProgress tip = (TaskInProgress) tasks.get(toCloseIds[i]); - tip.jobHasFinished(); + try { + tip.jobHasFinished(); + } catch (IOException ie) { + LOG.info("problem finishing task: " + + StringUtils.stringifyException(ie)); + } } } } - lastHeartbeat = now; + } catch (IOException ie) { + LOG.info("Problem getting closed tasks: " + + StringUtils.stringifyException(ie)); + } } return 0; From hadoop-commits-returnbkim@example.net Tue May 02 18:49:19 2006 Return-Path: <hadoop-commits-returnbkim@example.net> Delivered-To: bkim@example.net Received: (qmail 31025 invoked from network); 2 May 2006 18:49:18 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 May 2006 18:49:18 -0000 Received: (qmail 39273 invoked by uid 500); 2 May 2006 18:49:17 -0000 Delivered-To: bkim@example.net Received: (qmail 39257 invoked by uid 500); 2 May 2006 18:49:17 -0000 Mailing-List: contact bkim@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:bkim@example.net> List-Unsubscribe: <mailto:bkim@example.net> List-Post: <mailto:bkim@example.net> List-Id: <hadoop-commits.lucene.apache.org> Reply-To: bkim@example.net Delivered-To: mailing list bkim@example.net Received: (qmail 39248 invoked by uid 99); 2 May 2006 18:49:17 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 May 2006 11:49:17 -0700 X-ASF-Spam-Status: No, hits=-9.4 required=10.0 tests=ALL_TRUSTED,NO_REAL_NAME X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO minotaur.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with SMTP; Tue, 02 May 2006 11:49:16 -0700 Received: (qmail 30736 invoked by uid 65534); 2 May 2006 18:48:56 -0000 Message-ID: <bkim@example.net> Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r399011 - in /lucene/hadoop/trunk: CHANGES.txt src/examples/org/apache/hadoop/examples/ExampleDriver.java src/examples/org/apache/hadoop/examples/RandomWriter.java src/examples/org/apache/hadoop/examples/Sort.java Date: Tue, 02 May 2006 18:48:55 -0000 To: bkim@example.net From: bkim@example.net X-Mailer: svnmailer-1.0.8 X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N Author: cutting Date: Tue May 2 11:48:52 2006 New Revision: 399011 URL: http://svn.apache.org/viewcvs?rev=399011&view=rev Log: HADOOP-187. Add RandomWriter and Sort examples. Contributed by Owen. Added: lucene/hadoop/trunk/src/examples/org/apache/hadoop/examples/RandomWriter.java lucene/hadoop/trunk/src/examples/org/apache/hadoop/examples/Sort.java Modified: lucene/hadoop/trunk/CHANGES.txt lucene/hadoop/trunk/src/examples/org/apache/hadoop/examples/ExampleDriver.java Modified: lucene/hadoop/trunk/CHANGES.txt URL: http://svn.apache.org/viewcvs/lucene/hadoop/trunk/CHANGES.txt?rev=399011&r1=399010&r2=399011&view=diff ============================================================================== --- lucene/hadoop/trunk/CHANGES.txt (original) +++ lucene/hadoop/trunk/CHANGES.txt Tue May 2 11:48:52 2006 @@ -147,6 +147,10 @@ loop. Also improve calculation of time to send next heartbeat. (omalley via cutting) +39. HADOOP-187. Add two MapReduce examples/benchmarks. One creates + files containing random data. The second sorts the output of the + first. (omalley via cutting) + Release 0.1.1 - 2006-04-08 1. Added CHANGES.txt, logging all significant changes to Hadoop. (cutting) Modified: lucene/hadoop/trunk/src/examples/org/apache/hadoop/examples/ExampleDriver.java URL: http://svn.apache.org/viewcvs/lucene/hadoop/trunk/src/examples/org/apache/hadoop/examples/ExampleDriver.java?rev=399011&r1=399010&r2=399011&view=diff ============================================================================== --- lucene/hadoop/trunk/src/examples/org/apache/hadoop/examples/ExampleDriver.java (original) +++ lucene/hadoop/trunk/src/examples/org/apache/hadoop/examples/ExampleDriver.java Tue May 2 11:48:52 2006 @@ -103,6 +103,10 @@ "A map/reduce program that counts the words in the input files.")); programs.put("grep", new ProgramDescription(Grep.class, "A map/reduce program that counts the matches of a regex in the input.")); + programs.put("sort", new ProgramDescription(Sort.class, + "Sort binary keys and values.")); + programs.put("writer", new ProgramDescription(RandomWriter.class, + "Write random binary key/value pairs")); // Make sure they gave us a program name. if (args.length == 0) { Added: lucene/hadoop/trunk/src/examples/org/apache/hadoop/examples/RandomWriter.java URL: http://svn.apache.org/viewcvs/lucene/hadoop/trunk/src/examples/org/apache/hadoop/examples/RandomWriter.java?rev=399011&view=auto ============================================================================== --- lucene/hadoop/trunk/src/examples/org/apache/hadoop/examples/RandomWriter.java (added) +++ lucene/hadoop/trunk/src/examples/org/apache/hadoop/examples/RandomWriter.java Tue May 2 11:48:52 2006 @@ -0,0 +1,210 @@ +/** + * Copyright 2006 The Apache Software Foundation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.examples; + +import java.io.IOException; +import java.text.NumberFormat; +import java.util.Date; +import java.util.Iterator; +import java.util.Random; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FSDataOutputStream; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.io.BytesWritable; +import org.apache.hadoop.io.SequenceFile; +import org.apache.hadoop.io.UTF8; +import org.apache.hadoop.io.Writable; +import org.apache.hadoop.io.WritableComparable; +import org.apache.hadoop.mapred.ClusterStatus; +import org.apache.hadoop.mapred.JobClient; +import org.apache.hadoop.mapred.JobConf; +import org.apache.hadoop.mapred.Mapper; +import org.apache.hadoop.mapred.MapReduceBase; +import org.apache.hadoop.mapred.OutputCollector; +import org.apache.hadoop.mapred.Reducer; +import org.apache.hadoop.mapred.Reporter; + +/** + * This program uses map/reduce to just run a distributed job where there is + * no interaction between the tasks and each task write a large unsorted + * random binary sequence file of BytesWritable. + * + * @author Owen O'Malley + */ +public class RandomWriter extends MapReduceBase implements Reducer { + + public static class Map extends MapReduceBase implements Mapper { + private FileSystem fileSys = null; + private long numBytesToWrite; + private int minKeySize; + private int keySizeRange; + private int minValueSize; + private int valueSizeRange; + private Random random = new Random(); + private BytesWritable randomKey = new BytesWritable(); + private BytesWritable randomValue = new BytesWritable(); + + private void randomizeBytes(byte[] data, int offset, int length) { + for(int i=offset + length - 1; i >= offset; --i) { + data[i] = (byte) random.nextInt(256); + } + } + + /** + * Given an output filename, write a bunch of random records to it. + */ + public void map(WritableComparable key, + Writable value, + OutputCollector output, + Reporter reporter) throws IOException { + String filename = ((UTF8) value).toString(); + SequenceFile.Writer writer = + new SequenceFile.Writer(fileSys, new Path(filename), + BytesWritable.class, BytesWritable.class); + int itemCount = 0; + while (numBytesToWrite > 0) { + int keyLength = random.nextInt(keySizeRange) + minKeySize; + randomKey.setSize(keyLength); + randomizeBytes(randomKey.get(), 0, randomKey.getSize()); + int valueLength = random.nextInt(valueSizeRange) + minValueSize; + randomValue.setSize(valueLength); + randomizeBytes(randomValue.get(), 0, randomValue.getSize()); + writer.append(randomKey, randomValue); + numBytesToWrite -= keyLength + valueLength; + if (++itemCount % 200 == 0) { + reporter.setStatus("wrote record " + itemCount + ". " + + numBytesToWrite + " bytes left."); + } + } + reporter.setStatus("done with " + itemCount + " records."); + writer.close(); + } + + /** + * Save the values out of the configuaration that we need to write + * the data. + */ + public void configure(JobConf job) { + try { + fileSys = FileSystem.get(job); + } catch (IOException e) { + throw new RuntimeException("Can't get default file system", e); + } + numBytesToWrite = job.getLong("test.randomwrite.bytes_per_map", + 1*1024*1024*1024); + minKeySize = job.getInt("test.randomwrite.min_key", 10); + keySizeRange = + job.getInt("test.randomwrite.max_key", 1000) - minKeySize; + minValueSize = job.getInt("test.randomwrite.min_value", 0); + valueSizeRange = + job.getInt("test.randomwrite.max_value", 20000) - minValueSize; + } + + } + + public void reduce(WritableComparable key, + Iterator values, + OutputCollector output, + Reporter reporter) throws IOException { + // nothing + } + + /** + * This is the main routine for launching a distributed random write job. + * It runs 10 maps/node and each node writes 1 gig of data to a DFS file. + * The reduce doesn't do anything. + * + * This program uses a useful pattern for dealing with Hadoop's constraints + * on InputSplits. Since each input split can only consist of a file and + * byte range and we want to control how many maps there are (and we don't + * really have any inputs), we create a directory with a set of artificial + * files that each contain the filename that we want a given map to write + * to. Then, using the text line reader and this "fake" input directory, we + * generate exactly the right number of maps. Each map gets a single record + * that is the filename it is supposed to write its output to. + * @throws IOException + */ + public static void main(String[] args) throws IOException { + Configuration defaults = new Configuration(); + if (args.length == 0) { + System.out.println("Usage: writer <out-dir> [<config>]"); + return; + } + Path outDir = new Path(args[0]); + if (args.length >= 2) { + defaults.addFinalResource(new Path(args[1])); + } + + JobConf jobConf = new JobConf(defaults, RandomWriter.class); + jobConf.setJobName("random-writer"); + + // turn off speculative execution, because DFS doesn't handle + // multiple writers to the same file. + jobConf.setSpeculativeExecution(false); + jobConf.setOutputKeyClass(BytesWritable.class); + jobConf.setOutputValueClass(BytesWritable.class); + + jobConf.setMapperClass(Map.class); + jobConf.setReducerClass(RandomWriter.class); + + JobClient client = new JobClient(jobConf); + ClusterStatus cluster = client.getClusterStatus(); + int numMaps = cluster.getTaskTrackers() * + jobConf.getInt("test.randomwriter.maps_per_host", 10); + jobConf.setNumMapTasks(numMaps); + System.out.println("Running " + numMaps + " maps."); + jobConf.setNumReduceTasks(1); + + Path tmpDir = new Path("random-work"); + Path inDir = new Path(tmpDir, "in"); + Path fakeOutDir = new Path(tmpDir, "out"); + FileSystem fileSys = FileSystem.get(jobConf); + fileSys.delete(tmpDir); + fileSys.delete(outDir); + fileSys.mkdirs(inDir); + NumberFormat numberFormat = NumberFormat.getInstance(); + numberFormat.setMinimumIntegerDigits(6); + numberFormat.setGroupingUsed(false); + + for(int i=0; i < numMaps; ++i) { + Path file = new Path(inDir, "part"+i); + FSDataOutputStream writer = fileSys.create(file); + writer.writeBytes(outDir + "/part" + numberFormat.format(i)+ "\n"); + writer.close(); + } + jobConf.setInputPath(inDir); + jobConf.setOutputPath(fakeOutDir); + + // Uncomment to run locally in a single process + //job_conf.set("mapred.job.tracker", "local"); + + Date startTime = new Date(); + System.out.println("Job started: " + startTime); + try { + JobClient.runJob(jobConf); + Date endTime = new Date(); + System.out.println("Job ended: " + endTime); + System.out.println("The job took " + + (endTime.getTime() - startTime.getTime()) /1000 + " seconds."); + } finally { + fileSys.delete(tmpDir); + } + } + +} Added: lucene/hadoop/trunk/src/examples/org/apache/hadoop/examples/Sort.java URL: http://svn.apache.org/viewcvs/lucene/hadoop/trunk/src/examples/org/apache/hadoop/examples/Sort.java?rev=399011&view=auto ============================================================================== --- lucene/hadoop/trunk/src/examples/org/apache/hadoop/examples/Sort.java (added) +++ lucene/hadoop/trunk/src/examples/org/apache/hadoop/examples/Sort.java Tue May 2 11:48:52 2006 @@ -0,0 +1,122 @@ +/** + * Copyright 2006 The Apache Software Foundation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.examples; + +import java.io.*; +import java.util.*; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.io.BytesWritable; +import org.apache.hadoop.mapred.*; +import org.apache.hadoop.mapred.lib.*; +import org.apache.hadoop.fs.*; + +/** + * This is the trivial map/reduce program that does absolutely nothing + * other than use the framework to fragment and sort the input values. + * + * To run: bin/hadoop jar build/hadoop-examples.jar sort + * [-m <i>maps</i>] [-r <i>reduces</i>] <i>in-dir</i> <i>out-dir</i> + * + * @author Owen O'Malley + */ +public class Sort { + + static void printUsage() { + System.out.println("sort [-m <maps>] [-r <reduces>] <input> <output>"); + System.exit(1); + } + + /** + * The main driver for sort program. + * Invoke this method to submit the map/reduce job. + * @throws IOException When there is communication problems with the + * job tracker. + */ + public static void main(String[] args) throws IOException { + Configuration defaults = new Configuration(); + + JobConf jobConf = new JobConf(defaults, Sort.class); + jobConf.setJobName("sorter"); + + jobConf.setInputFormat(SequenceFileInputFormat.class); + jobConf.setOutputFormat(SequenceFileOutputFormat.class); + + jobConf.setInputKeyClass(BytesWritable.class); + jobConf.setInputValueClass(BytesWritable.class); + jobConf.setOutputKeyClass(BytesWritable.class); + jobConf.setOutputValueClass(BytesWritable.class); + + jobConf.setMapperClass(IdentityMapper.class); + jobConf.setReducerClass(IdentityReducer.class); + + JobClient client = new JobClient(jobConf); + ClusterStatus cluster = client.getClusterStatus(); + int num_maps = cluster.getTaskTrackers() * + jobConf.getInt("test.sort.maps_per_host", 10); + int num_reduces = cluster.getTaskTrackers() * + jobConf.getInt("test.sort.reduces_per_host", 10); + List otherArgs = new ArrayList(); + for(int i=0; i < args.length; ++i) { + try { + if ("-m".equals(args[i])) { + num_maps = Integer.parseInt(args[++i]); + } else if ("-r".equals(args[i])) { + num_reduces = Integer.parseInt(args[++i]); + } else { + otherArgs.add(args[i]); + } + } catch (NumberFormatException except) { + System.out.println("ERROR: Integer expected instead of " + args[i]); + printUsage(); + } catch (ArrayIndexOutOfBoundsException except) { + System.out.println("ERROR: Required parameter missing from " + + args[i-1]); + printUsage(); // exits + } + } + + jobConf.setNumMapTasks(num_maps); + jobConf.setNumReduceTasks(num_reduces); + + // Make sure there are exactly 2 parameters left. + if (otherArgs.size() != 2) { + System.out.println("ERROR: Wrong number of parameters: " + + otherArgs.size() + " instead of 2."); + printUsage(); + } + jobConf.setInputPath(new Path((String) otherArgs.get(0))); + jobConf.setOutputPath(new Path((String) otherArgs.get(1))); + + // Uncomment to run locally in a single process + //job_conf.set("mapred.job.tracker", "local"); + + System.out.println("Running on " + + cluster.getTaskTrackers() + + " nodes to sort from " + + jobConf.getInputPaths()[0] + " into " + + jobConf.getOutputPath() + "."); + Date startTime = new Date(); + System.out.println("Job started: " + startTime); + JobClient.runJob(jobConf); + Date end_time = new Date(); + System.out.println("Job ended: " + end_time); + System.out.println("The job took " + + (end_time.getTime() - startTime.getTime()) /1000 + " seconds."); + } + +} From hadoop-commits-returnbkim@example.net Tue May 02 20:08:12 2006 Return-Path: <hadoop-commits-returnbkim@example.net> Delivered-To: bkim@example.net Received: (qmail 80540 invoked from network); 2 May 2006 20:08:11 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 May 2006 20:08:11 -0000 Received: (qmail 57862 invoked by uid 500); 2 May 2006 20:08:11 -0000 Delivered-To: bkim@example.net Received: (qmail 57845 invoked by uid 500); 2 May 2006 20:08:10 -0000 Mailing-List: contact bkim@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:bkim@example.net> List-Unsubscribe: <mailto:bkim@example.net> List-Post: <mailto:bkim@example.net> List-Id: <hadoop-commits.lucene.apache.org> Reply-To: bkim@example.net Delivered-To: mailing list bkim@example.net Received: (qmail 57836 invoked by uid 99); 2 May 2006 20:08:10 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 May 2006 13:08:10 -0700 X-ASF-Spam-Status: No, hits=-8.6 required=10.0 tests=ALL_TRUSTED,INFO_TLD,NO_REAL_NAME X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO minotaur.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with SMTP; Tue, 02 May 2006 13:08:10 -0700 Received: (qmail 80419 invoked by uid 65534); 2 May 2006 20:07:49 -0000 Message-ID: <bkim@example.net> Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r399030 - /lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/JobClient.java Date: Tue, 02 May 2006 20:07:49 -0000 To: bkim@example.net From: bkim@example.net X-Mailer: svnmailer-1.0.8 X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N Author: cutting Date: Tue May 2 13:07:47 2006 New Revision: 399030 URL: http://svn.apache.org/viewcvs?rev=399030&view=rev Log: HADOOP-188. More fixes to to JobClient, following on HADOOP-174. Contributed by Owen. Modified: lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/JobClient.java Modified: lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/JobClient.java URL: http://svn.apache.org/viewcvs/lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/JobClient.java?rev=399030&r1=399029&r2=399030&view=diff ============================================================================== --- lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/JobClient.java (original) +++ lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/JobClient.java Tue May 2 13:07:47 2006 @@ -308,26 +308,30 @@ running = jc.submitJob(job); String jobId = running.getJobID(); LOG.info("Running job: " + jobId); - while (!running.isComplete()) { + while (true) { try { Thread.sleep(1000); } catch (InterruptedException e) {} try { + if (running.isComplete()) { + break; + } running = jc.getJob(jobId); + String report = (" map " + Math.round(running.mapProgress()*100)+ + "% reduce " + + Math.round(running.reduceProgress()*100) + "%"); + if (!report.equals(lastReport)) { + LOG.info(report); + lastReport = report; + } retries = MAX_RETRIES; } catch (IOException ie) { if (--retries == 0) { - LOG.info("Final attempt failed, killing job."); + LOG.warning("Final attempt failed, killing job."); throw ie; } LOG.info("Communication problem with server: " + StringUtils.stringifyException(ie)); - } - String report = null; - report = " map "+Math.round(running.mapProgress()*100)+"% reduce " + Math.round(running.reduceProgress()*100)+"%"; - if (!report.equals(lastReport)) { - LOG.info(report); - lastReport = report; } } if (!running.isSuccessful()) { From hadoop-commits-returnbkim@example.net Tue May 02 22:05:33 2006 Return-Path: <hadoop-commits-returnbkim@example.net> Delivered-To: bkim@example.net Received: (qmail 36275 invoked from network); 2 May 2006 22:05:33 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 May 2006 22:05:33 -0000 Received: (qmail 72296 invoked by uid 500); 2 May 2006 22:05:32 -0000 Delivered-To: bkim@example.net Received: (qmail 72276 invoked by uid 500); 2 May 2006 22:05:31 -0000 Mailing-List: contact bkim@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:bkim@example.net> List-Unsubscribe: <mailto:bkim@example.net> List-Post: <mailto:bkim@example.net> List-Id: <hadoop-commits.lucene.apache.org> Reply-To: bkim@example.net Delivered-To: mailing list bkim@example.net Received: (qmail 72267 invoked by uid 99); 2 May 2006 22:05:31 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 May 2006 15:05:31 -0700 X-ASF-Spam-Status: No, hits=-8.6 required=10.0 tests=ALL_TRUSTED,INFO_TLD,NO_REAL_NAME X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO minotaur.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with SMTP; Tue, 02 May 2006 15:05:30 -0700 Received: (qmail 36161 invoked by uid 65534); 2 May 2006 22:05:10 -0000 Message-ID: <bkim@example.net> Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r399065 - in /lucene/hadoop/trunk: CHANGES.txt src/java/org/apache/hadoop/mapred/JobTracker.java Date: Tue, 02 May 2006 22:05:09 -0000 To: bkim@example.net From: bkim@example.net X-Mailer: svnmailer-1.0.8 X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N Author: cutting Date: Tue May 2 15:05:08 2006 New Revision: 399065 URL: http://svn.apache.org/viewcvs?rev=399065&view=rev Log: HADOOP-185. Fix so that, if a task tracker times out making the RPC asking for a new task to run, the job tracker does not think that it is actually running the task returned (but never received). Contributed by Owen. Modified: lucene/hadoop/trunk/CHANGES.txt lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/JobTracker.java Modified: lucene/hadoop/trunk/CHANGES.txt URL: http://svn.apache.org/viewcvs/lucene/hadoop/trunk/CHANGES.txt?rev=399065&r1=399064&r2=399065&view=diff ============================================================================== --- lucene/hadoop/trunk/CHANGES.txt (original) +++ lucene/hadoop/trunk/CHANGES.txt Tue May 2 15:05:08 2006 @@ -151,6 +151,10 @@ files containing random data. The second sorts the output of the first. (omalley via cutting) +40. HADOOP-185. Fix so that, when a task tracker times out making the + RPC asking for a new task to run, the job tracker does not think + that it is actually running the task returned. (omalley via cutting) + Release 0.1.1 - 2006-04-08 1. Added CHANGES.txt, logging all significant changes to Hadoop. (cutting) Modified: lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/JobTracker.java URL: http://svn.apache.org/viewcvs/lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/JobTracker.java?rev=399065&r1=399064&r2=399065&view=diff ============================================================================== --- lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/JobTracker.java (original) +++ lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/JobTracker.java Tue May 2 15:05:08 2006 @@ -76,6 +76,93 @@ return tracker; } + /** + * A thread to timeout tasks that have been assigned to task trackers, + * but that haven't reported back yet. + * Note that I included a stop() method, even though there is no place + * where JobTrackers are cleaned up. + * @author Owen O'Malley + */ + private class ExpireLaunchingTasks implements Runnable { + private volatile boolean shouldRun = true; + /** + * This is a map of the tasks that have been assigned to task trackers, + * but that have not yet been seen in a status report. + * map: task-id (String) -> time-assigned (Long) + */ + private Map launchingTasks = new LinkedHashMap(); + private static final String errorMsg = "Error launching task"; + private static final String errorHost = "n/a"; + + public void run() { + try { + while (shouldRun) { + // Every 3 minutes check for any tasks that are overdue + Thread.sleep(TASKTRACKER_EXPIRY_INTERVAL/3); + long now = System.currentTimeMillis(); + LOG.fine("Starting launching task sweep"); + synchronized (launchingTasks) { + Iterator itr = launchingTasks.entrySet().iterator(); + while (itr.hasNext()) { + Map.Entry pair = (Map.Entry) itr.next(); + String taskId = (String) pair.getKey(); + long age = now - ((Long) pair.getValue()).longValue(); + LOG.fine(taskId + " is " + age + " ms old."); + if (age > TASKTRACKER_EXPIRY_INTERVAL) { + LOG.info("Launching task " + taskId + " timed out."); + TaskInProgress tip = null; + synchronized (JobTracker.this) { + tip = (TaskInProgress) taskidToTIPMap.get(taskId); + } + if (tip != null) { + synchronized (tip) { + JobInProgress job = tip.getJob(); + // record why the job failed, so that the user can + // see the problem + TaskStatus status = + new TaskStatus(taskId, + tip.isMapTask(), + 0.0f, + TaskStatus.FAILED, + errorMsg, + errorMsg, + errorHost); + tip.updateStatus(status); + job.failedTask(tip, taskId, errorHost); + } + } + itr.remove(); + } else { + // the tasks are sorted by start time, so once we find + // one that we want to keep, we are done for this cycle. + break; + } + } + } + } + } catch (InterruptedException ie) { + // all done + } + } + + public void addNewTask(String taskName) { + synchronized (launchingTasks) { + launchingTasks.put(taskName, + Long.valueOf(System.currentTimeMillis())); + } + } + + public void removeTask(String taskName) { + synchronized (launchingTasks) { + launchingTasks.remove(taskName); + } + } + + public void stop() { + shouldRun = false; + } + } + /////////////////////////////////////////////////////// // Used to expire TaskTrackers that have gone down /////////////////////////////////////////////////////// @@ -277,7 +364,9 @@ ExpireTrackers expireTrackers = new ExpireTrackers(); RetireJobs retireJobs = new RetireJobs(); JobInitThread initJobs = new JobInitThread(); - + ExpireLaunchingTasks expireLaunchingTasks = new ExpireLaunchingTasks(); + Thread expireLaunchingTaskThread = new Thread(expireLaunchingTasks); + /** * It might seem like a bug to maintain a TreeSet of status objects, * which can be updated at any time. But that's not what happens! We @@ -346,12 +435,12 @@ this.port = addr.getPort(); this.interTrackerServer = RPC.getServer(this, addr.getPort(), 10, false, conf); this.interTrackerServer.start(); - Properties p = System.getProperties(); - for (Iterator it = p.keySet().iterator(); it.hasNext(); ) { - String key = (String) it.next(); - String val = (String) p.getProperty(key); - LOG.info("Property '" + key + "' is " + val); - } + Properties p = System.getProperties(); + for (Iterator it = p.keySet().iterator(); it.hasNext(); ) { + String key = (String) it.next(); + String val = (String) p.getProperty(key); + LOG.info("Property '" + key + "' is " + val); + } this.infoPort = conf.getInt("mapred.job.tracker.info.port", 50030); this.infoServer = new JobTrackerInfoServer(this, infoPort); @@ -362,6 +451,7 @@ new Thread(this.expireTrackers).start(); new Thread(this.retireJobs).start(); new Thread(this.initJobs).start(); + expireLaunchingTaskThread.start(); } public static InetSocketAddress getAddress(Configuration conf) { @@ -622,7 +712,8 @@ Task t = job.obtainNewMapTask(taskTracker, tts); if (t != null) { - return t; + expireLaunchingTasks.addNewTask(t.getTaskId()); + return t; } // @@ -656,7 +747,8 @@ Task t = job.obtainNewReduceTask(taskTracker, tts); if (t != null) { - return t; + expireLaunchingTasks.addNewTask(t.getTaskId()); + return t; } // @@ -878,10 +970,12 @@ for (Iterator it = status.taskReports(); it.hasNext(); ) { TaskStatus report = (TaskStatus) it.next(); report.setHostname(status.getHost()); - TaskInProgress tip = (TaskInProgress) taskidToTIPMap.get(report.getTaskId()); + String taskId = report.getTaskId(); + TaskInProgress tip = (TaskInProgress) taskidToTIPMap.get(taskId); if (tip == null) { LOG.info("Serious problem. While updating status, cannot find taskid " + report.getTaskId()); } else { + expireLaunchingTasks.removeTask(taskId); JobInProgress job = tip.getJob(); job.updateTaskStatus(tip, report); From hadoop-commits-returnbkim@example.net Tue May 02 22:53:56 2006 Return-Path: <hadoop-commits-returnbkim@example.net> Delivered-To: bkim@example.net Received: (qmail 54316 invoked from network); 2 May 2006 22:53:56 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 May 2006 22:53:56 -0000 Received: (qmail 37103 invoked by uid 500); 2 May 2006 22:53:56 -0000 Delivered-To: bkim@example.net Received: (qmail 37052 invoked by uid 500); 2 May 2006 22:53:55 -0000 Mailing-List: contact bkim@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:bkim@example.net> List-Unsubscribe: <mailto:bkim@example.net> List-Post: <mailto:bkim@example.net> List-Id: <hadoop-commits.lucene.apache.org> Reply-To: bkim@example.net Delivered-To: mailing list bkim@example.net Received: (qmail 37036 invoked by uid 99); 2 May 2006 22:53:55 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 May 2006 15:53:55 -0700 X-ASF-Spam-Status: No, hits=-9.4 required=10.0 tests=ALL_TRUSTED,NO_REAL_NAME X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO minotaur.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with SMTP; Tue, 02 May 2006 15:53:55 -0700 Received: (qmail 54199 invoked by uid 65534); 2 May 2006 22:53:34 -0000 Message-ID: <bkim@example.net> Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r399074 - in /lucene/hadoop/trunk: CHANGES.txt src/java/org/apache/hadoop/mapred/TaskTracker.java Date: Tue, 02 May 2006 22:53:34 -0000 To: bkim@example.net From: bkim@example.net X-Mailer: svnmailer-1.0.8 X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N Author: cutting Date: Tue May 2 15:53:31 2006 New Revision: 399074 URL: http://svn.apache.org/viewcvs?rev=399074&view=rev Log: Fix HADOOP-19. If a child process hangs after it has reported completion its output should not be lost. Modified: lucene/hadoop/trunk/CHANGES.txt lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/TaskTracker.java Modified: lucene/hadoop/trunk/CHANGES.txt URL: http://svn.apache.org/viewcvs/lucene/hadoop/trunk/CHANGES.txt?rev=399074&r1=399073&r2=399074&view=diff ============================================================================== --- lucene/hadoop/trunk/CHANGES.txt (original) +++ lucene/hadoop/trunk/CHANGES.txt Tue May 2 15:53:31 2006 @@ -155,6 +155,10 @@ RPC asking for a new task to run, the job tracker does not think that it is actually running the task returned. (omalley via cutting) +41. HADOOP-190. If a child process hangs after it has reported + completion, its output should not be lost. (Stack via cutting) + + Release 0.1.1 - 2006-04-08 1. Added CHANGES.txt, logging all significant changes to Hadoop. (cutting) Modified: lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/TaskTracker.java URL: http://svn.apache.org/viewcvs/lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/TaskTracker.java?rev=399074&r1=399073&r2=399074&view=diff ============================================================================== --- lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/TaskTracker.java (original) +++ lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/TaskTracker.java Tue May 2 15:53:31 2006 @@ -573,7 +573,8 @@ } runstate = TaskStatus.FAILED; } - needCleanup = wasKilled || runstate == TaskStatus.FAILED; + + needCleanup = runstate == TaskStatus.FAILED; } // From hadoop-commits-returnbkim@example.net Tue May 02 23:07:59 2006 Return-Path: <hadoop-commits-returnbkim@example.net> Delivered-To: bkim@example.net Received: (qmail 60911 invoked from network); 2 May 2006 23:07:59 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 May 2006 23:07:59 -0000 Received: (qmail 50665 invoked by uid 500); 2 May 2006 23:07:59 -0000 Delivered-To: bkim@example.net Received: (qmail 50647 invoked by uid 500); 2 May 2006 23:07:58 -0000 Mailing-List: contact bkim@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:bkim@example.net> List-Unsubscribe: <mailto:bkim@example.net> List-Post: <mailto:bkim@example.net> List-Id: <hadoop-commits.lucene.apache.org> Reply-To: bkim@example.net Delivered-To: mailing list bkim@example.net Received: (qmail 50638 invoked by uid 99); 2 May 2006 23:07:58 -0000 Received: from asf.osuosl.org (HELO
{'source': 'Apache Software Foundation Public Mail Archives'}