When using Oracle JDK 1.8.0_60 on two centos 7.1 systems one Jenkins slave fails to collect artifacts with NoClassDefFoundError
Question: Riddle me this:

How can two nearly identical jenkins slaves running with Oracle JDK 1.8 u60
Java have one fail to collect artifact with a classdef error while the other
is fine? In both case the following are identical

  * system information shown from jenkins
  * boot classpath
  * jdk
  * jdk files (these are deployed using SVN so we know they are identical)
  * yum installed (mostly see below)

**Note** I can replicate this by going to script console and running the
following

    
    
    import org.apache.tools.ant.Location
    Location l = new Location()
    

works on the good and fails in same way as a real build on the bad.

**Error**

    
    
    java.io.IOException: remote file operation failed:... at 
    hudson.remoting.Channel@35f2fb2:linengbld50: java.io.IOException: 
    Remote call on linengbld50 failed
    ...
    Caused by: java.io.IOException: Remote call on linengbld50 failed
    ...
    Caused by: java.lang.NoClassDefFoundError: Could not initialize class
     org.apache.tools.ant.Location
    ...
    

**Yum difference**

Good v Bad

  * hiera.noarch 1:1.3.4-5 vs 1.3.4-1
  * net-snmp-* 1:5.7.2-24.el7_2.1 vs 1:5.7.2-24.el7
  * ruby-augeas 0.5.0-1 vs 0.4.1-3
  * zabbix 2.4.8 vs 2.4.7

Only ON Bad

  * python-chardet.noarch 2.2.1-1.el7_1
  * python-kitchen.noarch 1.1.1-5.el7
  * lsof
  * yum-utils.noarch 1.1.31-34.el


Answer: Using Script console was a great debugging tools as it executes in the right
environment. getting output of `which java` on each node showed a difference.
@mmasi found that bad showed a path whereas good returned null.

Even though we run slave using specific JDK java it used first java found on
path for its activities (archiving).

  * modify alternatives to set java to 1.8 (centos)

sudo alternatives --install /usr/bin/java java \
/opt/tools/Java/jdk1.8.0_60/bin/java 1;java -version

  * Disconnect and reconnect node (restart the slave)

  * Repeat the tiny script console check _OK_

  * Repeat Build test (tiny build that archives a file) _OK_

