title: Deserializing an array that contains some non-deserializable objects using Kryo (salvaging the deserializable parts)
body: <h2>Background</h2>
  
  <p>I am attempting to write a Kryo deserialization in such a way that if an array of objects contains <strong>some</strong> objects that (due to code change) can't be deserialized then those references in the array will become null rather than throwing an exception; allowing the remainder of the object to be salvaged. I have previously been using Java's inbuilt serialization and within that I have been able to achieve this by writing a "known good" integer between each item in the array and then looking for that in the stream if an error occurs to find the start of the next object. This is detailed the question <a href="https://stackoverflow.com/questions/20914158/deserializing-an-array-that-contains-some-non-deserializable-objects-salvaging">Deserializing an array that contains some non-deserializable objects (salvaging the deserializable parts)</a>. </p>
  
  <p>I have now moved over to Kryo serialization for reasons of efficiency and have attempted to recreate this approach, however within Kryo this error recovery seems to work once, but after that it does not recover correctly.</p>
  
  <h2>What I've tried</h2>
  
  <p>I have attempted to write a known good integer(<code>END_OF_APPLE_MAGIC</code>) between each object in the array of <code>Apple</code>s during serialization. During deserialization when a <code>BadApple</code> is found which cannot be deserialized it is replaced by an <code>ErrorApple</code> (analogy is getting weak) and the <code>END_OF_APPLE_MAGIC</code> is searched for to find where to look for the next apple. This works if there is a single <code>BadApple</code> in the array and the <code>BadApple</code> is not the first entry. But fails in assorted ways (see detailed analysis) if more than 1 <code>BadApple</code> is in the array or the first <code>Apple</code> is a <code>BadApple</code> </p>
  
  <pre><code>public class AppleHolder implements Serializable,KryoSerializable{
      static int END_OF_APPLE_MAGIC=1234467895; //if this just "turns up" in the stream we will have a bad day; however this is only the case in recovery mode, so is an acceptable risk
  
      int numberOfApples=6;
      Apple[] apples=new Apple[numberOfApples];
      double otherData=15;
  
      //these are just for debug
      int dividers=0; //counts number of times END_OF_APPLE_MAGIC is found
      int problems=0; //counts number of times an apple fails to load
      int badIntegers=0; //counts number of times END_OF_APPLE_MAGIC is looked for and a different Integer is found (I have never seen this happen)
  
      public AppleHolder(){
          Apple goodApple=new Apple("GoodApple","tastyGood");
          BadApple badApple=new BadApple("BadApple","untastyBad");
  
          apples[0]=goodApple;
          apples[1]=badApple;
          apples[2]=goodApple;
          apples[3]=goodApple; // multiple references to same object intentional
          apples[4]=goodApple;
          apples[5]=goodApple;
  
  
      }
  
  
      public void write (Kryo kryo, Output output) {
          for(int i=0;i&lt;apples.length;i++){
  
              //kryo.writeObject(output, apples[i]);
              kryo.writeClassAndObject(output, apples[i]);
              kryo.writeClassAndObject(output, END_OF_APPLE_MAGIC);
          }
  
          kryo.writeObject(output,otherData);
      }
  
  
      public void read (Kryo kryo, Input input) {
          try{
  
              apples =new Apple[numberOfApples];
              for(int i=0;i&lt;apples.length;i++){
  
                  try{
                      Object ob=kryo.readClassAndObject(input);
                      apples[i]=(Apple)ob;
                  }catch(Exception e){
                      apples[i]=new ErrorApple();
                      problems++;
                  }
  
                  //Search for next Apple Boundary (except in recovery mode 
                  //it will be the next entry)
                  boolean atBoundary=false;
                  while (atBoundary==false){ //should probably put a limit on this just in case
                      try{
                          int appleMagic =(Integer)kryo.readClassAndObject(input);
                          if (appleMagic == END_OF_APPLE_MAGIC){
                              atBoundary=true;
                              dividers++;
                          }else{
                              badIntegers++;
                          }
                      }catch(Exception e){
                          //painful byte reading mode only entered in recovery mode; under good 
                          //situations it does not represent an efficiency problem
                          input.skip(1); //consume byte of bad input
                          //Where buffer underflow exceptions occur they occur here
                      }
                  }
  
  
              }
              otherData = kryo.readObject(input, Double.class);
  
          }catch(Exception e){
              //something when wrong (always a Buffer underflow so far), print what we have
  
  
              for(int i=0;i&lt;apples.length;i++){
                  System.out.println(apples[i]);
  
              }
  
              throw e;
          }
  
      }
  
  
      public static void main(String[] args)
              throws Exception {
  
          /*
           * (1) First run serialize()
           * (2) Rename/delete badApple such that it cannot be found for deserialization
           * (3) Run deSerialize(()
           */
  
  
          serialize();
  
  
          //deSerialize();
  
      }
  
      public static void serialize() throws Exception{
          AppleHolder testWrite = new AppleHolder();
          /*FileOutputStream fos = new FileOutputStream("testfile");
          ObjectOutputStream oos = new ObjectOutputStream(fos);
          oos.writeObject(testWrite);
          oos.flush();
          oos.close();
          */
  
          Kryo kryo = new Kryo();
          Output output = new Output(new FileOutputStream("testfile"));
          kryo.writeObject(output, testWrite);
          output.close();
  
      }
  
      public static void deSerialize() throws Exception{
          /*AppleHolder testRead;
          FileInputStream fis = new FileInputStream("testfile");
          ObjectInputStream ois = new ObjectInputStream(fis);
          testRead = (AppleHolder) ois.readObject();
          ois.close();
          */
  
          Kryo kryo = new Kryo();
          Input input = new Input(new FileInputStream("testfile"));
          AppleHolder testRead = kryo.readObject(input, AppleHolder.class);
          input.close();
  
          for(int i=0;i&lt;testRead.apples.length;i++){
              System.out.println(testRead.apples[i]);
  
          }
  
          System.out.println("otherData: " + testRead.otherData);
  
      }
  
  }
  
  public class Apple implements Serializable {
      private String propertyOne;
      private String propertyTwo;
  
      public Apple(){}
  
      public Apple(String propertyOne, String propertyTwo) {
          this.propertyOne = propertyOne;
          this.propertyTwo = propertyTwo;
          validate();
      }
  
      private void writeObject(ObjectOutputStream o)
              throws IOException {
  
          o.writeObject(propertyOne);
          o.writeObject(propertyTwo);
      }
  
      private void readObject(ObjectInputStream o)
              throws IOException, ClassNotFoundException {
  
          propertyOne = (String) o.readObject();
          propertyTwo = (String) o.readObject();
          validate();
      }
  
      private void validate(){
          if(propertyOne == null ||
                  propertyOne.length() == 0 ||
                  propertyTwo == null ||
                  propertyTwo.length() == 0){
  
              throw new IllegalArgumentException();
          }
      }
  
      public String getPropertyOne() {
          return propertyOne;
      }
  
      public String getPropertyTwo() {
          return propertyTwo;
      }
  
      @Override
      public String toString() {
          return "goodApple";
      }
  
  
  
  }
  
  public class BadApple extends Apple {
  
  
      public BadApple(){}
  
      public BadApple(String propertyOne, String propertyTwo) {
          super(propertyOne, propertyTwo);
      }
  
      @Override
      public String toString() {
          return "badApple";
      }
  }
  
  public class ErrorApple extends Apple {
  
  
      public ErrorApple(){}
  
      public ErrorApple(String propertyOne, String propertyTwo) {
          super(propertyOne, propertyTwo);
      }
  
      @Override
      public String toString() {
          return "errorApple";
      }
  
  }
  </code></pre>
  
  <h2>Question</h2>
  
  <p>How can I salvage a Kyro serialized array in which only some of the objects are deserializable? Thereby getting an array with <code>ErrorApple</code> entries for the non deserializable parts. Within my array I have several references to the same object in a single array, it is essential that this is preserved in the deserialization process.</p>
  
  <p>So going into serialization I have</p>
  
  <pre><code>[GoodApple]
  [GoodApple]  
  [GoodApple]  
  [BadApple]
  [BadApple]
  [GoodApple] 
  </code></pre>
  
  <p>And coming out of deserialization I want (because badApple has changed and cannot be deserialised</p>
  
  <pre><code>[GoodApple]
  [GoodApple]  
  [GoodApple]  
  [ErrorApple]
  [ErrorApple]
  [GoodApple]  
  </code></pre>
  
  <p>I want this to provide a fallback where backwards compatibility cannot be achieved or a 3rd party modification to my program that was previously installed is removed</p>
  
  <h2>Detailed Analysis</h2>
  
  <p>This section outlines the ways in which the existing program fails. </p>
  
  <p>In general</p>
  
  <ul>
  <li>A single <code>BadApple</code> somewhere other than at the first position in the array will function correctly</li>
  <li>A <code>BadApple</code> at the first position in the array will lead to the next <code>Apple</code> reading correctly then <code>ErrorApples</code> from then on (even for good <code>Apple</code>s)</li>
  <li>Where there are more than 1 <code>BadApple</code> the first good <code>Apple</code> after the second <code>BadApple</code> will read correctly <strong>but may be moved forwards in the array</strong> and then <code>ErrorApples</code> from then on (even for good <code>Apple</code>s). There will be a <code>KryoException: Buffer underflow</code> and there may be null entries at the end of the array.</li>
  </ul>
  
  <p>The inputs and outputs I used are shown below:</p>
  
  <pre>
      In          Out
      [goodApple] [goodApple]  
      [goodApple] [goodApple]  
      [badApple]  [badApple]  
      [goodApple] [goodApple]  
      [goodApple] [goodApple]  
      [goodApple] [goodApple]  
  
      In          Out
      [badApple]  [errorApple]  
      [goodApple] [goodApple]  
      [goodApple] [errorApple]  
      [goodApple] [errorApple]  
      [goodApple] [errorApple]  
      [goodApple] [errorApple]  
  
      In          Out
      [goodApple] [goodApple]  
      [badApple]  [errorApple]  
      [goodApple] [goodApple]  
      [badApple]  [errorApple]  
      [goodApple] [goodApple]  
      [goodApple] [errorApple]  
      KryoException: Buffer underflow. (occures at input.skip(1);)  
  
      In          Out
      [goodApple] [goodApple]  
      [goodApple] [goodApple]  
      [badApple]  [errorApple]  
      [badApple]  [errorApple]  
      [goodApple] [goodApple]  
      [goodApple] [errorApple]  
      KryoException: Buffer underflow (occures at input.skip(1);)  
  
      In          Out
      [goodApple] [goodApple]  
      [badApple]  [errorApple]  
      [badApple]  [errorApple]  
      [badApple]  [goodApple]  
      [goodApple] [errorApple]  
      [goodApple] [null]  
      KryoException: Buffer underflow. (occures at input.skip(1);)  
  
  </pre>