10. A non-transitory computer readable medium containing a computer program comprising instructions that when executed by a computer perform functions that are useful in learning a data format, the computer program comprising: instructions to input an initial description of a data format and a batch of data comprising data in a new data format not covered by the initial description, instructions to use the first description to parse the records in the data source, instructions to discard records in the input data that parse successfully, instructions to collect records that fail to parse, instructions to accumulate a quantity, M of records that fail to parse, instructions to return a modified description that extends the initial description to cover the new data, instructions to transform the first description, D into a second description D′ to accommodate differences between the input data format and the first description D by introducing options where a piece of data was missing in the input data and introducing unions where a new type of data was found in the input data; and instructions to use a non-incremental format inference system to infer descriptions for the aggregated portions of input data that did not parse using the first description D; and instructions to rank the parses by a metric that measures their quality and return only the top quantity k of the parses, wherein the metric comprises a triple: m=(e, s, c), where e is a quantity of errors, s is a quantity of characters skipped during Sync token recovery, and c is a quantity of characters correctly parsed.
10. A non-transitory computer readable medium containing a computer program comprising instructions that when executed by a computer perform functions that are useful in learning a data format, the computer program comprising: instructions to input an initial description of a data format and a batch of data comprising data in a new data format not covered by the initial description, instructions to use the first description to parse the records in the data source, instructions to discard records in the input data that parse successfully, instructions to collect records that fail to parse, instructions to accumulate a quantity, M of records that fail to parse, instructions to return a modified description that extends the initial description to cover the new data, instructions to transform the first description, D into a second description D′ to accommodate differences between the input data format and the first description D by introducing options where a piece of data was missing in the input data and introducing unions where a new type of data was found in the input data; and instructions to use a non-incremental format inference system to infer descriptions for the aggregated portions of input data that did not parse using the first description D; and instructions to rank the parses by a metric that measures their quality and return only the top quantity k of the parses, wherein the metric comprises a triple: m=(e, s, c), where e is a quantity of errors, s is a quantity of characters skipped during Sync token recovery, and c is a quantity of characters correctly parsed. 13. The medium of claim 10 , the medium further comprising a learn node data structure embedded in the non-transitory computer readable medium containing data indicating an accumulation of the data that did not parse using the first description that needs to be learned.