1. A method for learning a data format, the method comprising: inputting an initial description of a data format and a batch of data comprising data in a new data format not covered by the initial description, instructions to use the first description to parse the records in the data source; discarding records in the input data that parse successfully, instructions to collect records that fail to parse, instructions to accumulate a quantity, M of records that fail to parse, returning a modified description that extends the initial description to cover the new data, transforming the first description, D into a second description D′ to accommodate differences between the input data format and the first description D by introducing options where a piece of data was missing in the input data and introducing unions where a new type of data was found in the input data; and using a non-incremental format inference system to infer descriptions for the aggregated portions of input data that did not parse using the first description D; and ranking the parses by a metric that measures their quality and return only the top quantity k of the parses metric comprises a triple: m=(e, s, c), where e is a quantity of errors, s is a quantity of characters skipped during Sync token recovery, and c is a quantity of characters correctly parsed.
1. A method for learning a data format, the method comprising: inputting an initial description of a data format and a batch of data comprising data in a new data format not covered by the initial description, instructions to use the first description to parse the records in the data source; discarding records in the input data that parse successfully, instructions to collect records that fail to parse, instructions to accumulate a quantity, M of records that fail to parse, returning a modified description that extends the initial description to cover the new data, transforming the first description, D into a second description D′ to accommodate differences between the input data format and the first description D by introducing options where a piece of data was missing in the input data and introducing unions where a new type of data was found in the input data; and using a non-incremental format inference system to infer descriptions for the aggregated portions of input data that did not parse using the first description D; and ranking the parses by a metric that measures their quality and return only the top quantity k of the parses metric comprises a triple: m=(e, s, c), where e is a quantity of errors, s is a quantity of characters skipped during Sync token recovery, and c is a quantity of characters correctly parsed. 9. The method of claim 1 , the method further comprising: maintaining a global memo table data structure embedded in the non-transitory computer readable medium, wherein the data structure stores data used by the processor and indexed by the pair of a description D and the beginning position for parsing D which stores the result for parsing D at the specific position; and bounding the total number of aggregates the produces by selecting the top quantity, k aggregates with the fewest number of Opt and Learn nodes.