text: wq.io's
  Parser mixin classes facilitate parsing data from a loaded
  file object into a
  list of
  dicts. A parser is essentially just a class with
  parse() and
  dump() methods defined. In general, a parser class should just provide a wrapper around a third party API (e.g. csv, xml.etree or xlrd). A hypothetical parser class would look like this:
  from some_library import some_api class HypotheticalParser(BaseParser): def parse(self): self.data = some_api.load(self.file) def dump(self, file): some_api.dump(self.data, file)
  As can be seen in the above example, the
  parse() function takes no arguments, instead assuming
  self.file has already been defined by a Loader mixin. The data object should be defined as a
  list of
  dicts (e.g.
  [{"id":1},{"id":2}]. If the result returned by the API has some other structure, it should be processed to match the expected format. The
  dump() function should accept a writable file handle as an argument and use the API to write the data object back to the file.
  There are two main ways in which parser classes are customized. One way is to define a completely new class to support a file format or API not currently supported by the built-in wq.io parsers. The other way, which is much more common, is to extend or change the behavior of an existing parser. With that in mind, each of the built-in parser classes is discussed below together with common customization options and techniques.
  Two of the built-in parsers are used for file formats that are not inherently tabular and can describe arbitrary data structures. While these file data formats are not inherently tabular, they often are used represent table-like data. These parsers directly extend
  BaseParser and have the
  tabular property set to
  False.
  Non-tabular file formats allow for some records to have more fields than others. By default, wq.io only searches the first record when automatically determining field names. This can cause issues with TupleMapper in particular which expects consistent field names throughout the dataset. If this happens to you, set
  scan_fields = Trueon your class to tell wq.io scan the entire dataset when determining field names.
  The JSON parser is a simple wrapper around Python's built-in json API.
  JsonParser assumes the result of
  json.load(self.file) will either be an array or an object with an array somewhere in an inner property (in which case
  namespace should be set). Each item in the array is assumed to be a relatively flat key-value mapping. The keys of the first item in the array will be assumed to be the same for the rest of the items.
  JsonParser supports the following options, specified as properties on the class or instance:
  The XML parser is a simple wrapper around Python's built-in xml.etree API. While it can be adapted to work with arbitrary XML documents, it assumes a basic structure like the following:
  <root> <item> <id>1</id> <value>val</value> </item> <item> <id>2</id> <value>val</value> </item> </root>
  In addition to the
  parse() and
  dump() methods,
  XmlParser provides row-level methods, described below.
  The tabular parsers are geared toward handling spreadsheets and other tabular data formats. These formats are differentiated from the non-tabular formats in that there is typically a single grid structure encompassing the entire file, and the field names / column headings are listed only once (usually, but not always, in the first row of the file).
  The tabular parsers extend wq.io.base.TableParser, which defines the following properties:
  CsvParser utilizes Python's csv module to provide a CSV-capable
  TableParser.
  CsvParser leverages SkipPreludeReader, a customized DictReader that adds support for files that have extra "prelude" text before the actual header row.
  WorkbookParser)
  ExcelParser provides support for both "old" (.xls) and "new" (.xlsx) files via the xlrd module. Write support can be enabled by installing the xlwt and/or xlsxwrite modules.
  ExcelParser extends a somewhat more generic
  WorkbookParser, with the idea that the latter could eventually be extended to other "workbook" style formats like ODS.
  The methods listed above are called in turn by
  parse(), which is defined by
  WorkbookParser. Working implementations of the methods are defined in
  ExcelParser.
  Last modified on 2015-08-10 01:06 PM
  Edit this Page | Suggest Improvement
  © 2013-2019 by S. Andrew Sheppard