anchor.diff:                password="password"
                 host="imap.gmail.com"
                 protocol="imaps"
  -              fetchMailsSince="2009-09-20 00:00:00"
  +              fetchMailsSince="2014-06-30 00:00:00"
                 batchSize="20"
                 folders="inbox"
                 processAttachement="false"
  -              name="sample_entity"/>
  +              name="mail_entity"/>
     </document>
   </dataConfig>
   ----
   
  -The entity attributes unique to the MailEntityProcessor are shown below.
  -
  -// TODO: Change column width to %autowidth.spread when https://github.com/asciidoctor/asciidoctor-pdf/issues/599 is fixed
  -
  -[cols="30,70",options="header"]
  -|===
  -|Attribute |Use
  -|processor |Required. Must be set to "MailEntityProcessor".
  -|user |Required. Username for authenticating to the IMAP server; this is typically the email address of the mailbox owner.
  -|password |Required. Password for authenticating to the IMAP server.
  -|host |Required. The IMAP server to connect to.
  -|protocol |Required. The IMAP protocol to use, valid values are: imap, imaps, gimap, and gimaps.
  -|fetchMailsSince |Optional. Date/time used to set a filter to import messages that occur after the specified date; expected format is: `yyyy-MM-dd HH:mm:ss`.
  -|folders |Required. Comma-delimited list of folder names to pull messages from, such as "inbox".
  -|recurse |Optional (default is true). Flag to indicate if the processor should recurse all child folders when looking for messages to import.
  -|include |Optional. Comma-delimited list of folder patterns to include when processing folders (can be a literal value or regular expression).
  -|exclude |Optional. Comma-delimited list of folder patterns to exclude when processing folders (can be a literal value or regular expression); excluded folder patterns take precedence over include folder patterns.
  -a|
  -processAttachement
  -
  -or
  -
  -processAttachments
  -
  - |Optional (default is true). Use Tika to process message attachments.
  -|includeContent |Optional (default is true). Include the message body when constructing Solr documents for indexing.
  -|===
  -
  -[[UploadingStructuredDataStoreDatawiththeDataImportHandler-ImportingNewEmailsOnly]]
   ==== Importing New Emails Only
   
  -After running a full import, the MailEntityProcessor keeps track of the timestamp of the previous import so that subsequent imports can use the fetchMailsSince filter to only pull new messages from the mail server. This occurs automatically using the Data Import Handler dataimport.properties file (stored in conf). For instance, if you set `fetchMailsSince="2014-08-22 00:00:00"` in your `mail-data-config.xml`, then all mail messages that occur after this date will be imported on the first run of the importer. Subsequent imports will use the date of the previous import as the fetchMailsSince filter, so that only new emails since the last import are indexed each time.
  -
  -[[UploadingStructuredDataStoreDatawiththeDataImportHandler-GMailExtensions]]
  +After running a full import, the MailEntityProcessor keeps track of the timestamp of the previous import so that subsequent imports can use the fetchMailsSince filter to only pull new messages from the mail server. This occurs automatically using the DataImportHandler `dataimport.properties` file (stored in `conf`).
  +
  +For instance, if you set `fetchMailsSince="2014-08-22 00:00:00"` in your `mail-data-config.xml`, then all mail messages that occur after this date will be imported on the first run of the importer. Subsequent imports will use the date of the previous import as the `fetchMailsSince` filter, so that only new emails since the last import are indexed each time.
  +
   ==== GMail Extensions
   
  -When connecting to a GMail account, you can improve the efficiency of the MailEntityProcessor by setting the protocol to *gimap* or *gimaps*. This allows the processor to send the fetchMailsSince filter to the GMail server to have the date filter applied on the server, which means the processor only receives new messages from the server. However, GMail only supports date granularity, so the server-side filter may return previously seen messages if run more than once a day.
  -
  -[[UploadingStructuredDataStoreDatawiththeDataImportHandler-TheTikaEntityProcessor]]
  +When connecting to a GMail account, you can improve the efficiency of the MailEntityProcessor by setting the protocol to *gimap* or *gimaps*.
  +
  +This allows the processor to send the `fetchMailsSince` filter to the GMail server to have the date filter applied on the server, which means the processor only receives new messages from the server. However, GMail only supports date granularity, so the server-side filter may return previously seen messages if run more than once a day.
  +
   === The TikaEntityProcessor
   
  -The TikaEntityProcessor uses Apache Tika to process incoming documents. This is similar to <<uploading-data-with-solr-cell-using-apache-tika.adoc#uploading-data-with-solr-cell-using-apache-tika,Uploading Data with Solr Cell using Apache Tika>>, but using the DataImportHandler options instead.
  -
  -Here is an example from the "```tika```" collection of the `dih` example (`example/example-DIH/tika/conf/tika-data-config.xml`):
  +The TikaEntityProcessor uses Apache Tika to process incoming documents. This is similar to <<uploading-data-with-solr-cell-using-apache-tika.adoc#uploading-data-with-solr-cell-using-apache-tika,Uploading Data with Solr Cell using Apache Tika>>, but using DataImportHandler options instead.
  +
  +The parameters for this processor are described in the table below. These are in addition to the attributes common to all entity processors described above.
  +
  +dataSource::
  +This parameter defines the data source and an optional name which can be referred to in later parts of the configuration if needed. This is the same `dataSource` explained in the description of general entity processor attributes above.
  ++
  +The available data source types for this processor are:
  ++
  +* BinURLDataSource: used for HTTP resources, but can also be used for files.
  +* BinContentStreamDataSource: used for uploading content as a stream.
  +* BinFileDataSource: used for content on the local filesystem.
  +
  +url::
  +Required. The path to the source file(s), as a file path or a traditional internet URL.
  +
  +htmlMapper::
  +Optional. Allows control of how Tika parses HTML. If this parameter is defined, it must be either *default* or *identity*; if it is absent, "default" is assumed.
  ++
  +The "default" mapper strips much of the HTML from documents while the "identity" mapper passes all HTML as-is with no modifications.
  +
  +format::
  +The output format. The options are *text*, *xml*, *html* or *none*. The default is "text" if not defined. The format "none" can be used if metadata only should be indexed and not the body of the documents.
  +
  +parser::
  +Optional. The default parser is `org.apache.tika.parser.AutoDetectParser`. If a custom or other parser should be used, it should be entered as a fully-qualified name of the class and path.
  +
  +fields::
  +The list of fields from the input documents and how they should be mapped to Solr fields. If the attribute `meta` is defined as "true", the field will be obtained from the metadata of the document and not parsed from the body of the main text.
  +
  +extractEmbedded::
  +Instructs the TikaEntityProcessor to extract embedded documents or attachments when *true*. If false, embedded documents and attachments will be ignored.
  +
  +onError::
  +By default, the TikaEntityProcessor will stop processing documents if it finds one that generates an error. If you define `onError` to "skip", the TikaEntityProcessor will instead skip documents that fail processing and log a message that the document was skipped.
  +
  +Here is an example from the `tika` collection of the `dih` example (data-config file found in `example/example-DIH/tika/conf/tika-data-config.xml`):
   
   [source,xml]
   ----
   <dataConfig>
  -  <dataSource type="BinFileDataSource" />
  +  <dataSource type="BinFileDataSource"/>
     <document>
  -    <entity name="tika-test" processor="TikaEntityProcessor"
  -            url="../contrib/extraction/src/test-files/extraction/solr-word.pdf" format="text">
  -      <field column="Author" name="author" meta="true"/>
  -      <field column="title" name="title" meta="true"/>
  -      <field column="text" name="text"/>
  +    <entity name="file" processor="FileListEntityProcessor" dataSource="null"
  +            baseDir="${solr.install.dir}/example/exampledocs" fileName=".*pdf"
  +            rootEntity="false">
  +
  +      <field column="file" name="id"/>
  +
  +      <entity name="pdf" processor="TikaEntityProcessor"
  +              url="${file.fileAbsolutePath}" format="text">
  +
  +        <field column="Author" name="author" meta="true"/>
  +        <!-- in the original PDF, the Author meta-field name is upper-cased,
  +          but in Solr schema it is lower-cased
  +         -->
  +
  +        <field column="title" name="title" meta="true"/>
  +        <field column="dc:format" name="format" meta="true"/>
  +
  +        <field column="text" name="text"/>
  +
  +      </entity>
       </entity>
     </document>
   </dataConfig>
   ----
   
  -The parameters for this processor are described in the table below:
  -
  -// TODO: Change column width to %autowidth.spread when https://github.com/asciidoctor/asciidoctor-pdf/issues/599 is fixed
  -
  -[cols="30,70",options="header"]
  -|===
  -|Attribute |Use
  -|dataSource a|
  -This parameter defines the data source and an optional name which can be referred to in later parts of the configuration if needed. This is the same dataSource explained in the description of general entity processor attributes above. The available data source types for this processor are:
  -
  -* BinURLDataSource: used for HTTP resources, but can also be used for files.
  -* BinContentStreamDataSource: used for uploading content as a stream.
  -* BinFileDataSource: used for content on the local filesystem.
  -
  -|url |The path to the source file(s), as a file path or a traditional internet URL. This parameter is required.
  -|htmlMapper |Allows control of how Tika parses HTML. The "default" mapper strips much of the HTML from documents while the "identity" mapper passes all HTML as-is with no modifications. If this parameter is defined, it must be either *default* or *identity*; if it is absent, "default" is assumed.
  -|format |The output format. The options are *text*, *xml*, *html* or *none*. The default is "text" if not defined. The format "none" can be used if metadata only should be indexed and not the body of the documents.
  -|parser |The default parser is `org.apache.tika.parser.AutoDetectParser`. If a custom or other parser should be used, it should be entered as a fully-qualified name of the class and path.
  -|fields |The list of fields from the input documents and how they should be mapped to Solr fields. If the attribute `meta` is defined as "true", the field will be obtained from the metadata of the document and not parsed from the body of the main text.
  -|extractEmbedded |Instructs the TikaEntityProcessor to extract embedded documents or attachments when *true*. If false, embedded documents and attachments will be ignored.
  -|onError |By default, the TikaEntityProcessor will stop processing documents if it finds one that generates an error. If you define `onError` to "skip", the TikaEntityProcessor will instead skip documents that fail processing and log a message that the document was skipped.
  -|===
  -
  -[[UploadingStructuredDataStoreDatawiththeDataImportHandler-TheFileListEntityProcessor]]
   === The FileListEntityProcessor
   
  -This processor is basically a wrapper, and is designed to generate a set of files satisfying conditions specified in the attributes which can then be passed to another processor, such as the <<UploadingStructuredDataStoreDatawiththeDataImportHandler-TheXPathEntityProcessor,XPathEntityProcessor>>. The entity information for this processor would be nested within the FileListEnitity entry. It generates five implicit fields: `fileAbsolutePath, ``fileDir, fileSize, ``fileLastModified, ``file,` which can be used in the nested processor. This processor does not use a data source.
  +This processor is basically a wrapper, and is designed to generate a set of files satisfying conditions specified in the attributes which can then be passed to another processor, such as the <<The XPathEntityProcessor,XPathEntityProcessor>>.
  +
  +The entity information for this processor would be nested within the FileListEntity entry. It generates five implicit fields: `fileAbsolutePath`, `fileDir`, `fileSize`, `fileLastModified`, and `file`, which can be used in the nested processor. This processor does not use a data source.
   
   The attributes specific to this processor are described in the table below:
   
  -// TODO: Change column width to %autowidth.spread when https://github.com/asciidoctor/asciidoctor-pdf/issues/599 is fixed
  -
  -[cols="30,70",options="header"]
  -|===
  -|Attribute |Use
  -|fileName |Required. A regular expression pattern to identify files to be included.
  -|basedir |Required. The base directory (absolute path).
  -|recursive |Whether to search directories recursively. Default is 'false'.
  -|excludes |A regular expression pattern to identify files which will be excluded.
  -|newerThan |A date in the format `yyyy-MM-ddHH:mm:ss` or a date math expression (`NOW - 2YEARS`).
  -|olderThan |A date, using the same formats as newerThan.
  -|rootEntity |This should be set to false. This ensures that each row (filepath) emitted by this processor is considered to be a document.
  -|dataSource |Must be set to null.
  -|===
  +fileName::
  +Required. A regular expression pattern to identify files to be included.
  +
  +basedir::
  +Required. The base directory (absolute path).
  +
  +recursive::
  +Whether to search directories recursively. Default is 'false'.
  +
  +excludes::
  +A regular expression pattern to identify files which will be excluded.
  +
  +newerThan::
  +A date in the format `yyyy-MM-ddHH:mm:ss` or a date math expression (`NOW - 2YEARS`).
  +
  +olderThan::
  +A date, using the same formats as newerThan.
  +
  +rootEntity::
  +This should be set to false. This ensures that each row (filepath) emitted by this processor is considered to be a document.
  +
  +dataSource::
  +Must be set to null.
   
   The example below shows the combination of the FileListEntityProcessor with another processor which will generate a set of fields from each file found.
   
  
positive.code: 
  The URLDataSource type accepts these optional parameters:
  
  // TODO: Change column width to %autowidth.spread when https://github.com/asciidoctor/asciidoctor-pdf/issues/599 is fixed
  
  [cols="30,70",options="header"]
  |===
  |Optional Parameter |Description
  |baseURL |Specifies a new baseURL for pathnames. You can use this to specify host/port changes between Dev/QA/Prod environments. Using this attribute isolates the changes to be made to the `solrconfig.xml`
  |connectionTimeout |Specifies the length of time in milliseconds after which the connection should time out. The default value is 5000ms.
  |encoding |By default the encoding in the response header is used. You can use this property to override the default encoding.
  |readTimeout |Specifies the length of time in milliseconds after which a read operation should time out. The default value is 10000ms.
  |===
  
  [[UploadingStructuredDataStoreDatawiththeDataImportHandler-EntityProcessors]]
  == Entity Processors
  
  Entity processors extract data, transform it, and add it to a Solr index. Examples of entities include views or tables in a data store.
  
  Each processor has its own set of attributes, described in its own section below. In addition, there are non-specific attributes common to all entities which may be specified.
  
  // TODO: Change column width to %autowidth.spread when https://github.com/asciidoctor/asciidoctor-pdf/issues/599 is fixed
  
  [cols="30,70",options="header"]
  |===
  |Attribute |Use
  |dataSource |The name of a data source. If there are multiple data sources defined, use this attribute with the name of the data source for this entity.
  |name |Required. The unique name used to identify an entity.
  |pk |The primary key for the entity. It is optional, and required only when using delta-imports. It has no relation to the uniqueKey defined in `schema.xml` but they can both be the same. It is mandatory if you do delta-imports and then refers to the column name in `${dataimporter.delta.<column-name>`} which is used as the primary key.
  |processor |Default is SqlEntityProcessor. Required only if the datasource is not RDBMS.
  |onError |Permissible values are (abort|skip|continue) . The default value is 'abort'. 'Skip' skips the current document. 'Continue' ignores the error and processing continues.
  |preImportDeleteQuery |Before a full-import command, use this query this to cleanup the index instead of using '*:*'. This is honored only on an entity that is an immediate sub-child of `<document>`.
  |postImportDeleteQuery |Similar to the above, but executed after the import has completed.
  |rootEntity |By default the entities immediately under the `<document>` are root entities. If this attribute is set to false, the entity directly falling under that entity will be treated as the root entity (and so on). For every row returned by the root entity, a document is created in Solr.
  |transformer |Optional. One or more transformers to be applied on this entity.
  |cacheImpl |Optional. A class (which must implement `DIHCache`) to use for caching this entity when doing lookups from an entity which wraps it. Provided implementation is "```SortedMapBackedCache```".
  |cacheKey |The name of a property of this entity to use as a cache key if `cacheImpl` is specified.
  |cacheLookup |An entity + property name that will be used to lookup cached instances of this entity if `cacheImpl` is specified.
  |where |an alternative way to specify `cacheKey` and `cacheLookup` concatenated with '='. eg `where="CODE=People.COUNTRY_CODE"` is equal to `cacheKey="CODE" cacheLookup="People.COUNTRY_CODE"`
  |child="true" |Enables indexing document blocks aka <<uploading-data-with-index-handlers.adoc#uploading-data-with-index-handlers,Nested Child Documents>> for searching with <<other-parsers.adoc#other-parsers,Block Join Query Parsers>>. It can be only specified on `<entity>` under another root entity. It switches from default behavior (merging field values) to nesting documents as children documents. Note: parent `<entity>` should add a field which is used as a parent filter in query time.
  |join="zipper" |Enables merge join aka "zipper" algorithm for joining parent and child entities without cache. It should be specified at child (nested) `<entity>`. It implies that parent and child queries return results ordered by keys, otherwise it throws an exception. Keys should be specified either with `where` attribute or with `cacheKey` and `cacheLookup`.
  |===
  
  Caching of entities in DIH is provided to avoid repeated lookups for same entities again and again. The default `SortedMapBackedCache` is a `HashMap` where a key is a field in the row and the value is a bunch of rows for that same key.
  
  In the example below, each `manufacturer` entity is cached using the '`id`' property as a cache key. Cache lookups will be performed for each `product` entity based on the product's "```manu```" property. When the cache has no data for a particular key, the query is run and the cache is populated
  
  [source,xml]
  ----
  
negative.code:   </document>
  </dataConfig>
  ----
  <1> The first element is the `dataSource`, in this case an HSQLDB database. The path to the JDBC driver and the JDBC URL and login credentials are all specified here. Other permissible attributes include whether or not to autocommit to Solr, the batchsize used in the JDBC connection, and a `readOnly` flag. The password attribute is optional if there is no password set for the DB.
  <2> Alternately the password can be encrypted as follows. This is the value obtained as a result of the command `openssl enc -aes-128-cbc -a -salt -in pwd.txt password="U2FsdGVkX18QMjY0yfCqlfBMvAB4d3XkwY96L7gfO2o="`. When the password is encrypted, you must provide an extra attribute `encryptKeyFile="/location/of/encryptionkey"`. This file should a text file with a single line containing the encrypt/decrypt password.
  <3> A `document` element follows, containing multiple `entity` elements. Note that `entity` elements can be nested, and this allows the entity relationships in the sample database to be mirrored here, so that we can generate a denormalized Solr record which may include multiple features for one item, for instance.
  <4> The possible attributes for the `entity` element are described in later sections. Entity elements may contain one or more `field` elements, which map the data source field names to Solr fields, and optionally specify per-field transformations. This entity is the `root` entity.
  <5> This entity is nested and reflects the one-to-many relationship between an item and its multiple features. Note the use of variables; `${item.ID}` is the value of the column 'ID' for the current item (`item` referring to the entity name).
  
  Datasources can still be specified in `solrconfig.xml`. These must be specified in the defaults section of the handler in `solrconfig.xml`. However, these are not parsed until the main configuration is loaded.
  
  