anchor.diff:                password="password"
                 host="imap.gmail.com"
                 protocol="imaps"
  -              fetchMailsSince="2009-09-20 00:00:00"
  +              fetchMailsSince="2014-06-30 00:00:00"
                 batchSize="20"
                 folders="inbox"
                 processAttachement="false"
  -              name="sample_entity"/>
  +              name="mail_entity"/>
     </document>
   </dataConfig>
   ----
   
  -The entity attributes unique to the MailEntityProcessor are shown below.
  -
  -// TODO: Change column width to %autowidth.spread when https://github.com/asciidoctor/asciidoctor-pdf/issues/599 is fixed
  -
  -[cols="30,70",options="header"]
  -|===
  -|Attribute |Use
  -|processor |Required. Must be set to "MailEntityProcessor".
  -|user |Required. Username for authenticating to the IMAP server; this is typically the email address of the mailbox owner.
  -|password |Required. Password for authenticating to the IMAP server.
  -|host |Required. The IMAP server to connect to.
  -|protocol |Required. The IMAP protocol to use, valid values are: imap, imaps, gimap, and gimaps.
  -|fetchMailsSince |Optional. Date/time used to set a filter to import messages that occur after the specified date; expected format is: `yyyy-MM-dd HH:mm:ss`.
  -|folders |Required. Comma-delimited list of folder names to pull messages from, such as "inbox".
  -|recurse |Optional (default is true). Flag to indicate if the processor should recurse all child folders when looking for messages to import.
  -|include |Optional. Comma-delimited list of folder patterns to include when processing folders (can be a literal value or regular expression).
  -|exclude |Optional. Comma-delimited list of folder patterns to exclude when processing folders (can be a literal value or regular expression); excluded folder patterns take precedence over include folder patterns.
  -a|
  -processAttachement
  -
  -or
  -
  -processAttachments
  -
  - |Optional (default is true). Use Tika to process message attachments.
  -|includeContent |Optional (default is true). Include the message body when constructing Solr documents for indexing.
  -|===
  -
  -[[UploadingStructuredDataStoreDatawiththeDataImportHandler-ImportingNewEmailsOnly]]
   ==== Importing New Emails Only
   
  -After running a full import, the MailEntityProcessor keeps track of the timestamp of the previous import so that subsequent imports can use the fetchMailsSince filter to only pull new messages from the mail server. This occurs automatically using the Data Import Handler dataimport.properties file (stored in conf). For instance, if you set `fetchMailsSince="2014-08-22 00:00:00"` in your `mail-data-config.xml`, then all mail messages that occur after this date will be imported on the first run of the importer. Subsequent imports will use the date of the previous import as the fetchMailsSince filter, so that only new emails since the last import are indexed each time.
  -
  -[[UploadingStructuredDataStoreDatawiththeDataImportHandler-GMailExtensions]]
  +After running a full import, the MailEntityProcessor keeps track of the timestamp of the previous import so that subsequent imports can use the fetchMailsSince filter to only pull new messages from the mail server. This occurs automatically using the DataImportHandler `dataimport.properties` file (stored in `conf`).
  +
  +For instance, if you set `fetchMailsSince="2014-08-22 00:00:00"` in your `mail-data-config.xml`, then all mail messages that occur after this date will be imported on the first run of the importer. Subsequent imports will use the date of the previous import as the `fetchMailsSince` filter, so that only new emails since the last import are indexed each time.
  +
   ==== GMail Extensions
   
  -When connecting to a GMail account, you can improve the efficiency of the MailEntityProcessor by setting the protocol to *gimap* or *gimaps*. This allows the processor to send the fetchMailsSince filter to the GMail server to have the date filter applied on the server, which means the processor only receives new messages from the server. However, GMail only supports date granularity, so the server-side filter may return previously seen messages if run more than once a day.
  -
  -[[UploadingStructuredDataStoreDatawiththeDataImportHandler-TheTikaEntityProcessor]]
  +When connecting to a GMail account, you can improve the efficiency of the MailEntityProcessor by setting the protocol to *gimap* or *gimaps*.
  +
  +This allows the processor to send the `fetchMailsSince` filter to the GMail server to have the date filter applied on the server, which means the processor only receives new messages from the server. However, GMail only supports date granularity, so the server-side filter may return previously seen messages if run more than once a day.
  +
   === The TikaEntityProcessor
   
  -The TikaEntityProcessor uses Apache Tika to process incoming documents. This is similar to <<uploading-data-with-solr-cell-using-apache-tika.adoc#uploading-data-with-solr-cell-using-apache-tika,Uploading Data with Solr Cell using Apache Tika>>, but using the DataImportHandler options instead.
  -
  -Here is an example from the "```tika```" collection of the `dih` example (`example/example-DIH/tika/conf/tika-data-config.xml`):
  +The TikaEntityProcessor uses Apache Tika to process incoming documents. This is similar to <<uploading-data-with-solr-cell-using-apache-tika.adoc#uploading-data-with-solr-cell-using-apache-tika,Uploading Data with Solr Cell using Apache Tika>>, but using DataImportHandler options instead.
  +
  +The parameters for this processor are described in the table below. These are in addition to the attributes common to all entity processors described above.
  +
  +dataSource::
  +This parameter defines the data source and an optional name which can be referred to in later parts of the configuration if needed. This is the same `dataSource` explained in the description of general entity processor attributes above.
  ++
  +The available data source types for this processor are:
  ++
  +* BinURLDataSource: used for HTTP resources, but can also be used for files.
  +* BinContentStreamDataSource: used for uploading content as a stream.
  +* BinFileDataSource: used for content on the local filesystem.
  +
  +url::
  +Required. The path to the source file(s), as a file path or a traditional internet URL.
  +
  +htmlMapper::
  +Optional. Allows control of how Tika parses HTML. If this parameter is defined, it must be either *default* or *identity*; if it is absent, "default" is assumed.
  ++
  +The "default" mapper strips much of the HTML from documents while the "identity" mapper passes all HTML as-is with no modifications.
  +
  +format::
  +The output format. The options are *text*, *xml*, *html* or *none*. The default is "text" if not defined. The format "none" can be used if metadata only should be indexed and not the body of the documents.
  +
  +parser::
  +Optional. The default parser is `org.apache.tika.parser.AutoDetectParser`. If a custom or other parser should be used, it should be entered as a fully-qualified name of the class and path.
  +
  +fields::
  +The list of fields from the input documents and how they should be mapped to Solr fields. If the attribute `meta` is defined as "true", the field will be obtained from the metadata of the document and not parsed from the body of the main text.
  +
  +extractEmbedded::
  +Instructs the TikaEntityProcessor to extract embedded documents or attachments when *true*. If false, embedded documents and attachments will be ignored.
  +
  +onError::
  +By default, the TikaEntityProcessor will stop processing documents if it finds one that generates an error. If you define `onError` to "skip", the TikaEntityProcessor will instead skip documents that fail processing and log a message that the document was skipped.
  +
  +Here is an example from the `tika` collection of the `dih` example (data-config file found in `example/example-DIH/tika/conf/tika-data-config.xml`):
   
   [source,xml]
   ----
   <dataConfig>
  -  <dataSource type="BinFileDataSource" />
  +  <dataSource type="BinFileDataSource"/>
     <document>
  -    <entity name="tika-test" processor="TikaEntityProcessor"
  -            url="../contrib/extraction/src/test-files/extraction/solr-word.pdf" format="text">
  -      <field column="Author" name="author" meta="true"/>
  -      <field column="title" name="title" meta="true"/>
  -      <field column="text" name="text"/>
  +    <entity name="file" processor="FileListEntityProcessor" dataSource="null"
  +            baseDir="${solr.install.dir}/example/exampledocs" fileName=".*pdf"
  +            rootEntity="false">
  +
  +      <field column="file" name="id"/>
  +
  +      <entity name="pdf" processor="TikaEntityProcessor"
  +              url="${file.fileAbsolutePath}" format="text">
  +
  +        <field column="Author" name="author" meta="true"/>
  +        <!-- in the original PDF, the Author meta-field name is upper-cased,
  +          but in Solr schema it is lower-cased
  +         -->
  +
  +        <field column="title" name="title" meta="true"/>
  +        <field column="dc:format" name="format" meta="true"/>
  +
  +        <field column="text" name="text"/>
  +
  +      </entity>
       </entity>
     </document>
   </dataConfig>
   ----
   
  -The parameters for this processor are described in the table below:
  -
  -// TODO: Change column width to %autowidth.spread when https://github.com/asciidoctor/asciidoctor-pdf/issues/599 is fixed
  -
  -[cols="30,70",options="header"]
  -|===
  -|Attribute |Use
  -|dataSource a|
  -This parameter defines the data source and an optional name which can be referred to in later parts of the configuration if needed. This is the same dataSource explained in the description of general entity processor attributes above. The available data source types for this processor are:
  -
  -* BinURLDataSource: used for HTTP resources, but can also be used for files.
  -* BinContentStreamDataSource: used for uploading content as a stream.
  -* BinFileDataSource: used for content on the local filesystem.
  -
  -|url |The path to the source file(s), as a file path or a traditional internet URL. This parameter is required.
  -|htmlMapper |Allows control of how Tika parses HTML. The "default" mapper strips much of the HTML from documents while the "identity" mapper passes all HTML as-is with no modifications. If this parameter is defined, it must be either *default* or *identity*; if it is absent, "default" is assumed.
  -|format |The output format. The options are *text*, *xml*, *html* or *none*. The default is "text" if not defined. The format "none" can be used if metadata only should be indexed and not the body of the documents.
  -|parser |The default parser is `org.apache.tika.parser.AutoDetectParser`. If a custom or other parser should be used, it should be entered as a fully-qualified name of the class and path.
  -|fields |The list of fields from the input documents and how they should be mapped to Solr fields. If the attribute `meta` is defined as "true", the field will be obtained from the metadata of the document and not parsed from the body of the main text.
  -|extractEmbedded |Instructs the TikaEntityProcessor to extract embedded documents or attachments when *true*. If false, embedded documents and attachments will be ignored.
  -|onError |By default, the TikaEntityProcessor will stop processing documents if it finds one that generates an error. If you define `onError` to "skip", the TikaEntityProcessor will instead skip documents that fail processing and log a message that the document was skipped.
  -|===
  -
  -[[UploadingStructuredDataStoreDatawiththeDataImportHandler-TheFileListEntityProcessor]]
   === The FileListEntityProcessor
   
  -This processor is basically a wrapper, and is designed to generate a set of files satisfying conditions specified in the attributes which can then be passed to another processor, such as the <<UploadingStructuredDataStoreDatawiththeDataImportHandler-TheXPathEntityProcessor,XPathEntityProcessor>>. The entity information for this processor would be nested within the FileListEnitity entry. It generates five implicit fields: `fileAbsolutePath, ``fileDir, fileSize, ``fileLastModified, ``file,` which can be used in the nested processor. This processor does not use a data source.
  +This processor is basically a wrapper, and is designed to generate a set of files satisfying conditions specified in the attributes which can then be passed to another processor, such as the <<The XPathEntityProcessor,XPathEntityProcessor>>.
  +
  +The entity information for this processor would be nested within the FileListEntity entry. It generates five implicit fields: `fileAbsolutePath`, `fileDir`, `fileSize`, `fileLastModified`, and `file`, which can be used in the nested processor. This processor does not use a data source.
   
   The attributes specific to this processor are described in the table below:
   
  -// TODO: Change column width to %autowidth.spread when https://github.com/asciidoctor/asciidoctor-pdf/issues/599 is fixed
  -
  -[cols="30,70",options="header"]
  -|===
  -|Attribute |Use
  -|fileName |Required. A regular expression pattern to identify files to be included.
  -|basedir |Required. The base directory (absolute path).
  -|recursive |Whether to search directories recursively. Default is 'false'.
  -|excludes |A regular expression pattern to identify files which will be excluded.
  -|newerThan |A date in the format `yyyy-MM-ddHH:mm:ss` or a date math expression (`NOW - 2YEARS`).
  -|olderThan |A date, using the same formats as newerThan.
  -|rootEntity |This should be set to false. This ensures that each row (filepath) emitted by this processor is considered to be a document.
  -|dataSource |Must be set to null.
  -|===
  +fileName::
  +Required. A regular expression pattern to identify files to be included.
  +
  +basedir::
  +Required. The base directory (absolute path).
  +
  +recursive::
  +Whether to search directories recursively. Default is 'false'.
  +
  +excludes::
  +A regular expression pattern to identify files which will be excluded.
  +
  +newerThan::
  +A date in the format `yyyy-MM-ddHH:mm:ss` or a date math expression (`NOW - 2YEARS`).
  +
  +olderThan::
  +A date, using the same formats as newerThan.
  +
  +rootEntity::
  +This should be set to false. This ensures that each row (filepath) emitted by this processor is considered to be a document.
  +
  +dataSource::
  +Must be set to null.
   
   The example below shows the combination of the FileListEntityProcessor with another processor which will generate a set of fields from each file found.
   
  
positive.code: </entity>
  ----
  
  [[UploadingStructuredDataStoreDatawiththeDataImportHandler-TheSQLEntityProcessor]]
  === The SQL Entity Processor
  
  The SqlEntityProcessor is the default processor. The associated <<UploadingStructuredDataStoreDatawiththeDataImportHandler-JdbcDataSource,data source>> should be a JDBC URL.
  
  The entity attributes specific to this processor are shown in the table below.
  
  // TODO: Change column width to %autowidth.spread when https://github.com/asciidoctor/asciidoctor-pdf/issues/599 is fixed
  
  [cols="30,70",options="header"]
  |===
  |Attribute |Use
  |query |Required. The SQL query used to select rows.
  |deltaQuery |SQL query used if the operation is delta-import. This query selects the primary keys of the rows which will be parts of the delta-update. The pks will be available to the deltaImportQuery through the variable `${dataimporter.delta.<column-name>`}.
  |parentDeltaQuery |SQL query used if the operation is delta-import.
  |deletedPkQuery |SQL query used if the operation is delta-import.
  |deltaImportQuery |SQL query used if the operation is delta-import. If this is not present, DIH tries to construct the import query by(after identifying the delta) modifying the 'query' (this is error prone). There is a namespace `${dataimporter.delta.<column-name>`} which can be used in this query. For example, `select * from tbl where id=${dataimporter.delta.id`}.
  |===
  
  [[UploadingStructuredDataStoreDatawiththeDataImportHandler-TheXPathEntityProcessor]]
  === The XPathEntityProcessor
  
  This processor is used when indexing XML formatted data. The data source is typically <<UploadingStructuredDataStoreDatawiththeDataImportHandler-URLDataSource,URLDataSource>> or <<UploadingStructuredDataStoreDatawiththeDataImportHandler-FileDataSource,FileDataSource>>. Xpath can also be used with the <<UploadingStructuredDataStoreDatawiththeDataImportHandler-TheFileListEntityProcessor,FileListEntityProcessor>> described below, to generate a document from each file.
  
  The entity attributes unique to this processor are shown below.
  
  // TODO: Change column width to %autowidth.spread when https://github.com/asciidoctor/asciidoctor-pdf/issues/599 is fixed
  
  [cols="30,70",options="header"]
  |===
  |Attribute |Use
  |Processor |Required. Must be set to "XpathEntityProcessor".
  |url |Required. HTTP URL or file location.
  |stream |Optional: Set to true for a large file or download.
  |forEach |Required unless you define `useSolrAddSchema`. The Xpath expression which demarcates each record. This will be used to set up the processing loop.
  |xsl |Optional: Its value (a URL or filesystem path) is the name of a resource used as a preprocessor for applying the XSL transformation.
  |useSolrAddSchema |Set this to true if the content is in the form of the standard Solr update XML schema.
  |===
  
  Each field element in the entity can have the following attributes as well as the default ones.
  
  // TODO: Change column width to %autowidth.spread when https://github.com/asciidoctor/asciidoctor-pdf/issues/599 is fixed
  
  [cols="30,70",options="header"]
  |===
  |Attribute |Use
  |xpath |Required. The XPath expression which will extract the content from the record for this field. Only a subset of Xpath syntax is supported.
  |commonField |Optional. If true, then when this field is encountered in a record it will be copied to future records when creating a Solr document.
  |flatten a|
  Optional: If set to true, then any children text nodes are collected to form the value of a field. image::images/icons/emoticons/warning.png[(warning)]
   The default value is false, meaning that
  
  if there are any sub-elements of the node pointed to by the XPath expression, they will be quietly omitted.
  
  |===
  
  Here is an example from the "```rss```" collection in the `dih` example (`example/example-DIH/solr/rss/conf/rss-data-config.xml`):
  
  [source,xml]
  ----
  <!-- slashdot RSS Feed --->
  <dataConfig>
    <dataSource type="HttpDataSource" />
      <document>
        <entity name="slashdot"
                pk="link"
                url="http://rss.slashdot.org/Slashdot/slashdot"
                processor="XPathEntityProcessor"
                transformer="DateFormatTransformer"
                forEach="/RDF/channel | /RDF/item" >
            <!-- NOTE: forEach sets up a processing loop ; here there are two expressions -->
        <field column="source" xpath="/RDF/channel/title" commonField="true" />
        <field column="source-link" xpath="/RDF/channel/link" commonField="true"/>
        <field column="subject" xpath="/RDF/channel/subject" commonField="true" />
        <field column="title" xpath="/RDF/item/title" />
        <field column="link" xpath="/RDF/item/link" />
        <field column="description" xpath="/RDF/item/description" />
        <field column="creator" xpath="/RDF/item/creator" />
        <field column="item-subject" xpath="/RDF/item/subject" />
        <field column="date" xpath="/RDF/item/date"
               dateTimeFormat="yyyy-MM-dd'T'hh:mm:ss" />
        <field column="slash-department" xpath="/RDF/item/department" />
        <field column="slash-section" xpath="/RDF/item/section" />
        <field column="slash-comments" xpath="/RDF/item/comments" />
      </entity>
    </document>
  </dataConfig>
  ----
  
  [[UploadingStructuredDataStoreDatawiththeDataImportHandler-TheMailEntityProcessor]]
  === The MailEntityProcessor
  
  The MailEntityProcessor uses the Java Mail API to index email messages using the IMAP protocol. The MailEntityProcessor works by connecting to a specified mailbox using a username and password, fetching the email headers for each message, and then fetching the full email contents to construct a document (one document for each mail message).
  
  Here is an example from the "```mail```" collection of the `dih` example (`example/example-DIH/mail/conf/mail-data-config.xml`):
  
  [source,xml]
  ----
  
negative.code: 	    password=${dataimporter.request.jdbcpassword} />
  ----
  
  These parameters can then be passed to the `full-import` command or defined in the `<defaults>` section in `solrconfig.xml`. This example shows the parameters with the full-import command:
  
  [source,bash]
  http://localhost:8983/solr/dih/dataimport?command=full-import&jdbcurl=jdbc:hsqldb:./example-DIH/hsqldb/ex&jdbcuser=sa&jdbcpassword=secret
  
  == DataImportHandler Commands
  
  DIH commands are sent to Solr via an HTTP request. The following operations are supported.
  
  abort::
  Aborts an ongoing operation. For example: `\http://localhost:8983/solr/dih/dataimport?command=abort`.
  
  delta-import::
  For incremental imports and change detection. Only the <<The SQL Entity Processor,SqlEntityProcessor>> supports delta imports.
  +
  For example: `\http://localhost:8983/solr/dih/dataimport?command=delta-import`.
  +
  This command supports the same `clean`, `commit`, `optimize` and `debug` parameters as `full-import` command described below.
  
  full-import::
  A Full Import operation can be started with a URL such as `\http://localhost:8983/solr/dih/dataimport?command=full-import`. The command returns immediately.
  +
  The operation will be started in a new thread and the _status_ attribute in the response should be shown as _busy_. The operation may take some time depending on the size of dataset. Queries to Solr are not blocked during full-imports.
  +
  When a `full-import` command is executed, it stores the start time of the operation in a file located at `conf/dataimport.properties`. This stored timestamp is used when a `delta-import` operation is executed.
  +
  Commands available to `full-import` are:
  +
  clean:::
  Default is true. Tells whether to clean up the index before the indexing is started.
  commit:::
  Default is true. Tells whether to commit after the operation.
  debug:::
  Default is false. Runs the command in debug mode and is used by the interactive development mode.
  +
  Note that in debug mode, documents are never committed automatically. If you want to run debug mode and commit the results too, add `commit=true` as a request parameter.
  entity:::
  The name of an entity directly under the `<document>` tag in the configuration file. Use this to execute one or more entities selectively.
  +
  Multiple "entity" parameters can be passed on to run multiple entities at once. If nothing is passed, all entities are executed.
  optimize:::
  Default is true. Tells Solr whether to optimize after the operation.
  synchronous:::
  Blocks request until import is completed. Default is false.
  
  reload-config::
  If the configuration file has been changed and you wish to reload it without restarting Solr, run the command `\http://localhost:8983/solr/dih/dataimport?command=reload-config`
  
  status::
  This command returns statistics on the number of documents created, deleted, queries run, rows fetched, status, and so on. For example:  `\http://localhost:8983/solr/dih/dataimport?command=status`.
  
  show-config::
  This command responds with configuration: `\http://localhost:8983/solr/dih/dataimport?command=show-config`.
  
  
  == Property Writer
  
  The `propertyWriter` element defines the date format and locale for use with delta queries. It is an optional configuration. Add the element to the DIH configuration file, directly under the `dataConfig` element.
  