Fixed examples.

Fixed examples.

xquery version "3.0";

(:
 : Copyright 2006-2011 The FLWOR Foundation.
 :
 : Licensed under the Apache License, Version 2.0 (the "License");
 : you may not use this file except in compliance with the License.
 : You may obtain a copy of the License at
 :
 : http://www.apache.org/licenses/LICENSE-2.0
 :
 : Unless required by applicable law or agreed to in writing, software
 : distributed under the License is distributed on an "AS IS" BASIS,
 : WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 : See the License for the specific language governing permissions and
 : limitations under the License.
 :)

(:===========================================================================:)

(:~
 : This module provides an XQuery API to full-text functions.
 : For general information about Zorba's implementation of the
 : XQuery and XPath Full Text 1.0 specification
 : as well as instructions for building an installing a thesaurus,
 : see the Full Text Thesaurus documentation.
 :
Notes on languages

 : To refer to paricular human languages,
 : Zorba uses both the
 : ISO 639-1
 : and
 : ISO 639-2
 : languages codes.
 : Note that Zorba supports only a subset of the
 : complete list of language codes
 : and not every function supports the same subset.
 :

 : Most functions in this module take a language as a parameter
 : using the
 : xs:language
 : XML schema data type.
 :
Notes on stemming

 : The stem() functions return the
 : stem
 : of a word.
 : In Zorba,
 : the stem of a word itself, however, is not guaranteed to be a word.
 : It is best to consider a stem as an opaque byte sequence.
 : All that is guaranteed about a stem is that,
 : for a given word,
 : the stem of that word will always be the same byte sequence.
 : Hence,
 : you sould never compare the result of one of the stem()
 : functions against a non-stemmed string,
 : for example:
 :

 :  if ( ft:stem( "apples" ) eq "apple" )             ** WRONG **
 :

 : Instead do:
 :

 :  if ( ft:stem( "apples" ) eq ft:stem( "apple" ) )  ** CORRECT **
 :

 :
Notes on the thesaurus

 : The thesaurus-lookup() functions have "levels"
 : and "relationship" parameters.
 : The values for these are implementation-defined.
 : Zorba's default implementation uses the
 : WordNet lexical database,
 : version 3.0.
 :

 : In WordNet,
 : the number of "levels" that two phrases are apart
 : are how many hierarchical meanings apart they are.
 : For example,
 : "canary" is 5 levels away from "vertebrate"
 : (carary > finch > oscine > passerine > bird > vertebrate).
 :

 : When using the WordNet implementation,
 : Zorba supports all of the relationships (and their abbreviations)
 : specified by
 : ISO 2788
 : and
 : ANSI/NISO Z39.19-2005
 : with the exceptions of "HN" (history note)
 : and "X SN" (see scope note for).
 : These relationships are:
 :

 :

 :      Rel.
 :      Meaning
 :      WordNet Rel.
 :

 :

 :      BT
 :      broader term
 :      hypernym
 :

 :

 :      BTG
 :      broader term generic
 :      hypernym
 :

 :

 :      BTI
 :      broader term instance
 :      instance hypernym
 :

 :

 :      BTP
 :      broader term partitive
 :      part meronym
 :

 :

 :      NT
 :      narrower term
 :      hyponym
 :

 :

 :      NTG
 :      narrower term generic
 :      hyponym
 :

 :

 :      NTI
 :      narrower term instance
 :      instance hyponym
 :

 :

 :      NTP
 :      narrower term partitive
 :      part holonym
 :

 :

 :      RT
 :      related term
 :      also see
 :

 :

 :      SN
 :      scope note
 :      n/a
 :

 :

 :      TT
 :      top term
 :      hypernym
 :

 :

 :      UF
 :      non-preferred term
 :      n/a
 :

 :

 :      USE
 :      preferred term
 :      n/a
 :

 :

 : Note that you can specify relationships
 : either by their abbreviation
 : or their meaning.
 : Relationships are case-insensitive.
 :
 : In addition to the
 : ISO 2788
 : and
 : ANSI/NISO Z39.19-2005
 : relationships,
 : Zorba also supports all of the relationships offered by WordNet.
 : These relationships are:
 :

 :

 :      Relationship
 :      Meaning
 :

 :

 :      also see
 :
 :        A word that is related to another,
 :        e.g., for "varnished" (furniture)
 :        one should also see "finished."
 :
 :

 :

 :      antonym
 :
 :        A word opposite in meaning to another,
 :        e.g., "light" is an antonym for "heavy."
 :
 :

 :

 :      attribute
 :
 :        A noun for which adjectives express values,
 :        e.g., "weight" is an attribute
 :        for which the adjectives "light" and "heavy"
 :        express values.
 :
 :

 :

 :      cause
 :
 :        A verb that causes another,
 :        e.g., "show" is a cause of "see."
 :
 :

 :

 :      derivationally related form
 :
 :        A word that is derived from a root word,
 :        e.g., "metric" is a derivationally related form of "meter."
 :
 :

 :

 :      derived from adjective
 :
 :        An adverb that is derived from an adjective,
 :        e.g., "correctly" is derived from the adjective "correct."
 :
 :

 :

 :      entailment
 :
 :        A verb that presupposes another,
 :        e.g., "snoring" entails "sleeping."
 :
 :

 :

 :      hypernym
 :
 :        A word with a broad meaning that more specific words fall under,
 :        e.g., "meal" is a hypernym of "breakfast."
 :
 :

 :

 :      hyponym
 :
 :        A word of more specific meaning than a general term applicable to it,
 :        e.g., "breakfast" is a hyponym of "meal."
 :
 :

 :

 :      instance hypernym
 :
 :        A word that denotes a category of some specific instance,
 :        e.g., "author" is an instance hypernym of "Asimov."
 :
 :

 :

 :      instance hyponym
 :
 :        A term that donotes a specific instance of some general category,
 :        e.g., "Asimov" is an instance hyponym of "author."
 :
 :

 :

 :      member holonym
 :
 :        A word that denotes a collection of individuals,
 :        e.g., "faculty" is a member holonym of "professor."
 :
 :

 :

 :      member meronym
 :
 :        A word that denotes a member of a larger group,
 :        e.g., a "person" is a member meronym of a "crowd."
 :
 :

 :

 :      part holonym
 :
 :        A word that denotes a larger whole comprised of some part,
 :        e.g., "car" is a part holonym of "engine."
 :
 :

 :

 :      part meronym
 :
 :        A word that denotes a part of a larger whole,
 :        e.g., an "engine" is part meronym of a "car."
 :
 :

 :

 :      participle of verb
 :
 :        An adjective that is the participle of some verb,
 :        e.g., "breaking" is the participle of the verb "break."
 :
 :

 :

 :      pertainym
 :
 :        An adjective that classifies its noun,
 :        e.g., "musical" is a pertainym in "musical instrument."
 :
 :

 :

 :      similar to
 :
 :        Similar, though not necessarily interchangeable, adjectives.
 :        For example, "shiny" is similar to "bright",
 :        but they have subtle differences.
 :
 :

 :

 :      substance holonym
 :
 :        A word that denotes a larger whole containing some constituent
 :        substance, e.g., "bread" is a substance holonym of "flour."
 :
 :

 :

 :      substance meronym
 :
 :        A word that denotes a constituant substance of some larger whole,
 :        e.g., "flour" is a substance meronym of "bread."
 :
 :

 :

 :      verb group
 :
 :        A verb that is a member of a group of similar verbs,
 :        e.g., "live" is in the verb group
 :        of "dwell", "live", "inhabit", etc.
 :
 :

 :

 :
Notes on tokenization

 : For general information about Zorba's implementation of tokenization,
 : including what constitutes a token,
 : see the Full Text Tokenizer documentation.
 : @author Paul J. Lucas
 :)
module namespace ft = "http://www.zorba-xquery.com/modules/full-text";

import schema namespace ft-schema =
  "http://www.zorba-xquery.com/modules/full-text";

declare namespace err = "http://www.w3.org/2005/xqt-errors";
declare namespace zerr = "http://www.zorba-xquery.com/errors";

declare namespace ver = "http://www.zorba-xquery.com/options/versioning";
declare option ver:module-version "2.0";

(:===========================================================================:)

(:~
 : Predeclared constant for the Danish
 : xs:language.
 :)
declare variable $ft:lang-da as xs:language := xs:language("da");

(:~
 : Predeclared constant for the German
 : xs:language.
 :)
declare variable $ft:lang-de as xs:language := xs:language("de");

(:~
 : Predeclared constant for the English
 : xs:language.
 :)
declare variable $ft:lang-en as xs:language := xs:language("en");

(:~
 : Predeclared constant for the Spanish
 : xs:language.
 :)
declare variable $ft:lang-es as xs:language := xs:language("es");

(:~
 : Predeclared constant for the Finnish
 : xs:language.
 :)
declare variable $ft:lang-fi as xs:language := xs:language("fi");

(:~
 : Predeclared constant for the French
 : xs:language.
 :)
declare variable $ft:lang-fr as xs:language := xs:language("fr");

(:~
 : Predeclared constant for the Hungarian
 : xs:language.
 :)
declare variable $ft:lang-hu as xs:language := xs:language("hu");

(:~
 : Predeclared constant for the Italian
 : xs:language.
 :)
declare variable $ft:lang-it as xs:language := xs:language("it");

(:~
 : Predeclared constant for the Dutch
 : xs:language.
 :)
declare variable $ft:lang-nl as xs:language := xs:language("nl");

(:~
 : Predeclared constant for the Norwegian
 : xs:language.
 :)
declare variable $ft:lang-no as xs:language := xs:language("no");

(:~
 : Predeclared constant for the Portuguese
 : xs:language.
 :)
declare variable $ft:lang-pt as xs:language := xs:language("pt");

(:~
 : Predeclared constant for the Romanian
 : xs:language.
 :)
declare variable $ft:lang-ro as xs:language := xs:language("ro");

(:~
 : Predeclared constant for the Russian
 : xs:language.
 :)
declare variable $ft:lang-ru as xs:language := xs:language("ru");

(:~
 : Predeclared constant for the Swedish
 : xs:language.
 :)
declare variable $ft:lang-sv as xs:language := xs:language("sv");

(:~
 : Predeclared constant for the Turkish
 : xs:language.
 :)
declare variable $ft:lang-tr as xs:language := xs:language("tr");

(:===========================================================================:)

(:~
 : Gets the current compare options.
 :
 : @return said compare options.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-current-compare-options-1.xq
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-current-compare-options-2.xq
 :)
declare function ft:current-compare-options()
  as element(ft-schema:compare-options) external;

(:~
 : Gets the current
 : language:
 : either the language specified by the
 : declare ft-option using
 : language
 : statement (if any)
 : or the one returned by ft:host-lang() (if none).
 :
 : @return said language.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-current-lang-true-1.xq
 :)
declare function ft:current-lang()
  as xs:language external;

(:~
 : Gets the host's current
 : language.
 : The "host" is the computer on which Zorba is running.
 : The host's current language is obtained as follows:
 :

 :

 :      For *nix systems:
 :

 :

 :          If setlocale(3) returns non-null,
 :          the language corresponding to that locale is used.
 :

 :

 :          Else, if the LANG environment variable is set,
 :          that language is ued.
 :

 :

 :          Otherwise, there is no default language.
 :

 :

 :

 :

 :      For Windows systems,
 :      the language corresponding to the locale returned by the
 :      GetLocaleInfo()
 :      function is used.
 :

 :

 :
 : @return said language.
 :)
declare function ft:host-lang()
  as xs:language external;

(:~
 : Checks whether the given
 : language
 : is supported for stemming.
 :
 : @param $lang The language to check.
 : @return true only if the language is supported.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stem-lang-es-supported-true.xq
 :)
declare function ft:is-stem-lang-supported( $lang as xs:language )
  as xs:boolean external;

(:~
 : Checks whether the given word is a stop-word.
 :
 : @param $word The word to check.
 : @param $lang The
 : language
 : of $word.
 : @return true only if $word is a stop-word.
 : @error err:FTST0009 if $lang is not supported.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-true-1.xq
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-true-3.xq
 :)
declare function ft:is-stop-word( $word as xs:string, $lang as xs:language )
  as xs:boolean external;

(:~
 : Checks whether the given word is a stop-word.
 :
 : @param $word The word to check.
 : The word's language
 : is assumed to be the one returned by ft:current-lang().
 : @return true only if $word is a stop-word.
 : @error err:FTST0009 if ft:current-lang() is not supported.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-true-2.xq
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-true-4.xq
 :)
declare function ft:is-stop-word( $word as xs:string )
  as xs:boolean external;

(:~
 : Checks whether the given
 : language
 : is supported for stop words.
 :
 : @param $lang The language to check.
 : @return true only if the language is supported.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-lang-en-supported-true.xq
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-lang-supported-false-1.xq
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-lang-supported-false-2.xq
 :)
declare function ft:is-stop-word-lang-supported( $lang as xs:language )
  as xs:boolean external;

(:~
 : Checks whether the given
 : language
 : is supported for look-up using the default thesaurus.
 :
 : @param $lang The language to check.
 : @return true only if the language is supported.
 :)
declare function ft:is-thesaurus-lang-supported( $lang as xs:language )
  as xs:boolean external;

(:~
 : Checks whether the given
 : language
 : is supported for look-up using the thesaurus specified by the given URI.
 :
 : @param $uri The URI specifying the thesaurus to use.
 : @param $lang The language to check.
 : @return true only if the language is supported.
 : @error err:FTST0018 if $uri refers to a thesaurus
 : that is not found in the statically known thesauri.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-thesaurus-lang-supported-true-1.xq
 :)
declare function ft:is-thesaurus-lang-supported( $uri as xs:string,
                                                 $lang as xs:language )
  as xs:boolean external;

(:~
 : Checks whether the given
 : language
 : is supported for tokenization.
 :
 : @param $lang The language to check.
 : @return true only if the language is supported.
 :)
declare function ft:is-tokenizer-lang-supported( $lang as xs:language )
  as xs:boolean external;

(:~
 : Stems the given word.
 :
 : @param $word The word to stem.
 : @param $lang The
 : language
 : of $word.
 : @return the stem of $word.
 : @error err:FTST0009 if $lang is not supported.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-stem-1.xq
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-stem-2.xq
 :)
declare function ft:stem( $word as xs:string, $lang as xs:language )
  as xs:string external;

(:~
 : Stems the given word.
 :
 : @param $word The word to stem.
 : The word's language
 : is assumed to be the one returned by ft:current-lang().
 : @return the stem of $word.
 : @error err:FTST0009 if ft:current-lang() is not supported.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-stem-3.xq
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-stem-4.xq
 :)
declare function ft:stem( $word as xs:string )
  as xs:string external;

(:~
 : Strips all diacritical marks from all characters.
 :
 : @param $string The string to strip diacritical marks from.
 : @return $string with diacritical marks stripped.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-strip-diacritics-1.xq
 :)
declare function ft:strip-diacritics( $string as xs:string )
  as xs:string external;

(:~
 : Looks-up the given phrase in the default thesaurus.
 :
 : @param $phrase The phrase to look up.
 : The phrase's
 : language
 : is assumed to be the one returned by ft:current-lang().
 : @return the original and related phrases.
 : @error err:FTST0009 if ft:current-lang() is not supported.
 : @error zerr:ZXQP8401 if the thesaurus data file's version is not supported
 : by the currently running version of Zorba.
 : @error zerr:ZXQP8402 if the thesaurus data file's endianness does not match
 : that of the CPU on which Zorba is currently running.
 : @error zerr:ZXQP8403 if there was an error reading the thesaurus data.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-thesaurus-lookup-1.xq
 :)
declare function ft:thesaurus-lookup( $phrase as xs:string )
  as xs:string+ external;

(:~
 : Looks-up the given phrase in the thesaurus specified by the given URI.
 :
 : @param $uri The URI specifying the thesaurus to use.
 : @param $phrase The phrase to look up.
 : @param $lang The
 : language
 : of $phrase.
 : @return the original and related phrases.
 : @error err:FTST0009 if $lang is not supported.
 : @error err:FTST0018 if $uri refers to a thesaurus
 : that is not found in the statically known thesauri.
 : @error zerr:ZOSE0001 if the thesaurus data file could not be found.
 : @error zerr:ZOSE0002 if the thesaurus data file is not a plain file.
 : @error zerr:ZXQP8401 if the thesaurus data file's version is not supported
 : by the currently running version of Zorba.
 : @error zerr:ZXQP8402 if the thesaurus data file's endianness does not match
 : that of the CPU on which Zorba is currently running.
 : @error zerr:ZXQP8403 if there was an error reading the thesaurus data file.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-thesaurus-lookup-2.xq
 :)
declare function ft:thesaurus-lookup( $uri as xs:string, $phrase as xs:string,
                                      $lang as xs:language )
  as xs:string+ external;

(:~
 : Looks-up the given phrase in a thesaurus.
 :
 : @param $uri The URI specifying the thesaurus to use.
 : @param $phrase The phrase to look up.
 : The phrase's
 : language
 : is assumed to be the one the one returned by ft:current-lang().
 : @return the original and related phrases.
 : @error err:FTST0009 if ft:current-lang() is unsupported.
 : @error err:FTST0018 if $uri refers to a thesaurus
 : that is not found in the statically known thesauri.
 : @error zerr:ZOSE0001 if the thesaurus data file could not be found.
 : @error zerr:ZOSE0002 if the thesaurus data file is not a plain file.
 : @error zerr:ZXQP8401 if the thesaurus data file's version is not supported
 : by the currently running version of Zorba.
 : @error zerr:ZXQP8402 if the thesaurus data file's endianness does not match
 : that of the CPU on which Zorba is currently running.
 : @error zerr:ZXQP8403 if there was an error reading the thesaurus data file.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-thesaurus-lookup-3.xq
 :)
declare function ft:thesaurus-lookup( $uri as xs:string, $phrase as xs:string )
  as xs:string+ external;

(:~
 : Looks-up the given phrase in a thesaurus.
 :
 : @param $uri The URI specifying the thesaurus to use.
 : @param $phrase The phrase to look up.
 : @param $lang The
 : language
 : of $phrase.
 : @param $relationship The relationship the results are to have to
 : $phrase.
 : @return the original and related phrases.
 : @error err:FTST0018 if $uri refers to a thesaurus
 : that is not found in the statically known thesauri.
 : @error err:FTST0009 if $lang is not supported.
 : @error zerr:ZOSE0001 if the thesaurus data file could not be found.
 : @error zerr:ZOSE0002 if the thesaurus data file is not a plain file.
 : @error zerr:ZXQP8401 if the thesaurus data file's version is not supported
 : by the currently running version of Zorba.
 : @error zerr:ZXQP8402 if the thesaurus data file's endianness does not match
 : that of the CPU on which Zorba is currently running.
 : @error zerr:ZXQP8403 if there was an error reading the thesaurus data file.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-thesaurus-lookup-4.xq
 :)
declare function ft:thesaurus-lookup( $uri as xs:string, $phrase as xs:string,
                                      $lang as xs:language,
                                      $relationship as xs:string )
  as xs:string+ external;

(:~
 : Looks-up the given phrase in a thesaurus.
 :
 : @param $uri The URI specifying the thesaurus to use.
 : @param $phrase The phrase to look up.
 : @param $lang The
 : language
 : of $phrase.
 : @param $relationship The relationship the results are to have to
 : $phrase.
 : @param $level-least The minimum number of levels within the thesaurus to be
 : travers$ed.
 : @param $level-most The maximum number of levels within the thesaurus to be
 : traversed.
 : @return the original and related phrases.
 : @error err:FOCA0003 if either $level-least or
 : $level-most is either negative or too large.
 : @error err:FTST0018 if $uri refers to a thesaurus
 : that is not found in the statically known thesauri.
 : @error err:FTST0009 if $lang is not supported.
 : @error zerr:ZOSE0001 if the thesaurus data file could not be found.
 : @error zerr:ZOSE0002 if the thesaurus data file is not a plain file.
 : @error zerr:ZXQP8401 if the thesaurus data file's version is not supported
 : by the currently running version of Zorba.
 : @error zerr:ZXQP8402 if the thesaurus data file's endianness does not match
 : that of the CPU on which Zorba is currently running.
 : @error zerr:ZXQP8403 if there was an error reading the thesaurus data file.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-thesaurus-lookup-5.xq
 :)
declare function ft:thesaurus-lookup( $uri as xs:string, $phrase as xs:string,
                                      $lang as xs:language,
                                      $relationship as xs:string,
                                      $level-least as xs:integer,
                                      $level-most as xs:integer )
  as xs:string+ external;

(:~
 : Tokenizes the given node and all of its descendants.
 :
 : @param $node The node to tokenize.
 : @param $lang The default
 : language
 : of $node.
 : @return a (possibly empty) sequence of tokens.
 : @error err:FTST0009 if $lang is not supported in general.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-node-1.xq
 :)
declare function ft:tokenize-node( $node as node(), $lang as xs:language )
  as element(ft-schema:token)* external;

(:~
 : Tokenizes the given node and all of its descendants.
 :
 : @param $node The node to tokenize.
 : The document's default
 : language
 : is assumed to be the one returned by ft:current-lang().
 : @return a (possibly empty) sequence of tokens.
 : @error err:FTST0009 if ft:current-lang() is not supported in
 : general.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-node-2.xq
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-node-3.xq
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-node-4.xq
 :)
declare function ft:tokenize-node( $node as node() )
  as element(ft-schema:token)* external;

(:~
 : Tokenizes the given string.
 :
 : @param $string The string to tokenize.
 : @param $lang The default
 : language
 : of $string.
 : @return a (possibly empty) sequence of tokens.
 : @error err:FTST0009 if $lang is not supported.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-string-1.xq
 :)
declare function ft:tokenize-string( $string as xs:string,
                                     $lang as xs:language )
  as xs:string* external;

(:~
 : Tokenizes the given string.
 :
 : @param $string The string to tokenize.
 : The string's default
 : language
 : is assumed to be the one returned by ft:current-lang().
 : @return a (possibly empty) sequence of tokens.
 : @error err:FTST0009 if ft:current-lang() is not supported.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-string-2.xq
 :)
declare function ft:tokenize-string( $string as xs:string )
  as xs:string* external;

(:~
 : Gets properties of the tokenizer for the given
 : language.
 :
 : @param $lang The language of the tokenizer to get the properties of.
 : @return said properties.
 : @error err:FTST0009 if $lang is not supported.
 : tokenization specifically.
 :)
declare function ft:tokenizer-properties( $lang as xs:language )
  as element(ft-schema:tokenizer-properties) external;

(:~
 : Gets properties of the tokenizer for the
 : language
 : returned by ft:current-lang().
 :
 : @return said properties.
 : @error err:FTST0009 if ft:current-lang() is not supported.
 :)
declare function ft:tokenizer-properties()
  as element(ft-schema:tokenizer-properties) external;

(:===========================================================================:)

(: vim:set et sw=2 ts=2: :)

xquery version "3.0";

(:
 : Copyright 2006-2011 The FLWOR Foundation.
 :
 : Licensed under the Apache License, Version 2.0 (the "License");
 : you may not use this file except in compliance with the License.
 : You may obtain a copy of the License at
 :
 : http://www.apache.org/licenses/LICENSE-2.0
 :
 : Unless required by applicable law or agreed to in writing, software
 : distributed under the License is distributed on an "AS IS" BASIS,
 : WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 : See the License for the specific language governing permissions and
 : limitations under the License.
 :)

(:===========================================================================:)

(:~
 : This module provides an XQuery API to full-text functions.
 : For general information about Zorba's implementation of the
 : XQuery and XPath Full Text 1.0 specification
 : as well as instructions for building an installing a thesaurus,
 : see the Full Text Thesaurus documentation.
 :
Notes on languages

 : To refer to paricular human languages,
 : Zorba uses both the
 : ISO 639-1
 : and
 : ISO 639-2
 : languages codes.
 : Note that Zorba supports only a subset of the
 : complete list of language codes
 : and not every function supports the same subset.
 :

 : Most functions in this module take a language as a parameter
 : using the
 : xs:language
 : XML schema data type.
 :
Notes on stemming

 : The stem() functions return the
 : stem
 : of a word.
 : In Zorba,
 : the stem of a word itself, however, is not guaranteed to be a word.
 : It is best to consider a stem as an opaque byte sequence.
 : All that is guaranteed about a stem is that,
 : for a given word,
 : the stem of that word will always be the same byte sequence.
 : Hence,
 : you sould never compare the result of one of the stem()
 : functions against a non-stemmed string,
 : for example:
 :

 :  if ( ft:stem( "apples" ) eq "apple" )             ** WRONG **
 :

 : Instead do:
 :

 :  if ( ft:stem( "apples" ) eq ft:stem( "apple" ) )  ** CORRECT **
 :

 :
Notes on the thesaurus

 : The thesaurus-lookup() functions have "levels"
 : and "relationship" parameters.
 : The values for these are implementation-defined.
 : Zorba's default implementation uses the
 : WordNet lexical database,
 : version 3.0.
 :

 : In WordNet,
 : the number of "levels" that two phrases are apart
 : are how many hierarchical meanings apart they are.
 : For example,
 : "canary" is 5 levels away from "vertebrate"
 : (carary > finch > oscine > passerine > bird > vertebrate).
 :

 : When using the WordNet implementation,
 : Zorba supports all of the relationships (and their abbreviations)
 : specified by
 : ISO 2788
 : and
 : ANSI/NISO Z39.19-2005
 : with the exceptions of "HN" (history note)
 : and "X SN" (see scope note for).
 : These relationships are:
 :

 :

 :      Rel.
 :      Meaning
 :      WordNet Rel.
 :

 :

 :      BT
 :      broader term
 :      hypernym
 :

 :

 :      BTG
 :      broader term generic
 :      hypernym
 :

 :

 :      BTI
 :      broader term instance
 :      instance hypernym
 :

 :

 :      BTP
 :      broader term partitive
 :      part meronym
 :

 :

 :      NT
 :      narrower term
 :      hyponym
 :

 :

 :      NTG
 :      narrower term generic
 :      hyponym
 :

 :

 :      NTI
 :      narrower term instance
 :      instance hyponym
 :

 :

 :      NTP
 :      narrower term partitive
 :      part holonym
 :

 :

 :      RT
 :      related term
 :      also see
 :

 :

 :      SN
 :      scope note
 :      n/a
 :

 :

 :      TT
 :      top term
 :      hypernym
 :

 :

 :      UF
 :      non-preferred term
 :      n/a
 :

 :

 :      USE
 :      preferred term
 :      n/a
 :

 :

 : Note that you can specify relationships
 : either by their abbreviation
 : or their meaning.
 : Relationships are case-insensitive.
 :
 : In addition to the
 : ISO 2788
 : and
 : ANSI/NISO Z39.19-2005
 : relationships,
 : Zorba also supports all of the relationships offered by WordNet.
 : These relationships are:
 :

 :

 :      Relationship
 :      Meaning
 :

 :

 :      also see
 :
 :        A word that is related to another,
 :        e.g., for "varnished" (furniture)
 :        one should also see "finished."
 :
 :

 :

 :      antonym
 :
 :        A word opposite in meaning to another,
 :        e.g., "light" is an antonym for "heavy."
 :
 :

 :

 :      attribute
 :
 :        A noun for which adjectives express values,
 :        e.g., "weight" is an attribute
 :        for which the adjectives "light" and "heavy"
 :        express values.
 :
 :

 :

 :      cause
 :
 :        A verb that causes another,
 :        e.g., "show" is a cause of "see."
 :
 :

 :

 :      derivationally related form
 :
 :        A word that is derived from a root word,
 :        e.g., "metric" is a derivationally related form of "meter."
 :
 :

 :

 :      derived from adjective
 :
 :        An adverb that is derived from an adjective,
 :        e.g., "correctly" is derived from the adjective "correct."
 :
 :

 :

 :      entailment
 :
 :        A verb that presupposes another,
 :        e.g., "snoring" entails "sleeping."
 :
 :

 :

 :      hypernym
 :
 :        A word with a broad meaning that more specific words fall under,
 :        e.g., "meal" is a hypernym of "breakfast."
 :
 :

 :

 :      hyponym
 :
 :        A word of more specific meaning than a general term applicable to it,
 :        e.g., "breakfast" is a hyponym of "meal."
 :
 :

 :

 :      instance hypernym
 :
 :        A word that denotes a category of some specific instance,
 :        e.g., "author" is an instance hypernym of "Asimov."
 :
 :

 :

 :      instance hyponym
 :
 :        A term that donotes a specific instance of some general category,
 :        e.g., "Asimov" is an instance hyponym of "author."
 :
 :

 :

 :      member holonym
 :
 :        A word that denotes a collection of individuals,
 :        e.g., "faculty" is a member holonym of "professor."
 :
 :

 :

 :      member meronym
 :
 :        A word that denotes a member of a larger group,
 :        e.g., a "person" is a member meronym of a "crowd."
 :
 :

 :

 :      part holonym
 :
 :        A word that denotes a larger whole comprised of some part,
 :        e.g., "car" is a part holonym of "engine."
 :
 :

 :

 :      part meronym
 :
 :        A word that denotes a part of a larger whole,
 :        e.g., an "engine" is part meronym of a "car."
 :
 :

 :

 :      participle of verb
 :
 :        An adjective that is the participle of some verb,
 :        e.g., "breaking" is the participle of the verb "break."
 :
 :

 :

 :      pertainym
 :
 :        An adjective that classifies its noun,
 :        e.g., "musical" is a pertainym in "musical instrument."
 :
 :

 :

 :      similar to
 :
 :        Similar, though not necessarily interchangeable, adjectives.
 :        For example, "shiny" is similar to "bright",
 :        but they have subtle differences.
 :
 :

 :

 :      substance holonym
 :
 :        A word that denotes a larger whole containing some constituent
 :        substance, e.g., "bread" is a substance holonym of "flour."
 :
 :

 :

 :      substance meronym
 :
 :        A word that denotes a constituant substance of some larger whole,
 :        e.g., "flour" is a substance meronym of "bread."
 :
 :

 :

 :      verb group
 :
 :        A verb that is a member of a group of similar verbs,
 :        e.g., "live" is in the verb group
 :        of "dwell", "live", "inhabit", etc.
 :
 :

 :

 :
Notes on tokenization

 : For general information about Zorba's implementation of tokenization,
 : including what constitutes a token,
 : see the Full Text Tokenizer documentation.
 : @author Paul J. Lucas
 :)
module namespace ft = "http://www.zorba-xquery.com/modules/full-text";

import schema namespace ft-schema =
  "http://www.zorba-xquery.com/modules/full-text";

declare namespace err = "http://www.w3.org/2005/xqt-errors";
declare namespace zerr = "http://www.zorba-xquery.com/errors";

declare namespace ver = "http://www.zorba-xquery.com/options/versioning";
declare option ver:module-version "2.0";

(:===========================================================================:)

(:~
 : Predeclared constant for the Danish
 : xs:language.
 :)
declare variable $ft:lang-da as xs:language := xs:language("da");

(:~
 : Predeclared constant for the German
 : xs:language.
 :)
declare variable $ft:lang-de as xs:language := xs:language("de");

(:~
 : Predeclared constant for the English
 : xs:language.
 :)
declare variable $ft:lang-en as xs:language := xs:language("en");

(:~
 : Predeclared constant for the Spanish
 : xs:language.
 :)
declare variable $ft:lang-es as xs:language := xs:language("es");

(:~
 : Predeclared constant for the Finnish
 : xs:language.
 :)
declare variable $ft:lang-fi as xs:language := xs:language("fi");

(:~
 : Predeclared constant for the French
 : xs:language.
 :)
declare variable $ft:lang-fr as xs:language := xs:language("fr");

(:~
 : Predeclared constant for the Hungarian
 : xs:language.
 :)
declare variable $ft:lang-hu as xs:language := xs:language("hu");

(:~
 : Predeclared constant for the Italian
 : xs:language.
 :)
declare variable $ft:lang-it as xs:language := xs:language("it");

(:~
 : Predeclared constant for the Dutch
 : xs:language.
 :)
declare variable $ft:lang-nl as xs:language := xs:language("nl");

(:~
 : Predeclared constant for the Norwegian
 : xs:language.
 :)
declare variable $ft:lang-no as xs:language := xs:language("no");

(:~
 : Predeclared constant for the Portuguese
 : xs:language.
 :)
declare variable $ft:lang-pt as xs:language := xs:language("pt");

(:~
 : Predeclared constant for the Romanian
 : xs:language.
 :)
declare variable $ft:lang-ro as xs:language := xs:language("ro");

(:~
 : Predeclared constant for the Russian
 : xs:language.
 :)
declare variable $ft:lang-ru as xs:language := xs:language("ru");

(:~
 : Predeclared constant for the Swedish
 : xs:language.
 :)
declare variable $ft:lang-sv as xs:language := xs:language("sv");

(:~
 : Predeclared constant for the Turkish
 : xs:language.
 :)
declare variable $ft:lang-tr as xs:language := xs:language("tr");

(:===========================================================================:)

(:~
 : Gets the current compare options.
 :
 : @return said compare options.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-current-compare-options-1.xq
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-current-compare-options-2.xq
 :)
declare function ft:current-compare-options()
  as element(ft-schema:compare-options) external;

(:~
 : Gets the current
 : language:
 : either the language specified by the
 : declare ft-option using
 : language
 : statement (if any)
 : or the one returned by ft:host-lang() (if none).
 :
 : @return said language.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-current-lang-true-1.xq
 :)
declare function ft:current-lang()
  as xs:language external;

(:~
 : Gets the host's current
 : language.
 : The "host" is the computer on which Zorba is running.
 : The host's current language is obtained as follows:
 :

 :

 :      For *nix systems:
 :

 :

 :          If setlocale(3) returns non-null,
 :          the language corresponding to that locale is used.
 :

 :

 :          Else, if the LANG environment variable is set,
 :          that language is ued.
 :

 :

 :          Otherwise, there is no default language.
 :

 :

 :

 :

 :      For Windows systems,
 :      the language corresponding to the locale returned by the
 :      GetLocaleInfo()
 :      function is used.
 :

 :

 :
 : @return said language.
 :)
declare function ft:host-lang()
  as xs:language external;

(:~
 : Checks whether the given
 : language
 : is supported for stemming.
 :
 : @param $lang The language to check.
 : @return true only if the language is supported.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stem-lang-es-supported-true.xq
 :)
declare function ft:is-stem-lang-supported( $lang as xs:language )
  as xs:boolean external;

(:~
 : Checks whether the given word is a stop-word.
 :
 : @param $word The word to check.
 : @param $lang The
 : language
 : of $word.
 : @return true only if $word is a stop-word.
 : @error err:FTST0009 if $lang is not supported.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-true-1.xq
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-true-3.xq
 :)
declare function ft:is-stop-word( $word as xs:string, $lang as xs:language )
  as xs:boolean external;

(:~
 : Checks whether the given word is a stop-word.
 :
 : @param $word The word to check.
 : The word's language
 : is assumed to be the one returned by ft:current-lang().
 : @return true only if $word is a stop-word.
 : @error err:FTST0009 if ft:current-lang() is not supported.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-true-2.xq
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-true-4.xq
 :)
declare function ft:is-stop-word( $word as xs:string )
  as xs:boolean external;

(:~
 : Checks whether the given
 : language
 : is supported for stop words.
 :
 : @param $lang The language to check.
 : @return true only if the language is supported.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-lang-en-supported-true.xq
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-lang-supported-false-1.xq
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-lang-supported-false-2.xq
 :)
declare function ft:is-stop-word-lang-supported( $lang as xs:language )
  as xs:boolean external;

(:~
 : Checks whether the given
 : language
 : is supported for look-up using the default thesaurus.
 :
 : @param $lang The language to check.
 : @return true only if the language is supported.
 :)
declare function ft:is-thesaurus-lang-supported( $lang as xs:language )
  as xs:boolean external;

(:~
 : Checks whether the given
 : language
 : is supported for look-up using the thesaurus specified by the given URI.
 :
 : @param $uri The URI specifying the thesaurus to use.
 : @param $lang The language to check.
 : @return true only if the language is supported.
 : @error err:FTST0018 if $uri refers to a thesaurus
 : that is not found in the statically known thesauri.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-thesaurus-lang-supported-true-1.xq
 :)
declare function ft:is-thesaurus-lang-supported( $uri as xs:string,
                                                 $lang as xs:language )
  as xs:boolean external;

(:~
 : Checks whether the given
 : language
 : is supported for tokenization.
 :
 : @param $lang The language to check.
 : @return true only if the language is supported.
 :)
declare function ft:is-tokenizer-lang-supported( $lang as xs:language )
  as xs:boolean external;

(:~
 : Stems the given word.
 :
 : @param $word The word to stem.
 : @param $lang The
 : language
 : of $word.
 : @return the stem of $word.
 : @error err:FTST0009 if $lang is not supported.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-stem-1.xq
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-stem-2.xq
 :)
declare function ft:stem( $word as xs:string, $lang as xs:language )
  as xs:string external;

(:~
 : Stems the given word.
 :
 : @param $word The word to stem.
 : The word's language
 : is assumed to be the one returned by ft:current-lang().
 : @return the stem of $word.
 : @error err:FTST0009 if ft:current-lang() is not supported.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-stem-3.xq
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-stem-4.xq
 :)
declare function ft:stem( $word as xs:string )
  as xs:string external;

(:~
 : Strips all diacritical marks from all characters.
 :
 : @param $string The string to strip diacritical marks from.
 : @return $string with diacritical marks stripped.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-strip-diacritics-1.xq
 :)
declare function ft:strip-diacritics( $string as xs:string )
  as xs:string external;

(:~
 : Looks-up the given phrase in the default thesaurus.
 :
 : @param $phrase The phrase to look up.
 : The phrase's
 : language
 : is assumed to be the one returned by ft:current-lang().
 : @return the original and related phrases.
 : @error err:FTST0009 if ft:current-lang() is not supported.
 : @error zerr:ZXQP8401 if the thesaurus data file's version is not supported
 : by the currently running version of Zorba.
 : @error zerr:ZXQP8402 if the thesaurus data file's endianness does not match
 : that of the CPU on which Zorba is currently running.
 : @error zerr:ZXQP8403 if there was an error reading the thesaurus data.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-thesaurus-lookup-1.xq
 :)
declare function ft:thesaurus-lookup( $phrase as xs:string )
  as xs:string+ external;

(:~
 : Looks-up the given phrase in the thesaurus specified by the given URI.
 :
 : @param $uri The URI specifying the thesaurus to use.
 : @param $phrase The phrase to look up.
 : @param $lang The
 : language
 : of $phrase.
 : @return the original and related phrases.
 : @error err:FTST0009 if $lang is not supported.
 : @error err:FTST0018 if $uri refers to a thesaurus
 : that is not found in the statically known thesauri.
 : @error zerr:ZOSE0001 if the thesaurus data file could not be found.
 : @error zerr:ZOSE0002 if the thesaurus data file is not a plain file.
 : @error zerr:ZXQP8401 if the thesaurus data file's version is not supported
 : by the currently running version of Zorba.
 : @error zerr:ZXQP8402 if the thesaurus data file's endianness does not match
 : that of the CPU on which Zorba is currently running.
 : @error zerr:ZXQP8403 if there was an error reading the thesaurus data file.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-thesaurus-lookup-2.xq
 :)
declare function ft:thesaurus-lookup( $uri as xs:string, $phrase as xs:string,
                                      $lang as xs:language )
  as xs:string+ external;

(:~
 : Looks-up the given phrase in a thesaurus.
 :
 : @param $uri The URI specifying the thesaurus to use.
 : @param $phrase The phrase to look up.
 : The phrase's
 : language
 : is assumed to be the one the one returned by ft:current-lang().
 : @return the original and related phrases.
 : @error err:FTST0009 if ft:current-lang() is unsupported.
 : @error err:FTST0018 if $uri refers to a thesaurus
 : that is not found in the statically known thesauri.
 : @error zerr:ZOSE0001 if the thesaurus data file could not be found.
 : @error zerr:ZOSE0002 if the thesaurus data file is not a plain file.
 : @error zerr:ZXQP8401 if the thesaurus data file's version is not supported
 : by the currently running version of Zorba.
 : @error zerr:ZXQP8402 if the thesaurus data file's endianness does not match
 : that of the CPU on which Zorba is currently running.
 : @error zerr:ZXQP8403 if there was an error reading the thesaurus data file.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-thesaurus-lookup-3.xq
 :)
declare function ft:thesaurus-lookup( $uri as xs:string, $phrase as xs:string )
  as xs:string+ external;

(:~
 : Looks-up the given phrase in a thesaurus.
 :
 : @param $uri The URI specifying the thesaurus to use.
 : @param $phrase The phrase to look up.
 : @param $lang The
 : language
 : of $phrase.
 : @param $relationship The relationship the results are to have to
 : $phrase.
 : @return the original and related phrases.
 : @error err:FTST0018 if $uri refers to a thesaurus
 : that is not found in the statically known thesauri.
 : @error err:FTST0009 if $lang is not supported.
 : @error zerr:ZOSE0001 if the thesaurus data file could not be found.
 : @error zerr:ZOSE0002 if the thesaurus data file is not a plain file.
 : @error zerr:ZXQP8401 if the thesaurus data file's version is not supported
 : by the currently running version of Zorba.
 : @error zerr:ZXQP8402 if the thesaurus data file's endianness does not match
 : that of the CPU on which Zorba is currently running.
 : @error zerr:ZXQP8403 if there was an error reading the thesaurus data file.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-thesaurus-lookup-4.xq
 :)
declare function ft:thesaurus-lookup( $uri as xs:string, $phrase as xs:string,
                                      $lang as xs:language,
                                      $relationship as xs:string )
  as xs:string+ external;

(:~
 : Looks-up the given phrase in a thesaurus.
 :
 : @param $uri The URI specifying the thesaurus to use.
 : @param $phrase The phrase to look up.
 : @param $lang The
 : language
 : of $phrase.
 : @param $relationship The relationship the results are to have to
 : $phrase.
 : @param $level-least The minimum number of levels within the thesaurus to be
 : travers$ed.
 : @param $level-most The maximum number of levels within the thesaurus to be
 : traversed.
 : @return the original and related phrases.
 : @error err:FOCA0003 if either $level-least or
 : $level-most is either negative or too large.
 : @error err:FTST0018 if $uri refers to a thesaurus
 : that is not found in the statically known thesauri.
 : @error err:FTST0009 if $lang is not supported.
 : @error zerr:ZOSE0001 if the thesaurus data file could not be found.
 : @error zerr:ZOSE0002 if the thesaurus data file is not a plain file.
 : @error zerr:ZXQP8401 if the thesaurus data file's version is not supported
 : by the currently running version of Zorba.
 : @error zerr:ZXQP8402 if the thesaurus data file's endianness does not match
 : that of the CPU on which Zorba is currently running.
 : @error zerr:ZXQP8403 if there was an error reading the thesaurus data file.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-thesaurus-lookup-5.xq
 :)
declare function ft:thesaurus-lookup( $uri as xs:string, $phrase as xs:string,
                                      $lang as xs:language,
                                      $relationship as xs:string,
                                      $level-least as xs:integer,
                                      $level-most as xs:integer )
  as xs:string+ external;

(:~
 : Tokenizes the given node and all of its descendants.
 :
 : @param $node The node to tokenize.
 : @param $lang The default
 : language
 : of $node.
 : @return a (possibly empty) sequence of tokens.
 : @error err:FTST0009 if $lang is not supported in general.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-1.xq
 :)
declare function ft:tokenize-node( $node as node(), $lang as xs:language )
  as element(ft-schema:token)* external;

(:~
 : Tokenizes the given node and all of its descendants.
 :
 : @param $node The node to tokenize.
 : The document's default
 : language
 : is assumed to be the one returned by ft:current-lang().
 : @return a (possibly empty) sequence of tokens.
 : @error err:FTST0009 if ft:current-lang() is not supported in
 : general.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-2.xq
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-3.xq
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-4.xq
 :)
declare function ft:tokenize-node( $node as node() )
  as element(ft-schema:token)* external;

(:~
 : Tokenizes the given string.
 :
 : @param $string The string to tokenize.
 : @param $lang The default
 : language
 : of $string.
 : @return a (possibly empty) sequence of tokens.
 : @error err:FTST0009 if $lang is not supported.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-string-1.xq
 :)
declare function ft:tokenize-string( $string as xs:string,
                                     $lang as xs:language )
  as xs:string* external;

(:~
 : Tokenizes the given string.
 :
 : @param $string The string to tokenize.
 : The string's default
 : language
 : is assumed to be the one returned by ft:current-lang().
 : @return a (possibly empty) sequence of tokens.
 : @error err:FTST0009 if ft:current-lang() is not supported.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-string-2.xq
 :)
declare function ft:tokenize-string( $string as xs:string )
  as xs:string* external;

(:~
 : Gets properties of the tokenizer for the given
 : language.
 :
 : @param $lang The language of the tokenizer to get the properties of.
 : @return said properties.
 : @error err:FTST0009 if $lang is not supported.
 : tokenization specifically.
 :)
declare function ft:tokenizer-properties( $lang as xs:language )
  as element(ft-schema:tokenizer-properties) external;

(:~
 : Gets properties of the tokenizer for the
 : language
 : returned by ft:current-lang().
 :
 : @return said properties.
 : @error err:FTST0009 if ft:current-lang() is not supported.
 :)
declare function ft:tokenizer-properties()
  as element(ft-schema:tokenizer-properties) external;

(:===========================================================================:)

(: vim:set et sw=2 ts=2: :)


Added documentation to the helper functions of the http-client

(:
 : Copyright 2006-2009 The FLWOR Foundation.
 :
 : Licensed under the Apache License, Version 2.0 (the "License");
 : you may not use this file except in compliance with the License.
 : You may obtain a copy of the License at
 :
 : http://www.apache.org/licenses/LICENSE-2.0
 :
 : Unless required by applicable law or agreed to in writing, software
 : distributed under the License is distributed on an "AS IS" BASIS,
 : WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 : See the License for the specific language governing permissions and
 : limitations under the License.
:)

(:~
 : This module defines a set of functions to send HTTP and HTTPS requests
 : and handle responses. This is quite a low-level functionality, allowing
 : one to deal with most aspects of the HTTP protocol.
 :
 : @see http://www.expath.org/modules/http-client/
 : @version Candidate 13 April 2009
 : @author Markus Pilman
 :)
module namespace http = "http://www.expath.org/mod/http-client";

import module namespace httpclientimpl = "http://www.expath.org/mod/httpclientimpl";

import schema namespace https = "http://www.expath.org/mod/http-client";

(:~
 : This function sends an HTTP request and returns the corresponding
 : response. It supports HTTP multi-part messages.
 :
 : @param $request Contains the various parameters of the request. See the
 :  documentation of the type requestType in the schema definition.
 : @param $href is the HTTP or HTTPS URI to send the request to. It is an
 :  xs:anyURI, but is declared as a string to be able to pass literal strings
 :  (without requiring to explicitly cast it to an xs:anyURI.)
 : @param $content is the request body content, for HTTP methods that can
 :  contain a body in the request (POST and PUT.) It is an error, if this
 :  param is not the empty sequence for methods other then DELETE, GET, HEAD
 :  and OPTIONS.
 : @return a sequence of items, where the first item is a element of type
 :  http:responseType and describes the response of the http request. If there
 :  is one (or several, in case of multipart) response body, the response bodies
 :  are the next items in the sequence.
:)
declare function http:send-request(
    $request as element(https:request, https:requestType)?,
    $href as xs:string?,
    $content as item()?,
    $serial as item()?) as item()+ {
    if (fn:empty($href) and fn:empty($request)) then
        fn:error(fn:QName("http://www.expath.org/mod/http-client", "WrongArgs"),
            "Not all required arguments are set.")
    else
        if ($href eq "") then
            fn:error(fn:QName("http://www.expath.org/mod/http-client", "HrefEmpty"),
                "The href value is set to the empty string")
        else
            httpclientimpl:http-send-request-impl($request,
                $href,
                $content,
                $serial)
};

(:~
 : Function for convenience.
 : @see documentation of send-request with four parameters.
 :
 : @see http://www.expath.org/mod/http-client;send-request
 :
 : @param $request @see parameter $request of
 :  http://www.expath.org/mod/http-client;send-request
 : @return @see return of
 :  http://www.expath.org/mod/http-client;send-request
:)
declare function http:send-request (
    $request as element(https:request, https:requestType)) as item()+ {
    http:send-request($request,(),(),())
};

(:~
 : Function for convenience.
 : @see documentation of send-request with four parameters.
 :
 : @see http://www.expath.org/mod/http-client;send-request
 :
 : @param $request @see parameter $request of
 :  http://www.expath.org/mod/http-client;send-request
 : @param href @see parameter $href of
 :  http://www.expath.org/mod/http-client;send-request
 : @return @see return of
 :  http://www.expath.org/mod/http-client;send-request
:)
declare function http:send-request(
    $request as element(https:request, https:requestType)?,
    $href as xs:string?) as item()+ {
    http:send-request($request,$href,(),())
};

(:~
 : Function for convenience.
 : @see documentation of send-request with four parameters.
 :
 : @see http://www.expath.org/mod/http-client;send-request
 :
 : @param $request @see parameter $request of
 :  http://www.expath.org/mod/http-client;send-request
 : @param $href @see parameter $href of
 :  http://www.expath.org/mod/http-client;send-request
 : @param $content @see parameter $content of
 :  http://www.expath.org/mod/http-client;send-request
 : @return @see return of
 :  http://www.expath.org/mod/http-client;send-request
:)
declare function http:send-request(
    $request as element(https:request, https:requestType)?,
    $href as xs:string?,
    $content as item()?) as item()+ {
    http:send-request($request,$href,$content,())
};

(:
 : Copyright 2006-2009 The FLWOR Foundation.
 :
 : Licensed under the Apache License, Version 2.0 (the "License");
 : you may not use this file except in compliance with the License.
 : You may obtain a copy of the License at
 :
 : http://www.apache.org/licenses/LICENSE-2.0
 :
 : Unless required by applicable law or agreed to in writing, software
 : distributed under the License is distributed on an "AS IS" BASIS,
 : WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 : See the License for the specific language governing permissions and
 : limitations under the License.
:)

(:~
 : This module defines a set of functions to send HTTP and HTTPS requests
 : and handle responses. This is quite a low-level functionality, allowing
 : one to deal with most aspects of the HTTP protocol.
 :
 : @see http://www.expath.org/modules/http-client/
 : @version Candidate 13 April 2009
 : @author Markus Pilman
 :)
module namespace http = "http://www.expath.org/mod/http-client";

import module namespace httpclientimpl = "http://www.expath.org/mod/httpclientimpl";

import schema namespace https = "http://www.expath.org/mod/http-client";

(:~
 : This function sends an HTTP request and returns the corresponding
 : response. It supports HTTP multi-part messages.
 :
 : @param $request Contains the various parameters of the request. See the
 :  documentation of the type requestType in the schema definition.
 : @param $href is the HTTP or HTTPS URI to send the request to. It is an
 :  xs:anyURI, but is declared as a string to be able to pass literal strings
 :  (without requiring to explicitly cast it to an xs:anyURI.)
 : @param $content is the request body content, for HTTP methods that can
 :  contain a body in the request (POST and PUT.) It is an error, if this
 :  param is not the empty sequence for methods other then DELETE, GET, HEAD
 :  and OPTIONS.
:)
declare function http:send-request(
    $request as element(https:request, https:requestType)?,
    $href as xs:string?,
    $content as item()?,
    $serial as item()?) as item()+ {
    if (fn:empty($href) and fn:empty($request)) then
        fn:error(fn:QName("http://www.expath.org/mod/http-client", "WrongArgs"),
            "Not all required arguments are set.")
    else
        if ($href eq "") then
            fn:error(fn:QName("http://www.expath.org/mod/http-client", "HrefEmpty"),
                "The href value is set to the empty string")
        else
            httpclientimpl:http-send-request-impl($request,
                $href,
                $content,
                $serial)
};

(:~
 : Function for convenience.
 : @see documentation of send-request with four parameters.
 :
 : @see http://www.expath.org/mod/http-client;send-request
:)
declare function http:send-request (
    $request as element(https:request, https:requestType)) as item()+ {
    http:send-request($request,(),(),())
};

(:~
 : Function for convenience.
 : @see documentation of send-request with four parameters.
 :
 : @see http://www.expath.org/mod/http-client;send-request
:)
declare function http:send-request(
    $request as element(https:request, https:requestType)?,
    $href as xs:string?) as item()+ {
    http:send-request($request,$href,(),())
};

(:~
 : Function for convenience.
 : @see documentation of send-request with four parameters.
 :
 : @see http://www.expath.org/mod/http-client;send-request
:)
declare function http:send-request(
    $request as element(https:request, https:requestType)?,
    $href as xs:string?,
    $content as item()?) as item()+ {
    http:send-request($request,$href,$content,())
};


Set BaseX v7.8 as a baseline for `%a:since` annotation.

xquery version "3.0";
(:~
: BaseX Database Module functions
:
: @see http://docs.basex.org/wiki/Database_Module
:)
module namespace db = "http://basex.org/modules/db";

import module namespace a = "http://reecedunn.co.uk/xquery/annotations" at "res://reecedunn.co.uk/xquery/annotations.xqy";

declare %a:since("basex", "7.8") function db:system() as element(system) external;
declare %a:since("basex", "9.0") function db:option($name as xs:string) as xs:string external;
declare %a:since("basex", "7.8") function db:info($db as xs:string) as element(database) external;
declare %a:since("basex", "8.6") function db:property($db as xs:string, $name as xs:string) as xs:anyAtomicType external;
declare %a:since("basex", "7.8") function db:list() as xs:string* external;
declare %a:since("basex", "7.8") function db:list($db as xs:string) as xs:string* external;
declare %a:since("basex", "7.8") function db:list($db as xs:string, $path as xs:string) as xs:string* external;
declare %a:since("basex", "7.8") function db:list-details() as element(database)* external;
declare %a:since("basex", "7.8") function db:list-details($db as xs:string) as element(resource)* external;
declare %a:since("basex", "7.8") function db:list-details($db as xs:string, $path as xs:string) as element(resource)* external;
declare %a:since("basex", "7.8") function db:backups() as element(backup)* external;
declare %a:since("basex", "7.8") function db:backups($db as xs:string) as element(backup)* external;
declare %a:since("basex", "7.8") function db:open($db as xs:string) as document-node()* external;
declare %a:since("basex", "7.8") function db:open($db as xs:string, $path as xs:string) as document-node()* external;
declare %a:since("basex", "7.8") function db:open-pre($db as xs:string, $pre as xs:integer) as node() external;
declare %a:since("basex", "7.8") function db:open-id($db as xs:string, $id as xs:integer) as node() external;
declare %a:since("basex", "7.8") function db:node-pre($nodes as node()*) as xs:integer* external;
declare %a:since("basex", "7.8") function db:node-id($nodes as node()*) as xs:integer* external;
declare %a:since("basex", "7.8") function db:retrieve($db as xs:string, $path as xs:string) as xs:base64Binary external;
declare %a:since("basex", "7.8") function db:retrieve($db as xs:string, $path as xs:string) as xs:base64Binary external;
declare %a:since("basex", "7.8") function db:text($db as xs:string, $strings as xs:string*) as text()* external;
declare %a:since("basex", "7.8") function db:text-range($db as xs:string, $min as xs:string, $max as xs:string) as text()* external;
declare %a:since("basex", "7.8") function db:attribute($db as xs:string, $strings as xs:string*) as attribute()* external;
declare %a:since("basex", "7.8") function db:attribute($db as xs:string, $strings as xs:strings*, $name as xs:string) as attribute()* external;
declare %a:since("basex", "7.8") function db:attribute-range($db as xs:string, $min as xs:string, $max as xs:string) as attribute()* external;
declare %a:since("basex", "7.8") function db:attribute-range($db as xs:string, $min as xs:string, $max as xs:string, $name as xs:string) as attribute()* external;
declare %a:since("basex", "8.4") function db:token($db as xs:string, $tokens as xs:string*) as attribute()* external;
declare %a:since("basex", "8.4") function db:token($db as xs:string, $tokens as xs:string*, $name as xs:string) as attribute()* external;
declare %a:since("basex", "7.8") function db:create($db as xs:string) as empty-sequence() external;
declare %a:since("basex", "7.8") function db:create($db as xs:string, $inputs as item()*) as empty-sequence() external;
declare %a:since("basex", "7.8") function db:create($db as xs:string, $inputs as item()*, $paths as xs:string*) as empty-sequence() external;
declare %a:since("basex", "7.8") function db:create($db as xs:string, $inputs as item()*, $paths as xs:string*, $options as map(*)?) as empty-sequence() external;
declare %a:since("basex", "7.8") function db:drop($db as xs:string) as empty-sequence() external;
declare %a:since("basex", "7.8") function db:add($db as xs:string, $input as item()) as empty-sequence() external;
declare %a:since("basex", "7.8") function db:add($db as xs:string, $input as item(), $path as xs:string) as empty-sequence() external;
declare %a:since("basex", "7.8") function db:add($db as xs:string, $input as item(), $path as xs:string, $options as map(*)?) as empty-sequence() external;
declare %a:since("basex", "7.8") function db:delete($db as xs:string, $path as xs:string) as empty-sequence() external;
declare %a:since("basex", "7.8.2") function db:copy($db as xs:string, $name as xs:string) as empty-sequence() external;
declare %a:since("basex", "7.8.2") function db:alter($db as xs:string, $name as xs:string) as empty-sequence() external;
declare %a:since("basex", "7.8.2") function db:create-backup($db as xs:string) as empty-sequence() external;
declare %a:since("basex", "7.8.2") function db:drop-backup($name as xs:string) as empty-sequence() external;
declare %a:since("basex", "7.8.2") function db:restore($name as xs:string) as empty-sequence() external;
declare %a:since("basex", "7.8") function db:optimize($db as xs:string) as empty-sequence() external;
declare %a:since("basex", "7.8") function db:optimize($db as xs:string, $all as xs:boolean) as empty-sequence() external;
declare %a:since("basex", "7.8") function db:optimize($db as xs:string, $all as xs:boolean, $options as map(*)?) as empty-sequence() external;
declare %a:since("basex", "7.8") function db:rename($db as xs:string, $source as xs:string, $target as xs:string) as empty-sequence() external;
declare %a:since("basex", "7.8") function db:replace($db as xs:string, $path as xs:string, $input as item()) as empty-sequence() external;
declare %a:since("basex", "7.8") function db:replace($db as xs:string, $path as xs:string, $input as item(), $options as map(*)?) as empty-sequence() external;
declare %a:since("basex", "7.8") function db:store($db as xs:string, $path as xs:string, $input as item()) as empty-sequence() external;
declare %a:since("basex", "7.8") function db:flush($db as xs:string) as empty-sequence() external;
declare %a:since("basex", "7.8") function db:name($node as node()) as xs:string external;
declare %a:since("basex", "7.8") function db:path($node as node()) as xs:string external;
declare %a:since("basex", "7.8") function db:exists($db as xs:string) as xs:boolean external;
declare %a:since("basex", "7.8") function db:exists($db as xs:string, $path as xs:string) as xs:boolean external;
declare %a:since("basex", "7.8") function db:is-raw($db as xs:string, $path as xs:string) as xs:boolean external;
declare %a:since("basex", "7.8") function db:is-xml($db as xs:string, $path as xs:string) as xs:boolean external;
declare %a:since("basex", "7.8") function db:content-type($db as xs:string, $path as xs:string) as xs:string external;
declare %a:since("basex", "7.8") %a:until("basex", "9.0") %a:see-also("basex", "9.0", "update:output") function db:output() external;
declare %a:since("basex", "8.2") %a:until("basex", "9.0") %a:see-also("basex", "9.0", "update:cache") function db:output-cache() external;
declare %a:since("basex", "7.8") function db:export() external;
declare %a:since("basex", "7.8") %a:until("basex", "8.2") function db:event($name as xs:string, $query as item()) as empty-sequence() external;

xquery version "3.0";
(:~
: BaseX Database Module functions
:
: @see http://docs.basex.org/wiki/Database_Module
:)
module namespace db = "http://basex.org/modules/db";

import module namespace a = "http://reecedunn.co.uk/xquery/annotations" at "res://reecedunn.co.uk/xquery/annotations.xqy";

declare %a:since("basex", "7.0") function db:system() as element(system) external;
declare %a:since("basex", "9.0") function db:option($name as xs:string) as xs:string external;
declare %a:since("basex", "7.0") function db:info($db as xs:string) as element(database) external;
declare %a:since("basex", "8.6") function db:property($db as xs:string, $name as xs:string) as xs:anyAtomicType external;
declare %a:since("basex", "7.0") function db:list() as xs:string* external;
declare %a:since("basex", "7.0") function db:list($db as xs:string) as xs:string* external;
declare %a:since("basex", "7.0") function db:list($db as xs:string, $path as xs:string) as xs:string* external;
declare %a:since("basex", "7.1") function db:list-details() as element(database)* external;
declare %a:since("basex", "7.1") function db:list-details($db as xs:string) as element(resource)* external;
declare %a:since("basex", "7.1") function db:list-details($db as xs:string, $path as xs:string) as element(resource)* external;
declare %a:since("basex", "7.5") function db:backups() as element(backup)* external;
declare %a:since("basex", "7.5") function db:backups($db as xs:string) as element(backup)* external;
declare %a:since("basex", "7.0") function db:open($db as xs:string) as document-node()* external;
declare %a:since("basex", "7.0") function db:open($db as xs:string, $path as xs:string) as document-node()* external;
declare %a:since("basex", "7.0") function db:open-pre($db as xs:string, $pre as xs:integer) as node() external;
declare %a:since("basex", "7.0") function db:open-id($db as xs:string, $id as xs:integer) as node() external;
declare %a:since("basex", "7.0") function db:node-pre($nodes as node()*) as xs:integer* external;
declare %a:since("basex", "7.0") function db:node-id($nodes as node()*) as xs:integer* external;
declare %a:since("basex", "7.0") function db:retrieve($db as xs:string, $path as xs:string) as xs:base64Binary external;
declare %a:since("basex", "7.0") function db:retrieve($db as xs:string, $path as xs:string) as xs:base64Binary external;
declare %a:since("basex", "7.0") function db:text($db as xs:string, $strings as xs:string*) as text()* external;
declare %a:since("basex", "7.2.1") function db:text-range($db as xs:string, $min as xs:string, $max as xs:string) as text()* external;
declare %a:since("basex", "7.0") function db:attribute($db as xs:string, $strings as xs:string*) as attribute()* external;
declare %a:since("basex", "7.0") function db:attribute($db as xs:string, $strings as xs:strings*, $name as xs:string) as attribute()* external;
declare %a:since("basex", "7.2.1") function db:attribute-range($db as xs:string, $min as xs:string, $max as xs:string) as attribute()* external;
declare %a:since("basex", "7.2.1") function db:attribute-range($db as xs:string, $min as xs:string, $max as xs:string, $name as xs:string) as attribute()* external;
declare %a:since("basex", "8.4") function db:token($db as xs:string, $tokens as xs:string*) as attribute()* external;
declare %a:since("basex", "8.4") function db:token($db as xs:string, $tokens as xs:string*, $name as xs:string) as attribute()* external;
declare %a:since("basex", "7.5") function db:create($db as xs:string) as empty-sequence() external;
declare %a:since("basex", "7.5") function db:create($db as xs:string, $inputs as item()*) as empty-sequence() external;
declare %a:since("basex", "7.5") function db:create($db as xs:string, $inputs as item()*, $paths as xs:string*) as empty-sequence() external;
declare %a:since("basex", "7.5") function db:create($db as xs:string, $inputs as item()*, $paths as xs:string*, $options as map(*)?) as empty-sequence() external;
declare %a:since("basex", "7.5") function db:drop($db as xs:string) as empty-sequence() external;
declare %a:since("basex", "7.0") function db:add($db as xs:string, $input as item()) as empty-sequence() external;
declare %a:since("basex", "7.0") function db:add($db as xs:string, $input as item(), $path as xs:string) as empty-sequence() external;
declare %a:since("basex", "7.0") function db:add($db as xs:string, $input as item(), $path as xs:string, $options as map(*)?) as empty-sequence() external;
declare %a:since("basex", "7.0") function db:delete($db as xs:string, $path as xs:string) as empty-sequence() external;
declare %a:since("basex", "7.8.2") function db:copy($db as xs:string, $name as xs:string) as empty-sequence() external;
declare %a:since("basex", "7.8.2") function db:alter($db as xs:string, $name as xs:string) as empty-sequence() external;
declare %a:since("basex", "7.8.2") function db:create-backup($db as xs:string) as empty-sequence() external;
declare %a:since("basex", "7.8.2") function db:drop-backup($name as xs:string) as empty-sequence() external;
declare %a:since("basex", "7.8.2") function db:restore($name as xs:string) as empty-sequence() external;
declare %a:since("basex", "7.0") function db:optimize($db as xs:string) as empty-sequence() external;
declare %a:since("basex", "7.0") function db:optimize($db as xs:string, $all as xs:boolean) as empty-sequence() external;
declare %a:since("basex", "7.0") function db:optimize($db as xs:string, $all as xs:boolean, $options as map(*)?) as empty-sequence() external;
declare %a:since("basex", "7.0") function db:rename($db as xs:string, $source as xs:string, $target as xs:string) as empty-sequence() external;
declare %a:since("basex", "7.0") function db:replace($db as xs:string, $path as xs:string, $input as item()) as empty-sequence() external;
declare %a:since("basex", "7.0") function db:replace($db as xs:string, $path as xs:string, $input as item(), $options as map(*)?) as empty-sequence() external;
declare %a:since("basex", "7.0") function db:store($db as xs:string, $path as xs:string, $input as item()) as empty-sequence() external;
declare %a:since("basex", "7.3") function db:flush($db as xs:string) as empty-sequence() external;
declare %a:since("basex", "7.7") function db:name($node as node()) as xs:string external;
declare %a:since("basex", "7.7") function db:path($node as node()) as xs:string external;
declare %a:since("basex", "7.0") function db:exists($db as xs:string) as xs:boolean external;
declare %a:since("basex", "7.0") function db:exists($db as xs:string, $path as xs:string) as xs:boolean external;
declare %a:since("basex", "7.0") function db:is-raw($db as xs:string, $path as xs:string) as xs:boolean external;
declare %a:since("basex", "7.0") function db:is-xml($db as xs:string, $path as xs:string) as xs:boolean external;
declare %a:since("basex", "7.1") function db:content-type($db as xs:string, $path as xs:string) as xs:string external;
declare %a:since("basex", "7.2.1") %a:until("basex", "9.0") %a:see-also("basex", "9.0", "update:output") function db:output() external;
declare %a:since("basex", "8.2") %a:until("basex", "9.0") %a:see-also("basex", "9.0", "update:cache") function db:output-cache() external;
declare %a:since("basex", "7.7") %a:until("basex", "..") function db:export() external;
declare %a:since("basex", "7.0") %a:until("basex", "7.8") %a:see-also("basex", "7.8", "ft:search") function db:fulltext() external;
declare %a:since("basex", "..") %a:until("basex", "8.2") function db:event() external;


[feature] Add a facility to collapse or drop text nodes which only contain whitespace

(:~
    KWIC module: formats query results to display keywords in context (KWIC). A configurable
    amount of text is displayed to the left and right of a matching keyword (or phrase).

    The module works with all indexes that support match highlighting (matches are tagged
    with an <exist:match> element or exist:matches attribute). This includes the old
    full text index, the new Lucene-based full text index, as well as the NGram index.

    The kwic:summarize() function represents the main entry point into the module.
    To have more control over the text extraction context, you can also call
    kwic:get-summary() instead. For example, the following snippet will only print the
    first match within a given set of context nodes ($ancestor):

    let $matches := kwic:get-matches($hit)

    for $ancestor in $matches/ancestor::para | $matches/ancestor::title | $matches/ancestor::td

    return

        kwic:get-summary($ancestor, ($ancestor//exist:match, $ancestor//*[@exist:matches])[1], $config)

    The configuration element used by some functions has the following structure:

    <config
        width="character width"
        table="yes|no"
        link="URL to which the match is linked"
        whitespace-text-nodes="collapse|drop|leave"/>
:)
xquery version "3.0";

module namespace kwic = "http://exist-db.org/xquery/kwic";

import module namespace util = "http://exist-db.org/xquery/util";

declare namespace exist = "http://exist.sourceforge.net/NS/exist";

declare variable $kwic:MODE_BEFORE := "before";
declare variable $kwic:MODE_AFTER := "after";

declare %private variable $kwic:CHARS_SUMMARY := 120;
declare %private variable $kwic:CHARS_KWIC := 40;

(:~
	Like fn:substring, but takes a node argument. If the node is an element,
	a new element is created with the same node-name as the old one and the
	shortened text content.
:)
declare
    %private
function kwic:substring($node as item(), $start as xs:integer, $count as xs:integer) as item()? {
	let $str := substring($node, $start, $count)
	return
		if ($node instance of element()) then
			element { node-name($node) } { $str }
		else
			text { $str }
};

declare
    %private
function kwic:display-text($text as text()?) as node()? {
    if ($text/parent::exist:match) then
    	{$text}
    else
        $text
};

declare
    %private
function kwic:callback($callback as (function(text(), xs:string) as text()?)?, $node as text(), $mode as xs:string) as text()? {
    if (exists($callback)) then
        $callback($node, $mode)
    else
        $node
};

(:~
	Generate the left-hand context of the match. Returns a sequence of nodes
	and strings, whose total string length is less than or equal to $max characters.

	Note: this function calls itself recursively until $nodes is empty or
	the returned sequence has the desired total string length.
:)
declare
    %private
function kwic:truncate-previous($root as node(), $node as node()?, $truncated as item()*,
	   $max as xs:integer, $chars as xs:integer, $callback as (function(text(), xs:string) as text()?)?) {
	if ($node) then
		let $next := $node/preceding::text()[1] intersect $root//text()
		return
			if (empty($next)) then
				$truncated
			else if ($chars + string-length($next) gt $max) then
			    let $str := kwic:callback($callback, $next, $kwic:MODE_BEFORE)
			    return
    			    if (exists($str)) then
        				let $remaining := $max - $chars
        				return (
        				    text { "..." },
        				    kwic:substring($str, string-length($str) - $remaining + 1, $remaining),
        				    $truncated
        			    )
        			else
        			    kwic:truncate-previous($root, $next, $truncated, $max, $chars, $callback)
			else
				kwic:truncate-previous($root, $next,
				    (kwic:callback($callback, $next, $kwic:MODE_BEFORE), $truncated),
					$max, $chars + string-length($next), $callback)
	else
		$truncated
};

(:~
	Generate the right-hand context of the match. Returns a sequence of nodes
	and strings, whose total string length is less than or equal to $max characters.

	Note: this function calls itself recursively until $nodes is empty or
	the returned sequence has the desired total string length.
:)
declare
    %private
function kwic:truncate-following($root as node(), $node as node()?, $truncated as item()*,
	$max as xs:integer, $chars as xs:integer, $callback as (function(text(), xs:string) as text()?)?) {
	if ($node) then
		let $next := $node/following::text()[1] intersect $root//text()
		return
			if (empty($next)) then
				$truncated
			else if ($chars + string-length($next) gt $max) then
			    let $str := kwic:callback($callback, $next, $kwic:MODE_AFTER)
			    return
    			    if (exists($str)) then
        				let $remaining := $max - $chars
        				return (
        				    $truncated, kwic:substring($next, 1, $remaining), text { "..." }
        			    )
        			else
        			    kwic:truncate-following($root, $next, $truncated, $max, $chars, $callback)
			else
				kwic:truncate-following($root, $next,
				    ($truncated, kwic:callback($callback, $next, $kwic:MODE_AFTER)),
					$max, $chars + string-length($next), $callback)
	else
		$truncated
};

(:~
	Computes the total string length of the nodes in the argument sequence
:)
declare
    %private
function kwic:string-length($nodes as item()*) as xs:integer {
	if (exists($nodes)) then
		sum($nodes ! string-length(.))
	else
		0
};

declare
    %private
function kwic:collapse-whitespace-fn($callback as (function(text(), xs:string) as text()?)?) as (function(text(), xs:string) as text()) {
    function($text as text(), $mode as xs:string) as text()? {
        let $text :=
            if(matches($text, "\s+")) then
                text { " " }
            else
                $text
        return
            if(exists($callback))then
                $callback($text, $mode)
            else
                $text
    }
};

declare
    %private
function kwic:drop-whitespace-fn($callback as (function(text(), xs:string) as text()?)?) as (function(text(), xs:string) as text()?) {
    function($text as text(), $mode as xs:string) as text()? {
        let $text :=
            if(matches($text, "\s+")) then
                ()
            else
                $text
        return
            if(exists($text) and exists($callback))then
                $callback($text, $mode)
            else
                $text
    }
};

declare function kwic:get-summary($root as node(), $node as element(),
	$config as element(config)?) as element() {
	kwic:get-summary($root, $node, $config, ())
};

(:~
	Print a summary of the match in $node. Output a predefined amount of text to
	the left and the right of the match.

	@param $root root element which should be used as context for the match. It defines the
	    boundaries for the text extraction. Text will be taken from this context.
	@param $node the exist:match element or the element with @exist:macthes attribute to process.
	@param $config configuration element which determines the behaviour of the function
	@param $callback (optional) reference to a callback function which will be called
	once for every text node before it is appended to the displayed text. The function
	should accept 2 parameters: 1) a single text node, 2) a string indicating the
	current direction in which text is appended, i.e. $kwic:MODE_BEFORE or $kwic:MODE_AFTER.
	The function may return the empty sequence if the current node should be ignore
	(e.g. if it belongs to a "footnote" which should not be displayed). Otherwise it should
	return a single string.
:)
declare function kwic:get-summary($root as node(), $node as element(),
	$config as element(config)?,
    $callback as (function(text(), xs:string) as text()?)?
) as element() {
	let $chars := xs:integer(($config/@width, $kwic:CHARS_KWIC)[1])
	let $table := $config/@table = ('yes', 'true')
	let $whitespace-text-nodes := string($config/@whitespace-text-nodes)

	let $callback :=
	   if($whitespace-text-nodes = "collapse") then
	       kwic:collapse-whitespace-fn($callback)
	   else if($whitespace-text-nodes = "drop") then
	       kwic:drop-whitespace-fn($callback)
	   else
	       $callback

	let $prevTrunc := kwic:truncate-previous($root, $node, (), $chars, 0, $callback)
	let $remain :=
		if (not($table)) then
			$chars * 2 - kwic:string-length($prevTrunc)
		else
			$chars
	let $followingTrunc := kwic:truncate-following($root, $node, (), $remain, 0, $callback)
	return
		if (not($table)) then

				{$prevTrunc}
				{
					if ($config/@link) then
						{ $node/text() }
					else
						{ $node/text() }
				}
				{$followingTrunc}

		else

				{$prevTrunc}

				{
					if ($config/@link) then
						{$node/text()}
					else
						$node/text()
				}

				{$followingTrunc}

};

(:~
    Expand the element in $hit. Creates an in-memory copy of the element and marks
    all element matches with an exist:match tag and attribute matches with an
    exist:matches tag, which will be used by all other functions in
    this module. You need to call kwic:expand before kwic:get-summary.
    kwic:summarize will call it automatically.
:)
declare function kwic:expand($hit as element()) as element() {
    util:expand($hit, "expand-xincludes=no highlight-matches=both")
};

(:~
    Return all matches within the specified element, $hit. Matches are returned as
    exist:match elements or elements with exist:matches attribute. The returned nodes
    are part of a new document whose root element is a copy of the specified $hit element.

    @param $hit an arbitrary XML element which has been selected by one of the full text
		operations or an ngram search.
:)
declare function kwic:get-matches($hit as element()) as element()* {
    let $expanded := kwic:expand($hit)
	return ($expanded//exist:match, $expanded//*[@exist:matches])
};

declare function kwic:summarize($hit as element(), $config as element(config)?) as element()* {
    kwic:summarize($hit, $config, ())
};

(:~
	Main function of the KWIC module: takes the passed element and returns an
	XHTML fragment containing a chunk of text before and after the first full text
	match in the node.

	The optional config parameter is used to configure the behaviour of the function:

	<config width="character width" table="yes|no" link="URL to which the match is linked"/>

	By default, kwic:summarize returns an XHTML fragment with the following structure:

	<p xmlns="http://www.w3.org/1999/xhtml">
		<span class="previous">Text before match</span>
		<a href="passed URL if any" class="hi">The highlighted term</a>
		<span class="following">Text after match</span>
	</p>

	If table=yes is passed with the config element, a tr table row will be returned instead
	of a span (using the same class names).

	@param $hit an arbitrary XML element which has been selected by one of the full text
		operations or an ngram search.
	@param $config configuration element to configure the behaviour of the function
:)
declare function kwic:summarize($hit as element(), $config as element(config)?,
    $callback as (function(text(), xs:string) as text()?)?) as element()* {
    let $expanded := util:expand($hit, "expand-xincludes=no highlight-matches=both")
	for $match in ($expanded//exist:match, $expanded//*[@exist:matches])
	return
		kwic:get-summary($expanded, $match, $config, $callback)
};

(:
declare function kwic:summarize-all($hits as element()*, $config as element(config),
    $wrap as function(node(), function(node()) as element()) as element()) {
    let $hits := util:expand($hits, "expand-xincludes=no")
    for $hit in $hits
    return
        $wrap($hit, function($node) {
            for $match in $node//exist:match
            return
        		kwic:get-summary($node, $match, $config, ())
        })
};
:)

(:~
    KWIC module: formats query results to display keywords in context (KWIC). A configurable
    amount of text is displayed to the left and right of a matching keyword (or phrase).

    The module works with all indexes that support match highlighting (matches are tagged
    with an <exist:match> element or exist:matches attribute). This includes the old
    full text index, the new Lucene-based full text index, as well as the NGram index.

    The kwic:summarize() function represents the main entry point into the module.
    To have more control over the text extraction context, you can also call
    kwic:get-summary() instead. For example, the following snippet will only print the
    first match within a given set of context nodes ($ancestor):

    let $matches := kwic:get-matches($hit)

    for $ancestor in $matches/ancestor::para | $matches/ancestor::title | $matches/ancestor::td

    return

        kwic:get-summary($ancestor, ($ancestor//exist:match, $ancestor//*[@exist:matches])[1], $config)

    The configuration element used by some functions has the following structure:

    <config
        width="character width"
        table="yes|no"
        link="URL to which the match is linked"/>
:)
xquery version "3.0";

module namespace kwic = "http://exist-db.org/xquery/kwic";

import module namespace util = "http://exist-db.org/xquery/util";

declare namespace exist = "http://exist.sourceforge.net/NS/exist";

declare variable $kwic:MODE_BEFORE := "before";
declare variable $kwic:MODE_AFTER := "after";

declare %private variable $kwic:CHARS_SUMMARY := 120;
declare %private variable $kwic:CHARS_KWIC := 40;

(:~
	Like fn:substring, but takes a node argument. If the node is an element,
	a new element is created with the same node-name as the old one and the
	shortened text content.
:)
declare
    %private
function kwic:substring($node as item(), $start as xs:integer, $count as xs:integer) as item()? {
	let $str := substring($node, $start, $count)
	return
		if ($node instance of element()) then
			element { node-name($node) } { $str }
		else
			text { $str }
};

declare
    %private
function kwic:display-text($text as text()?) as node()? {
    if ($text/parent::exist:match) then
    	{$text}
    else
        $text
};

declare
    %private
function kwic:callback($callback as (function(text(), xs:string) as text()?)?, $node as text(), $mode as xs:string) as text()? {
    if (exists($callback)) then
        $callback($node, $mode)
    else
        $node
};

(:~
	Generate the left-hand context of the match. Returns a sequence of nodes
	and strings, whose total string length is less than or equal to $max characters.

	Note: this function calls itself recursively until $nodes is empty or
	the returned sequence has the desired total string length.
:)
declare
    %private
function kwic:truncate-previous($root as node(), $node as node()?, $truncated as item()*,
	   $max as xs:integer, $chars as xs:integer, $callback as (function(text(), xs:string) as text()?)?) {
	if ($node) then
		let $next := $node/preceding::text()[1] intersect $root//text()
		return
			if (empty($next)) then
				$truncated
			else if ($chars + string-length($next) gt $max) then
			    let $str := kwic:callback($callback, $next, $kwic:MODE_BEFORE)
			    return
    			    if (exists($str)) then
        				let $remaining := $max - $chars
        				return (
        				    text { "..." },
        				    kwic:substring($str, string-length($str) - $remaining + 1, $remaining),
        				    $truncated
        			    )
        			else
        			    kwic:truncate-previous($root, $next, $truncated, $max, $chars, $callback)
			else
				kwic:truncate-previous($root, $next,
				    (kwic:callback($callback, $next, $kwic:MODE_BEFORE), $truncated),
					$max, $chars + string-length($next), $callback)
	else
		$truncated
};

(:~
	Generate the right-hand context of the match. Returns a sequence of nodes
	and strings, whose total string length is less than or equal to $max characters.

	Note: this function calls itself recursively until $nodes is empty or
	the returned sequence has the desired total string length.
:)
declare
    %private
function kwic:truncate-following($root as node(), $node as node()?, $truncated as item()*,
	$max as xs:integer, $chars as xs:integer, $callback as (function(text(), xs:string) as text()?)?) {
	if ($node) then
		let $next := $node/following::text()[1] intersect $root//text()
		return
			if (empty($next)) then
				$truncated
			else if ($chars + string-length($next) gt $max) then
			    let $str := kwic:callback($callback, $next, $kwic:MODE_AFTER)
			    return
    			    if (exists($str)) then
        				let $remaining := $max - $chars
        				return (
        				    $truncated, kwic:substring($next, 1, $remaining), text { "..." }
        			    )
        			else
        			    kwic:truncate-following($root, $next, $truncated, $max, $chars, $callback)
			else
				kwic:truncate-following($root, $next,
				    ($truncated, kwic:callback($callback, $next, $kwic:MODE_AFTER)),
					$max, $chars + string-length($next), $callback)
	else
		$truncated
};

(:~
	Computes the total string length of the nodes in the argument sequence
:)
declare
    %private
function kwic:string-length($nodes as item()*) as xs:integer {
	if (exists($nodes)) then
		sum($nodes ! string-length(.))
	else
		0
};

declare function kwic:get-summary($root as node(), $node as element(),
	$config as element(config)?) as element() {
	kwic:get-summary($root, $node, $config, ())
};

(:~
	Print a summary of the match in $node. Output a predefined amount of text to
	the left and the right of the match.

	@param $root root element which should be used as context for the match. It defines the
	    boundaries for the text extraction. Text will be taken from this context.
	@param $node the exist:match element or the element with @exist:macthes attribute to process.
	@param $config configuration element which determines the behaviour of the function
	@param $callback (optional) reference to a callback function which will be called
	once for every text node before it is appended to the displayed text. The function
	should accept 2 parameters: 1) a single text node, 2) a string indicating the
	current direction in which text is appended, i.e. $kwic:MODE_BEFORE or $kwic:MODE_AFTER.
	The function may return the empty sequence if the current node should be ignore
	(e.g. if it belongs to a "footnote" which should not be displayed). Otherwise it should
	return a single string.
:)
declare function kwic:get-summary($root as node(), $node as element(),
	$config as element(config)?,
    $callback as (function(text(), xs:string) as text()?)?
) as element() {
	let $chars := xs:integer(($config/@width, $kwic:CHARS_KWIC)[1])
	let $table := $config/@table = ('yes', 'true')
	let $prevTrunc := kwic:truncate-previous($root, $node, (), $chars, 0, $callback)
	let $remain :=
		if (not($table)) then
			$chars * 2 - kwic:string-length($prevTrunc)
		else
			$chars
	let $followingTrunc := kwic:truncate-following($root, $node, (), $remain, 0, $callback)
	return
		if (not($table)) then

				{$prevTrunc}
				{
					if ($config/@link) then
						{ $node/text() }
					else
						{ $node/text() }
				}
				{$followingTrunc}

		else

				{$prevTrunc}

				{
					if ($config/@link) then
						{$node/text()}
					else
						$node/text()
				}

				{$followingTrunc}

};

(:~
    Expand the element in $hit. Creates an in-memory copy of the element and marks
    all element matches with an exist:match tag and attribute matches with an
    exist:matches tag, which will be used by all other functions in
    this module. You need to call kwic:expand before kwic:get-summary.
    kwic:summarize will call it automatically.
:)
declare function kwic:expand($hit as element()) as element() {
    util:expand($hit, "expand-xincludes=no highlight-matches=both")
};

(:~
    Return all matches within the specified element, $hit. Matches are returned as
    exist:match elements or elements with exist:matches attribute. The returned nodes
    are part of a new document whose root element is a copy of the specified $hit element.

    @param $hit an arbitrary XML element which has been selected by one of the full text
		operations or an ngram search.
:)
declare function kwic:get-matches($hit as element()) as element()* {
    let $expanded := kwic:expand($hit)
	return ($expanded//exist:match, $expanded//*[@exist:matches])
};

declare function kwic:summarize($hit as element(), $config as element(config)?) as element()* {
    kwic:summarize($hit, $config, ())
};

(:~
	Main function of the KWIC module: takes the passed element and returns an
	XHTML fragment containing a chunk of text before and after the first full text
	match in the node.

	The optional config parameter is used to configure the behaviour of the function:

	<config width="character width" table="yes|no" link="URL to which the match is linked"/>

	By default, kwic:summarize returns an XHTML fragment with the following structure:

	<p xmlns="http://www.w3.org/1999/xhtml">
		<span class="previous">Text before match</span>
		<a href="passed URL if any" class="hi">The highlighted term</a>
		<span class="following">Text after match</span>
	</p>

	If table=yes is passed with the config element, a tr table row will be returned instead
	of a span (using the same class names).

	@param $hit an arbitrary XML element which has been selected by one of the full text
		operations or an ngram search.
	@param $config configuration element to configure the behaviour of the function
:)
declare function kwic:summarize($hit as element(), $config as element(config)?,
    $callback as (function(text(), xs:string) as text()?)?) as element()* {
    let $expanded := util:expand($hit, "expand-xincludes=no highlight-matches=both")
	for $match in ($expanded//exist:match, $expanded//*[@exist:matches])
	return
		kwic:get-summary($expanded, $match, $config, $callback)
};

(:
declare function kwic:summarize-all($hits as element()*, $config as element(config),
    $wrap as function(node(), function(node()) as element()) as element()) {
    let $hits := util:expand($hits, "expand-xincludes=no")
    for $hit in $hits
    return
        $wrap($hit, function($node) {
            for $match in $node//exist:match
            return
        		kwic:get-summary($node, $match, $config, ())
        })
};
:)