Added WordNet relationships.

Added WordNet relationships.

xquery version "3.0";

(:
 : Copyright 2006-2011 The FLWOR Foundation.
 :
 : Licensed under the Apache License, Version 2.0 (the "License");
 : you may not use this file except in compliance with the License.
 : You may obtain a copy of the License at
 :
 : http://www.apache.org/licenses/LICENSE-2.0
 :
 : Unless required by applicable law or agreed to in writing, software
 : distributed under the License is distributed on an "AS IS" BASIS,
 : WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 : See the License for the specific language governing permissions and
 : limitations under the License.
 :)

(:===========================================================================:)

(:~
 : This module provides an XQuery API to full-text functions.
 : For general information about Zorba's implementation of the
 : XQuery and XPath Full Text 1.0 specification
 : as well as instructions for building an installing a thesaurus,
 : see the Full Text Thesaurus documentation.
 :
Notes on languages

 : To refer to paricular human languages,
 : Zorba uses both the
 : ISO 639-1
 : and
 : ISO 639-2
 : languages codes.
 : Note that Zorba supports only a subset of the
 : complete list of language codes
 : and not every function supports the same subset.
 :

 : Most functions in this module take a language as a parameter
 : using the
 : xs:language
 : XML schema data type.
 :
Notes on stemming

 : The stem() functions return the
 : stem
 : of a word.
 : In Zorba,
 : the stem of a word itself, however, is not guaranteed to be a word.
 : It is best to consider a stem as an opaque byte sequence.
 : All that is guaranteed about a stem is that,
 : for a given word,
 : the stem of that word will always be the same byte sequence.
 : Hence,
 : you sould never compare the result of one of the stem()
 : functions against a non-stemmed string,
 : for example:
 :

 :  if ( ft:stem( "apples" ) eq "apple" )             ** WRONG **
 :

 : Instead do:
 :

 :  if ( ft:stem( "apples" ) eq ft:stem( "apple" ) )  ** CORRECT **
 :

 :
Notes on the thesaurus

 : The thesaurus-lookup() functions have "levels"
 : and "relationship" parameters.
 : The values for these are implementation-defined.
 : Zorba's default implementation uses the
 : WordNet lexical database,
 : version 3.0.
 :

 : In WordNet,
 : the number of "levels" that two phrases are apart
 : are how many hierarchical meanings apart they are.
 : For example,
 : "canary" is 5 levels away from "vertebrate"
 : (carary > finch > oscine > passerine > bird > vertebrate).
 :

 : When using the WordNet implementation,
 : Zorba supports all of the relationships (and their abbreviations)
 : specified by
 : ISO 2788
 : and
 : ANSI/NISO Z39.19-2005
 : with the exceptions of "HN" (history note)
 : and "X SN" (see scope note for).
 : These relationships are:
 :

 :

 :      Rel.
 :      Meaning
 :      WordNet Rel.
 :

 :

 :      BT
 :      broader term
 :      hypernym
 :

 :

 :      BTG
 :      broader term generic
 :      hypernym
 :

 :

 :      BTI
 :      broader term instance
 :      instance hypernym
 :

 :

 :      BTP
 :      broader term partitive
 :      part meronym
 :

 :

 :      NT
 :      narrower term
 :      hyponym
 :

 :

 :      NTG
 :      narrower term generic
 :      hyponym
 :

 :

 :      NTI
 :      narrower term instance
 :      instance hyponym
 :

 :

 :      NTP
 :      narrower term partitive
 :      part holonym
 :

 :

 :      RT
 :      related term
 :      also see
 :

 :

 :      SN
 :      scope note
 :      n/a
 :

 :

 :      TT
 :      top term
 :      hypernym
 :

 :

 :      UF
 :      non-preferred term
 :      n/a
 :

 :

 :      USE
 :      preferred term
 :      n/a
 :

 :

 : Note that you can specify relationships
 : either by their abbreviation
 : or their meaning.
 : Relationships are case-insensitive.
 :
 : In addition to the
 : ISO 2788
 : and
 : ANSI/NISO Z39.19-2005
 : relationships,
 : Zorba also supports all of the relationships offered by WordNet.
 : These relationships are:
 :

 :

 :      Relationship
 :      Meaning
 :

 :

 :      also see
 :
 :        A word that is related to another,
 :        e.g., for "varnished" (furniture)
 :        one should also see "finished."
 :
 :

 :

 :      antonym
 :
 :        A word opposite in meaning to another,
 :        e.g., "light" is an antonym for "heavy."
 :
 :

 :

 :      attribute
 :
 :        A noun for which adjectives express values,
 :        e.g., "weight" is an attribute
 :        for which the adjectives "light" and "heavy"
 :        express values.
 :
 :

 :

 :      cause
 :
 :        A verb that causes another,
 :        e.g., "show" is a cause of "see."
 :
 :

 :

 :      derivationally related form
 :
 :        A word that is derived from a root word,
 :        e.g., "metric" is a derivationally related form of "meter."
 :
 :

 :

 :      derived from adjective
 :
 :        An adverb that is derived from an adjective,
 :        e.g., "correctly" is derived from the adjective "correct."
 :
 :

 :

 :      entailment
 :
 :        A verb that presupposes another,
 :        e.g., "snoring" entails "sleeping."
 :
 :

 :

 :      hypernym
 :
 :        A word with a broad meaning that more specific words fall under,
 :        e.g., "meal" is a hypernym of "breakfast."
 :
 :

 :

 :      hyponym
 :
 :        A word of more specific meaning than a general term applicable to it,
 :        e.g., "breakfast" is a hyponym of "meal."
 :
 :

 :

 :      instance hypernym
 :
 :        A word that denotes a category of some specific instance,
 :        e.g., "author" is an instance hypernym of "Asimov."
 :
 :

 :

 :      instance hyponym
 :
 :        A term that donotes a specific instance of some general category,
 :        e.g., "Asimov" is an instance hyponym of "author."
 :
 :

 :

 :      member holonym
 :
 :        A word that denotes a collection of individuals,
 :        e.g., "faculty" is a member holonym of "professor."
 :
 :

 :

 :      member meronym
 :
 :        A word that denotes a member of a larger group,
 :        e.g., a "person" is a member meronym of a "crowd."
 :
 :

 :

 :      part holonym
 :
 :        A word that denotes a larger whole comprised of some part,
 :        e.g., "car" is a part holonym of "engine."
 :
 :

 :

 :      part meronym
 :
 :        A word that denotes a part of a larger whole,
 :        e.g., an "engine" is part meronym of a "car."
 :
 :

 :

 :      participle of verb
 :
 :        An adjective that is the participle of some verb,
 :        e.g., "breaking" is the participle of the verb "break."
 :
 :

 :

 :      pertainym
 :
 :        An adjective that classifies its noun,
 :        e.g., "musical" is a pertainym in "musical instrument."
 :
 :

 :

 :      similar to
 :
 :        Similar, though not necessarily interchangeable, adjectives.
 :        For example, "shiny" is similar to "bright",
 :        but they have subtle differences.
 :
 :

 :

 :      substance holonym
 :
 :        A word that denotes a larger whole containing some constituent
 :        substance, e.g., "bread" is a substance holonym of "flour."
 :
 :

 :

 :      substance meronym
 :
 :        A word that denotes a constituant substance of some larger whole,
 :        e.g., "flour" is a substance meronym of "bread."
 :
 :

 :

 :      verb group
 :
 :        A verb that is a member of a group of similar verbs,
 :        e.g., "live" is in the verb group
 :        of "dwell", "live", "inhabit", etc.
 :
 :

 :

 :
Notes on tokenization

 : For general information about Zorba's implementation of tokenization,
 : including what constitutes a token,
 : see the Full Text Tokenizer documentation.
 :)

(:===========================================================================:)

module namespace ft = "http://www.zorba-xquery.com/modules/full-text";

import schema namespace ft-schema =
  "http://www.zorba-xquery.com/modules/full-text";

declare namespace err = "http://www.w3.org/2005/xqt-errors";
declare namespace zerr = "http://www.zorba-xquery.com/errors";

declare namespace ver = "http://www.zorba-xquery.com/options/versioning";
declare option ver:module-version "2.0";

(:===========================================================================:)

(:~
 : Predeclared constant for the Danish
 : xs:language.
 :)
declare variable $ft:lang-da as xs:language := xs:language("da");

(:~
 : Predeclared constant for the German
 : xs:language.
 :)
declare variable $ft:lang-de as xs:language := xs:language("de");

(:~
 : Predeclared constant for the English
 : xs:language.
 :)
declare variable $ft:lang-en as xs:language := xs:language("en");

(:~
 : Predeclared constant for the Spanish
 : xs:language.
 :)
declare variable $ft:lang-es as xs:language := xs:language("es");

(:~
 : Predeclared constant for the Finnish
 : xs:language.
 :)
declare variable $ft:lang-fi as xs:language := xs:language("fi");

(:~
 : Predeclared constant for the French
 : xs:language.
 :)
declare variable $ft:lang-fr as xs:language := xs:language("fr");

(:~
 : Predeclared constant for the Hungarian
 : xs:language.
 :)
declare variable $ft:lang-hu as xs:language := xs:language("hu");

(:~
 : Predeclared constant for the Italian
 : xs:language.
 :)
declare variable $ft:lang-it as xs:language := xs:language("it");

(:~
 : Predeclared constant for the Dutch
 : xs:language.
 :)
declare variable $ft:lang-nl as xs:language := xs:language("nl");

(:~
 : Predeclared constant for the Norwegian
 : xs:language.
 :)
declare variable $ft:lang-no as xs:language := xs:language("no");

(:~
 : Predeclared constant for the Portuguese
 : xs:language.
 :)
declare variable $ft:lang-pt as xs:language := xs:language("pt");

(:~
 : Predeclared constant for the Romanian
 : xs:language.
 :)
declare variable $ft:lang-ro as xs:language := xs:language("ro");

(:~
 : Predeclared constant for the Russian
 : xs:language.
 :)
declare variable $ft:lang-ru as xs:language := xs:language("ru");

(:~
 : Predeclared constant for the Swedish
 : xs:language.
 :)
declare variable $ft:lang-sv as xs:language := xs:language("sv");

(:~
 : Predeclared constant for the Turkish
 : xs:language.
 :)
declare variable $ft:lang-tr as xs:language := xs:language("tr");

(:===========================================================================:)

(:~
 : Gets the current
 : language:
 : either the langauge specified by the
 : declare ft-option using
 : language
 : statement (if any)
 : or the one returned by ft:host-lang() (if none).
 :
 : @return said language.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-current-lang-true-1.xq
 :)
declare function ft:current-lang()
  as xs:language external;

(:~
 : Gets the host's current
 : language.
 : The "host" is the computer on which Zorba is running.
 : The host's current language is obtained as follows:
 :

 :

 :      For *nix systems:
 :

 :

 :          If setlocale(3) returns non-null,
 :          the language corresponding to that locale is used.
 :

 :

 :          Else, if the LANG environment variable is set,
 :          that language is ued.
 :

 :

 :          Otherwise, there is no default language.
 :

 :

 :

 :

 :      For Windows systems,
 :      the language corresponding to the locale returned by the
 :      GetLocaleInfo()
 :      function is used.
 :

 :

 :
 : @return said language.
 :)
declare function ft:host-lang()
  as xs:language external;

(:~
 : Checks whether the given
 : language
 : is supported for stemming.
 :
 : @param $lang The language to check.
 : @return true only if the language is supported.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stem-lang-es-supported-true.xq
 :)
declare function ft:is-stem-lang-supported( $lang as xs:language )
  as xs:boolean external;

(:~
 : Checks whether the given word is a stop-word.
 :
 : @param $word The word to check.
 : @param $lang The
 : language
 : of $word.
 : @return true only if $word is a stop-word.
 : @error zerr:ZXQP8405 if $lang is not supported for stop-words.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-true-1.xq
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-true-3.xq
 :)
declare function ft:is-stop-word( $word as xs:string, $lang as xs:language )
  as xs:boolean external;

(:~
 : Checks whether the given word is a stop-word.
 :
 : @param $word The word to check.
 : The word's language
 : is assumed to be the one returned by ft:current-lang().
 : @return true only if $word is a stop-word.
 : @error err:FTST0009 if ft:current-lang() is not supported in
 : general.
 : @error zerr:ZXQP8405 if ft:current-lang() is not supported for
 : stop-words specifically.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-true-2.xq
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-true-4.xq
 :)
declare function ft:is-stop-word( $word as xs:string )
  as xs:boolean external;

(:~
 : Checks whether the given
 : language
 : is supported for stop words.
 :
 : @param $lang The language to check.
 : @return true only if the language is supported.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-lang-en-supported-true.xq
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-lang-supported-false-1.xq
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-lang-supported-false-2.xq
 :)
declare function ft:is-stop-word-lang-supported( $lang as xs:language )
  as xs:boolean external;

(:~
 : Checks whether the given
 : language
 : is supported for look-up using the default thesaurus.
 :
 : @param $lang The language to check.
 : @return true only if the language is supported.
 :)
declare function ft:is-thesaurus-lang-supported( $lang as xs:language )
  as xs:boolean external;

(:~
 : Checks whether the given
 : language
 : is supported for look-up using the thesaurus specified by the given URI.
 :
 : @param $uri The URI specifying the thesaurus to use.
 : @param $lang The language to check.
 : @return true only if the language is supported.
 : @error err:FTST0018 if $uri refers to a thesaurus
 : that is not found in the statically known thesauri.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-thesaurus-lang-supported-true-1.xq
 :)
declare function ft:is-thesaurus-lang-supported( $uri as xs:string,
                                                 $lang as xs:language )
  as xs:boolean external;

(:~
 : Stems the given word.
 :
 : @param $word The word to stem.
 : @param $lang The
 : language
 : of $word.
 : @return the stem of $word.
 : @error err:FTST0009 if $lang is not supported in general.
 : @error zerr:ZXQP8404 if $lang is not supported for stemming
 : specifically.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-stem-1.xq
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-stem-2.xq
 :)
declare function ft:stem( $word as xs:string, $lang as xs:language )
  as xs:string external;

(:~
 : Stems the given word.
 :
 : @param $word The word to stem.
 : The word's language
 : is assumed to be the one returned by ft:current-lang().
 : @return the stem of $word.
 : @error err:FTST0009 if ft:current-lang() is not supported in
 : general.
 : @error zerr:ZXQP8404 if ft:current-lang() is not supported for
 : stemming specifically.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-stem-3.xq
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-stem-4.xq
 :)
declare function ft:stem( $word as xs:string )
  as xs:string external;

(:~
 : Strips all diacritical marks from all characters.
 :
 : @param $string The string to strip diacritical marks from.
 : @return $string with diacritical marks stripped.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-strip-diacritics-1.xq
 :)
declare function ft:strip-diacritics( $string as xs:string )
  as xs:string external;

(:~
 : Looks-up the given phrase in the default thesaurus.
 :
 : @param $phrase The phrase to look up.
 : The phrase's
 : language
 : is assumed to be the one returned by ft:current-lang().
 : @return the original and related phrases.
 : @error err:FTST0009 if ft:current-lang() is not supported in
 : general.
 : @error zerr:ZXQP8401 if the thesaurus data file's version is not supported
 : by the currently running version of Zorba.
 : @error zerr:ZXQP8402 if the thesaurus data file's endianness does not match
 : that of the CPU on which Zorba is currently running.
 : @error zerr:ZXQP8403 if there was an error reading the thesaurus data.
 : @error zerr:ZXQP8405 if ft:current-lang() is not supported for
 : thesaurus look-up specifically.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-thesaurus-lookup-1.xq
 :)
declare function ft:thesaurus-lookup( $phrase as xs:string )
  as xs:string+ external;

(:~
 : Looks-up the given phrase in the thesaurus specified by the given URI.
 :
 : @param $uri The URI specifying the thesaurus to use.
 : @param $phrase The phrase to look up.
 : @param $lang The
 : language
 : of $phrase.
 : @return the original and related phrases.
 : @error err:FTST0009 if $lang is not supported in general.
 : @error err:FTST0018 if $uri refers to a thesaurus
 : that is not found in the statically known thesauri.
 : @error zerr:ZOSE0001 if the thesaurus data file could not be found.
 : @error zerr:ZOSE0002 if the thesaurus data file is not a plain file.
 : @error zerr:ZXQP8401 if the thesaurus data file's version is not supported
 : by the currently running version of Zorba.
 : @error zerr:ZXQP8402 if the thesaurus data file's endianness does not match
 : that of the CPU on which Zorba is currently running.
 : @error zerr:ZXQP8403 if there was an error reading the thesaurus data file.
 : @error zerr:ZXQP8405 if $lang is not supported for thesaurus
 : look-up specifically.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-thesaurus-lookup-2.xq
 :)
declare function ft:thesaurus-lookup( $uri as xs:string, $phrase as xs:string,
                                      $lang as xs:language )
  as xs:string+ external;

(:~
 : Looks-up the given phrase in a thesaurus.
 :
 : @param $uri The URI specifying the thesaurus to use.
 : @param $phrase The phrase to look up.
 : The phrase's
 : language
 : is assumed to be the one the one returned by ft:current-lang().
 : @return the original and related phrases.
 : @error err:FTST0009 if ft:current-lang() is unsupported in
 : general.
 : @error err:FTST0018 if $uri refers to a thesaurus
 : that is not found in the statically known thesauri.
 : @error zerr:ZOSE0001 if the thesaurus data file could not be found.
 : @error zerr:ZOSE0002 if the thesaurus data file is not a plain file.
 : @error zerr:ZXQP8401 if the thesaurus data file's version is not supported
 : by the currently running version of Zorba.
 : @error zerr:ZXQP8402 if the thesaurus data file's endianness does not match
 : that of the CPU on which Zorba is currently running.
 : @error zerr:ZXQP8403 if there was an error reading the thesaurus data file.
 : @error zerr:ZXQP8405 if ft:current-lang() is not supported for
 : thesaurus look-up specifically.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-thesaurus-lookup-3.xq
 :)
declare function ft:thesaurus-lookup( $uri as xs:string, $phrase as xs:string )
  as xs:string+ external;

(:~
 : Looks-up the given phrase in a thesaurus.
 :
 : @param $uri The URI specifying the thesaurus to use.
 : @param $phrase The phrase to look up.
 : @param $lang The
 : language
 : of $phrase.
 : @param $relationship The relationship the results are to have to
 : $phrase.
 : @return the original and related phrases.
 : @error err:FTST0018 if $uri refers to a thesaurus
 : that is not found in the statically known thesauri.
 : @error err:FTST0009 if $lang is not supported in general.
 : @error zerr:ZOSE0001 if the thesaurus data file could not be found.
 : @error zerr:ZOSE0002 if the thesaurus data file is not a plain file.
 : @error zerr:ZXQP8401 if the thesaurus data file's version is not supported
 : by the currently running version of Zorba.
 : @error zerr:ZXQP8402 if the thesaurus data file's endianness does not match
 : that of the CPU on which Zorba is currently running.
 : @error zerr:ZXQP8403 if there was an error reading the thesaurus data file.
 : @error zerr:ZXQP8405 if $lang is not supported for thesaurus
 : look-up specifically.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-thesaurus-lookup-4.xq
 :)
declare function ft:thesaurus-lookup( $uri as xs:string, $phrase as xs:string,
                                      $lang as xs:language,
                                      $relationship as xs:string )
  as xs:string+ external;

(:~
 : Looks-up the given phrase in a thesaurus.
 :
 : @param $uri The URI specifying the thesaurus to use.
 : @param $phrase The phrase to look up.
 : @param $lang The
 : language
 : of $phrase.
 : @param $relationship The relationship the results are to have to
 : $phrase.
 : @param $level-least The minimum number of levels within the thesaurus to be
 : travers$ed.
 : @param $level-most The maximum number of levels within the thesaurus to be
 : traversed.
 : @return the original and related phrases.
 : @error err:FOCA0003 if either $level-least or
 : $level-most is either negative or too large.
 : @error err:FTST0018 if $uri refers to a thesaurus
 : that is not found in the statically known thesauri.
 : @error err:FTST0009 if $lang is not supported in general.
 : @error zerr:ZOSE0001 if the thesaurus data file could not be found.
 : @error zerr:ZOSE0002 if the thesaurus data file is not a plain file.
 : @error zerr:ZXQP8401 if the thesaurus data file's version is not supported
 : by the currently running version of Zorba.
 : @error zerr:ZXQP8402 if the thesaurus data file's endianness does not match
 : that of the CPU on which Zorba is currently running.
 : @error zerr:ZXQP8403 if there was an error reading the thesaurus data file.
 : @error zerr:ZXQP8405 if $lang is not supported for thesaurus
 : look-up specifically.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-thesaurus-lookup-5.xq
 :)
declare function ft:thesaurus-lookup( $uri as xs:string, $phrase as xs:string,
                                      $lang as xs:language,
                                      $relationship as xs:string,
                                      $level-least as xs:integer,
                                      $level-most as xs:integer )
  as xs:string+ external;

(:~
 : Tokenizes the given document.
 :
 : @param $node The node to tokenize.
 : @param $lang The default
 : language
 : of $node.
 : @return a (possibly empty) sequence of tokens.
 : @error err:FTST0009 if $lang is not supported in general.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-1.xq
 :)
declare function ft:tokenize( $node as node(), $lang as xs:language )
  as element(ft-schema:token)* external;

(:~
 : Tokenizes the given document.
 :
 : @param $node The node to tokenize.
 : The document's default
 : language
 : is assumed to be the one returned by ft:current-lang().
 : @return a (possibly empty) sequence of tokens.
 : @error err:FTST0009 if ft:current-lang() is not supported in
 : general.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-2.xq
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-3.xq
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-4.xq
 :)
declare function ft:tokenize( $node as node() )
  as element(ft-schema:token)* external;

(:~
 : Tokenizes the given string.
 :
 : @param $string The string to tokenize.
 : @param $lang The default
 : language
 : of $string.
 : @return a (possibly empty) sequence of tokens.
 : @error err:FTST0009 if $lang is not supported in general.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-string-1.xq
 :)
declare function ft:tokenize-string( $string as xs:string,
                                     $lang as xs:language )
  as xs:string* external;

(:~
 : Tokenizes the given string.
 :
 : @param $string The string to tokenize.
 : The string's default
 : language
 : is assumed to be the one returned by ft:current-lang().
 : @return a (possibly empty) sequence of tokens.
 : @error err:FTST0009 if ft:current-lang() is not supported in
 : general.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-string-2.xq
 :)
declare function ft:tokenize-string( $string as xs:string )
  as xs:string* external;

(:~
 : Gets properties of the tokenizer for the given
 : language.
 :
 : @param $lang The langauage of the tokenizer to get the properties of.
 : @return said properties.
 : @error err:FTST0009 if $lang is not supported in general.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenizer-properties-1.xq
 :)
declare function ft:tokenizer-properties( $lang as xs:language )
  as element(ft-schema:tokenizer-properties) external;

(:~
 : Gets properties of the tokenizer for the
 : language
 : returned by ft:current-lang().
 :
 : @return said properties.
 : @error err:FTST0009 if ft:current-lang() is not supported in
 : general.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenizer-properties-2.xq
 :)
declare function ft:tokenizer-properties()
  as element(ft-schema:tokenizer-properties) external;

(:===========================================================================:)

(: vim:set et sw=2 ts=2: :)

xquery version "3.0";

(:
 : Copyright 2006-2011 The FLWOR Foundation.
 :
 : Licensed under the Apache License, Version 2.0 (the "License");
 : you may not use this file except in compliance with the License.
 : You may obtain a copy of the License at
 :
 : http://www.apache.org/licenses/LICENSE-2.0
 :
 : Unless required by applicable law or agreed to in writing, software
 : distributed under the License is distributed on an "AS IS" BASIS,
 : WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 : See the License for the specific language governing permissions and
 : limitations under the License.
 :)

(:===========================================================================:)

(:~
 : This module provides an XQuery API to full-text functions.
 : For general information about Zorba's implementation of the
 : XQuery and XPath Full Text 1.0 specification
 : as well as instructions for building an installing a thesaurus,
 : see the Full Text Thesaurus documentation.
 :
Notes on languages

 : To refer to paricular human languages,
 : Zorba uses both the
 : ISO 639-1
 : and
 : ISO 639-2
 : languages codes.
 : Note that Zorba supports only a subset of the
 : complete list of language codes
 : and not every function supports the same subset.
 :

 : Most functions in this module take a language as a parameter
 : using the
 : xs:language
 : XML schema data type.
 :
Notes on stemming

 : The stem() functions return the
 : stem
 : of a word.
 : In Zorba,
 : the stem of a word itself, however, is not guaranteed to be a word.
 : It is best to consider a stem as an opaque byte sequence.
 : All that is guaranteed about a stem is that,
 : for a given word,
 : the stem of that word will always be the same byte sequence.
 : Hence,
 : you sould never compare the result of one of the stem()
 : functions against a non-stemmed string,
 : for example:
 :

 :  if ( ft:stem( "apples" ) eq "apple" )             ** WRONG **
 :

 : Instead do:
 :

 :  if ( ft:stem( "apples" ) eq ft:stem( "apple" ) )  ** CORRECT **
 :

 :
Notes on the thesaurus

 : The thesaurus-lookup() functions have "levels"
 : and "relationship" parameters.
 : The values for these are implementation-defined.
 : Zorba's default implementation uses the
 : WordNet lexical database,
 : version 3.0.
 :

 : In WordNet,
 : the number of "levels" that two phrases are apart
 : are how many hierarchical meanings apart they are.
 : For example,
 : "canary" is 5 levels away from "vertebrate"
 : (carary > finch > oscine > passerine > bird > vertebrate).
 :

 : When using the WordNet implementation,
 : Zorba supports all of the relationships (and their abbreviations)
 : specified by
 : ISO 2788
 : and
 : ANSI/NISO Z39.19-2005
 : with the exceptions of "HN" (history note)
 : and "X SN" (see scope note for).
 : Additionally, Zorba also supports all WordNet relationships.
 :

 :

 :     Rel.
 :     Meaning
 :     WordNet Rel.
 :

 :

 :     BT
 :     broader term
 :     hypernym
 :

 :

 :     BTG
 :     broader term generic
 :     hypernym
 :

 :

 :     BTI
 :     broader term instance
 :     instance hypernym
 :

 :

 :     BTP
 :     broader term partitive
 :     part meronym
 :

 :

 :     NT
 :     narrower term
 :     hyponym
 :

 :

 :     NTG
 :     narrower term generic
 :     hyponym
 :

 :

 :     NTI
 :     narrower term instance
 :     instance hyponym
 :

 :

 :     NTP
 :     narrower term partitive
 :     part holonym
 :

 :

 :     RT
 :     related term
 :     also see
 :

 :

 :     SN
 :     scope note
 :     n/a
 :

 :

 :     TT
 :     top term
 :     hypernym
 :

 :

 :     UF
 :     non-preferred term
 :     n/a
 :

 :

 :     USE
 :     preferred term
 :     n/a
 :

 :

 : Relationships are case-insensitive.
 :
Notes on tokenization

 : For general information about Zorba's implementation of tokenization,
 : including what constitutes a token,
 : see the Full Text Tokenizer documentation.
 :)

(:===========================================================================:)

module namespace ft = "http://www.zorba-xquery.com/modules/full-text";

import schema namespace ft-schema =
  "http://www.zorba-xquery.com/modules/full-text";

declare namespace err = "http://www.w3.org/2005/xqt-errors";
declare namespace zerr = "http://www.zorba-xquery.com/errors";

declare namespace ver = "http://www.zorba-xquery.com/options/versioning";
declare option ver:module-version "2.0";

(:===========================================================================:)

(:~
 : Predeclared constant for the Danish
 : xs:language.
 :)
declare variable $ft:lang-da as xs:language := xs:language("da");

(:~
 : Predeclared constant for the German
 : xs:language.
 :)
declare variable $ft:lang-de as xs:language := xs:language("de");

(:~
 : Predeclared constant for the English
 : xs:language.
 :)
declare variable $ft:lang-en as xs:language := xs:language("en");

(:~
 : Predeclared constant for the Spanish
 : xs:language.
 :)
declare variable $ft:lang-es as xs:language := xs:language("es");

(:~
 : Predeclared constant for the Finnish
 : xs:language.
 :)
declare variable $ft:lang-fi as xs:language := xs:language("fi");

(:~
 : Predeclared constant for the French
 : xs:language.
 :)
declare variable $ft:lang-fr as xs:language := xs:language("fr");

(:~
 : Predeclared constant for the Hungarian
 : xs:language.
 :)
declare variable $ft:lang-hu as xs:language := xs:language("hu");

(:~
 : Predeclared constant for the Italian
 : xs:language.
 :)
declare variable $ft:lang-it as xs:language := xs:language("it");

(:~
 : Predeclared constant for the Dutch
 : xs:language.
 :)
declare variable $ft:lang-nl as xs:language := xs:language("nl");

(:~
 : Predeclared constant for the Norwegian
 : xs:language.
 :)
declare variable $ft:lang-no as xs:language := xs:language("no");

(:~
 : Predeclared constant for the Portuguese
 : xs:language.
 :)
declare variable $ft:lang-pt as xs:language := xs:language("pt");

(:~
 : Predeclared constant for the Romanian
 : xs:language.
 :)
declare variable $ft:lang-ro as xs:language := xs:language("ro");

(:~
 : Predeclared constant for the Russian
 : xs:language.
 :)
declare variable $ft:lang-ru as xs:language := xs:language("ru");

(:~
 : Predeclared constant for the Swedish
 : xs:language.
 :)
declare variable $ft:lang-sv as xs:language := xs:language("sv");

(:~
 : Predeclared constant for the Turkish
 : xs:language.
 :)
declare variable $ft:lang-tr as xs:language := xs:language("tr");

(:===========================================================================:)

(:~
 : Gets the current
 : language:
 : either the langauge specified by the
 : declare ft-option using
 : language
 : statement (if any)
 : or the one returned by ft:host-lang() (if none).
 :
 : @return said language.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-current-lang-true-1.xq
 :)
declare function ft:current-lang()
  as xs:language external;

(:~
 : Gets the host's current
 : language.
 : The "host" is the computer on which Zorba is running.
 : The host's current language is obtained as follows:
 :

 :

 :      For *nix systems:
 :

 :

 :          If setlocale(3) returns non-null,
 :          the language corresponding to that locale is used.
 :

 :

 :          Else, if the LANG environment variable is set,
 :          that language is ued.
 :

 :

 :          Otherwise, there is no default language.
 :

 :

 :

 :

 :      For Windows systems,
 :      the language corresponding to the locale returned by the
 :      GetLocaleInfo()
 :      function is used.
 :

 :

 :
 : @return said language.
 :)
declare function ft:host-lang()
  as xs:language external;

(:~
 : Checks whether the given
 : language
 : is supported for stemming.
 :
 : @param $lang The language to check.
 : @return true only if the language is supported.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stem-lang-es-supported-true.xq
 :)
declare function ft:is-stem-lang-supported( $lang as xs:language )
  as xs:boolean external;

(:~
 : Checks whether the given word is a stop-word.
 :
 : @param $word The word to check.
 : @param $lang The
 : language
 : of $word.
 : @return true only if $word is a stop-word.
 : @error zerr:ZXQP8405 if $lang is not supported for stop-words.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-true-1.xq
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-true-3.xq
 :)
declare function ft:is-stop-word( $word as xs:string, $lang as xs:language )
  as xs:boolean external;

(:~
 : Checks whether the given word is a stop-word.
 :
 : @param $word The word to check.
 : The word's language
 : is assumed to be the one returned by ft:current-lang().
 : @return true only if $word is a stop-word.
 : @error err:FTST0009 if ft:current-lang() is not supported in
 : general.
 : @error zerr:ZXQP8405 if ft:current-lang() is not supported for
 : stop-words specifically.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-true-2.xq
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-true-4.xq
 :)
declare function ft:is-stop-word( $word as xs:string )
  as xs:boolean external;

(:~
 : Checks whether the given
 : language
 : is supported for stop words.
 :
 : @param $lang The language to check.
 : @return true only if the language is supported.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-lang-en-supported-true.xq
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-lang-supported-false-1.xq
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-lang-supported-false-2.xq
 :)
declare function ft:is-stop-word-lang-supported( $lang as xs:language )
  as xs:boolean external;

(:~
 : Checks whether the given
 : language
 : is supported for look-up using the default thesaurus.
 :
 : @param $lang The language to check.
 : @return true only if the language is supported.
 :)
declare function ft:is-thesaurus-lang-supported( $lang as xs:language )
  as xs:boolean external;

(:~
 : Checks whether the given
 : language
 : is supported for look-up using the thesaurus specified by the given URI.
 :
 : @param $uri The URI specifying the thesaurus to use.
 : @param $lang The language to check.
 : @return true only if the language is supported.
 : @error err:FTST0018 if $uri refers to a thesaurus
 : that is not found in the statically known thesauri.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-thesaurus-lang-supported-true-1.xq
 :)
declare function ft:is-thesaurus-lang-supported( $uri as xs:string,
                                                 $lang as xs:language )
  as xs:boolean external;

(:~
 : Stems the given word.
 :
 : @param $word The word to stem.
 : @param $lang The
 : language
 : of $word.
 : @return the stem of $word.
 : @error err:FTST0009 if $lang is not supported in general.
 : @error zerr:ZXQP8404 if $lang is not supported for stemming
 : specifically.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-stem-1.xq
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-stem-2.xq
 :)
declare function ft:stem( $word as xs:string, $lang as xs:language )
  as xs:string external;

(:~
 : Stems the given word.
 :
 : @param $word The word to stem.
 : The word's language
 : is assumed to be the one returned by ft:current-lang().
 : @return the stem of $word.
 : @error err:FTST0009 if ft:current-lang() is not supported in
 : general.
 : @error zerr:ZXQP8404 if ft:current-lang() is not supported for
 : stemming specifically.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-stem-3.xq
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-stem-4.xq
 :)
declare function ft:stem( $word as xs:string )
  as xs:string external;

(:~
 : Strips all diacritical marks from all characters.
 :
 : @param $string The string to strip diacritical marks from.
 : @return $string with diacritical marks stripped.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-strip-diacritics-1.xq
 :)
declare function ft:strip-diacritics( $string as xs:string )
  as xs:string external;

(:~
 : Looks-up the given phrase in the default thesaurus.
 :
 : @param $phrase The phrase to look up.
 : The phrase's
 : language
 : is assumed to be the one returned by ft:current-lang().
 : @return the original and related phrases.
 : @error err:FTST0009 if ft:current-lang() is not supported in
 : general.
 : @error zerr:ZXQP8401 if the thesaurus data file's version is not supported
 : by the currently running version of Zorba.
 : @error zerr:ZXQP8402 if the thesaurus data file's endianness does not match
 : that of the CPU on which Zorba is currently running.
 : @error zerr:ZXQP8403 if there was an error reading the thesaurus data.
 : @error zerr:ZXQP8405 if ft:current-lang() is not supported for
 : thesaurus look-up specifically.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-thesaurus-lookup-1.xq
 :)
declare function ft:thesaurus-lookup( $phrase as xs:string )
  as xs:string+ external;

(:~
 : Looks-up the given phrase in the thesaurus specified by the given URI.
 :
 : @param $uri The URI specifying the thesaurus to use.
 : @param $phrase The phrase to look up.
 : @param $lang The
 : language
 : of $phrase.
 : @return the original and related phrases.
 : @error err:FTST0009 if $lang is not supported in general.
 : @error err:FTST0018 if $uri refers to a thesaurus
 : that is not found in the statically known thesauri.
 : @error zerr:ZOSE0001 if the thesaurus data file could not be found.
 : @error zerr:ZOSE0002 if the thesaurus data file is not a plain file.
 : @error zerr:ZXQP8401 if the thesaurus data file's version is not supported
 : by the currently running version of Zorba.
 : @error zerr:ZXQP8402 if the thesaurus data file's endianness does not match
 : that of the CPU on which Zorba is currently running.
 : @error zerr:ZXQP8403 if there was an error reading the thesaurus data file.
 : @error zerr:ZXQP8405 if $lang is not supported for thesaurus
 : look-up specifically.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-thesaurus-lookup-2.xq
 :)
declare function ft:thesaurus-lookup( $uri as xs:string, $phrase as xs:string,
                                      $lang as xs:language )
  as xs:string+ external;

(:~
 : Looks-up the given phrase in a thesaurus.
 :
 : @param $uri The URI specifying the thesaurus to use.
 : @param $phrase The phrase to look up.
 : The phrase's
 : language
 : is assumed to be the one the one returned by ft:current-lang().
 : @return the original and related phrases.
 : @error err:FTST0009 if ft:current-lang() is unsupported in
 : general.
 : @error err:FTST0018 if $uri refers to a thesaurus
 : that is not found in the statically known thesauri.
 : @error zerr:ZOSE0001 if the thesaurus data file could not be found.
 : @error zerr:ZOSE0002 if the thesaurus data file is not a plain file.
 : @error zerr:ZXQP8401 if the thesaurus data file's version is not supported
 : by the currently running version of Zorba.
 : @error zerr:ZXQP8402 if the thesaurus data file's endianness does not match
 : that of the CPU on which Zorba is currently running.
 : @error zerr:ZXQP8403 if there was an error reading the thesaurus data file.
 : @error zerr:ZXQP8405 if ft:current-lang() is not supported for
 : thesaurus look-up specifically.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-thesaurus-lookup-3.xq
 :)
declare function ft:thesaurus-lookup( $uri as xs:string, $phrase as xs:string )
  as xs:string+ external;

(:~
 : Looks-up the given phrase in a thesaurus.
 :
 : @param $uri The URI specifying the thesaurus to use.
 : @param $phrase The phrase to look up.
 : @param $lang The
 : language
 : of $phrase.
 : @param $relationship The relationship the results are to have to
 : $phrase.
 : @return the original and related phrases.
 : @error err:FTST0018 if $uri refers to a thesaurus
 : that is not found in the statically known thesauri.
 : @error err:FTST0009 if $lang is not supported in general.
 : @error zerr:ZOSE0001 if the thesaurus data file could not be found.
 : @error zerr:ZOSE0002 if the thesaurus data file is not a plain file.
 : @error zerr:ZXQP8401 if the thesaurus data file's version is not supported
 : by the currently running version of Zorba.
 : @error zerr:ZXQP8402 if the thesaurus data file's endianness does not match
 : that of the CPU on which Zorba is currently running.
 : @error zerr:ZXQP8403 if there was an error reading the thesaurus data file.
 : @error zerr:ZXQP8405 if $lang is not supported for thesaurus
 : look-up specifically.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-thesaurus-lookup-4.xq
 :)
declare function ft:thesaurus-lookup( $uri as xs:string, $phrase as xs:string,
                                      $lang as xs:language,
                                      $relationship as xs:string )
  as xs:string+ external;

(:~
 : Looks-up the given phrase in a thesaurus.
 :
 : @param $uri The URI specifying the thesaurus to use.
 : @param $phrase The phrase to look up.
 : @param $lang The
 : language
 : of $phrase.
 : @param $relationship The relationship the results are to have to
 : $phrase.
 : @param $level-least The minimum number of levels within the thesaurus to be
 : travers$ed.
 : @param $level-most The maximum number of levels within the thesaurus to be
 : traversed.
 : @return the original and related phrases.
 : @error err:FOCA0003 if either $level-least or
 : $level-most is either negative or too large.
 : @error err:FTST0018 if $uri refers to a thesaurus
 : that is not found in the statically known thesauri.
 : @error err:FTST0009 if $lang is not supported in general.
 : @error zerr:ZOSE0001 if the thesaurus data file could not be found.
 : @error zerr:ZOSE0002 if the thesaurus data file is not a plain file.
 : @error zerr:ZXQP8401 if the thesaurus data file's version is not supported
 : by the currently running version of Zorba.
 : @error zerr:ZXQP8402 if the thesaurus data file's endianness does not match
 : that of the CPU on which Zorba is currently running.
 : @error zerr:ZXQP8403 if there was an error reading the thesaurus data file.
 : @error zerr:ZXQP8405 if $lang is not supported for thesaurus
 : look-up specifically.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-thesaurus-lookup-5.xq
 :)
declare function ft:thesaurus-lookup( $uri as xs:string, $phrase as xs:string,
                                      $lang as xs:language,
                                      $relationship as xs:string,
                                      $level-least as xs:integer,
                                      $level-most as xs:integer )
  as xs:string+ external;

(:~
 : Tokenizes the given document.
 :
 : @param $node The node to tokenize.
 : @param $lang The default
 : language
 : of $node.
 : @return a (possibly empty) sequence of tokens.
 : @error err:FTST0009 if $lang is not supported in general.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-1.xq
 :)
declare function ft:tokenize( $node as node(), $lang as xs:language )
  as element(ft-schema:token)* external;

(:~
 : Tokenizes the given document.
 :
 : @param $node The node to tokenize.
 : The document's default
 : language
 : is assumed to be the one returned by ft:current-lang().
 : @return a (possibly empty) sequence of tokens.
 : @error err:FTST0009 if ft:current-lang() is not supported in
 : general.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-2.xq
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-3.xq
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-4.xq
 :)
declare function ft:tokenize( $node as node() )
  as element(ft-schema:token)* external;

(:~
 : Tokenizes the given string.
 :
 : @param $string The string to tokenize.
 : @param $lang The default
 : language
 : of $string.
 : @return a (possibly empty) sequence of tokens.
 : @error err:FTST0009 if $lang is not supported in general.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-string-1.xq
 :)
declare function ft:tokenize-string( $string as xs:string,
                                     $lang as xs:language )
  as xs:string* external;

(:~
 : Tokenizes the given string.
 :
 : @param $string The string to tokenize.
 : The string's default
 : language
 : is assumed to be the one returned by ft:current-lang().
 : @return a (possibly empty) sequence of tokens.
 : @error err:FTST0009 if ft:current-lang() is not supported in
 : general.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-string-2.xq
 :)
declare function ft:tokenize-string( $string as xs:string )
  as xs:string* external;

(:~
 : Gets properties of the tokenizer for the given
 : language.
 :
 : @param $lang The langauage of the tokenizer to get the properties of.
 : @return said properties.
 : @error err:FTST0009 if $lang is not supported in general.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenizer-properties-1.xq
 :)
declare function ft:tokenizer-properties( $lang as xs:language )
  as element(ft-schema:tokenizer-properties) external;

(:~
 : Gets properties of the tokenizer for the
 : language
 : returned by ft:current-lang().
 :
 : @return said properties.
 : @error err:FTST0009 if ft:current-lang() is not supported in
 : general.
 : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenizer-properties-2.xq
 :)
declare function ft:tokenizer-properties()
  as element(ft-schema:tokenizer-properties) external;

(:===========================================================================:)

(: vim:set et sw=2 ts=2: :)


html-wrapper with repo-utils:serialise-as

xquery version "1.0";
module namespace cmdcheck = "http://clarin.eu/cmd/check";
(: checking (trying to ensure) consistency of the IDs in CMD-records (MdSelfLink vs. ResourceProxies vs. IsPartOf)

TODO: check (and/or generate) the inverse links in IsPartOf (vs. ResourceProxies)
:)

(:import module namespace cmd  = "http://clarin.eu/cmd/collections" at "cmd-collections.xqm";:)
import module namespace repo-utils = "http://aac.ac.at/content_repository/utils" at  "/db/cr/repo-utils.xqm";
import module namespace fcs = "http://clarin.eu/fcs/1.0" at "/db/cr/fcs.xqm";
import module namespace cr-admin  = "http://aac.ac.at/content_repository/admin" at "/db/cr/admin.xqm";

declare namespace sru = "http://www.loc.gov/zing/srw/";
declare namespace cmd = "http://www.clarin.eu/cmd/";

(:~ default namespace: cmd - declared statically, because dynamic ns-declaration did not work
cmd is the default namespace in (not) all the CMD records
:)
declare default element namespace "http://www.clarin.eu/cmd/";
(:~ default namespace - not declared explicitely, it is declared dynamically where necessary (function: addIsPartOf() :)
declare variable $cmdcheck:default-ns := "http://www.clarin.eu/cmd/";
declare variable $cmdcheck:root-coll := "root";

(:~ runs cmd.profile-scan for all contexts defined in the mappings :)

declare function cmdcheck:run-stats($config-path as xs:string) as item()* {

       let $config := doc($config-path),
           $mappings := doc(repo-utils:config-value($config, 'mappings'))

        let $opt := util:declare-option("exist:serialize", "media-type=text/html method=xhtml")

      for $map in $mappings//map[@key]
                let $map-key := $map/xs:string(@key),
                    $map-dbcoll-path := $map/xs:string(@path),
                    $scan-cmd-profile := fcs:scan('cmd.profile', $map-key, 1, 50, 1, 1, "text", $config)
                return $scan-cmd-profile
};

(:~ init-function meant to call individual functions actually doing something.
at least it resolves x-context to a nodeset
:)
declare function cmdcheck:check($x-context as xs:string+, $config as node() ) as item()* {

    let $log-file-name := concat('log_checks_', repo-utils:sanitize-name($x-context), '.xml')
    let $log-path := repo-utils:config-value($config, 'log.path')

    let $start-time := util:system-dateTime()

    let $data-collection := repo-utils:context-to-collection($x-context, $config),
        $stat-profiles := cmdcheck:stat-profiles($data-collection),
(:        $check-linking := cmdcheck:check-linking($data-collection),:)
        $duration := util:system-dateTime() - $start-time

    let $result-data := <checks context="{$x-context}" on="{$start-time}" duration="{$duration}" >
                    <profiles>{$stat-profiles}</profiles>
                    <check-linking>{$check-linking}</check-linking>
                  </checks>
     let $store := xmldb:store($log-path ,  $log-file-name, $result-data)
    return $store
};

(:~ extracts CMD-Profiles from given nodeset

TODO: profile-name deduction not reliable  - needs match with cmd-terms and diagnostics
:)
declare function cmdcheck:stat-profiles($context as node()*) as item()* {
      (: try- to handle namespace problem - primitively :)
    let $ns-uri := namespace-uri($context[1]/*)
            (: dynamically declare a namespace for the next step, if one is defined in current context
       $dummy := if (exists($ns-uri)) then util:declare-namespace("",$ns-uri) else () :)
       (: this is now trying to overcome the default-ns issue, by accepting with and without ns :)

(:        $dummy := util:declare-namespace("",xs:anyURI(""))       :)
(:    let $profiles := util:eval("$context//(MdProfile|cmd:MdProfile)/text()"):)
    let $profiles := $context//cmd:CMD/concat(cmd:Header/cmd:MdProfile/text(), '#', cmd:Components/*/name())
    let $distinct-profiles := distinct-values($profiles)
    let $profiles-summary := for $profile in $distinct-profiles
(:                                let $profile-name := util:eval("$context[.//(MdProfile|cmd:MdProfile)/text() = $profile][1]//(Components|cmd:Components)/*/name()"):)
                                let $profile-name := substring-after($profile, '#')
                                let $profile-id := substring-before($profile, '#')
                                let $cnt := count($profiles[. eq $profile])
                                return <sru:term>
                                           <sru:value>{$profile-id}</sru:value>
                                           <sru:numberOfRecords>{$cnt }</sru:numberOfRecords>
                                           <sru:displayTerm>{$profile-name}</sru:displayTerm>
                                         </sru:term>
    let $count-all := count($context)
    return
        <sru:scanResponse xmlns:sru="http://www.loc.gov/zing/srw/" xmlns:fcs="http://clarin.eu/fcs/1.0">
              <sru:version>1.2</sru:version>
              <sru:terms>
                {$profiles-summary }
               </sru:terms>
               <sru:extraResponseData>
                     <fcs:countTerms>{$count-all}</fcs:countTerms>
                 </sru:extraResponseData>
                 <sru:echoedScanRequest>
                      <sru:scanClause>cmd.profile</sru:scanClause>
                  </sru:echoedScanRequest>
           </sru:scanResponse>
};

declare function cmdcheck:display-overview($config-path as xs:string) as item()* {

   let $config := doc($config-path),
       $dummy := util:declare-namespace("",xs:anyURI("")),
       $mappings := doc(repo-utils:config-value($config, 'mappings')),
       $baseurl := repo-utils:config-value($config, 'base.url'),
       $baseadminurl := repo-utils:config-value($config, 'admin.url')

   let $opt := util:declare-option("exist:serialize", "media-type=text/html method=xhtml")

(:    {for $target in $config//target return {xs:string($target/@key)}	}
:)
let $overview :=

collection	path	size	(:ns	root-elem	:)base-elem	indexes	tests	profiles

           { for $map in util:eval("$mappings//map[@key]")
                    let $map-key := $map/xs:string(@key),
                        $map-dbcoll-path := $map/xs:string(@path),
                        $map-dbcoll:= if ($map-dbcoll-path ne '' and xmldb:collection-available (($map-dbcoll-path,"")[1])) then collection($map-dbcoll-path) else (),

                        $queries-doc-name := cr-admin:check-queries-doc-name($config, $map-key),
                        $invoke-check-queries-href := concat($baseadminurl, '?x-context=', repo-utils:sanitize-name($map-key) ,'&config=', $config-path, '&operation=' ),
                        $queries := if (repo-utils:is-in-cache($queries-doc-name, $config)) then
                                                view
                                              else (),
                        $invoke-scan-cmd-profile-href := concat($baseurl, '?operation=scan&scanClause=cmd.profile&x-format=htmlpage&x-context=', $map-key )
                    return

                        {$map-key}
                        {$map-dbcoll-path}
                        {count($map-dbcoll)}
                        {$map/xs:string(@base_elem)}
                        {count($map/index)}
                        {$queries} [run]
                        [profiles]

                        }

        (:                        $root-elems := for $elem in distinct-values($map-dbcoll/*/name()) return $elem,
                        $ns-uris := for $ns in distinct-values($map-dbcoll/namespace-uri(*)) return $ns,
                                             {$ns-uris}
                        {$root-elems}	:)

let $profiles-overview :=

collection	profiles

           { for $map in util:eval("$mappings//map[@key]")
                    let $map-key := $map/xs:string(@key),
                        $map-dbcoll-path := $map/xs:string(@path),
                        $scan-cmd-profile := fcs:scan('cmd.profile', $map-key, 1, 50, 1, 1, "text", $config),
                        $scan-formatted := repo-utils:serialise-as( $scan-cmd-profile, 'htmldetail', 'scan', $config, ())
                    return

                        {$map-key}
                        { $scan-formatted }

                        }

            (: { for $profile in $scan-cmd-profile//sru:term
                            return ($profile/sru:value, $profile/sru:displayTerm, $profile/sru:numberOfRecords)
                        }:)

        return repo-utils:serialise-as(
{($overview, $profiles-overview)}
, 'htmlpage', 'html', $config, ())

};

(:~	WARNING - CHANGES DATA! adds IsPartOf to the CMDrecords (contained in the db-coll)
- expects CMD-records for collections in specific db-coll (with name of the db-coll matching the name of the collectionfile):
    ./_corpusstructure/collection_{db-coll-name}.cmdi
- logs the progress of the processing in a separate file
- can be applied repeatedly - deletes(!) old IsPartOfList, before inserting new

@param $x-context an identifier of a collection (as defined in mappings)
@param $config config-node - used to get log.path and collection-path

TODO: could also generate the CMD-records for collections (assuming db-colls as collections)
      (now done by the python script: dir2cmdicollection.py)
TODO: currently a hack: the collection-records also are marked as root elements - this has to be optional at least,
      and may even be dangerous (count(IsPartOf[@level=1])>1)!
TODO: not recursive yet!
:)
declare function cmdcheck:addIsPartOf-colls($x-context as xs:string, $config as node()*) as item()* {

let $log-file-name := concat('log_addIsPartOf_', $x-context, '.xml')
    let $log-path := repo-utils:config-value($config, 'log.path')
    let $log-doc-path := xmldb:store($log-path ,  $log-file-name, <result></result>)
    let $log-doc := doc($log-doc-path)

    (:let $root-dbcoll := repo-utils:context-to-collection($x-context, $config),:)
    let $root-dbcoll-path := repo-utils:context-to-collection-path($x-context, $config)

    let $coll-dbcoll := '_corpusstructure'

    (: beware empty path!! would return first-level collections and run over whole database :)
    let $colls := if ($root-dbcoll-path ne "") then xmldb:get-child-collections($root-dbcoll-path) else ()
    let $start_time := fn:current-dateTime()

    (: dynamic ns-declaration does not work for xpath
    let $declare-dummy := util:declare-namespace("",xs:anyURI($cmdcheck:default-ns))
    :)
    let $log-dummy := update insert <edit x-context="{$x-context}" root-dbcoll="{$root-dbcoll-path}"
                        coll-dbcoll="{$coll-dbcoll}" count-colls="{count($colls)}" time="{$start_time}" /> into $log-doc/result
    (:if (exists($colls)) then:)
    for $coll_name in $colls[not(.=$coll-dbcoll)]
    (: let $coll_name := 'HomeFamilyManagement' :)
    let $files := collection(concat($root-dbcoll-path,$coll_name))
    (: let $files := xmldb:xcollection($root_coll) :)
	let $coll_file := concat($root-dbcoll-path, $coll-dbcoll, '/collection_', $coll_name, '.cmdi')
	let $coll_doc := doc($coll_file)
	let $coll_id := $coll_doc//MdSelfLink/text()

(:	let $cmdi_files := $files[ends-with(util:document-name(.),".cmdi")]:)

    let $pre_time := util:system-dateTime()
	let $duration := $pre_time - $start_time
 return	( update insert <edit coll="{$coll_name}" collid="{$coll_id}" coll_file="{$coll_file}" count="{count($files)}" time="{$pre_time}" /> into $log-doc/result,
			update delete $files//IsPartOfList,
			update delete $coll_doc//IsPartOfList,
			update insert <IsPartOfList>
								<IsPartOf level="1">{$x-context}</IsPartOf>
								<IsPartOf level="1">{$cmdcheck:root-coll}</IsPartOf>
						  </IsPartOfList>
			  	into $coll_doc//Resources,
			update insert <IsPartOfList>
							<IsPartOf level="2">{$x-context}</IsPartOf>
							<IsPartOf level="1">{$coll_id}</IsPartOf>
						  </IsPartOfList>
				into $files//Resources,
		update value $log-doc/result/edit[last()] with (util:system-dateTime() - $pre_time)
        )
    (:
    return	($coll_name, count($cmdi_files), $coll_file,  update insert <IsPartOfList>
							<IsPartOf level="1">{$coll_id}</IsPartOf>
						  </IsPartOfList> into $cmdi_files//Resources ):)
};

(:~ recursive addIsPartOf
starts by finding "orphaned" mdrecords = expecting that to be the root records.
and continues  ?
:)
declare function cmdcheck:addIsPartOf($x-context as xs:string, $config as node()*) as item()* {

let $log-file-name := concat('log_addIsPartOf_', $x-context, '.xml')
    let $log-path := repo-utils:config-value($config, 'log.path')
    let $log-doc-path := xmldb:store($log-path ,  $log-file-name, <result></result>)
    let $log-doc := doc($log-doc-path)

    let $root-dbcoll := repo-utils:context-to-collection($x-context, $config),
        $root-dbcoll-path := repo-utils:context-to-collection-path($x-context, $config)

(:    let $coll-dbcoll := '_corpusstructure':)

    (: beware empty path!! would return first-level collections and run over whole database :)
    let $colls := if ($root-dbcoll-path ne "") then xmldb:get-child-collections($root-dbcoll-path) else ()
    let $start_time := fn:current-dateTime()

    let $log-dummy := update insert <edit x-context="{$x-context}" root-dbcoll="{$root-dbcoll-path}"
                         count-colls="{count($colls)}" time="{$start_time}" /> into $log-doc/result

    let $root := cmdcheck:addIsPartOf-root($root-dbcoll, $log-doc)
    let $updated := cmdcheck:addIsPartOf-r($root-dbcoll, $root, 1, $log-doc)

    return $log-doc
 };

 (:~ consider those, whose MdSelfLink isn't referenced anywhere (orphans) as the root-cmd-collections
 @returns the edited records :)
 declare function cmdcheck:addIsPartOf-root($context as node()*, $log-doc as node()*) as item()* {

    let $pre_time := util:system-dateTime()

    let $orphaned := $context//CMD[not(Header/MdSelfLink = $context//ResourceProxy[ResourceType eq 'Metadata']/ResourceRef)]

    let $update := ( update insert <edit collid="root" count="{count($orphaned)}" time="{$pre_time}" /> into $log-doc/result,
			update delete $orphaned//IsPartOfList,
			update insert <IsPartOfList>
			                 <IsPartOf>{$cmdcheck:root-coll}</IsPartOf>
						  </IsPartOfList>
				into $orphaned//Resources,
		  update value $log-doc/result/edit[last()] with (util:system-dateTime() - $pre_time)
        )

    return $orphaned

 };

 declare function cmdcheck:addIsPartOf-r($context as node()*, $parents as node()*, $level as xs:integer, $log-doc as node()*) as item()* {

    let $start_time := util:system-dateTime()
    let $log-dummy := update insert <edit level="{$level}"
                        count-colls="{count($parents)}" time="{$start_time}" /> into $log-doc/result

  let $update := for $cmd-record in $parents
                    let $parent-id := $cmd-record//MdSelfLink/text()
                    let $children := $context//CMD[Header/MdSelfLink = $cmd-record//ResourceRef]
                	let $ispartoflist := <IsPartOfList>{($cmd-record//IsPartOfList/IsPartOf[not(text()=$cmdcheck:root-coll)],
                								<IsPartOf >{$parent-id}</IsPartOf>) }
                						  </IsPartOfList>

                    let $pre_time := util:system-dateTime()
                	let $duration := $pre_time - $start_time
                    return	( update insert <edit collid="{$parent-id}" count="{count($children)}" time="{$pre_time}" /> into $log-doc/result,
                			update delete $children//IsPartOfList,
                			update insert $ispartoflist
                				into $children//Resources,
                		  update value $log-doc/result/edit[last()] with (util:system-dateTime() - $pre_time)
                        )

    (: go next level - we dont have to recurse in the update-loop,
     but we can do whole next level in one call :)
    let $next-level := $context//CMD[Header/MdSelfLink = $parents//ResourceRef]
    let $updated := if (exists($next-level)) then  cmdcheck:addIsPartOf-r($context, $next-level ,($level+1),$log-doc)
                        else ()
    return $update
};

(:~ checking consistency of the IDs in CMD-records (MdSelfLink vs. ResourceProxies)

this is now solved better (faster) with xpaths in the cmd-testset,
so this is probably OBSOLETED.
it has very bad performance!

TODO: check (and/or generate) the inverse links in IsPartOf (vs. ResourceProxies) :)
declare function cmdcheck:check-linking($context as node()*) as item()* {

let $mdselflinks := $context//MdSelfLink
let $resourceproxies := $context//ResourceProxy
let $resourceproxies-md := $resourceproxies[ResourceType eq 'Metadata']/ResourceRef
(: )let $diff :=  $resourceproxies-md[not(index-of($mdselflinks, self::*))]
let $diff2 :=  $mdselflinks[not($resourceproxies-md = text())]:)
let $diff := for $resource-proxy in $resourceproxies-md
                    let $id := xs:string($resource-proxy/text())
                return if ($mdselflinks[ft:query(.,<term>{$id}</term>)]) then () (: $mdselflinks[ft:query(.,<term>{$id}</term>)] :)
                 else <doc n="{util:document-name($resource-proxy)}" >{$resource-proxy}</doc>

let $diff2 := for $mdselflink in $mdselflinks
                    let $id := xs:string($mdselflink/text())
                return if ($resourceproxies-md[ft:query(.,<term>{$id}</term>)]) then () (: $mdselflinks[ft:query(.,<term>{$id}</term>)] :)
                 else <doc n="{util:document-name($mdselflink)}" >{$mdselflink}</doc>
return ($diff,$diff2)
};

xquery version "1.0";
module namespace cmdcheck = "http://clarin.eu/cmd/check";
(: checking (trying to ensure) consistency of the IDs in CMD-records (MdSelfLink vs. ResourceProxies vs. IsPartOf)

TODO: check (and/or generate) the inverse links in IsPartOf (vs. ResourceProxies)
:)

(:import module namespace cmd  = "http://clarin.eu/cmd/collections" at "cmd-collections.xqm";:)
import module namespace repo-utils = "http://aac.ac.at/content_repository/utils" at  "/db/cr/repo-utils.xqm";
import module namespace fcs = "http://clarin.eu/fcs/1.0" at "/db/cr/fcs.xqm";
import module namespace cr-admin  = "http://aac.ac.at/content_repository/admin" at "/db/cr/admin.xqm";

declare namespace sru = "http://www.loc.gov/zing/srw/";
declare namespace cmd = "http://www.clarin.eu/cmd/";

(:~ default namespace: cmd - declared statically, because dynamic ns-declaration did not work
cmd is the default namespace in (not) all the CMD records
:)
declare default element namespace "http://www.clarin.eu/cmd/";
(:~ default namespace - not declared explicitely, it is declared dynamically where necessary (function: addIsPartOf() :)
declare variable $cmdcheck:default-ns := "http://www.clarin.eu/cmd/";
declare variable $cmdcheck:root-coll := "root";

(:~ runs cmd.profile-scan for all contexts defined in the mappings :)

declare function cmdcheck:run-stats($config-path as xs:string) as item()* {

       let $config := doc($config-path),
           $mappings := doc(repo-utils:config-value($config, 'mappings'))

        let $opt := util:declare-option("exist:serialize", "media-type=text/html method=xhtml")

      for $map in $mappings//map[@key]
                let $map-key := $map/xs:string(@key),
                    $map-dbcoll-path := $map/xs:string(@path),
                    $scan-cmd-profile := fcs:scan('cmd.profile', $map-key, 1, 50, 1, 1, "text", $config)
                return $scan-cmd-profile
};

(:~ init-function meant to call individual functions actually doing something.
at least it resolves x-context to a nodeset
:)
declare function cmdcheck:check($x-context as xs:string+, $config as node() ) as item()* {

    let $log-file-name := concat('log_checks_', repo-utils:sanitize-name($x-context), '.xml')
    let $log-path := repo-utils:config-value($config, 'log.path')

    let $start-time := util:system-dateTime()

    let $data-collection := repo-utils:context-to-collection($x-context, $config),
        $stat-profiles := cmdcheck:stat-profiles($data-collection),
(:        $check-linking := cmdcheck:check-linking($data-collection),:)
        $duration := util:system-dateTime() - $start-time

    let $result-data := <checks context="{$x-context}" on="{$start-time}" duration="{$duration}" >
                    <profiles>{$stat-profiles}</profiles>
                    <check-linking>{$check-linking}</check-linking>
                  </checks>
     let $store := xmldb:store($log-path ,  $log-file-name, $result-data)
    return $store
};

(:~ extracts CMD-Profiles from given nodeset

TODO: profile-name deduction not reliable  - needs match with cmd-terms and diagnostics
:)
declare function cmdcheck:stat-profiles($context as node()*) as item()* {
      (: try- to handle namespace problem - primitively :)
    let $ns-uri := namespace-uri($context[1]/*)
            (: dynamically declare a namespace for the next step, if one is defined in current context
       $dummy := if (exists($ns-uri)) then util:declare-namespace("",$ns-uri) else () :)
       (: this is now trying to overcome the default-ns issue, by accepting with and without ns :)

(:        $dummy := util:declare-namespace("",xs:anyURI(""))       :)
(:    let $profiles := util:eval("$context//(MdProfile|cmd:MdProfile)/text()"):)
    let $profiles := $context//cmd:CMD/concat(cmd:Header/cmd:MdProfile/text(), '#', cmd:Components/*/name())
    let $distinct-profiles := distinct-values($profiles)
    let $profiles-summary := for $profile in $distinct-profiles
(:                                let $profile-name := util:eval("$context[.//(MdProfile|cmd:MdProfile)/text() = $profile][1]//(Components|cmd:Components)/*/name()"):)
                                let $profile-name := substring-after($profile, '#')
                                let $profile-id := substring-before($profile, '#')
                                let $cnt := count($profiles[. eq $profile])
                                return <sru:term>
                                           <sru:value>{$profile-id}</sru:value>
                                           <sru:numberOfRecords>{$cnt }</sru:numberOfRecords>
                                           <sru:displayTerm>{$profile-name}</sru:displayTerm>
                                         </sru:term>
    let $count-all := count($context)
    return
        <sru:scanResponse xmlns:sru="http://www.loc.gov/zing/srw/" xmlns:fcs="http://clarin.eu/fcs/1.0">
              <sru:version>1.2</sru:version>
              <sru:terms>
                {$profiles-summary }
               </sru:terms>
               <sru:extraResponseData>
                     <fcs:countTerms>{$count-all}</fcs:countTerms>
                 </sru:extraResponseData>
                 <sru:echoedScanRequest>
                      <sru:scanClause>cmd.profile</sru:scanClause>
                  </sru:echoedScanRequest>
           </sru:scanResponse>
};

declare function cmdcheck:display-overview($config-path as xs:string) as item()* {

   let $config := doc($config-path),
       $dummy := util:declare-namespace("",xs:anyURI("")),
       $mappings := doc(repo-utils:config-value($config, 'mappings')),
       $baseurl := repo-utils:config-value($config, 'base.url'),
       $baseadminurl := repo-utils:config-value($config, 'admin.url')

   let $opt := util:declare-option("exist:serialize", "media-type=text/html method=xhtml")

(:    {for $target in $config//target return {xs:string($target/@key)}	}
:)
let $overview :=

collection	path	size	(:ns	root-elem	:)base-elem	indexes	tests	profiles

           { for $map in util:eval("$mappings//map[@key]")
                    let $map-key := $map/xs:string(@key),
                        $map-dbcoll-path := $map/xs:string(@path),
                        $map-dbcoll:= if ($map-dbcoll-path ne '' and xmldb:collection-available (($map-dbcoll-path,"")[1])) then collection($map-dbcoll-path) else (),

                        $queries-doc-name := cr-admin:check-queries-doc-name($config, $map-key),
                        $invoke-check-queries-href := concat($baseadminurl, '?x-context=', repo-utils:sanitize-name($map-key) ,'&config=', $config-path, '&operation=' ),
                        $queries := if (repo-utils:is-in-cache($queries-doc-name, $config)) then
                                                view
                                              else (),
                        $invoke-scan-cmd-profile-href := concat($baseurl, '?operation=scan&scanClause=cmd.profile&x-format=htmlpage&x-context=', $map-key )
                    return

                        {$map-key}
                        {$map-dbcoll-path}
                        {count($map-dbcoll)}
                        {$map/xs:string(@base_elem)}
                        {count($map/index)}
                        {$queries} [run]
                        [profiles]

                        }

        (:                        $root-elems := for $elem in distinct-values($map-dbcoll/*/name()) return $elem,
                        $ns-uris := for $ns in distinct-values($map-dbcoll/namespace-uri(*)) return $ns,
                                             {$ns-uris}
                        {$root-elems}	:)

let $profiles-overview :=

collection	profiles

           { for $map in util:eval("$mappings//map[@key]")
                    let $map-key := $map/xs:string(@key),
                        $map-dbcoll-path := $map/xs:string(@path),
                        $scan-cmd-profile := fcs:scan('cmd.profile', $map-key, 1, 50, 1, 1, "text", $config)
                    return

                        {$map-key}

                        { for $profile in $scan-cmd-profile//sru:term
                            return ($profile/sru:value, $profile/sru:displayTerm, $profile/sru:numberOfRecords)
                        }

                        }

         return
{($overview, $profiles-overview)}

};

(:~	WARNING - CHANGES DATA! adds IsPartOf to the CMDrecords (contained in the db-coll)
- expects CMD-records for collections in specific db-coll (with name of the db-coll matching the name of the collectionfile):
    ./_corpusstructure/collection_{db-coll-name}.cmdi
- logs the progress of the processing in a separate file
- can be applied repeatedly - deletes(!) old IsPartOfList, before inserting new

@param $x-context an identifier of a collection (as defined in mappings)
@param $config config-node - used to get log.path and collection-path

TODO: could also generate the CMD-records for collections (assuming db-colls as collections)
      (now done by the python script: dir2cmdicollection.py)
TODO: currently a hack: the collection-records also are marked as root elements - this has to be optional at least,
      and may even be dangerous (count(IsPartOf[@level=1])>1)!
TODO: not recursive yet!
:)
declare function cmdcheck:addIsPartOf-colls($x-context as xs:string, $config as node()*) as item()* {

let $log-file-name := concat('log_addIsPartOf_', $x-context, '.xml')
    let $log-path := repo-utils:config-value($config, 'log.path')
    let $log-doc-path := xmldb:store($log-path ,  $log-file-name, <result></result>)
    let $log-doc := doc($log-doc-path)

    (:let $root-dbcoll := repo-utils:context-to-collection($x-context, $config),:)
    let $root-dbcoll-path := repo-utils:context-to-collection-path($x-context, $config)

    let $coll-dbcoll := '_corpusstructure'

    (: beware empty path!! would return first-level collections and run over whole database :)
    let $colls := if ($root-dbcoll-path ne "") then xmldb:get-child-collections($root-dbcoll-path) else ()
    let $start_time := fn:current-dateTime()

    (: dynamic ns-declaration does not work for xpath
    let $declare-dummy := util:declare-namespace("",xs:anyURI($cmdcheck:default-ns))
    :)
    let $log-dummy := update insert <edit x-context="{$x-context}" root-dbcoll="{$root-dbcoll-path}"
                        coll-dbcoll="{$coll-dbcoll}" count-colls="{count($colls)}" time="{$start_time}" /> into $log-doc/result
    (:if (exists($colls)) then:)
    for $coll_name in $colls[not(.=$coll-dbcoll)]
    (: let $coll_name := 'HomeFamilyManagement' :)
    let $files := collection(concat($root-dbcoll-path,$coll_name))
    (: let $files := xmldb:xcollection($root_coll) :)
	let $coll_file := concat($root-dbcoll-path, $coll-dbcoll, '/collection_', $coll_name, '.cmdi')
	let $coll_doc := doc($coll_file)
	let $coll_id := $coll_doc//MdSelfLink/text()

(:	let $cmdi_files := $files[ends-with(util:document-name(.),".cmdi")]:)

    let $pre_time := util:system-dateTime()
	let $duration := $pre_time - $start_time
 return	( update insert <edit coll="{$coll_name}" collid="{$coll_id}" coll_file="{$coll_file}" count="{count($files)}" time="{$pre_time}" /> into $log-doc/result,
			update delete $files//IsPartOfList,
			update delete $coll_doc//IsPartOfList,
			update insert <IsPartOfList>
								<IsPartOf level="1">{$x-context}</IsPartOf>
								<IsPartOf level="1">{$cmdcheck:root-coll}</IsPartOf>
						  </IsPartOfList>
			  	into $coll_doc//Resources,
			update insert <IsPartOfList>
							<IsPartOf level="2">{$x-context}</IsPartOf>
							<IsPartOf level="1">{$coll_id}</IsPartOf>
						  </IsPartOfList>
				into $files//Resources,
		update value $log-doc/result/edit[last()] with (util:system-dateTime() - $pre_time)
        )
    (:
    return	($coll_name, count($cmdi_files), $coll_file,  update insert <IsPartOfList>
							<IsPartOf level="1">{$coll_id}</IsPartOf>
						  </IsPartOfList> into $cmdi_files//Resources ):)
};

(:~ recursive addIsPartOf
starts by finding "orphaned" mdrecords = expecting that to be the root records.
and continues  ?
:)
declare function cmdcheck:addIsPartOf($x-context as xs:string, $config as node()*) as item()* {

let $log-file-name := concat('log_addIsPartOf_', $x-context, '.xml')
    let $log-path := repo-utils:config-value($config, 'log.path')
    let $log-doc-path := xmldb:store($log-path ,  $log-file-name, <result></result>)
    let $log-doc := doc($log-doc-path)

    let $root-dbcoll := repo-utils:context-to-collection($x-context, $config),
        $root-dbcoll-path := repo-utils:context-to-collection-path($x-context, $config)

(:    let $coll-dbcoll := '_corpusstructure':)

    (: beware empty path!! would return first-level collections and run over whole database :)
    let $colls := if ($root-dbcoll-path ne "") then xmldb:get-child-collections($root-dbcoll-path) else ()
    let $start_time := fn:current-dateTime()

    let $log-dummy := update insert <edit x-context="{$x-context}" root-dbcoll="{$root-dbcoll-path}"
                         count-colls="{count($colls)}" time="{$start_time}" /> into $log-doc/result

    let $root := cmdcheck:addIsPartOf-root($root-dbcoll, $log-doc)
    let $updated := cmdcheck:addIsPartOf-r($root-dbcoll, $root, 1, $log-doc)

    return $log-doc
 };

 (:~ consider those, whose MdSelfLink isn't referenced anywhere (orphans) as the root-cmd-collections
 @returns the edited records :)
 declare function cmdcheck:addIsPartOf-root($context as node()*, $log-doc as node()*) as item()* {

    let $pre_time := util:system-dateTime()

    let $orphaned := $context//CMD[not(Header/MdSelfLink = $context//ResourceProxy[ResourceType eq 'Metadata']/ResourceRef)]

    let $update := ( update insert <edit collid="root" count="{count($orphaned)}" time="{$pre_time}" /> into $log-doc/result,
			update delete $orphaned//IsPartOfList,
			update insert <IsPartOfList>
			                 <IsPartOf>{$cmdcheck:root-coll}</IsPartOf>
						  </IsPartOfList>
				into $orphaned//Resources,
		  update value $log-doc/result/edit[last()] with (util:system-dateTime() - $pre_time)
        )

    return $orphaned

 };

 declare function cmdcheck:addIsPartOf-r($context as node()*, $parents as node()*, $level as xs:integer, $log-doc as node()*) as item()* {

    let $start_time := util:system-dateTime()
    let $log-dummy := update insert <edit level="{$level}"
                        count-colls="{count($parents)}" time="{$start_time}" /> into $log-doc/result

  let $update := for $cmd-record in $parents
                    let $parent-id := $cmd-record//MdSelfLink/text()
                    let $children := $context//CMD[Header/MdSelfLink = $cmd-record//ResourceRef]
                	let $ispartoflist := <IsPartOfList>{($cmd-record//IsPartOfList/IsPartOf[not(text()=$cmdcheck:root-coll)],
                								<IsPartOf >{$parent-id}</IsPartOf>) }
                						  </IsPartOfList>

                    let $pre_time := util:system-dateTime()
                	let $duration := $pre_time - $start_time
                    return	( update insert <edit collid="{$parent-id}" count="{count($children)}" time="{$pre_time}" /> into $log-doc/result,
                			update delete $children//IsPartOfList,
                			update insert $ispartoflist
                				into $children//Resources,
                		  update value $log-doc/result/edit[last()] with (util:system-dateTime() - $pre_time)
                        )

    (: go next level - we dont have to recurse in the update-loop,
     but we can do whole next level in one call :)
    let $next-level := $context//CMD[Header/MdSelfLink = $parents//ResourceRef]
    let $updated := if (exists($next-level)) then  cmdcheck:addIsPartOf-r($context, $next-level ,($level+1),$log-doc)
                        else ()
    return $update
};

(:~ checking consistency of the IDs in CMD-records (MdSelfLink vs. ResourceProxies)

this is now solved better (faster) with xpaths in the cmd-testset,
so this is probably OBSOLETED.
it has very bad performance!

TODO: check (and/or generate) the inverse links in IsPartOf (vs. ResourceProxies) :)
declare function cmdcheck:check-linking($context as node()*) as item()* {

let $mdselflinks := $context//MdSelfLink
let $resourceproxies := $context//ResourceProxy
let $resourceproxies-md := $resourceproxies[ResourceType eq 'Metadata']/ResourceRef
(: )let $diff :=  $resourceproxies-md[not(index-of($mdselflinks, self::*))]
let $diff2 :=  $mdselflinks[not($resourceproxies-md = text())]:)
let $diff := for $resource-proxy in $resourceproxies-md
                    let $id := xs:string($resource-proxy/text())
                return if ($mdselflinks[ft:query(.,<term>{$id}</term>)]) then () (: $mdselflinks[ft:query(.,<term>{$id}</term>)] :)
                 else <doc n="{util:document-name($resource-proxy)}" >{$resource-proxy}</doc>

let $diff2 := for $mdselflink in $mdselflinks
                    let $id := xs:string($mdselflink/text())
                return if ($resourceproxies-md[ft:query(.,<term>{$id}</term>)]) then () (: $mdselflinks[ft:query(.,<term>{$id}</term>)] :)
                 else <doc n="{util:document-name($mdselflink)}" >{$mdselflink}</doc>
return ($diff,$diff2)
};