Clarify examples, especially regarding how HTML is handled.

xquery version "3.0";

(:
 : Copyright 2006-2009 The FLWOR Foundation.
 :
 : Licensed under the Apache License, Version 2.0 (the "License");
 : you may not use this file except in compliance with the License.
 : You may obtain a copy of the License at
 :
 : http://www.apache.org/licenses/LICENSE-2.0
 :
 : Unless required by applicable law or agreed to in writing, software
 : distributed under the License is distributed on an "AS IS" BASIS,
 : WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 : See the License for the specific language governing permissions and
 : limitations under the License.
:)

(:~
 :
Introduction

 :

 : This module provides provides simple functions for performing HTTP requests
 : (GET, POST, DELETE etc.), as well as a more flexible general
 : purpose function (send-request()).
 :

 :
 :
Examples of how to use this module

 :
 :
Simple GET Request

 :
 :

 : import module namespace http="http://www.zorba-xquery.com/modules/http-client";
 : declare namespace svg="http://www.w3.org/2000/svg";
 : http:get("http://www.w3.org/Graphics/SVG/svglogo.svg")[2]/svg:svg/svg:title
 :

 :
 :

 : This example downloads an XML resource from the web (in this case,
 : an SVG file, which is an XML-based image format) and returns it as
 : a document node. Since the XML is in a namespace, we declare that
 : namespace; we can then perform a path expression directly on the
 : return value of http:get().
 :

 :
 :
Simple GET Request (retrieving XHTML)

 :
 :

 :   import module namespace http="http://www.zorba-xquery.com/modules/http-client";
 :   declare namespace xhtml="http://www.w3.org/1999/xhtml";
 :
 :   http:get-node( "http://www.w3.org" )[2]//xhtml:body
 :

 :
 :

 : This example shows how to retrieve an XHTML resource. XHTML is
 : XML, so the http:get-node() function will return it as a document node
 : and you can operate on it with the full power of XQuery. As above, since this
 : XML is in a particular namespace, the above query defines that namespace
 : with the prefix "xhtml" so it can easily perform path expressions, etc.
 :

 :
 :

 : Note: many webservers, include www.w3.org, return XHTML with the
 : HTTP Content-Type "text/html". Zorba cannot assume that "text/html"
 : is actually XHTML, and so http:get() would have returned raw text
 : rather than a document node. That is why the example above uses
 : http:get-node(), which overrides the server's Content-Type and tells
 : Zorba to attempt to parse the result as XML.
 :

 :
 :
Simple GET Request (retrieving HTML as text)

 :
 :

 : Note that XQuery does not understand plain HTML, and so if the URL
 : you retrieve contains plain HTML data (not XHTML), it will be treated as
 : plain text as shown in the next example. If you want to operate on the HTML
 : with XQuery, you should use the HTML language module which can transform
 : HTML to XHTML. The HTML module is supported by the Zorba team, but it is
 : not a "core module", meaning that it is not shipped with every Zorba
 : installation and may not be available. See
 : the Zorba downloads
 : page for information about obtaining this module if you do not
 : have it.

 :
 :

 : import module namespace http="http://www.zorba-xquery.com/modules/http-client";
 : http:get("http://www.example.com")[2]
 :

 :
 : returns
 :
 :

 :   <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
 :   <html>
 :     <head>
 :       <meta http-equiv="Content-Type"
 :       content="text/html; charset=utf-8" />
 :       <title>Example Web Page</title>
 :     </head>
 :     <body>
 :       <p>You have reached this web page by typing "example.com",
 :       "example.net", or "example.org" into your web browser.</p>
 :       <p>These domain names are reserved for use in documentation and are
 :       Not available for registration. See
 :       <a href="http://www.rfc-editor.org/rfc/rfc2606.txt">RFC 2606</a>,
 :       Section 3.</p>
 :     </body>
 :   </html>
 :

 :
 :
Note that the response data above is a simple
 : xs:string value containing the HTML data, not actual XML data. If you
 : executed the above query using the Zorba command-line client, you would
 : have actually seen data like the following:

 :
 :

 :   &lt;!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"&gt;
 :   &lt;html&gt;
 :      ...
 :

 :
 :
because Zorba would attempt to serialize it as XML data, and would
 : escape all the raw angle brackets in the original xs:string.

 :
 :
Simple POST Request

 :
 :

 : Here is a simple example which sends text content by making an HTTP POST
 : request.
 :

 :
 :

 : import module namespace http="http://www.zorba-xquery.com/modules/http-client";
 : http:post( "...", "Hello World" )
 :

 :
 :
Return Values

 :
 :
Most functions in this module (all except
 : options()) return one or more items.
 : (head() returns exactly one.) For all of these,
 : the first item returned will be a <http-schema:response>
 : element, as seen in the examples above. This element has "status" and
 : "message" attributes, representing the result of the HTTP call. It
 : also has any number of <http-schema:header> child elements that
 : encode the HTTP headers returned by the HTTP server. Finally, it
 : will generally contain a <http-schema:body> child element with
 : a "media-type" attribute that identifies the content-type of the
 : result data.

 :
 :
The full schema of this <http-schema:response> element is
 : part of the EXPath
 : HTTP Client module. You can see the schema
 : here.

 :
 :
Any items in function return values after the initial
 : <http-schema:response> element are the body/bodies of the HTTP
 : response from the server. (MIME Multi-part responses will have
 : more than one body.) The type of these items depends on the
 : Content-Type for each body. Each item will be:

 :
 :

 :

 :    an element node, if the returned content type is one of:
 :

 :
text/xml

 :
application/xml

 :
text/xml-external-parsed-entity

 :
application/xml-external-parsed-entity

 :
or if the Content-Type ends with "+xml".

 :

 :

 :

 :    an xs:string, if the content type starts with "text/" and does not match the
 :    above XML content types strings.
 :

 :
xs:base64Binary for all other content types.

 :

 :
 :
This return value - a sequence of items comprising one
 : <http-schema:response> element followed by zero or more
 : response items - is referred to as the "standard http-client
 : return type" in the function declarations below.

 :
 :
$href Arguments to Functions

 :
 : All functions in this module accept a URL argument named $href. In
 : all cases, the value passed to $href must be a valid xs:anyURI.
 : However, all functions declare $href to be of type xs:string. This
 : is for convenience, since you can pass a string literal value (that
 : is, a URL in double-quotes spelled out explicitly in your query)
 : to an xs:string parameter.
 :
 :
Important Notice Regarding get() Functions

 :
 : All of the get() functions in this module -
 : get(), get-node(),
 : get-text(), and
 : get-binary() - are declared to be
 : nondeterministic, which means that Zorba will not cache
 : their results. However, they are not declared to be
 : sequential, which means that Zorba may re-order them
 : as part of its query optimization. According to the HTTP RFC,
 : GET requests should only return data, and should not have any
 : side-effects. However, in practice it is not uncommon for GET
 : requests to have side-effects. If your application depends on
 : the ordering of side-effects from making GET requests, you should
 : either use the more complex send-request()
 : function (which is declared sequential), or alterately
 : wrap each call to get() in your own sequential function, to ensure
 : that Zorba does not place the GET requests out of order.
 :
 :
Relation to the EXPath http-client module

 :
 : EXPath defines its own http-client
 : module, which is available separately for Zorba as a non-core module.
 : There are two primary differences between EXPath's http-client and
 : Zorba's core http-client (this module):
 :
 :

 :
EXPath defines only the send-request() function, although it
 : does include convenient 1- and 2-argument forms in addition to the
 : full 3-argument form. EXPath does not include the simpler get(),
 : post(), put(), delete(), head(), and options() functions defined by
 : this module.

 :
EXPath specifies that all HTML content returned from the
 : HTTP server will be tidied up into valid XML, and then parsed
 : into an element. As this required an additional third-party library
 : dependency, Zorba's http-client module does not perform this tidying.
 : Instead, HTML content is returned as a string (with special XML
 : characters replaced with XML entity references, as shown in the
 : above examples).

 :

 :
 : See the full spec
 : of the EXPath http-client module for more information.
 :
 :
 : @author Markus Pilman
 : @see XQuery 1.1: Function Declaration
 : @library cURL Library
 : @project external
 :)
module namespace http = "http://www.zorba-xquery.com/modules/http-client";

import module namespace error = "http://expath.org/ns/error";

import schema namespace http-schema = "http://expath.org/ns/http-client";

declare namespace ann = "http://www.zorba-xquery.com/annotations";
declare namespace ver = "http://www.zorba-xquery.com/options/versioning";
declare namespace err = "http://www.w3.org/2005/xqt-errors";

declare option ver:module-version "2.0";

(:~
 : This function sends an HTTP request and returns the corresponding response.
 : Its inputs, outputs, and behavior are identical to the
 : EXPath http-client's
 : send-request() function (except that HTML responses are not tidied
 : into XML - see the note above). It
 : is provided here for use in Zorba installations that do not have
 : the EXPath module available. If you have the option of using the
 : EXPath module instead of this function, please do so, as it will
 : allow your application to be more interoperable between different
 : XQuery engines.
 :
 : Full documentation of the $request parameter can be found in
 : the EXPath
 : specification.
 :
 : @param $request Contains the various parameters of the request (see above).
 : @param $href The URL to which the request will be made (see
 :  note above). If this
 :  parameter is specified, it will override the "href" attribute of
 :  $request.
 : @param $bodies is the request body content, for HTTP methods that can
 :  contain a body in the request (i.e. POST and PUT). It is an error if this
 :  param is not the empty sequence for methods
 : @return standard http-client return type.
 : @error error:HC001 An HTTP error occurred.
 : @error error:HC002 Error parsing the response content as XML.
 : @error error:HC003 With a multipart response, the override-media-type must be either a multipart media type or application/octet-stream.
 : @error error:HC004 The src attribute on the body element is mutually exclusive with all other attribute (except the media-type).
 : @error error:HC005 The input request element is not valid.
 : @error error:HC006 A timeout occurred waiting for the response.
 :)
declare %ann:sequential function http:send-request(
  $request as element(http-schema:request)?,
  $href as xs:string?,
  $bodies as item()*) as item()+
{
  if (http:check-params($request, $href, $bodies))
  then
    {
      variable $req := if ($request)
                      then
                        try
                        {
                          validate { http:set-content-type($request) }
                        }
                        catch XQDY0027
                        {
                          fn:error($error:HC005, "The request element is not valid.")
                        }
                      else
                       ();

      variable $result := http:http-sequential-impl($req, $href, $bodies);

     $result
    }
  else
    ()
};

(:~
 : This function makes a GET request to a given URL.
 :
 : @see Notice about get() functions
 : @param $href The URL to which the request will be made (see
 :  note above).
 : @return standard http-client return type.
 : @error error:HC001 An HTTP error occurred.
 : @error error:HC002 Error parsing the response content as XML.
 : @error error:HC006 A timeout occurred waiting for the response.
 :)
declare %ann:nondeterministic function http:get($href as xs:string) as item()+
{
  http:http-nondeterministic-impl(validate {<http-schema:request method="GET" href="{$href}" follow-redirect="true"/>}, (), ())
};

(:~
 : This function makes a GET request to a given URL. All returned bodies
 : are forced to be interpreted as XML and parsed into elements.
 :
 : @see Notice about get() functions
 : @param $href The URL to which the request will be made (see
 :  note above).
 : @return standard http-client return type.
 : @error error:HC001 An HTTP error occurred.
 : @error error:HC002 Error parsing the response content as XML.
 : @error error:HC006 A timeout occurred waiting for the response.
 :)
declare %ann:nondeterministic function http:get-node($href as xs:string) as item()+
{
  http:http-nondeterministic-impl(validate {<http-schema:request method="GET" href="{$href}" follow-redirect="true" override-media-type="text/xml"/>}, (), ())
};

(:~
 : This function makes a GET request to a given URL. All returned bodies
 : are forced to be interpreted as plain strings, and will be returned
 : as xs:string items.
 :
 : @see Notice about get() functions
 : @param $href The URL to which the request will be made (see
 :  note above).
 : @return standard http-client return type.
 : @error error:HC001 An HTTP error occurred.
 : @error error:HC002 Error parsing the response content as XML.
 : @error error:HC006 A timeout occurred waiting for the response.
 :)
declare %ann:nondeterministic function http:get-text($href as xs:string) as item()+
{
  http:http-nondeterministic-impl(validate {<http-schema:request method="GET" href="{$href}" follow-redirect="true" override-media-type="text/plain"/>}, (), ())
};

(:~
 : This function makes a GET request on a given URL. All returned bodies
 : are forced to be interpreted as binary data, and will be returned
 : as xs:base64Binary items.
 :
 : @see Notice about get() functions
 : @param $href The URL to which the request will be made (see
 :  note above).
 : @return standard http-client return type.
 : @error error:HC001 An HTTP error occurred.
 : @error error:HC002 Error parsing the response content as XML.
 : @error error:HC006 A timeout occurred waiting for the response.
 :)
declare %ann:nondeterministic function http:get-binary($href as xs:string) as item()+
{
  http:http-nondeterministic-impl(validate {<http-schema:request method="GET" href="{$href}" follow-redirect="true" override-media-type="binary"/>}, (), ())
};

(:~
 : This function makes an HTTP HEAD request on a given URL.
 :
 : @param $href The URL to which the request will be made (see
 :  note above).
 : @return standard http-client return type
 :  (since HEAD never returns any body data, only the
 :  <http-schema:response> element will be returned).
 : @error error:HC001 An HTTP error occurred.
 : @error error:HC006 A timeout occurred waiting for the response.
 :)
declare %ann:nondeterministic function http:head($href as xs:string) as item() {
  http:http-nondeterministic-impl(
    validate {
      <http-schema:request method="HEAD" href="{$href}">
      </http-schema:request>
    }, (), ())
};

(:~
 : This function makes an HTTP OPTIONS request, which asks the server
 : which operations it supports.
 :
 : @param $href The URL to which the request will be made (see
 :  note above).
 : @return A sequence of xs:string values of the allowed operations.
 : @error error:HC001 An HTTP error occurred.
 : @error error:HC006 A timeout occurred waiting for the response.
 :)
declare %ann:nondeterministic function http:options($href as xs:string) as xs:string* {
  let $resp := http:http-nondeterministic-impl(
    validate {
      <http-schema:request method="OPTIONS" href="{$href}">
      </http-schema:request>
    }, (), ())[1]
  return
    fn:tokenize(fn:data($resp/http-schema:header[@name = "Allow"]/@value), ",")
};

(:~
 : This function makes an HTTP PUT request to a given URL. If the body
 : passed to this function is an element, it will be serialized to XML
 : to be sent to the server, and the Content-Type sent to the server will
 : be "text/xml". Otherwise, the body will be converted to
 : a plain string, and the Content-Type will be "text/plain".
 :
 : @param $href The URL to which the request will be made (see
 :  note above).
 : @param $body The body which will be sent to the server.
 : @return standard http-client return type.
 : @error error:HC001 An HTTP error occurred.
 : @error error:HC002 Error parsing the response content as XML.
 : @error error:HC006 A timeout occurred waiting for the response.
 :)
declare %ann:sequential function http:put($href as xs:string, $body as item()) as item()+
{
  variable $media-type as xs:string+ :=
    typeswitch($body)
      case xs:string return ("text/plain","text")
      case element() return ("text/xml", "xml")
      default return ("text/plain","xml");

  variable $result := http:put($href, $body, $media-type[1]);

  $result
};

(:~
 : This function makes an HTTP PUT request to a given URL. If the body
 : passed to this function is an element, it will be serialized
 : according to the $content-type parameter as follows:
 :

 :
If $content-type is "text/xml", "application/xml",
 : "text/xml-external-parsed-entity", or
 : "application/xml-external-parsed-entity", or if it ends with "+xml",
 : $body will be serialized to XML.

 :
If $content-type starts with "text/html", $body will be
 : serialized to HTML.

 :
Otherwise, $body will be serialized to text.

 :

 : If $body is not an element, $body will be serialized to text
 : regardless of $content-type.
 :
 :
In any case, Content-Type of the request sent to the server will
 : be $content-type.

 :
 : @param $href The URL to which the request will be made (see
 :  note above).
 : @param $body The body which will be sent to the server.
 : @param $content-type The content type of $body as described above.
 : @return standard http-client return type.
 : @error error:HC001 An HTTP error occurred.
 : @error error:HC002 Error parsing the response content as XML.
 : @error error:HC006 A timeout occurred waiting for the response.
 :)
declare %ann:sequential function http:put($href as xs:string, $body as item(), $content-type as xs:string) as item()+
{
  variable $method :=
    typeswitch ($body)
      case element() return
        if ($content-type = ("text/xml", "application/xml",
                                         "text/xml-external-parsed-entity",
                                         "application/xml-external-parsed-entity")
          or fn:ends-with($content-type, "+xml")) then
            "xml"
        else if (fn:starts-with($content-type, "text/html")) then
          "html"
        else if (fn:starts-with($content-type, "text/")) then
          "text"
        else
          "binary"
      case xs:base64Binary return "binary"
      default return "text";

  variable $result := http:http-sequential-impl(validate {
    <http-schema:request href="{$href}" method="PUT">
      <http-schema:body media-type="{$content-type}" method="{$method}">{$body}</http-schema:body>
    </http-schema:request>}
    , (), ());

  $result
};

(:~
 : This function makes an HTTP DELETE request to a given URL.
 :
 : @param $href The URL to which the request will be made (see
 :  note above).
 : @return standard http-client return type.
 : @error error:HC001 An HTTP error occurred.
 : @error error:HC002 Error parsing the response content as XML.
 : @error error:HC006 A timeout occurred waiting for the response.
 :)
declare %ann:sequential function http:delete($href as xs:string) as item()+
{
  http:http-sequential-impl(
    validate {
      <http-schema:request method="DELETE" href="{$href}">
      </http-schema:request>
    }, (), ())
};

(:~
 : This function makes an HTTP POST request to a given URL. If the body
 : passed to this function is an element, it will be serialized to XML
 : to be sent to the server, and the Content-Type sent to the server will
 : be "text/xml". Otherwise, the body will be converted to
 : a plain string, and the Content-Type will be "text/plain".
 :
 : @param $href The URL to which the request will be made (see
 :  note above).
 : @param $body The body which will be sent to the server.
 : @return standard http-client return type.
 : @error error:HC001 An HTTP error occurred.
 : @error error:HC002 Error parsing the response content as XML.
 : @error error:HC006 A timeout occurred waiting for the response.
 :)
declare %ann:sequential function http:post($href as xs:string, $body as item()) as item()+
{
  variable $media-type as xs:string :=
    typeswitch($body)
      case xs:string return "text/plain"
      case element() return "text/xml"
      default return "text/plain";

  variable $result := http:post($href, $body, $media-type);

  $result
};

(:~
 : This function makes an HTTP POST request to a given URL. If the body
 : passed to this function is an element, it will be serialized
 : according to the $content-type parameter as follows:
 :

 :
If $content-type is "text/xml", "application/xml",
 : "text/xml-external-parsed-entity", or
 : "application/xml-external-parsed-entity", or if it ends with "+xml",
 : $body will be serialized to XML.

 :
If $content-type starts with "text/html", $body will be
 : serialized to HTML.

 :
Otherwise, $body will be serialized to text.

 :

 : If $body is not an element, $body will be serialized to text
 : regardless of $content-type.
 :
 :
In any case, Content-Type of the request sent to the server will
 : be $content-type.

 :
 : @param $href The URL to which the request will be made (see
 :  note above).
 : @param $body The body which will be sent to the server
 : @param $content-type The content type of the body as described above.
 : @return The first element of the result is the metadata (like
 :         headers, status etc), the next elements are the response
 : @error error:HC001 An HTTP error occurred.
 : @error error:HC002 Error parsing the response content as XML.
 : @error error:HC006 A timeout occurred waiting for the response.
 :)
declare %ann:sequential function http:post($href as xs:string, $body as item(), $content-type as xs:string) as item()+
{
  variable $method :=
    typeswitch ($body)
      case element() return
        if ($content-type = ("text/xml", "application/xml",
                                         "text/xml-external-parsed-entity",
                                         "application/xml-external-parsed-entity")
          or fn:ends-with($content-type, "+xml")) then
            "xml"
        else if (fn:starts-with($content-type, "text/html")) then
          "html"
        else if (fn:starts-with($content-type, "text/")) then
          "text"
        else
          "binary"
      case xs:base64Binary return "binary"
      default return "text";

  variable $result := http:http-sequential-impl(validate {
    <http-schema:request href="{$href}" method="POST">
      <http-schema:body media-type="{$content-type}" method="{$method}">{$body}</http-schema:body>
    </http-schema:request>}
    , (), ());

  $result
};

declare %private %ann:sequential function http:http-sequential-impl(
  $request as schema-element(http-schema:request)?,
  $href as xs:string?,
  $bodies as item()*) as item()+ external;

declare %private %ann:nondeterministic function http:http-nondeterministic-impl(
  $request as schema-element(http-schema:request)?,
  $href as xs:string?,
  $bodies as item()*) as item()+ external;

(:~
 : This function takes an http-schema:body element, copies it, and
 : adds a method attribute to the copy if there is none
 : in the original element.
 :
 : @param $body is the original http-schema:body element
 : @return a http-schema:body element with a corresponding @method
 :   attribute
 :)
declare %private function http:create-body (
  $body as element(http-schema:body))
    as element(http-schema:body) {
  <http-schema:body>{$body/@*}
  {
    if ($body/@method) then
      ()
    else
      attribute method {
        if ($body/@media-type eq "text/xml" or
            $body/@media-type eq "application/xml" or
            $body/@media-type eq "text/xml-external-parsed-entity" or
            $body/@media-type eq "application/xml-external-parsed-entity") then
              "xml"
         else if ($body/@media-type eq "text/html") then "html"
         else if (fn:starts-with($body/@media-type/data(.), "text/")) then "text"
         else "binary"
      }
  }
  {$body/node()}
  </http-schema:body>
};

(:~
 : This function takes an http-schema:multipart element, copies it and
 : adds a @method attribute to all body elements which don't have
 : one.
 :
 : @param $multipart the original http-schema:multipart
 : @return a copy of $multipart with all $multipart/body/@method set
 :)
declare %private function http:create-multipart (
  $multipart as element(http-schema:multipart))
    as element(http-schema:multipart) {
    <http-schema:multipart>{$multipart/@*}
    {
    for $x in $multipart/node()
    return
      typeswitch($x)
        case element(http-schema:body) return http:create-body($x)
        default return $x
    }
    </http-schema:multipart>
};

(:~
 : This adds a default method attribute to all body elements which
 : don't contain a method attribute.
 :
 : @param $request The request which need to be copied.
 : @return A copy of $request with all request//body/@method set.
 :)
declare %private function http:set-content-type(
  $request as element(http-schema:request)?)
    as element(http-schema:request)? {
  if ($request) then
    <http-schema:request>{$request/@*}
    {
    for $x in $request/node()
    return
      typeswitch($x)
        case element(http-schema:body) return http:create-body($x)
        case element(http-schema:multipart) return http:create-multipart($x)
        default return $x
    }
    </http-schema:request>
  else ()
};

(:~
 : Private function used internally by this module.
 :
 : This function checks if the request, href, and bodies parameters
 : are consistent.
 :
 : @param $request The request which needs to be checked.
 : @param $href The href which needs to be checked.
 : @param $bodies The bodies element which needs to be checked.
 : @return true if the parameters are consistent. Otherwise,
 :         this function raises an error.
 :)
declare %private function http:check-params(
  $request as element(http-schema:request)?,
  $href as xs:string?,
  $bodies as item()*) as xs:boolean {
  let $multipart := $request/http:multipart
  let $override := $request/@override-media-type/data(.)
  let $should-be-empty :=
    for $x in $request//http-schema:body
    return
      if ($x/@src and fn:not((fn:count($x/@*) eq 2))) then 1
      else ()
  return
    if (fn:empty($href) and fn:empty($request)) then
      fn:error($error:HC005, "The request element is not valid.")
    else if ($href eq "") then
      fn:error($error:HC005,
        "The request element is not valid.")
    else if (not(count($request//http-schema:body[not(exists(node())) and not(exists(@src))]) eq count($bodies))) then
      fn:error($error:HC005,
        "The request element is not valid.")
    else if ($should-be-empty) then
      fn:error($error:HC004, "The src attribute on the body element is mutually exclusive with all other attribute (except the media-type).")
    else
      fn:true()
};

xquery version "3.0";

(:
 : Copyright 2006-2009 The FLWOR Foundation.
 :
 : Licensed under the Apache License, Version 2.0 (the "License");
 : you may not use this file except in compliance with the License.
 : You may obtain a copy of the License at
 :
 : http://www.apache.org/licenses/LICENSE-2.0
 :
 : Unless required by applicable law or agreed to in writing, software
 : distributed under the License is distributed on an "AS IS" BASIS,
 : WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 : See the License for the specific language governing permissions and
 : limitations under the License.
:)

(:~
 :
Introduction

 :

 : This module provides provides simple functions for performing HTTP requests
 : (GET, POST, DELETE etc.), as well as a more flexible general
 : purpose function (send-request()).
 :

 :
 :
Examples of how to use this module:

 :
 :
Simple GET Request (retrieving text)

 :
 :
http:get-text( "www.example.com" )

 :
 : returns
 :
 :

 :   <response xmlns="http://expath.org/ns/http-client" status="200" message="OK">
 :     <header name="Content-Type" value="text/html; charset=UTF-8"/>
 :     <header name="Content-Length" value="574"/>
 :     ...
 :     <body media-type="text/html"/>
 :   </response>
 :   &lt;!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"&gt;
 :   &lt;html&gt;
 :     &lt;head&gt;
 :       &lt;meta http-equiv="Content-Type"
 :       content="text/html; charset=utf-8" /&gt;
 :       &lt;title&gt;Example Web Page</title&gt;
 :     &lt;/head&gt;
 :     &lt;body&gt;
 :       &lt;p&gt;You have reached this web page by typing "example.com",
 :       "example.net", or "example.org" into your web browser.&lt;/p&gt;
 :       &lt;p&gt;These domain names are reserved for use in documentation and are
 :       Not available for registration. See
 :       &lt;a href="http://www.rfc-editor.org/rfc/rfc2606.txt"&gt;RFC 2606&lt;/a&gt;,
 :       Section 3.&lt;/p&gt;
 :     &lt;/body&gt;
 :   &lt;/html&gt;
 :

 :
 :
Simple GET Request (retrieving XHTML)

 :
 :

 :   declare namespace xhtml="http://www.w3.org/1999/xhtml";
 :
 :   http:get-node( "www.w3.org" )[2]//xhtml:body
 :

 :
 :

 : This example shows how to retrieve an XHTML resource. Note that the path
 : expression that is looking for the "body" element in the result requires the xhtml
 : namespace to be specified.
 :

 :
 :
Simple POST Request

 :
 :

 : Here is a simple example which sends text content by making an HTTP POST
 : request.
 :

 :
 :

 :   http:post( "...", "Hello World" )
 :

 :
 :
 :
 :
Return Values

 :
 :
Most functions in this module (all except
 : options()) return one or more items.
 : (head() returns exactly one.) For all of these,
 : the first item returned will be a <http-schema:response>
 : element, as seen in the examples above. This element has "status" and
 : "message" attributes, representing the result of the HTTP call. It
 : also has any number of <http-schema:header> child elements that
 : encode the HTTP headers returned by the HTTP server. Finally, it
 : will generally contain a <http-schema:body> child element with
 : a "media-type" attribute that identifies the content-type of the
 : result data.

 :
 :
The full schema of this <http-schema:response> element is
 : part of the EXPath
 : HTTP Client module. You can see the schema
 : here.

 :
 :
Any items in function return values after the initial
 : <http-schema:response> element are the body/bodies of the HTTP
 : response from the server. (MIME Multi-part responses will have
 : more than one body.) The type of these items depends on the
 : Content-Type for each body. Each item will be:

 :
 :

 :

 :    an element node, if the returned content type is one of:
 :

 :
text/xml

 :
application/xml

 :
text/xml-external-parsed-entity

 :
application/xml-external-parsed-entity

 :
or if the Content-Type ends with "+xml".

 :

 :

 :

 :    an xs:string, if the content type starts with "text/" and does not match the
 :    above XML content types strings.
 :

 :
xs:base64Binary for all other content types.

 :

 :
 :
This return value - a sequence of items comprising one
 : <http-schema:response> element followed by zero or more
 : response items - is referred to as the "standard http-client
 : return type" in the function declarations below.

 :
 :
$href Arguments to Functions

 :
 : All functions in this module accept a URL argument named $href. In
 : all cases, the value passed to $href must be a valid xs:anyURI.
 : However, all functions declare $href to be of type xs:string. This
 : is for convenience, since you can pass a string literal value (that
 : is, a URL in double-quotes spelled out explicitly in your query)
 : to an xs:string parameter.
 :
 :
Important Notice Regarding get() Functions

 :
 : All of the get() functions in this module -
 : get(), get-node(),
 : get-text(), and
 : get-binary() - are declared to be
 : nondeterministic, which means that Zorba will not cache
 : their results. However, they are not declared to be
 : sequential, which means that Zorba may re-order them
 : as part of its query optimization. According to the HTTP RFC,
 : GET requests should only return data, and should not have any
 : side-effects. However, in practice it is not uncommon for GET
 : requests to have side-effects. If your application depends on
 : the ordering of side-effects from making GET requests, you should
 : either use the more complex send-request()
 : function (which is declared sequential), or alterately
 : wrap each call to get() in your own sequential function, to ensure
 : that Zorba does not place the GET requests out of order.
 :
 :
Relation to the EXPath http-client module

 :
 : EXPath defines its own http-client
 : module, which is available separately for Zorba as a non-core module.
 : There are two primary differences between EXPath's http-client and
 : Zorba's core http-client (this module):
 :
 :

 :
EXPath defines only the send-request() function, although it
 : does include convenient 1- and 2-argument forms in addition to the
 : full 3-argument form. EXPath does not include the simpler get(),
 : post(), put(), delete(), head(), and options() functions defined by
 : this module.

 :
EXPath specifies that all HTML content returned from the
 : HTTP server will be tidied up into valid XML, and then parsed
 : into an element. As this required an additional third-party library
 : dependency, Zorba's http-client module does not perform this tidying.
 : Instead, HTML content is returned as a string (with special XML
 : characters replaced with XML entity references, as shown in the
 : above examples).

 :

 :
 : See the full spec
 : of the EXPath http-client module for more information.
 :
 :
 : @author Markus Pilman
 : @see XQuery 1.1: Function Declaration
 : @library cURL Library
 : @project external
 :)
module namespace http = "http://www.zorba-xquery.com/modules/http-client";

import module namespace error = "http://expath.org/ns/error";

import schema namespace http-schema = "http://expath.org/ns/http-client";

declare namespace ann = "http://www.zorba-xquery.com/annotations";
declare namespace ver = "http://www.zorba-xquery.com/options/versioning";
declare namespace err = "http://www.w3.org/2005/xqt-errors";

declare option ver:module-version "2.0";

(:~
 : This function sends an HTTP request and returns the corresponding response.
 : Its inputs, outputs, and behavior are identical to the
 : EXPath http-client's
 : send-request() function (except that HTML responses are not tidied
 : into XML - see the note above). It
 : is provided here for use in Zorba installations that do not have
 : the EXPath module available. If you have the option of using the
 : EXPath module instead of this function, please do so, as it will
 : allow your application to be more interoperable between different
 : XQuery engines.
 :
 : Full documentation of the $request parameter can be found in
 : the EXPath
 : specification.
 :
 : @param $request Contains the various parameters of the request (see above).
 : @param $href The URL to which the request will be made (see
 :  note above). If this
 :  parameter is specified, it will override the "href" attribute of
 :  $request.
 : @param $bodies is the request body content, for HTTP methods that can
 :  contain a body in the request (i.e. POST and PUT). It is an error if this
 :  param is not the empty sequence for methods
 : @return standard http-client return type.
 : @error error:HC001 An HTTP error occurred.
 : @error error:HC002 Error parsing the response content as XML.
 : @error error:HC003 With a multipart response, the override-media-type must be either a multipart media type or application/octet-stream.
 : @error error:HC004 The src attribute on the body element is mutually exclusive with all other attribute (except the media-type).
 : @error error:HC005 The input request element is not valid.
 : @error error:HC006 A timeout occurred waiting for the response.
 :)
declare %ann:sequential function http:send-request(
  $request as element(http-schema:request)?,
  $href as xs:string?,
  $bodies as item()*) as item()+
{
  if (http:check-params($request, $href, $bodies))
  then
    {
      variable $req := if ($request)
                      then
                        try
                        {
                          validate { http:set-content-type($request) }
                        }
                        catch XQDY0027
                        {
                          fn:error($error:HC005, "The request element is not valid.")
                        }
                      else
                       ();

      variable $result := http:http-sequential-impl($req, $href, $bodies);

     $result
    }
  else
    ()
};

(:~
 : This function makes a GET request to a given URL.
 :
 : @see Notice about get() functions
 : @param $href The URL to which the request will be made (see
 :  note above).
 : @return standard http-client return type.
 : @error error:HC001 An HTTP error occurred.
 : @error error:HC002 Error parsing the response content as XML.
 : @error error:HC006 A timeout occurred waiting for the response.
 :)
declare %ann:nondeterministic function http:get($href as xs:string) as item()+
{
  http:http-nondeterministic-impl(validate {<http-schema:request method="GET" href="{$href}" follow-redirect="true"/>}, (), ())
};

(:~
 : This function makes a GET request to a given URL. All returned bodies
 : are forced to be interpreted as XML and parsed into elements.
 :
 : @see Notice about get() functions
 : @param $href The URL to which the request will be made (see
 :  note above).
 : @return standard http-client return type.
 : @error error:HC001 An HTTP error occurred.
 : @error error:HC002 Error parsing the response content as XML.
 : @error error:HC006 A timeout occurred waiting for the response.
 :)
declare %ann:nondeterministic function http:get-node($href as xs:string) as item()+
{
  http:http-nondeterministic-impl(validate {<http-schema:request method="GET" href="{$href}" follow-redirect="true" override-media-type="text/xml"/>}, (), ())
};

(:~
 : This function makes a GET request to a given URL. All returned bodies
 : are forced to be interpreted as plain strings, and will be returned
 : as xs:string items.
 :
 : @see Notice about get() functions
 : @param $href The URL to which the request will be made (see
 :  note above).
 : @return standard http-client return type.
 : @error error:HC001 An HTTP error occurred.
 : @error error:HC002 Error parsing the response content as XML.
 : @error error:HC006 A timeout occurred waiting for the response.
 :)
declare %ann:nondeterministic function http:get-text($href as xs:string) as item()+
{
  http:http-nondeterministic-impl(validate {<http-schema:request method="GET" href="{$href}" follow-redirect="true" override-media-type="text/plain"/>}, (), ())
};

(:~
 : This function makes a GET request on a given URL. All returned bodies
 : are forced to be interpreted as binary data, and will be returned
 : as xs:base64Binary items.
 :
 : @see Notice about get() functions
 : @param $href The URL to which the request will be made (see
 :  note above).
 : @return standard http-client return type.
 : @error error:HC001 An HTTP error occurred.
 : @error error:HC002 Error parsing the response content as XML.
 : @error error:HC006 A timeout occurred waiting for the response.
 :)
declare %ann:nondeterministic function http:get-binary($href as xs:string) as item()+
{
  http:http-nondeterministic-impl(validate {<http-schema:request method="GET" href="{$href}" follow-redirect="true" override-media-type="binary"/>}, (), ())
};

(:~
 : This function makes an HTTP HEAD request on a given URL.
 :
 : @param $href The URL to which the request will be made (see
 :  note above).
 : @return standard http-client return type
 :  (since HEAD never returns any body data, only the
 :  <http-schema:response> element will be returned).
 : @error error:HC001 An HTTP error occurred.
 : @error error:HC006 A timeout occurred waiting for the response.
 :)
declare %ann:nondeterministic function http:head($href as xs:string) as item() {
  http:http-nondeterministic-impl(
    validate {
      <http-schema:request method="HEAD" href="{$href}">
      </http-schema:request>
    }, (), ())
};

(:~
 : This function makes an HTTP OPTIONS request, which asks the server
 : which operations it supports.
 :
 : @param $href The URL to which the request will be made (see
 :  note above).
 : @return A sequence of xs:string values of the allowed operations.
 : @error error:HC001 An HTTP error occurred.
 : @error error:HC006 A timeout occurred waiting for the response.
 :)
declare %ann:nondeterministic function http:options($href as xs:string) as xs:string* {
  let $resp := http:http-nondeterministic-impl(
    validate {
      <http-schema:request method="OPTIONS" href="{$href}">
      </http-schema:request>
    }, (), ())[1]
  return
    fn:tokenize(fn:data($resp/http-schema:header[@name = "Allow"]/@value), ",")
};

(:~
 : This function makes an HTTP PUT request to a given URL. If the body
 : passed to this function is an element, it will be serialized to XML
 : to be sent to the server, and the Content-Type sent to the server will
 : be "text/xml". Otherwise, the body will be converted to
 : a plain string, and the Content-Type will be "text/plain".
 :
 : @param $href The URL to which the request will be made (see
 :  note above).
 : @param $body The body which will be sent to the server.
 : @return standard http-client return type.
 : @error error:HC001 An HTTP error occurred.
 : @error error:HC002 Error parsing the response content as XML.
 : @error error:HC006 A timeout occurred waiting for the response.
 :)
declare %ann:sequential function http:put($href as xs:string, $body as item()) as item()+
{
  variable $media-type as xs:string+ :=
    typeswitch($body)
      case xs:string return ("text/plain","text")
      case element() return ("text/xml", "xml")
      default return ("text/plain","xml");

  variable $result := http:put($href, $body, $media-type[1]);

  $result
};

(:~
 : This function makes an HTTP PUT request to a given URL. If the body
 : passed to this function is an element, it will be serialized
 : according to the $content-type parameter as follows:
 :

 :
If $content-type is "text/xml", "application/xml",
 : "text/xml-external-parsed-entity", or
 : "application/xml-external-parsed-entity", or if it ends with "+xml",
 : $body will be serialized to XML.

 :
If $content-type starts with "text/html", $body will be
 : serialized to HTML.

 :
Otherwise, $body will be serialized to text.

 :

 : If $body is not an element, $body will be serialized to text
 : regardless of $content-type.
 :
 :
In any case, Content-Type of the request sent to the server will
 : be $content-type.

 :
 : @param $href The URL to which the request will be made (see
 :  note above).
 : @param $body The body which will be sent to the server.
 : @param $content-type The content type of $body as described above.
 : @return standard http-client return type.
 : @error error:HC001 An HTTP error occurred.
 : @error error:HC002 Error parsing the response content as XML.
 : @error error:HC006 A timeout occurred waiting for the response.
 :)
declare %ann:sequential function http:put($href as xs:string, $body as item(), $content-type as xs:string) as item()+
{
  variable $method :=
    typeswitch ($body)
      case element() return
        if ($content-type = ("text/xml", "application/xml",
                                         "text/xml-external-parsed-entity",
                                         "application/xml-external-parsed-entity")
          or fn:ends-with($content-type, "+xml")) then
            "xml"
        else if (fn:starts-with($content-type, "text/html")) then
          "html"
        else if (fn:starts-with($content-type, "text/")) then
          "text"
        else
          "binary"
      case xs:base64Binary return "binary"
      default return "text";

  variable $result := http:http-sequential-impl(validate {
    <http-schema:request href="{$href}" method="PUT">
      <http-schema:body media-type="{$content-type}" method="{$method}">{$body}</http-schema:body>
    </http-schema:request>}
    , (), ());

  $result
};

(:~
 : This function makes an HTTP DELETE request to a given URL.
 :
 : @param $href The URL to which the request will be made (see
 :  note above).
 : @return standard http-client return type.
 : @error error:HC001 An HTTP error occurred.
 : @error error:HC002 Error parsing the response content as XML.
 : @error error:HC006 A timeout occurred waiting for the response.
 :)
declare %ann:sequential function http:delete($href as xs:string) as item()+
{
  http:http-sequential-impl(
    validate {
      <http-schema:request method="DELETE" href="{$href}">
      </http-schema:request>
    }, (), ())
};

(:~
 : This function makes an HTTP POST request to a given URL. If the body
 : passed to this function is an element, it will be serialized to XML
 : to be sent to the server, and the Content-Type sent to the server will
 : be "text/xml". Otherwise, the body will be converted to
 : a plain string, and the Content-Type will be "text/plain".
 :
 : @param $href The URL to which the request will be made (see
 :  note above).
 : @param $body The body which will be sent to the server.
 : @return standard http-client return type.
 : @error error:HC001 An HTTP error occurred.
 : @error error:HC002 Error parsing the response content as XML.
 : @error error:HC006 A timeout occurred waiting for the response.
 :)
declare %ann:sequential function http:post($href as xs:string, $body as item()) as item()+
{
  variable $media-type as xs:string :=
    typeswitch($body)
      case xs:string return "text/plain"
      case element() return "text/xml"
      default return "text/plain";

  variable $result := http:post($href, $body, $media-type);

  $result
};

(:~
 : This function makes an HTTP POST request to a given URL. If the body
 : passed to this function is an element, it will be serialized
 : according to the $content-type parameter as follows:
 :

 :
If $content-type is "text/xml", "application/xml",
 : "text/xml-external-parsed-entity", or
 : "application/xml-external-parsed-entity", or if it ends with "+xml",
 : $body will be serialized to XML.

 :
If $content-type starts with "text/html", $body will be
 : serialized to HTML.

 :
Otherwise, $body will be serialized to text.

 :

 : If $body is not an element, $body will be serialized to text
 : regardless of $content-type.
 :
 :
In any case, Content-Type of the request sent to the server will
 : be $content-type.

 :
 : @param $href The URL to which the request will be made (see
 :  note above).
 : @param $body The body which will be sent to the server
 : @param $content-type The content type of the body as described above.
 : @return The first element of the result is the metadata (like
 :         headers, status etc), the next elements are the response
 : @error error:HC001 An HTTP error occurred.
 : @error error:HC002 Error parsing the response content as XML.
 : @error error:HC006 A timeout occurred waiting for the response.
 :)
declare %ann:sequential function http:post($href as xs:string, $body as item(), $content-type as xs:string) as item()+
{
  variable $method :=
    typeswitch ($body)
      case element() return
        if ($content-type = ("text/xml", "application/xml",
                                         "text/xml-external-parsed-entity",
                                         "application/xml-external-parsed-entity")
          or fn:ends-with($content-type, "+xml")) then
            "xml"
        else if (fn:starts-with($content-type, "text/html")) then
          "html"
        else if (fn:starts-with($content-type, "text/")) then
          "text"
        else
          "binary"
      case xs:base64Binary return "binary"
      default return "text";

  variable $result := http:http-sequential-impl(validate {
    <http-schema:request href="{$href}" method="POST">
      <http-schema:body media-type="{$content-type}" method="{$method}">{$body}</http-schema:body>
    </http-schema:request>}
    , (), ());

  $result
};

declare %private %ann:sequential function http:http-sequential-impl(
  $request as schema-element(http-schema:request)?,
  $href as xs:string?,
  $bodies as item()*) as item()+ external;

declare %private %ann:nondeterministic function http:http-nondeterministic-impl(
  $request as schema-element(http-schema:request)?,
  $href as xs:string?,
  $bodies as item()*) as item()+ external;

(:~
 : This function takes an http-schema:body element, copies it, and
 : adds a method attribute to the copy if there is none
 : in the original element.
 :
 : @param $body is the original http-schema:body element
 : @return a http-schema:body element with a corresponding @method
 :   attribute
 :)
declare %private function http:create-body (
  $body as element(http-schema:body))
    as element(http-schema:body) {
  <http-schema:body>{$body/@*}
  {
    if ($body/@method) then
      ()
    else
      attribute method {
        if ($body/@media-type eq "text/xml" or
            $body/@media-type eq "application/xml" or
            $body/@media-type eq "text/xml-external-parsed-entity" or
            $body/@media-type eq "application/xml-external-parsed-entity") then
              "xml"
         else if ($body/@media-type eq "text/html") then "html"
         else if (fn:starts-with($body/@media-type/data(.), "text/")) then "text"
         else "binary"
      }
  }
  {$body/node()}
  </http-schema:body>
};

(:~
 : This function takes an http-schema:multipart element, copies it and
 : adds a @method attribute to all body elements which don't have
 : one.
 :
 : @param $multipart the original http-schema:multipart
 : @return a copy of $multipart with all $multipart/body/@method set
 :)
declare %private function http:create-multipart (
  $multipart as element(http-schema:multipart))
    as element(http-schema:multipart) {
    <http-schema:multipart>{$multipart/@*}
    {
    for $x in $multipart/node()
    return
      typeswitch($x)
        case element(http-schema:body) return http:create-body($x)
        default return $x
    }
    </http-schema:multipart>
};

(:~
 : This adds a default method attribute to all body elements which
 : don't contain a method attribute.
 :
 : @param $request The request which need to be copied.
 : @return A copy of $request with all request//body/@method set.
 :)
declare %private function http:set-content-type(
  $request as element(http-schema:request)?)
    as element(http-schema:request)? {
  if ($request) then
    <http-schema:request>{$request/@*}
    {
    for $x in $request/node()
    return
      typeswitch($x)
        case element(http-schema:body) return http:create-body($x)
        case element(http-schema:multipart) return http:create-multipart($x)
        default return $x
    }
    </http-schema:request>
  else ()
};

(:~
 : Private function used internally by this module.
 :
 : This function checks if the request, href, and bodies parameters
 : are consistent.
 :
 : @param $request The request which needs to be checked.
 : @param $href The href which needs to be checked.
 : @param $bodies The bodies element which needs to be checked.
 : @return true if the parameters are consistent. Otherwise,
 :         this function raises an error.
 :)
declare %private function http:check-params(
  $request as element(http-schema:request)?,
  $href as xs:string?,
  $bodies as item()*) as xs:boolean {
  let $multipart := $request/http:multipart
  let $override := $request/@override-media-type/data(.)
  let $should-be-empty :=
    for $x in $request//http-schema:body
    return
      if ($x/@src and fn:not((fn:count($x/@*) eq 2))) then 1
      else ()
  return
    if (fn:empty($href) and fn:empty($request)) then
      fn:error($error:HC005, "The request element is not valid.")
    else if ($href eq "") then
      fn:error($error:HC005,
        "The request element is not valid.")
    else if (not(count($request//http-schema:body[not(exists(node())) and not(exists(@src))]) eq count($bodies))) then
      fn:error($error:HC005,
        "The request element is not valid.")
    else if ($should-be-empty) then
      fn:error($error:HC004, "The src attribute on the body element is mutually exclusive with all other attribute (except the media-type).")
    else
      fn:true()
};