Automatically Extracting Web API Specifications from HTML Documentation
Jinqiu Yang Erik Wittern Annie T. T. Ying Julian Dolby Lin Tan
cs.SE








































































































Automatically Extracting Web API Specifications from HTML Documentation
    Jinqiu Yang1,

Erik Wittern2,
Annie T.T. Ying32,
Julian Dolby2,
Lin Tan1
1 University of Waterloo, Waterloo, Canada
2 IBM Research, Yorktown Heights, NY, USA
3 EquitySim, Vancouver, Canada
j223yang@uwaterloo.ca, witternj@us.ibm.com, annie.ying@gmail.com, dolby@us.ibm.com, lintan@uwaterloo.ca

    December 30, 2023
=========================================================================================================================================================================================================================================================================================================






Web API specifications are machine-readable descriptions of APIs. These specifications, in combination with related tooling, simplify and support the consumption of APIs.
However, despite the increased distribution of web APIs, specifications are rare and their creation and maintenance heavily relies on manual efforts by third parties.
In this paper, we propose an automatic approach and an associated tool called for extracting specifications from web API documentation pages.
Given a seed online documentation page on an API, first crawls all documentation pages on the API, and then uses a set of machine-learning techniques to extract the base URL, path templates, and HTTP methods – collectively describing the endpoints of an API.

We evaluated whether can accurately extract endpoints from documentation on 120 web APIs. The results showed that achieved a precision of  in identifying base URLs, a precision of  and a recall of  in generating path templates, and a precision of  and a recall of  in extracting HTTP methods. 
In addition, we found that was useful when applied to APIs with pre-existing API specifications:  revealed many inconsistencies between web API documentation and their corresponding publicly available specifications.  Thus, can be used by web API providers to keep documentation and specifications in synchronization.

















§ INTRODUCTION


Web Application Programming Interfaces (web APIs or simply APIs from hereon) provide applications remote, programmatic access to resources such as data or functionalities.
For application developers, the proliferation of such APIs provides tremendous opportunities. Applications can take advantage of vast amount of existing data, like location-based information from the Google Places API[<https://developers.google.com/places>], hook into established and global social networks, using for example Facebook's[<https://developers.facebook.com/docs/graph-api>] or Twitter's[<https://developer.twitter.com/en/docs/api-reference-index>] APIs, or outsource critical and hard to implement functionalities, like payment processing using the Stripe API.[<https://stripe.com/docs/api>]

To consume APIs, though, developers face numerous challenges <cit.>:
The need to find and select the APIs meeting their requirements, both from a functional and non-functional point-of-view <cit.>.
They need to familiarize with the capabilities provided by an API and how to invoke these capabilities, which typically involves studying HTML-based documentation pages that vary across APIs. Compared to using library APIs, for example of a Java library, consumers of web APIs do not have interface signatures readily available or accessible via development tools.
In addition, web APIs are under the control of independent providers who can change the API in a way that can break client code <cit.>.
Even for supposedly standardized notions such as the APIs' URL structures, HTTP methods, or HTTP status codes, the semantics and implementation styles differ across APIs <cit.>.

One attempt to mitigate these problems is to describe APIs in a well-defined way using web API specifications.[
  To Julian: Can you please review this footnoteWe acknowledge that the term specification relates to a much more comprehensive description of an application's or system's syntax and semantics.
 
]
  Web API specification formats, like the OpenAPI Specification <cit.> or the RESTful API Modeling Language (RAML) <cit.>, describe the URL templates, HTTP methods, headers, parameters, and input and output data required to interact with an API. Being machine-understandable, web API specifications are the basis for various tools that support API consumption:
specification are input for generating consistent API documentation pages[e.g., <https://editor.swagger.io/> or <https://github.com/Rebilly/ReDoc>],
they are used to catalog APIs[e.g., <https://apiharmony-open.mybluemix.net/> or <https://any-api.com>],
to auto-generate software development kits that wrap APIs in various languages[e.g., <https://swagger.io/swagger-codegen> or <https://apimatic.docs.apiary.io>],
or even to statically check client code for possible errors <cit.>.

Unfortunately, client developers cannot leverage these advantages from such tooling unless web API specifications are available.
In contrast to over 19 thousand APIs listed in ProgrammableWeb <cit.> (an authoritative directory of web APIs referred as “The Journal of the API Economy”[https://techcrunch.com/2013/08/07/veteran-news-editor-david-berlind-joins-programmable-web-the-journal-of-the-api-economy/]), fewer than a thousand specifications are publicly available on APIs.guru <cit.>, the largest publicly available directory of web API specifications.
  
To Eric: Do we know how many of the APIs.guru's specs are specs that from the API provided and how many are generated?


Annie, unfortunately information on how many specs are made available by API providers is not available. I propose to fall back to stating that APIs.guru depends on 1) manual efforts by the community and 2) API-specific scripts to generate and maintain specs (see also commented-out text above).

The goal of this research is to provide client developers or API catalogs access to a much larger number of specifications by extracting them from much more prevalent, semi-structured online documentation (typically in form of HTML pages).
Many software engineering researchers have looked into a similar problem but in the traditional library API context, namely, identifying Java method signatures from API documentation <cit.>.
These approaches share the assumption that method signatures or code elements being extracted are written in Java, adhering to the specific Java grammar and conventions. Approaches capable of extracting web API endpoint descriptions not only require an adjustment to the search pattern for web API endpoints, but also need to account for the two common but distinct styles of web API documentation:
an example-based style (e.g., the GitHub API documentation as shown in Figure <ref> uses an example-based style, where the base URL  and the path template  are embedded in free-form text and a  command) and a more structured, reference-based documentation style (e.g., the Instagram API, Figure <ref>).






In this paper, with these two distinct styles of web API documentation in mind, we propose an approach, and the associated tool , to automatically extract a web API specification – more specifically, endpoint descriptions consisting of a base URL, path templates, and HTTP methods – from API documentation pages containing free-form text and arbitrary HTML structures.
Given a seed documentation page for an API, first crawls the associated documentation pages and then uses a set of machine learning techniques to extract the base URL of an API (e.g., ), the path templates (possibly containing path parameters, e.g., ), and HTTP methods (e.g., , ).
More specifically, uses classifiers and a hierarchical clustering algorithm to extract a base URL and path templates for an API, and searches the context of a path template to infer the HTTP method.

We evaluated whether can accurately extract endpoints from documentation on 120 web APIs. The results showed that achieves a precision of  in identifying base URLs, a precision of  and a recall of  in generating path templates, and a precision of  and a recall of  in extracting HTTP methods. 
In addition, from an evaluation on APIs with pre-existing API specifications, we found that was also useful: revealed many inconsistencies between web API documentation and their corresponding publicly available specifications. Thus, can be used by web API providers to keep documentation and specifications in synchronization.


currently does not infer the structure of data being sent to or received from an API, nor HTTP headers. However, 
since most requests use the  method <cit.> and thus do not expect any request payload, in those cases, the combination of a base URL and path template already allow for a successful API invocation.

Extending our work to extract the structure of data, for example through schema inference of provided example response data, is future work.

In the remainder of this paper, we present our approach to extract API specifications from documentation using a combination of machine learning techniques in Section <ref>. We present an empirical evaluation in Section <ref>.  

We discuss threats in Section <ref> and related work in Section <ref> before concluding in Section <ref>.












§ APPROACH



In this section, we describe how combines machine learning classification and hierarchical clustering to extract significant parts of web API specifications from online documentation. 
focuses on extracting three components of API specifications: base URLs, path templates, and descriptions (i.e., HTTP methods).

The web API's base URL is essential in a web API specification: any URL of a Web API request must contain the base URL and the relative path of the corresponding endpoint.
More formally, a base URL is a common prefix of all URLs for web API invocations, excluding other URLs such as documentation pages.
In OpenAPI specifications, a base URL is constructed via three fields: a scheme (e.g., ), the host (e.g., ), and optionally a base path (e.g., ).
In many APIs (e.g., Instagram API), the base URL is the longest common prefix of all the URLs for invoking the web API.  However, for other APIs, such as Microsoft's The DevTest Labs Client API, the longest common prefix is  while the actual base URL is , because  is defined to be part of the endpoint paths.  Whether a base URL is indeed the longest common prefix is a design decision of the API provider.

A path template defines fixed components of a URL as well as ones to be instantiated dynamically. For example, in the path template , the part  is a path parameter that needs to be instantiated with a concrete value of a user ID before performing a request. 
A path parameter is typically denoted via enclosing brackets (i.e., “{}”, “[]”, “<>”, or “()”) or a prefix “:”.

focuses on one type of description associated with the path template: the HTTP method. It reflects the type of interaction to be performed on a resource exposed by a web API. While many web APIs long relied on  and , now a much broader spectrum of methods is used <cit.>. As proposed in related work, we denote every valid combination of a path template and an HTTP method as an endpoint of the API <cit.>.



combines a set of techniques to infer the base URL, path templates, and HTTP methods, given a seed documentation page.
Figure <ref> provides an overview.
In the first step, uses a simple crawling approach to obtain complete documentation sources for an API. The crawler, starting from the provided seed page, iteratively downloads all linked sub pages. For crawling, uses the headless browser Splash[<https://scrapinghub.com/splash/>] to execute any JavaScript on each page before downloading it, as this may impact the resulting HTML structure.
In order to extract the base URL (see Section <ref>), first extracts all candidate URLs that can represent a web API call from the crawled documentation pages;

next leverages machine learning classification to determine for each candidate URL whether or not it is likely to represent an invocation to the documented web API.  Finally, selects the longest common prefix of these URLs.
For the path templates (see Section <ref>), leverages the URLs likely to be invocations of the API and extracts the URL paths (the part of the URL after removing the base URL). From these paths, using an agglomerative hierarchical clustering algorithm, infers path templates by identifying path parameters and aggregating paths.
then finds the descriptions co-located with the URL paths in the documentation, from which it extracts the HTTP method(s) (see Section <ref>) that can be combined with each path template (forming endpoints). 







 §.§ Base URL Extraction


Identifying the base URL in online documentation is not as straightforward as searching for keywords or templates such as “The base URL is <base URL>”; 
base URLs are often not explicitly mentioned in the documentation.
Rather, base URLs are often included as part of depicted examples of web API requests, as in the case of the GitHub documentation shown in Figure <ref>.

Thus, 's approach is to infer the base URL from all the URLs provided in online documentation.


Step 1 - Extracting URLs
As a first step, extracts all candidate URLs in the documentation that represent web API calls.
These candidate URLs consist of standard URLs (according to the W3C definition <cit.>) and URLs containing path parameters enclosed in “”, “”, “”, or “”.
We do not include in this list URL links within  attributes of link tags, nor inside  tags: URLs that represent web API calls are one of the main content in a documentation page to be communicated to the readers; hence, such URLs tend to be rendered in the documentation rather than appear as links or in scripts.
Even excluding such links, some of the URLs in the candidate list may not represent web API calls, e.g., URLs of related or even unrelated resources. In fact, we studied a set of 15 web APIs[This data set of 15 web APIs, which we studied  to design , is independent of our evaluation set.] and found that 42% of the contained URLs are not invocations of web APIs.

Step 2 - Identifying URLs of web API calls
To filter out spurious URLs that do not represent web API calls, we use supervised machine learning classification to determine whether each URL from Step 1 is likely to represent a web API call.
For each URL,



generates two categories of features: properties of documentation pages and properties of the URL itself. 
The first category consists of four features related to properties of the documentation pages from which URLs were extracted:

  
  * : True, if the URL is part of the link text enclosed in  tags with the “” attribute.

  
  * : True, if the URL appears inside <code> tags, and false otherwise.

  
  * : True, if the URL is inside valid JSON within a pair of matched HTML tags.

  
  * : True, if the URL has the same host name as the URL of the documentation page itself, and false otherwise.


The second category consists of four features related to properties of the URL itself:

  
  * : True, if the URL contains query parameters which are denoted by  and/or . For example, in the URL https://api.github.com/.../issues?state=closed,  is a parameter with the value . URLs with parameters are more likely to depict web API calls.

  
  * : The number of conventions exhibited by the URL indicates whether it likely corresponds to a web API call. We include three conventions described in previous work <cit.>:
  (1) whether the URL contains the term ;
  (2) whether the URL contains the term ; and
  (3) whether the URL contains version related information, including the terms  or . For example, if a URL satisfies all three conventions, the value of this feature is 3.

  
  * : True, if the URL contains a path parameter denoted by enclosing brackets () or a colon prefix ().

  
  * : A categorical feature representing the return value from invoking a  command on the URL.  We determine into which of the following categories the return value of the command falls:
(1) is in JSON format (the URL likely corresponds to a web API request);
(2) contains authentication errors[We defined an authentication error to be indicated by a HTTP status code of either 401 or 407, or by a return message that contains the string “Invalid certificate”.] (the URL may correspond to a web request without the proper authentication);
(3) everything else (e.g., in XML/HTML format which likely corresponds to learning resources as opposed to web API requests).



We built the classification model based on 15 web API documentation. This data set is independent of our evaluation data set. We manually identify a base URL for each web API in order to label whether each URL is indeed one web API call in the training set. Note that this manual process is only needed in order to build the training set; we do not need to manually identify base URLs when we apply on web API documentation. In our evaluation, URLs are automatically labeled by machine learning classification. The feature vector of the URLs is automatically created by .

uses the support vector machine (SVM) classifier from scikit-learn <cit.> with the default parameters, both to train the model and to use the trained model in the evaluation as well. The trained model achieves an accuracy[Accuracy is the percentage of correctly labeled (whether are indeed web API calls or not) URLs out of all the URLs extracted from the documentation.] of 0.97 and a F1-measure of 0.97 from the 10-fold cross-validation.


Step 3 - Extracting longest common prefix
Finally, identifies the base URL by computing the longest common prefix of the URLs classified as likely depicting calls to the web API from step 2. This approach is based on the assumption that web API requests are the most frequent type of URLs rendered in the documentation. This assumption works well in practice, as the reported results in Section <ref> show. This step is necessary because although the classification achieves high accuracy (97%), there are still URLs that do not target the web API.




 §.§ Path Template Generation


Having identified an API's base URL as described in Section <ref>, we can use it to extract the path templates of the API.



Paths of endpoints are typically presented in a documentation page in one of two ways:

Absolute URLs describe the whole URL used to perform an API request, for example, .

When identifying base URLs, already extracts absolute URLs and can obtain paths of endpoints by truncating the already determined base URL.

Alternatively, documentation pages may only provide relative path components without the base URL, for example .
In this case, extracts relative paths based on heuristics (i.e., using regular expressions). In the experiment, we did not observe a significant number of false endpoint paths caused by this approach. From a manual analysis, we found that unlike URLs, which often include links to external resources, relative paths often describe API endpoints, since they are otherwise not very meaningful to a human. 

In addition, path parameters are denoted in two ways:
A path parameter can be denoted explicitly via enclosing syntactic constructs (e.g., “”, “”, “”, or “”) or by prefixing a path parameter using “”.
Other documentation pages implicitly indicate path parameters via an example style (e.g., the GitHub example in Figure 1), with URLs where parameters are instantiated. For example, in the URL , “alice” is an instantiated value of the path parameter . 
Identifying path parameters expressed syntactically is straight-forward, while identifying path parameters in the example based documentation pages requires an algorithm that can determine from the path examples which of the path segments are instantiated values.



We propose an iterative algorithm to infer whether a path segment is a fixed segment of an endpoint, a path parameter, or an instantiated value. The algorithm consists of two main ideas. First, it uses clustering to group paths that we infer to invoke the same endpoint. For example, if we found four paths in the documentation for an API:

 

 
 
 

 

the clustering algorithm groups the first two into one cluster and the last two into the second cluster.
From the first cluster, we know that  is an instantiated value of .
Second, in subsequent iterations, the algorithm then leverages the fact that  is an already inferred instantiated value to improve the clustering in the next iteration, marking both  and  as two instantiated values.

uses hierarchical agglomerative clustering <cit.>, as described in Algorithm <ref>.
Given a set of paths with the same number of segments, the goal is to group the paths so that paths in a cluster invoke the same endpoint. 
We begin with one data point (i.e., one path) per cluster (line 2 in
Algorithm <ref>). At each iteration (lines 4-10), we calculate the
distance among all the pair-wise clusters and picks the pair with the
shortest distance (line 5) to merge (line 8). For our implementation, the
distance function (Algorithm <ref>) considers two paths the
“closest” if they have exactly the same segments – each matching
concrete (i.e. not a path parameter) segment i gets one point (Algorithm <ref>, line 8). Because two paths can never invoke the same endpoint when they have a different number of segments, the distance of such a pair is infinite (Algorithm <ref>, line 5). If the j-th segment of a path is a path parameter, the distance function considers the segment a match on the j-th segment of any other paths of the same length, with a discounted point of 0.8 instead of 1 (Algorithm <ref>, line 8). The clustering algorithm stops when the next pair of clusters to merge has the distance larger than a threshold T (Algorithm <ref>, lines 6, 7, and 11).  In our implementation, the threshold is set to 1, meaning that we allow paths in a cluster to have a single path segment different from each other. 



To leverage the instantiated values that are already inferred, such as
 in the  cluster in the
example, we adapt the standard hierarchical agglomerative algorithm as follows (Algorithm <ref>): The algorithm keeps track of a list of instantiated values of the path parameters per API (line 9), and stops when no additional instantiated values are found from the function infer_parameter_value (lines 10 and 12).  Each iteration starts by updating the paths with the currently known instantiated values (lines 5-7).  These paths are the input to the hierarchical agglomerative clustering algorithm (line 8). Clustering is performed after updating the newly instantiated values because when new path parameters are identified, the similarities will be updated.  Within each cluster, new values of path parameters are inferred (line 10, the call to infer_parameter_value).  This adapted algorithm can correctly cluster the four paths into two endpoints:  and .







 §.§ HTTP Methods



In web API documentation, the paths (whether or not path parameters are denoted explicitly or not) are typically co-located with other valuable information that aims to extract, namely valid HTTP methods to use with a path template (, , ...). 
We call the context in which this information exists a description block of a path template.
In this section, we first describe how we locate the description block associated with a path template, and then how extracts the HTTP method. 

uses the URLs from the original documentation page that match with the inferred path templates as anchors in documentation pages (in HTML) to locate the scope of the description block.
If there are multiple URLs in the page that match the path template, combines the contexts of all the URLs as the description block of the path template.  locates a description block for each path template as follows. 

First, parses the documentation page into a DOM tree (Figure <ref>), with each node representing the rendered text from the fragment of the HTML page enclosed in a pair of matched tags.
Second, marks the nodes whose rendered text contains at least one URL that matches a path template as gray, and locates the description block for each of these nodes. More specifically, for each gray node, combines its description block by expanding to include (1) the siblings of the node (starting with the closest siblings); then (2) the parent of the node. In (1), assuming the node is the n-th child of its parent, the expansion starts from n-1 to 0 and then n+1 to the farthest sibling; if a sibling is an ancestor of any other gray node, the expansion terminates entirely. This termination condition applies to 2) as well. The expansion stops if the parent is an ancestor of a node. An example description block is marked by the gray box in Figure <ref>.

In this work, focuses on extracting HTTP methods for each path template.






Having determined description blocks, searches in them for the seven possible method names, namely, , , , , , , and . If none of these names is found, uses  as the default value for method of endpoint, as the GET method is the most popular method for web APIs <cit.>.




§ EVALUATION


In this section, we present the evaluation of . We aim to answer two questions: 

RQ1: How accurately can extract web API specifications from documentation?
We are interested in a quantitative perspective on how accurately extracts base URL, path templates, and HTTP methods from the documentation.

RQ2: 
Can be used to identify inconsistencies between a pre-existing API specification and the API's documentation, pointing to the two being out of synchronization?
Some web APIs do have the associated specifications readily available. Third parties such as <cit.> (Section <ref>) curate specifications for many popular, public APIs. For other public or private APIs, the API providers themselves generate specifications. However, in either case, specifications can become out of synchronization with the documentation if, for example, a provider routinely maintains the one but not the other. We want to evaluate how well can detect mismatches between documentation and corresponding specifications.




 §.§ Data Collection


To address the two research questions, we need two types of information.
For RQ1, we need API documentation pages to which we can apply . We can then manually assess how accurate the resulting specifications are as compared to the documentation pages.
To address RQ2, we need APIs that feature both documentation pages and existing specifications. After applying to the documentation pages, we can assess whether the extracted specifications are in sync with the existing ones. We obtained these two types of information from two sources:

APIs.guru <cit.> is a third-party effort that curates OpenAPI specifications of popular public APIs. These specifications are either created and maintained through manual community effort, or using API-specific scripts, which translate other specification formats to the OpenAPI specification or are hard-coded to generate specifications from API-specific documentation pages. When we performed the evaluation, APIs.guru hosted specifications, each describing a different API with a unique base URL. Most of these APIs are provided by Google (127 APIs) or Microsoft (22 APIs), while the remaining 87 APIs are provided by other individual providers. We found documentation pages for all of the Google APIs, for none of the Microsoft APIs, and for of the remaining APIs. If we were to construct the evaluation dataset for the machine learning model by including all APIs for which we found documentation pages, we risk unfairly evaluating the model that biases the Google APIs, whose documentation pages have the same HTML structure while the documentation pages from the other individual providers have different structures among them. Ultimately, we include randomly selected Google APIs in our evaluation. Overall, from APIs.guru, the APIs used in the evaluation consist of APIs from the individual providers (we will call this set GuruIndividual) and Google APIs (we will call this set GuruGoogleSample), thus a total of 64 APIs.

API Harmony <cit.> is a catalog of web APIs that helps developers to find and choose web APIs, and learn how to use them. API Harmony collects information on public web APIs. When we performed the evaluation, API Harmony listed 1,019 web APIs in total, 772 of which contained links to the API's documentation page. We crawled the links to these documentation pages with the help of API Harmony's sitemap.xml file.

From these two sources, collectively, we obtained unique APIs – 681 from API Harmony (from 772, we excluded 91 APIs

that overlapped with APIs.guru). For RQ1 and RQ2, we will examine a subset of the as follows:



  * The GuruIndividual dataset consists of the APIs in APIs.guru that are not from Google or Microsoft.

  * The GuruGoogleSample dataset consists of the Google APIs from APIs.guru.

  * The HarmonySample dataset consists of a sample of 56 APIs from the 681 APIs from API Harmony. Section <ref> will describe how this sample was obtained.





 §.§ RQ1: Can accurately extract web API specifications from documentation?



Approach: To assess the accuracy of , we aim to determine how well the produced specifications match the input online documentation. 




To increase the generalizability of our results, we performed the evaluation in two stages:
First, we applied to all 64 APIs obtained from APIs.guru (GuruIndividual and GuruGoogleSample). These APIs do not contain the 15 APIs we used to train (see Section <ref>). We decided to use all APIs from APIs.guru in this evaluation because the required manual examination of them is also required to conduct the second research question, and there are limited APIs in APIs.guru to study. 
Second, we performed the evaluation on HarmonySample, which is a completely separate data source from APIs.guru. The results for these APIs thus better quantify how well can potentially generalize to other API documentation pages.
To create HarmonySample, we randomly selected 56 APIs from 560 APIs obtained from API Harmony for which was able to generate specifications – for the remaining 121 (681 – 560) APIs, failed to generate specifications because either no base URLs are described in documentation or determines all URLs in documentation are not web API invocations by classification (Section <ref>). We limited the number of APIs in HarmonySample because the evaluation requires significant manual effort.

Overall, for RQ1, we considered 120 APIs (GuruIndividual + GuruGoogleSample + HarmonySample). For all of them, we manually identified base URLs, path templates and HTTP methods from web API documentation. We then compared the manually extracted information with the specifications created by for the same API.

For base URLs, we calculated precision, which is the percentage the base URLs generated by that are correct. Since each API documentation describes only one base URL, and by design generates one base URL for each API documentation, recall is equal to precision for base URLs.

For path templates and HTTP methods, we consider precision to be the percentage the results generated by that are correct and recall to be the percentage of the given information type (e.g., path templates) in the documentation that correctly generates. Because path templates and HTTP methods can only be extracted if a base URL was previously detected (see Sections <ref> and <ref>), we focus on APIs for which was able to do so in these parts of the evaluation.


Results: recovered base URLs with a precision of , inferred path templates with a precision of  and a recall of , and extracted HTTP methods with a precision of  and a recall of . 

Table <ref> provides a break-down of the results.





  §.§.§ Base URL Results

When manually examining the documentation pages from the 120 APIs, we found that for eight APIs, the pages did not contain any base URLs. Obviously, could not generate correct base URLs from such documentation pages and we declared that is not applicable for such cases. Thus, the eight APIs are excluded in the precision calculation for base URLs extraction. For the remaining 112 web APIs, generated correct base URLs for 98 of them, yielding a precision of .  In the subsequent evaluation for path templates and HTTP methods for RQ1, the evaluation was based on the 98 APIs.


Upon manual inspection, we found that there were two reasons that generated incorrect base URLs.
First, when the documentation described multiple API versions, was unable to tell which one was preferred by the writer of the documentation. For example, in the documentation of the CityContext web API, two endpoints were described with  and . determined the base URL to be  by selecting the longest common prefix of these two URLs. However, the official documentation listed  as base URL.
Second, although the classification achieved a good precision, it is unable to remove all URLs that are not web API requests. Such URLs with the same prefix caused to generate incorrect base URLs when they outnumbered the true web API requests.



  §.§.§ Path Template Results
 was able to generate the majority ( recall) of path templates correctly ( precision) for the 98 web APIs whose base URLs are correctly identified by . There were in total  path templates described in the documentation. generated  path templates in total, and  of them were correct. Thus, the overall precision of path template extraction was , and the recall was . Figure <ref> illustrates stacked histograms on precision and recall of the path template extraction on the 98 APIs that can generate correct base URLs for. 

For example, Figure <ref> shows that for 57 (out of 98) web APIs, achieves a precision above 90%.





  §.§.§ HTTP Method Results

achieved a precision of  and a recall of  in extracting HTTP methods for the path templates in the evaluated 98 web APIs. In total, there were  endpoints with the associated HTTP methods described in the web API documentation evaluated; produced a result for  of them and  HTTP methods were correct.

failed to locate the correct HTTP method when its position in the documentation was far away from the path templates. For example, the Mandrillapp documentation has a consolidated description for all endpoints: “All API calls should be made with HTTP POST”, instead of listing the method  individually for each of the path template. Thus, failed to identify correct method names for Mandrillapp's path templates.





 §.§ RQ2: Can be used to identify inconsistencies between a pre-existing API specification and the API's documentation, pointing to the two being out of synchronization?


Approach: We focused on the 64 APIs from APIs.guru (GuruIndividual + GuruGoogleSample). For these APIs, we compared the specifications generated by (from hereon denoted as ToolSpecs) with the specifications provided by APIs.guru (from hereon denoted as GuruSpecs). Our comparison focused, again, on the three pieces of information extracted by , namely, base URLs, path templates, and HTTP methods.
For base URLs, we compared whether the ones extracted by per API match those defined in the OpenAPI specifications. We obtained base URLs from the specifications by concatenating the , , and  fields.
We then compared whether the extracted path templates and the associated HTTP methods match the ones in the specifications.


For each of the three extracted pieces of information, we counted the number of matches, and then manually inspected the mismatches to determine their origin.


Results: We found that mismatches between GuruSpecs and ToolSpecs were partly caused by limitations of , and partly by publicly-available specifications (i.e., the GuruSpecs) being out of synchronization with API documentation:

Our manual inspection showed that for base URLs and HTTP methods, GuruSpecs were often up-to-date with documentation, and mismatches between ToolSpecs and GuruSpecs were due to inaccuracies of .

However, for path templates, our manual inspection found that many mismatches were due to the documentation and GuruSpecs being out of synchronization with each other, or due to errors in the documentation. Specifically, for the 64 APIs evaluated, we identified 385 path templates from 21 APIs where GuruSpecs and the documentation were different. One reason for the mismatches is that as web APIs evolve, API providers tend to keep documentation up-to-date since it is, as a human-readable medium, often the first source that developers inquire to use APIs.
In the following, we present the results of manually examining the mismatches between path templates in GuruSpecs and ToolSpecs. We found that mismatches fall into four categories:






  
  * Inconsistencies were mismatches resulting from the documentation and specification in GuruSpecs being inconsistent with each other. Such inconsistencies were not caused by deficiencies of or by errors in the documentation, but indicated that the API provider should either update the documentation or the specification. For example, in the documentation of Slack, there were eight endpoints on getting information on members from a given Slack team. The paths of all eight endpoints start with /users.<action>. However, only one path template was listed in the specification–/users.list; the remaining seven (e.g., /users.info and /users.setPresence) were missing in the specification. In this case, we considered that there were seven inconsistencies between the documentation and the specification.

  
  * Errors in the documentation referred to obvious errors in the documentation, e.g., typos. Incorrect information in the documentation led to generate path templates that, while being labeled as correct with regard to RQ1, did not match the specifications. For example, in the documentation of the ClickMeter API, many path templates starting with /datatpoints were misspelled as /datapoints. Thus, generated several mismatched path templates compared to the official specification.

  
  * Partially correct path templates occurred if failed to infer path parameters correctly (i.e., the path templates generated by still contain path parameter values). A common reason for this problem was that the documentation contained only one instance of an instantiated value for a path parameter. In such cases, even though 's clustering-based algorithm can correctly place the path in its own cluster, could not distinguish which segments of the path were instantiated values and which ones are fixed segments.

  
  * Deficiencies in the algorithm occurred when failed to extract certain path templates or generated incorrect path templates because of deficiencies in its design. For instance, failed to extract certain path templates if the way the path templates appeared in the documentation was beyond the scope of the conventions used by . relies on the format of URL and relative path to extract path template information. If the path templates are not presented as such, will not extract them correctly. For example, HealthCare.gov's documentation describes a series of path templates as follows: “The following content types are available: articles, blog, questions, glossary, states, and topics. The request structure is https://.../api/:content-type.json.” extracts one path template–/api/{content-type. json} instead of six path templates (e.g., /api/articles and /api/blog). On the other hand, using the conventions mentioned above, also extracted false path templates which did not describe path templates in the documentation. For example, the documentation of dweet.io listed a file path – /play/definitions, which was not a true path template.


Figure <ref> visualizes the comparison of path templates from GuruSpecs and ToolSpecs. The breakdown of the mismatches from both aspects (ToolSpecs∩GuruSpecs and ToolSpecs∩GuruSpecs) is shown as well. In total, there are 862 path template matches between ToolSpecs and GuruSpecs. Among the 1407 path templates generated by , there are 545 mismatches with the path templates defined in the GuruSpecs. Our manual analysis shows that 385 (70.6%) of the mismatches are caused by de-synchronization, i.e., “inconsistencies” and “errors in the doc.”. The other two categories – “partially correct” and “deficiencies” – are due to limitations of .

Overall, while the manual examination of mismatches also pointed to some weaknesses of , it also highlights that can be used to find documentation and existing specifications being out of synchronization. To focus on this aspect, Figure <ref> shows a histogram on the percentage of mismatches that are caused by de-synchronization for each web API. It shows that, for example, for 10 web APIs, over 90% of the mismatches detected by indicate that documentation and pre-existing specifications were out of synchronization with each other.






































§ THREATS TO VALIDITY AND DISCUSSIONS



Generalizability. generates base URL from documentation by firstly identifying URLs that represent web API calls through a classification algorithm. The classification algorithm uses a set of pre-labeled URLs for training. We built the training set from a set of web API documentation that were independent of the ones used in the evaluation. The precision of in base URL extraction may be different if we use a different training set. However, we mitigated this bias by choosing a random set of web APIs for building the training set, and evaluating on a different set of APIs (the GuruIndividual and GuruGoogleSample datasets described in Section <ref>). In addition, we evaluated on a completely separate dataset (HarmonySample) and even achieved a better precision (97.5%) compared to a 80.0% precision on GuruGoogle and a 84.1% precision on GuruIndividual, demonstrating that our approach is likely to generalize to other unseen documentation pages. 

Thresholds Used in the Clustering-Based Path Template Extraction. leverages an iterative clustering-based algorithm to identify path parameters by inferring values of path parameters from similar web API calls. The proposed algorithm contains thresholds to control the hierarchical clustering (e.g., determining whether two web API calls are similar through a threshold T, see Algorithm <ref> in Section <ref>). In this evaluation, we set the thresholds based on our observations on the training set. We found that the chosen thresholds also worked well for the evaluation set. Nevertheless, future studies should investigate the effects of different thresholds on the path template extraction results. 
If increases T, the precision and recall of extracting path templates may change. 
For example, as T is increased, more web API calls that are less similar would be clustered together to infer more values of path parameters. If the inferred path parameter values are correct, the precision and recall will increase, and decrease if otherwise. 
In contrast, if T is decreased, less web API calls would be clustered together; thus less path parameter values would be inferred. If such missing path parameter values are actually incorrect, the precision and recall would increase, and decrease if otherwise. 


Documentation with Identical Structures. APIs from the same providers may have identical documentation structures (e.g., Google web APIs). Documentation structures may be different across different API providers. 
To show the generalizability of our approach, we applied our approach to APIs from different providers: Our evaluation set contains 120 APIs from 98 different web API providers. 




§ RELATED WORK



We discuss related work in using information extraction approaches in obtaining software or other types of entities, and other methods of inferring web API specifications.


Many software engineering researchers have looked into the problem of identifying code elements—more specifically, Java code elements such as method signatures and calls—from API documentation.
Dagenais and Robillard proposed an approach that extracts code elements from API documentation and links the elements to an index of known code elements, i.e., signatures from a Java library <cit.>.  
Subramanian et al. subsequently applied this approach to identify code elements on Stack Overflow posts and augmented the code elements in the posts with links to their official JavaDoc <cit.>. 
Rigby and Robillard use a light-weight, regular expression based approach to identify code elements that relaxes the requirement on a known index <cit.>.
Another line of work focuses on extracting more complex specifications on code entities from natural language descriptions. Pandita et al. <cit.> extract method pre-conditions and post-conditions from natural language API documentation.
Lin et al. <cit.> extract code contracts from comments and statically check for violations in the code.
Our work differs in two ways. First, we extract web API endpoints and related information as opposed to code elements. Second, there is arguably greater value in our recovered index (i.e., OpenAPI Specifications) because such an index is often not available or known to the clients; while clients of Java libraries (or other statically-typed languages) are always exposed at least to method signatures, but callers of web APIs often do not have such information.


There have been many efforts in information extraction on web pages <cit.>. 
For example, techniques for extracting product information from e-commerce sites <cit.> leverage the structure from the sites: the sites' organizational structure usually consists of a search page and a set of individual product pages, which typically have the same structure as they are generated from scripts.  These techniques exploit this common structure across the pages within the same site.

However, for extracting endpoints and other information from web API documentation pages, we cannot rely on such an assumption: There is no standard structure for API documentation.
For many API documentation the content is semi-structured at best, written by humans using free-form text and/or diverse HTML structures.
For example, the GitHub API documentation uses an example-based style, where the base URL  and the path template  are embedded in free-form text and a  command.
Other documentation uses a more structured, reference-based documentation style.



There have been other attempts at extracting OpenAPI Specifications. Wittern and Suter use dynamic traces in form of web-server logs <cit.>. Or, the SpyREST tool intercepts HTTP requests to an API using a proxy and then attempts to infer the API documentation <cit.>. Our work differs in that we use a different source of input, namely API documentation, as opposed to dynamic traces or observed traffic. The benefit of our approach is that API documentation is meant to be public while access to web logs are limited to those with access to the private web servers, and proxying may not be an option.





§ CONCLUSION


In this paper, we presented , a tool which extracts parts of web API specifications from documentation, including base URLs, path templates, and HTTP methods. is based on the three assumptions: (1) documentation includes multiple web API URLs (so that a base URL can be extracted); (2) path templates are either denoted explicitly (e.g., using brackets) or that multiple example URLs for paths exist from which templates can be inferred; and (3) descriptions close to the path templates contain information about HTTP methods. Our evaluation of shows that these assumptions hold mostly true when it comes to extracting base URLs, path templates, and HTTP methods. It furthermore shows that is not only useful for creating specifications from scratch, but also for checking existing ones for consistency with documentation. In the future, we aim to expand the scope of to also extract information on data structures, HTTP headers, and authentication methods.




abbrv



10

scikit
scikit-learn, 2016.
http://scikit-learn.org/stable/index.html.

url_definition
URL - W3C, 2016.
https://www.w3.org/TR/url-1/.

apiharmony
Api harmony, 2017.
https://apiharmony-open.mybluemix.net.

apiguru
Apis.guru - wikipedia for web apis, 2018.
https://apis.guru/openapi-directory.

swagger
Openapi specification, 2018.
https://github.com/OAI/OpenAPI-Specification.

ProgrammableWeb
Programmableweb api directory, 2018.
https://www.programmableweb.com/apis/directory/.

RAML
Restful api modeling language (raml), 2018.
https://raml.org/.

alvarez2008extracting
M. Álvarez, A. Pan, J. Raposo, F. Bellas, and F. Cacheda.
Extracting lists of data records from semi-structured web pages.
Data & Knowledge Engineering, 64(2):491–509, 2008.

crescenzi2001roadrunner
V. Crescenzi, G. Mecca, P. Merialdo, et al.
Roadrunner: Towards automatic data extraction from large web sites.
In Proceedings of the International Conference of Very Large
  Data Bases, pages 109–118, 2001.

recover2012
B. Dagenais and M. P. Robillard.
Recovering Traceability Links Between an API and Its Learning
  Resources.
In Proceedings of the 34th International Conference on Software
  Engineering, pages 47–57. IEEE Press, 2012.

Espinha:2015
T. Espinha, A. Zaidman, and H.-G. Gross.
Web api growing pains: Loosely coupled yet strongly tied.
Journal of Systems and Software, 100:27 – 43, 2015.

ferrara2014web
E. Ferrara, P. De Meo, G. Fiumara, and R. Baumgartner.
Web data extraction, applications and techniques: a survey.
Knowledge-based systems, 70:301–323, 2014.

laender2002brief
A. H. Laender, B. A. Ribeiro-Neto, A. S. da Silva, and J. S. Teixeira.
A brief survey of web data extraction tools.
ACM Sigmod Record, 31(2):84–93, 2002.

Li:2013
J. Li, Y. Xiong, X. Liu, and L. Zhang.
How does web service api evolution affect clients?
In 2013 IEEE 20th International Conference on Web Services
  (ICWS), pages 300–307, June 2013.

manning2009introduction
Manning.
Introduction to Information Retrieval.
Cambridge Press, 2009.

myllymaki2002effective
J. Myllymaki.
Effective web data extraction with standard XML technologies.
Computer Networks, 39(5):635–644, 2002.

pandita2012
R. Pandita, X. Xiao, H. Zhong, T. Xie, S. Oney, and A. Paradkar.
Inferring Method Specifications from Natural Language API
  Descriptions.
In Proceedings of the 34th International Conference on Software
  Engineering, pages 815–825, 2012.

rigby2013discovering
P. C. Rigby and M. P. Robillard.
Discovering essential code elements in informal documentation.
In Proceedings of the 35th International Conference on Software
  Engineering, pages 832–841, 2013.

Rodriguez:2016
C. Rodríguez, M. Baez, F. Daniel, F. Casati, J. C. Trabucco, L. Canali,
  and G. Percannella.
REST APIs: A Large-Scale Analysis of Compliance with Principles and
  Best Practices.
In Web Engineering, pages 21–39. Springer International
  Publishing, Cham, June 2016.

Sohan:2015dp
S. M. Sohan, C. Anslow, and F. Maurer.
SpyREST: Automated RESTful API Documentation Using an HTTP Proxy
  Server.
In Proceedings of the 30th IEEE/ACM International Conference on
  Automated Software Engineering, pages 271–276, 2015.

subramanian2014live
S. Subramanian, L. Inozemtseva, and R. Holmes.
Live API documentation.
In Proceedings of the 36th International Conference on Software
  Engineering, pages 643–652. ACM, 2014.

suter2015inferring
P. Suter and E. Wittern.
Inferring web API descriptions from usage data.
In Proceedings of the Third IEEE Workshop on Hot Topics in Web
  Systems and Technologies, pages 7–12, 2015.

icomment
L. Tan, D. Yuan, G. Krishna, and Y. Zhou.
/*Icomment: Bugs or Bad Comments?*/.
SIGOPS Oper. Syst. Rev., 41(6):145–158, Oct. 2007.

Wittern:2016
E. Wittern, V. Muthusamy, J. A. Laredo, M. Vukovic, A. A. Slominski,
  S. Rajagopalan, H. Jamjoom, and A. Natarajan.
Api harmony: Graph-based search and selection of apis in the cloud.
IBM Journal of Research and Development, 60(2-3):12:1–12:11,
  March 2016.

Wittern:2017
E. Wittern, A. T. T. Ying, Y. Zheng, J. Dolby, and J. A. Laredo.
Statically checking web API requests in JavaScript.
To appear in the 39th International Conference on Software
  Engineering, 2017.
http://arxiv.org/abs/1702.03906.

Wittern:2017b
E. Wittern, A. T. T. Ying, Y. Zheng, J. A. Laredo, J. Dolby, C. C. Young, and
  A. A. Slominski.
Opportunities in software engineering research for web api
  consumption.
In 2017 IEEE/ACM 1st International Workshop on API Usage and
  Evolution (WAPI), pages 7–10, May 2017.

zhai2005web
Y. Zhai and B. Liu.
Web data extraction based on partial tree alignment.
In Proceedings of the International Conference on World Wide
  Web, pages 76–85. ACM, 2005.

zhai2007extracting
Y. Zhai and B. Liu.
Extracting web data using instance-based learning.
World Wide Web, 10(2):113–132, 2007.


































