使用 requests 访问 Python 包索引（PyPI）的 JSON API
Python PyPI

> 
> PyPI 的 JSON API 是一种机器可直接使用的数据源，你可以访问和你浏览网站时相同类型的数据。
> 
> 
> 


![](/data/attachment/album/202105/03/111943du0lgbjj6br6sruu.jpg "Python programming language logo with question marks")


PyPI（Python 软件包索引）提供了有关其软件包信息的 JSON API。本质上，它是机器可以直接使用的数据源，与你在网站上直接访问是一样的的。例如，作为人类，我可以在浏览器中打开 [Numpy](https://pypi.org/project/numpy/) 项目页面，点击左侧相关链接，查看有哪些版本，哪些文件可用以及发行日期和支持的 Python 版本等内容：


![NumPy project page](/data/attachment/album/202105/03/112014gwknqnfnauunun6w.png "NumPy project page")


但是，如果我想编写一个程序来访问此数据，则可以使用 JSON API，而不必在这些页面上抓取和解析 HTML。


顺便说一句：在旧的 PyPI 网站上，还托管在 `pypi.python.org` 时，NumPy 的项目页面位于 `pypi.python.org/pypi/numpy`，访问其 JSON API 也很简单，只需要在最后面添加一个 `/json` ，即 `https://pypi.org/pypi/numpy/json`。现在，PyPI 网站托管在 `pypi.org`，NumPy 的项目页面是 `pypi.org/project/numpy`。新站点不会有单独的 JSON API URL，但它仍像以前一样工作。因此，你不必在 URL 后添加 `/json`，只要记住 URL 就够了。


你可以在浏览器中打开 NumPy 的 JSON API URL，Firefox 很好地渲染了数据：


![JSON rendered in Firefox](/data/attachment/album/202105/03/112014ze99pm77pmerzpr8.png "JSON rendered in Firefox")


你可以查看 `info`，`release` 和 `urls` 其中的内容。或者，你可以将其加载到 Python Shell 中，以下是几行入门教程：



```
import requests
url = "https://pypi.org/pypi/numpy/json"
r = requests.get(url)
data = r.json()

```

获得数据后（调用 `.json()` 提供了该数据的 [字典](https://docs.python.org/3/tutorial/datastructures.html#dictionaries)），你可以对其进行查看：


![Inspecting data](/data/attachment/album/202105/03/112014zzr80jv8yj5vyx0r.png "Inspecting data")


查看 `release` 中的键：


![Inspecting keys in releases](/data/attachment/album/202105/03/112014rcc5b8xc8vbscc6c.png "Inspecting keys in releases")


这表明 `release` 是一个以版本号为键的字典。选择一个并查看以下内容：


![Inspecting version](/data/attachment/album/202105/03/112015g9a4mxgaxmfnnla9.png "Inspecting version")


每个版本都包含一个列表，`release` 包含 24 项。但是每个项目是什么？由于它是一个列表，因此你可以索引第一项并进行查看：


![Indexing an item](/data/attachment/album/202105/03/112015u05ktukzcw16k0zc.png "Indexing an item")


这是一个字典，其中包含有关特定文件的详细信息。因此，列表中的 24 个项目中的每一个都与此特定版本号关联的文件相关，即在 <https://pypi.org/project/numpy/1.20.1/#files> 列出的 24 个文件。


你可以编写一个脚本在可用数据中查找内容。例如，以下的循环查找带有 sdist（源代码包）的版本，它们指定了 `requires_python` 属性并进行打印：



```
for version, files in data['releases'].items():
    for f in files:
        if f.get('packagetype') == 'sdist' and f.get('requires_python'):
            print(version, f['requires_python'])

```

![sdist files with requires_python attribute](/data/attachment/album/202105/03/112015lzuqnbu95ll52qlu.png "sdist files with requires_python attribute ")


### piwheels


去年，我在 piwheels 网站上[实现了类似的 API](https://blog.piwheels.org/requires-python-support-new-project-page-layout-and-a-new-json-api/)。[piwheels.org](https://www.piwheels.org/) 是一个 Python 软件包索引，为树莓派架构提供了 wheel（预编译的二进制软件包）。它本质上是 PyPI 软件包的镜像，但带有 Arm wheel，而不是软件包维护者上传到 PyPI 的文件。


由于 piwheels 模仿了 PyPI 的 URL 结构，因此你可以将项目页面 URL 的 `pypi.org` 部分更改为 `piwheels.org`。它将向你显示类似的项目页面，其中详细说明了构建的版本和可用的文件。由于我喜欢旧站点允许你在 URL 末尾添加 `/json` 的方式，所以我也支持这种方式。NumPy 在 PyPI 上的项目页面为 [pypi.org/project/numpy](https://pypi.org/project/numpy)，在 piwheels 上，它是 [piwheels.org/project/numpy](https://www.piwheels.org/project/numpy)，而 JSON API 是 [piwheels.org/project/numpy/json](https://www.piwheels.org/project/numpy/json) 页面。


没有必要重复 PyPI API 的内容，所以我们提供了 piwheels 上可用内容的信息，包括所有已知发行版的列表，一些基本信息以及我们拥有的文件列表：


![JSON files available in piwheels](/data/attachment/album/202105/03/112016nfk00450ukq75kku.png "JSON files available in piwheels")


与之前的 PyPI 例子类似，你可以创建一个脚本来分析 API 内容。例如，对于每个 NumPy 版本，其中有多少 piwheels 文件：



```
import requests

url = "https://www.piwheels.org/project/numpy/json"
package = requests.get(url).json()

for version, info in package['releases'].items():
    if info['files']:
        print('{}: {} files'.format(version, len(info['files'])))
    else:
        print('{}: No files'.format(version))

```

此外，每个文件都包含一些元数据：


![Metadata in JSON files in piwheels](/data/attachment/album/202105/03/112016j2a9mo2io23xab6b.png "Metadata in JSON files in piwheels")


方便的是 `apt_dependencies` 字段，它列出了使用该库所需的 Apt 软件包。本例中的 NumPy 文件，或者通过 `pip` 安装 Numpy，你还需要使用 Debian 的 `apt` 包管理器安装 `libatlas3-base` 和 `libgfortran`。


以下是一个示例脚本，显示了程序包的 Apt 依赖关系：



```
import requests

def get_install(package, abi):
    url = 'https://piwheels.org/project/{}/json'.format(package)
    r = requests.get(url)
    data = r.json()
    for version, release in sorted(data['releases'].items(), reverse=True):
        for filename, file in release['files'].items():
            if abi in filename:
                deps = ' '.join(file['apt_dependencies'])
                print("sudo apt install {}".format(deps))
                print("sudo pip3 install {}=={}".format(package, version))
                return

get_install('opencv-python', 'cp37m')
get_install('opencv-python', 'cp35m')
get_install('opencv-python-headless', 'cp37m')
get_install('opencv-python-headless', 'cp35m')

```

我们还为软件包列表提供了一个通用的 API 入口，其中包括每个软件包的下载统计：



```
import requests

url = "https://www.piwheels.org/packages.json"
packages = requests.get(url).json()
packages = {
    pkg: (d_month, d_all)
    for pkg, d_month, d_all, \*_ in packages
}

package = 'numpy'
d_month, d_all = packages[package]

print(package, "has had", d_month, "downloads in the last month")
print(package, "has had", d_all, "downloads in total")

```

### pip search


`pip search` 因为其 XMLRPC 接口过载而被禁用，因此人们一直在寻找替代方法。你可以使用 piwheels 的 JSON API 来搜索软件包名称，因为软件包的集合是相同的：



```
#!/usr/bin/python3
import sys

import requests

PIWHEELS_URL = 'https://www.piwheels.org/packages.json'

r = requests.get(PIWHEELS_URL)
packages = {p[0] for p in r.json()}

def search(term):
    for pkg in packages:
        if term in pkg:
            yield pkg

if __name__ == '__main__':
    if len(sys.argv) == 2:
        results = search(sys.argv[1].lower())
        for res in results:
            print(res)
    else:
        print("Usage: pip_search TERM")

```

有关更多信息，参考 piwheels 的 [JSON API 文档](https://www.piwheels.org/json.html).




---


*本文最初发表在 Ben Nuttall 的 [Tooling Tuesday 博客上](https://tooling.bennuttall.com/accessing-python-package-index-json-apis-with-requests/)，经许可转载使用。*




---


via: <https://opensource.com/article/21/3/python-package-index-json-apis-requests>


作者：[Ben Nuttall](https://opensource.com/users/bennuttall) 选题：[lujun9972](https://github.com/lujun9972) 译者：[MjSeven](https://github.com/MjSeven) 校对：[wxy](https://github.com/wxy)


本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出


OK
PyPI, the Python package index, provides a JSON API for information about its packages. This is essentially a machine-readable source of the same kind of data you can access while browsing the website. For example, as a human, I can head to the [ NumPy](https://pypi.org/project/numpy/) project page in my browser, click around, and see which versions there are, what files are available, and things like release dates and which Python versions are supported:

![NumPy project page NumPy project page](https://opensource.com/sites/default/files/uploads/numpy-project-page.png)

(Ben Nuttall, CC BY-SA 4.0)

But if I want to write a program to access this data, I can use the JSON API instead of having to scrape and parse the HTML on these pages.

As an aside: On the old PyPI website, when it was hosted at `pypi.python.org`

, the NumPy project page was at `pypi.python.org/pypi/numpy`

, and accessing the JSON was a simple matter of adding a `/json`

on the end, hence `https://pypi.org/pypi/numpy/json`

. Now the PyPI website is hosted at `pypi.org`

, and NumPy's project page is at `pypi.org/project/numpy`

. The new site doesn't include rendering the JSON, but it still runs as it was before. So now, rather than adding `/json`

to the URL, you have to remember the URL where they are.

You can open up the JSON for NumPy in your browser by heading to its URL. Firefox renders it nicely like this:

![JSON rendered in Firefox JSON rendered in Firefox](https://opensource.com/sites/default/files/uploads/pypi-json-firefox.png)

(Ben Nuttall, CC BY-SA 4.0)

You can open `info`

, `releases`

, and `urls`

to inspect the contents within. Or you can load it into a Python shell. Here are a few lines to get started:

```
import requests
url = "https://pypi.org/pypi/numpy/json"
r = requests.get(url)
data = r.json()
```


Once you have the data (calling `.json()`

provides a [ dictionary](https://docs.python.org/3/tutorial/datastructures.html#dictionaries) of the data), you can inspect it:

![Inspecting data Inspecting data](https://opensource.com/sites/default/files/uploads/pypi-json-notebook.png)

(Ben Nuttall, CC BY-SA 4.0)

Open `releases`

, and inspect the keys inside it:

![Inspecting releases Inspecting keys in releases](https://opensource.com/sites/default/files/uploads/pypi-json-releases.png)

(Ben Nuttall, CC BY-SA 4.0)

This shows that `releases`

is a dictionary with version numbers as keys. Pick one (say, the latest one) and inspect that:

![Inspecting version Inspecting version](https://opensource.com/sites/default/files/uploads/pypi-json-inspect.png)

(Ben Nuttall, CC BY-SA 4.0)

Each release is a list, and this one contains 24 items. But what is each item? Since it's a list, you can index the first one and take a look:

![Indexing an item](https://opensource.com/sites/default/files/uploads/pypi-json-release.png)

(Ben Nuttall, CC BY-SA 4.0)

This item is a dictionary containing details about a particular file. So each of the 24 items in the list relates to a file associated with this particular version number, i.e., the 24 files listed at [ https://pypi.org/project/numpy/1.20.1/#files](https://pypi.org/project/numpy/1.20.1/#files).

You could write a script that looks for something within the available data. For example, the following loop looks for versions with sdist (source distribution) files that specify a `requires_python`

attribute and prints them:

```
for version, files in data['releases'].items():
for f in files:
if f.get('packagetype') == 'sdist' and f.get('requires_python'):
print(version, f['requires_python'])
```



![sdist files with requires_python attribute sdist files with requires_python attribute](https://opensource.com/sites/default/files/uploads/pypi-json-requires-python.png)

(Ben Nuttall, CC BY-SA 4.0)

## piwheels

Last year I [ implemented a similar API](https://blog.piwheels.org/requires-python-support-new-project-page-layout-and-a-new-json-api/) on the piwheels website. [ piwheels.org](https://www.piwheels.org/) is a Python package index that provides wheels (precompiled binary packages) for the Raspberry Pi architecture. It's essentially a mirror of the package set on PyPI, but with Arm wheels instead of files uploaded to PyPI by package maintainers.

Since piwheels mimics the URL structure of PyPI, you can change the `pypi.org`

part of a project page's URL to `piwheels.org`

. It'll show you a similar kind of project page with details about which versions we have built and which files are available. Since I liked how the old site allowed you to add `/json`

to the end of the URL, I made ours work that way, so NumPy's project page on PyPI is [ pypi.org/project/numpy](https://pypi.org/project/numpy). On piwheels, it is [ piwheels.org/project/numpy](https://www.piwheels.org/project/numpy), and the JSON is at [ piwheels.org/project/numpy/json](https://www.piwheels.org/project/numpy/json).

There's no need to duplicate the contents of PyPI's API, so we provide information about what's available on piwheels and include a list of all known releases, some basic information, and a list of files we have:

![JSON files available in piwheels JSON files available in piwheels](https://opensource.com/sites/default/files/uploads/piwheels-json.png)

(Ben Nuttall, CC BY-SA 4.0)

Similar to the previous PyPI example, you could create a script to analyze the API contents, for example, to show the number of files piwheels has for each version of NumPy:

```
import requests
url = "https://www.piwheels.org/project/numpy/json"
package = requests.get(url).json()
for version, info in package['releases'].items():
if info['files']:
print('{}: {} files'.format(version, len(info['files'])))
else:
print('{}: No files'.format(version))
```


Also, each file contains some metadata:

![Metadata in JSON files in piwheels Metadata in JSON files in piwheels](https://opensource.com/sites/default/files/uploads/piwheels-json-numpy.png)

(Ben Nuttall, CC BY-SA 4.0)

One handy thing is the `apt_dependencies`

field, which lists the Apt packages needed to use the library. In the case of this NumPy file, as well as installing NumPy with pip, you'll also need to install `libatlas3-base`

and `libgfortran`

using Debian's Apt package manager.

Here is an example script that shows the Apt dependencies for a package:

```
import requests
def get_install(package, abi):
url = 'https://piwheels.org/project/{}/json'.format(package)
r = requests.get(url)
data = r.json()
for version, release in sorted(data['releases'].items(), reverse=True):
for filename, file in release['files'].items():
if abi in filename:
deps = ' '.join(file['apt_dependencies'])
print("sudo apt install {}".format(deps))
print("sudo pip3 install {}=={}".format(package, version))
return
get_install('opencv-python', 'cp37m')
get_install('opencv-python', 'cp35m')
get_install('opencv-python-headless', 'cp37m')
get_install('opencv-python-headless', 'cp35m')
```


We also provide a general API endpoint for the list of packages, which includes download stats for each package:

```
import requests
url = "https://www.piwheels.org/packages.json"
packages = requests.get(url).json()
packages = {
pkg: (d_month, d_all)
for pkg, d_month, d_all, *_ in packages
}
package = 'numpy'
d_month, d_all = packages[package]
print(package, "has had", d_month, "downloads in the last month")
print(package, "has had", d_all, "downloads in total")
```


## pip search

Since `pip search`

is currently disabled due to its XMLRPC interface being overloaded, people have been looking for alternatives. You can use the piwheels JSON API to search for package names instead since the set of packages is the same:

```
#!/usr/bin/python3
import sys
import requests
PIWHEELS_URL = 'https://www.piwheels.org/packages.json'
r = requests.get(PIWHEELS_URL)
packages = {p[0] for p in r.json()}
def search(term):
for pkg in packages:
if term in pkg:
yield pkg
if __name__ == '__main__':
if len(sys.argv) == 2:
results = search(sys.argv[1].lower())
for res in results:
print(res)
else:
print("Usage: pip_search TERM")
```


For more information, see the piwheels [ JSON API documentation](https://www.piwheels.org/json.html).

*This article originally appeared on Ben Nuttall's Tooling Tuesday blog and is reused with permission.*

## 1 Comment