使用 Sourcegraph 搜索 34000 多个 Fedora 仓库
Sourcegraph 代码搜索 软件仓库
![](/data/attachment/album/202204/05/151255yuq9zem6euei5ui5.jpg)


在 2021 年 10 月，一个 Fedora Linux 用户 [问了一个关于许可的问题](https://lists.fedoraproject.org/archives/list/legal@lists.fedoraproject.org/thread/CBCJHOSP36YXQKCVGWVL5MXU64LZ6NZA/)。Fedora 项目负责人 Matthew Miller [回复道](https://lists.fedoraproject.org/archives/list/legal@lists.fedoraproject.org/message/LTIQS2PX33FSCEIAPJS62UZXVPDT5JPB/)：“我不能快速地检查（这个问题），因为我们没有一个完整的、庞大的、可搜索的存储库来检索 Fedora 中所有软件包。”


[接着他说](https://lists.fedoraproject.org/archives/list/legal@lists.fedoraproject.org/message/5GEPBSRGUK5E2FLW4MQBVP6DI65XP2LQ/)：“……或许我们可以付钱给 Sourcegraph，让他们帮我们做这个。他们看起来乐于助人。” 他说的没错，我们（Sourcegraph）的确是乐于助人，而且我们还不想要你的钱，相反，我们想与 Fedora 社区合作。


Fedora 社区现在可以在他们的开源代码世界中尽情搜索 —— 目前有超过 34,000 个存储库，而且还在不断增加。


### 代码搜索简介


如果你还不熟悉 [代码搜索](https://codesearchguide.org/) 的概念，我现在就来告诉你。代码搜索可以让团队更快地掌握一个新的代码库，在里面找到答案，帮助团队识别安全风险，以及许多其他用例。Sourcegraph 已经在 GitHub 和 GitLab 等多个代码托管服务上，索引了 200 多万个存储库。本文只关注 [src.fedoraproject.org](http://src.fedoraproject.org) 的代码搜索。Sourcegraph 同时提供了一个 [Web 应用](https://sourcegraph.com/search?q=context:global+repo:%5Esrc.fedoraproject.org/&patternType=regexp) 和 [命令行客户端](https://docs.sourcegraph.com/cli/quickstart)。


### 使用 Web 应用


在使用 Sourcegraph [Web 应用](https://sourcegraph.com/search?q=context:global+repo:%5Esrc.fedoraproject.org/&patternType=regexp) 时，你需要先输入初始字符串 `repo:^src.fedoraprojects.org`，然后再开始查询。这个 [Web 应用](https://sourcegraph.com/search?q=context:global+repo:%5Esrc.fedoraproject.org/&patternType=regexp) 链接中包括了上面的初始字符串，点击这个链接后，搜索界面如下图所示：


![Sourcegraph Web 应用界面](/data/attachment/album/202204/05/151330xhtmqmt09bd7ymv0.png)


下面我将提供几个使用 Web 应用程序进行搜索的例子，大家可能会对它们感兴趣。


#### 查找使用流行的经 OSI 批准的许可证的存储库


下面的查询语句将扫描所有兼容 “<ruby> 开源定义 <rt>  Open Source Definition </rt></ruby>”（OSD） 的软件存储库。



```
repo:^src.fedoraproject.org/ lang:"RPM Spec" License: ^.*apache|bsd|gpl|lgpl|mit|mpl|cddl|epl.*$

```

![许可证搜索](/data/attachment/album/202204/05/151331eqyncmqz88h0it3u.png)



> 
> [试一下！](https://sourcegraph.com/search?q=context:global+repo:%5Esrc.fedoraproject.org/+lang:%22RPM+Spec%22+License:+%5E.*apache%7Cbsd%7Cgpl%7Clgpl%7Cmit%7Cmpl%7Ccddl%7Cepl.*%24&patternType=regexp)
> 
> 
> 


#### 查找带有 TODO 的文件


下面的查询语句将在 34,000 多个仓库中找到 `TODO` 文件。对于那些希望为需要帮助的项目做出贡献的人来说，是一个非常棒的功能。



```
repo:^src.fedoraproject.org/ "TODO"

```

![搜索 TODO](/data/attachment/album/202204/05/151332oopo65asyouyt0ds.png)



> 
> [试一下！](https://sourcegraph.com/search?q=context:global+repo:%5Esrc.fedoraproject.org/+%22TODO%22&patternType=regexp&case=yes)
> 
> 
> 


#### 查找 FTP 服务器上的文件


我的一个前同事告诉我 “FTP 是一个死协议”。真的是这样吗？你也可以在这个查询中加入任何其他协议，如 irc、https 等。



```
repo:^src.fedoraproject.org/ (?:ftp)://[A-Za-z0-9-]{0,63}(.[A-Za-z0-9-]{0,63})+(:d{1,4})?/*(/*[A-Za-z0-9-._]+/*)*(?.*)?(#.*)?

```

![搜索协议](/data/attachment/album/202204/05/151333kbbs444rvsw4f9nc.png)



> 
> [试一下！](https://sourcegraph.com/search?q=context:global+repo:%5Esrc.fedoraproject.org/+%28%3F:ftp%29:%5C/%5C/%5BA-Za-z0-9%5C-%5D%7B0%2C63%7D%28%5C.%5BA-Za-z0-9%5C-%5D%7B0%2C63%7D%29%2B%28:%5Cd%7B1%2C4%7D%29%3F%5C/*%28%5C/*%5BA-Za-z0-9%5C-._%5D%2B%5C/*%29*%28%5C%3F.*%29%3F%28%23.*%29%3F&patternType=regexp)
> 
> 
> 


#### 查找使用有漏洞的 Log4j 版本的文件


这个查询语句将找到任何可能存在 CVE-2021-44228（也就是 Log4j）漏洞的文件（可能会有误报）。你也可以搜索其他漏洞，然后报告给项目维护者。



```
repo:^src.fedoraproject.org/ org.apache.logging.log4j 2.((0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15)(.[0-9]+)) count:all

```

![搜索 log4j](/data/attachment/album/202204/05/151333w7oh2szkxzwz2uhd.png)



> 
> [试一下！](https://sourcegraph.com/search?q=context:global+repo:%5Esrc.fedoraproject.org/+org%5C.apache%5C.logging%5C.log4j+2.%28%280%7C1%7C2%7C3%7C4%7C5%7C6%7C7%7C8%7C9%7C10%7C11%7C12%7C13%7C14%7C15%29%28%5C.%5B0-9%5D%2B%29%29+count:all&patternType=regexp)
> 
> 
> 


### 使用命令行


Sourcegraph 还有一个叫做 [src](https://github.com/sourcegraph/src-cli#readme) 的命令行客户端，它可以让你完成我刚才提到的所有事情。此外，它还有其他一些有用的命令。比如说，它可以把结果用 JSON 格式输出，方便你在编程中使用。



```
src search -json 'repo:^src.fedoraproject.org/ lang:"RPM Spec" License: ^.*apache|bsd|gpl|lgpl|mit|mpl|cddl|epl.*$'

```

#### 输出 JSON


![输出 JSON](/data/attachment/album/202204/05/151334m2i3ajj6e6gs8t62.png)



> 
> [试一下！](https://sourcegraph.com/notebooks/Tm90ZWJvb2s6MzQ2)
> 
> 
> 


### 搜索语法


就入门而言，上面的例子是很好的起点，但 Sourcegraph 还支持更多的查询语句。你可以 [查看所有的搜索查询语法](https://docs.sourcegraph.com/code_search/reference/queries)，并根据需要创建你自己的查询语句。


### 总结


正如你所看到的，有了 Sourcegraph，Fedora Linux 社区现在可以快速搜索托管在 [src.fedoraproject.org](https://src.fedoraproject.org/) 上的所有代码，无论是使用普通查询还是复杂的正则查询。


感谢 Fedora Linux 社区的慷慨帮助和热情欢迎。如果你有任何想补充的内容或问题，我和我的团队都会在下面的评论区回复。你也可以 [在 Slack 上找到我们](https://srcgr.ph/wp-join-community-space)。


特别感谢 [Vanesa Ortiz](https://twitter.com/vanesacodes) 促成了这次合作，还有 [Ben Venker](https://handbook.sourcegraph.com/team/#ben-venker) 帮助修复了我的正则表达式（多次），以及 [Rebecca Dodd](https://handbook.sourcegraph.com/team/#rebecca-dodd) 和 [Nick Moore](https://twitter.com/nickwritesit) 在编辑上的帮助。




---


via: <https://fedoramagazine.org/using-sourcegraph-to-search-34000-fedora-repositories/>


作者：[Justin Dorfman](https://fedoramagazine.org/author/jdorfman/) 选题：[lujun9972](https://github.com/lujun9972) 译者：[lkxed](https://github.com/lkxed) 校对：[wxy](https://github.com/wxy)


本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出


OK
In October 2021, a Fedora Linux user [asked a question about licensing](https://lists.fedoraproject.org/archives/list/legal@lists.fedoraproject.org/thread/CBCJHOSP36YXQKCVGWVL5MXU64LZ6NZA/). Fedora Project Leader Matthew Miller [left a response](https://lists.fedoraproject.org/archives/list/legal@lists.fedoraproject.org/message/LTIQS2PX33FSCEIAPJS62UZXVPDT5JPB/): “Since we don’t have a complete, exploded, searchable repository of all of the packages in Fedora, I don’t have a quick way to check.”

[Followed by](https://lists.fedoraproject.org/archives/list/legal@lists.fedoraproject.org/message/5GEPBSRGUK5E2FLW4MQBVP6DI65XP2LQ/): “…or possibly pay Sourcegraph to do it for us. They seem like nice people.” He is correct, we (Sourcegraph) *are* nice people, but we don’t want your money. Instead, we wanted to team up with the Fedora community.

The Fedora Community can now search their universe of open source code—currently over 34,000 repositories and counting.

## Introduction to code search

For those who aren’t familiar with the concept of [code search](https://codesearchguide.org/), it enables teams to onboard to a new codebase and find answers faster, helps to identify security risks, and many other use cases. Sourcegraph has indexed over two-million repositories across multiple code hosts such as GitHub and GitLab. **This article is going to focus strictly on code search for src.fedoraproject.org. **Sourcegraph provides both a

[web app](https://sourcegraph.com/search?q=context:global+repo:%5Esrc.fedoraproject.org/&patternType=regexp)and

[CLI](https://docs.sourcegraph.com/cli/quickstart)interface.

## Using the Web app

When using the Sourcegraph [web app](https://sourcegraph.com/search?q=context:global+repo:%5Esrc.fedoraproject.org/&patternType=regexp) you will need to start each search with **repo:^src.fedoraprojects.org** before entering any search queries. Using this link to the [web app](https://sourcegraph.com/search?q=context:global+repo:%5Esrc.fedoraproject.org/&patternType=regexp) will include this initial string as shown here:

![](https://fedoramagazine.org/wp-content/uploads/2022/03/Image-2022-03-28-at-1.41.54-PM-1024x335.png)


![](https://fedoramagazine.org/wp-content/uploads/2022/03/Image-2022-03-28-at-1.41.54-PM-1024x335.png)

The following sections will provide some web app examples of searches that might be of interest.

### Find repositories using popular OSI-approved licenses

The following query will scan all the repositories for software that is compatible with the “Open Source Definition” (OSD).

repo:^src.fedoraproject.org/ lang:"RPM Spec" License: ^.*apache|bsd|gpl|lgpl|mit|mpl|cddl|epl.*$

![](https://fedoramagazine.org/wp-content/uploads/2022/03/license-1024x513.png)


![](https://fedoramagazine.org/wp-content/uploads/2022/03/license-1024x513.png)

### Find files with TODOs

The following query can find TODOs in 34k repositories. This is great for those looking to contribute to projects that need help.

repo:^src.fedoraproject.org/ "TODO"

![](https://fedoramagazine.org/wp-content/uploads/2022/03/todo-1024x605.png)


![](https://fedoramagazine.org/wp-content/uploads/2022/03/todo-1024x605.png)

### Find files being served via FTP

A co-worker of mine from back in the day told me “FTP is a dead protocol”. Is it? You can add to this query to find any other protocol such as irc, https, etc.

repo:^src.fedoraproject.org/ (?:ftp)://[A-Za-z0-9-]{0,63}(.[A-Za-z0-9-]{0,63})+(:d{1,4})?/*(/*[A-Za-z0-9-._]+/*)*(?.*)?(#.*)?

![](https://fedoramagazine.org/wp-content/uploads/2022/03/protocol-1024x457.png)


![](https://fedoramagazine.org/wp-content/uploads/2022/03/protocol-1024x457.png)

### Find files with a vulnerable version of Log4j

This query will find any files that are possibly vulnerable (false positives can happen) to CVE-2021-44228 aka Log4j. You can also search for other vulnerabilities that can then be reported to project maintainers.

repo:^src.fedoraproject.org/ org.apache.logging.log4j 2.((0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15)(.[0-9]+)) count:all

![](https://fedoramagazine.org/wp-content/uploads/2022/03/log4j-1024x295.png)


![](https://fedoramagazine.org/wp-content/uploads/2022/03/log4j-1024x295.png)

## Use the CLI

Sourcegraph also has a command-line interface tool called [src](https://github.com/sourcegraph/src-cli#readme), which allows you to do everything I just mentioned above, plus other useful commands like getting results in JSON for programmatic consumption.

src search -json 'repo:^src.fedoraproject.org/ lang:"RPM Spec" License: ^.*apache|bsd|g pl|lgpl|mit|mpl|cddl|epl.*$'

### JSON output

![](https://fedoramagazine.org/wp-content/uploads/2022/03/Image-2022-03-22-at-9.46.26-AM-1024x521.png)


![](https://fedoramagazine.org/wp-content/uploads/2022/03/Image-2022-03-22-at-9.46.26-AM-1024x521.png)

## Search Syntax

The examples shown may be a good starting point but are by no means the only queries that may be made. You can [view all search query syntaxes](https://docs.sourcegraph.com/code_search/reference/queries) and create your own as needed.

## Conclusion

As you can see, with Sourcegraph, the Fedora Linux community can now quickly search for all code hosted at [src.fedoraproject.org](https://src.fedoraproject.org/), regardless of whether they are literal or complex regex queries.

I appreciate the Fedora Linux community being so helpful and welcoming. If you have anything you want to add or questions, my team and I will be in the comments section below. You can also [join us on Slack](https://srcgr.ph/wp-join-community-space).

Special thanks to [Vanesa Ortiz](https://twitter.com/vanesacodes) for [making this collaboration happen](https://discussion.fedoraproject.org/t/fedora-sourcegraph-marketing-community-collaboration/36151), [Ben Venker](https://handbook.sourcegraph.com/team/#ben-venker) for his help fixing my broken regex (multiple times), as well as [Rebecca Dodd](https://handbook.sourcegraph.com/team/#rebecca-dodd) and [Nick Moore](https://twitter.com/nickwritesit) for their help with editing.

## wefrwe

how download (rsync, wget curl etc) all package to disc?

for install offline/ update

## Justin Dorfman

Try this:

--publish 7080:7080 --publish 127.0.0.1:3370:3370 --rm

--volume ~/.sourcegraph/config:/etc/sourcegraph

--volume ~/.sourcegraph/data:/var/opt/sourcegraph

sourcegraph/server:3.37.0

## Darvond

Neat. Seems like something the Debian Project could use, given the recent accusations of poor maintainership that have been heard across the winds.

https://unixsheikh.com/articles/the-delusions-of-debian.html

The arduous TL;DR being that Debian’s packaging has more abandoned, unmaintained, and orphaned than would initially be lead on. This creates headaches for everyone.

## Justin Dorfman

Thanks for pointing this out, I was not aware that the Debian community was struggling. Do you think it’s worth reaching out?

## Darvond

Sure, why not. It’d at the least be an option for them to consider.

## Michael Gruber

First of all thanks for this great service!

src.fedoraproject.org hosts

packagingsources, that is spec files, source identifiers (archive name and sha256 checksum), patches and possibly other “small items” checked into “dist-git” as we call it. Being able to search them in one place is quite useful already. (plus the tests namespace …)This does not include the actual

packagesource (upstream source) code as for the indexed GitHub and GitLab repos. This is somewhat more complicated (and more than somewhat more code), of course.## Justin Dorfman

Our pleasure!

Yes, I should have been more clear in the article. This is just the first step of indexing the Fedora universe. We will continue to work with the Fedora team to make the integration more useful.

Exactly. We don’t have a timeline yet but it is definitely on our radar.

## junaruga

Here is alternative way to search all the RPM spec files.

$ tar xf rpm-specs-latest.tar.xz

$ cd rpm-specs

$ grep 'something' *.spec

## fedorafree

sourcegraph itself should be open source and installable on fedora on disk and not just on the cloud. It should be freely downloadable from the fedora store, and its code must be open source and accessible.

## Nick Vlahos

Positively “AMAZING”.. !! Benefits the “Universe as a Whole”.. not just Fedora’s!! Keep up the Good Work !!