How To Delete Finder Search History? - "Downloaded From"
I believe this list is populated based on metadata stored inside files where you're searching. I would think mds/Spotlight would be indexing that data and allowing you to search on it, so I'm confused about your assertion that Spotlight is off .. but regardless.

If you're comfortable with the command line, you can inspect any given file's metadata with the xattr command. The file attribute you're looking for is com.apple.metadata:kMDItemWhereFroms. For example:

************@Themis  ~/Downloads  xattr -p -l com.apple.metadata:kMDItemWhereFroms conference_poster_5.zip
com.apple.metadata:kMDItemWhereFroms:
00000000  62 70 6C 69 73 74 30 30 A2 01 02 5F 10 54 68 74  |bplist00..._.Tht|
00000010  74 70 3A 2F 2F 77 77 77 2E 6C 61 74 65 78 74 65  |tp://www.latexte|
00000020  6D 70 6C 61 74 65 73 2E 63 6F 6D 2F 74 65 6D 70  |mplates.com/temp|
00000030  6C 61 74 65 73 2F 63 6F 6E 66 65 72 65 6E 63 65  |lates/conference|
00000040  5F 70 6F 73 74 65 72 73 2F 35 2F 63 6F 6E 66 65  |_posters/5/confe|
00000050  72 65 6E 63 65 5F 70 6F 73 74 65 72 5F 35 2E 7A  |rence_poster_5.z|
00000060  69 70 5F 10 40 68 74 74 70 3A 2F 2F 77 77 77 2E  |ip_.@http://www.|
00000070  6C 61 74 65 78 74 65 6D 70 6C 61 74 65 73 2E 63  |latextemplates.c|
00000080  6F 6D 2F 74 65 6D 70 6C 61 74 65 2F 61 30 70 6F  |om/template/a0po|
00000090  73 74 65 72 2D 6C 61 6E 64 73 63 61 70 65 2D 70  |ster-landscape-p|
000000A0  6F 73 74 65 72 08 0B 62 00 00 00 00 00 00 01 01  |oster..b........|
000000B0  00 00 00 00 00 00 00 03 00 00 00 00 00 00 00 00  |................|
000000C0  00 00 00 00 00 00 00 A5                          |........|
000000c8
************@Themis  ~/Downloads 

So you can see the URL used to obtain this file is embedded in the file itself.
To remove that metadata:

************@Themis  ~/Downloads  xattr -d com.apple.metadata:kMDItemWhereFroms conference_poster_5.zip
************@Themis  ~/Downloads  xattr -p -l com.apple.metadata:kMDItemWhereFroms conference_poster_5.zip
xattr: conference_poster_5.zip: No such xattr: com.apple.metadata:kMDItemWhereFroms
✘ ************@Themis  ~/Downloads 

That bit of metadata has been removed. Now you would repeat your search and see if you get the same results once you've removed the metadata. I'm guessing not, but if you do let us know.

If you're not comfortable with the command-line, you can try a GUI-based extended attributes editor, such as this one.