diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst
index 616b56101..93c91f18a 100644
--- a/docs/topics/downloader-middleware.rst
+++ b/docs/topics/downloader-middleware.rst
@@ -989,6 +989,12 @@ RobotsTxtMiddleware
     To make sure Scrapy respects robots.txt make sure the middleware is enabled
     and the :setting:`ROBOTSTXT_OBEY` setting is enabled.
 
+    The :setting:`ROBOTSTXT_USER_AGENT` setting can be used to specify the
+    user agent string to use for matching in the robots.txt_ file. If it
+    is ``None``, the User-Agent header you are sending with the request or the
+    :setting:`USER_AGENT` setting (in that order) will be used for determining
+    the user agent to use in the robots.txt_ file.
+
     This middleware has to be combined with a robots.txt_ parser.
 
     Scrapy ships with support for the following robots.txt_ parsers:
diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst
index 12606fe47..d3ad777f2 100644
--- a/docs/topics/settings.rst
+++ b/docs/topics/settings.rst
@@ -1151,6 +1151,18 @@ Default: ``'scrapy.robotstxt.PythonRobotParser'``
 The parser backend to use for parsing ``robots.txt`` files. For more information see
 :ref:`topics-dlmw-robots`.
 
+.. setting:: ROBOTSTXT_USER_AGENT
+
+ROBOTSTXT_USER_AGENT
+^^^^^^^^^^^^^^^^^^^^
+
+Default: ``None``
+
+The user agent string to use for matching in the robots.txt file. If ``None``,
+the User-Agent header you are sending with the request or the
+:setting:`USER_AGENT` setting (in that order) will be used for determining
+the user agent to use in the robots.txt file.
+
 .. setting:: SCHEDULER
 
 SCHEDULER
@@ -1409,7 +1421,10 @@ USER_AGENT
 
 Default: ``"Scrapy/VERSION (+https://scrapy.org)"``
 
-The default User-Agent to use when crawling, unless overridden.
+The default User-Agent to use when crawling, unless overridden. This user agent is
+also used by :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware`
+if :setting:`ROBOTSTXT_USER_AGENT` setting is ``None`` and
+there is no overridding User-Agent header specified for the request.
 
 
 Settings documented elsewhere:
diff --git a/scrapy/downloadermiddlewares/robotstxt.py b/scrapy/downloadermiddlewares/robotstxt.py
index c5a60d355..6a5dfb79c 100644
--- a/scrapy/downloadermiddlewares/robotstxt.py
+++ b/scrapy/downloadermiddlewares/robotstxt.py
@@ -26,6 +26,7 @@ class RobotsTxtMiddleware(object):
         if not crawler.settings.getbool('ROBOTSTXT_OBEY'):
             raise NotConfigured
         self._default_useragent = crawler.settings.get('USER_AGENT', 'Scrapy')
+        self._robotstxt_useragent = crawler.settings.get('ROBOTSTXT_USER_AGENT', None)
         self.crawler = crawler
         self._parsers = {}
         self._parserimpl = load_object(crawler.settings.get('ROBOTSTXT_PARSER'))
@@ -47,7 +48,10 @@ class RobotsTxtMiddleware(object):
     def process_request_2(self, rp, request, spider):
         if rp is None:
             return
-        useragent = request.headers.get(b'User-Agent', self._default_useragent)
+
+        useragent = self._robotstxt_useragent
+        if not useragent:
+            useragent = request.headers.get(b'User-Agent', self._default_useragent)
         if not rp.allowed(request.url, useragent):
             logger.debug("Forbidden by robots.txt: %(request)s",
                          {'request': request}, extra={'spider': spider})
diff --git a/scrapy/settings/default_settings.py b/scrapy/settings/default_settings.py
index 81fee543f..efb257e2b 100644
--- a/scrapy/settings/default_settings.py
+++ b/scrapy/settings/default_settings.py
@@ -246,6 +246,7 @@ RETRY_PRIORITY_ADJUST = -1
 
 ROBOTSTXT_OBEY = False
 ROBOTSTXT_PARSER = 'scrapy.robotstxt.PythonRobotParser'
+ROBOTSTXT_USER_AGENT = None
 
 SCHEDULER = 'scrapy.core.scheduler.Scheduler'
 SCHEDULER_DISK_QUEUE = 'scrapy.squeues.PickleLifoDiskQueue'

Using separate user agent for robots.txt
The current implementation of `RobotsTxtMiddleware` ([link](https://github.com/scrapy/scrapy/blob/master/scrapy/downloadermiddlewares/robotstxt.py)) uses `USER_AGENT` setting and `User-Agent` header of the request to determine the user agent to use for matching in `robots.txt`. There is no easy way for a user to send one user-agent header value but have a different user agent string evaluated to decide which pages to crawl. This use case seems to be more common that we thought previously for example - while [Googlebot](https://support.google.com/webmasters/answer/182072?hl=en) uses `"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"`  as `User-Agent` header of the request, it actually uses user agent `"Googlebot"` in `robots.txt` for deciding which pages to crawl. [Here is the an article describing this.](https://support.google.com/webmasters/answer/1061943?hl=en) The same seems to be the case with Bing - [link](https://www.bing.com/webmaster/help/which-crawlers-does-bing-use-8c184ec0). Infact, Google made this distinction very clear in their proposed [robots.txt specification](https://github.com/google/robotstxt/blob/master/protocol-draft/draft-rep-wg-topic-00.txt) where they described the user agent used in `robots.txt` as the "product token".

The proposed solutions are implementing a new `ROBOTSTXT_USER_AGENT` setting, implementing a `robotstxt_user_agent` meta key for overriding user agent per request or both. Here is a thread that has previous discussion on the topic - https://github.com/scrapy/scrapy/pull/3796#discussion_r300036852.

I'm creating this issue as a place for discussion regarding whether to add this functionality into Scrapy? If yes, which route we should take?

diff --git a/tests/test_downloadermiddleware_robotstxt.py b/tests/test_downloadermiddleware_robotstxt.py
index 79f172848..fbc46cba4 100644
--- a/tests/test_downloadermiddleware_robotstxt.py
+++ b/tests/test_downloadermiddleware_robotstxt.py
@@ -164,6 +164,15 @@ Disallow: /some/randome/page.html
         d.addCallback(lambda _: self.assertFalse(mw_module_logger.error.called))
         return d
 
+    def test_robotstxt_user_agent_setting(self):
+        crawler = self._get_successful_crawler()
+        crawler.settings.set('ROBOTSTXT_USER_AGENT', 'Examplebot')
+        crawler.settings.set('USER_AGENT', 'Mozilla/5.0 (X11; Linux x86_64)')
+        middleware = RobotsTxtMiddleware(crawler)
+        rp = mock.MagicMock(return_value=True)
+        middleware.process_request_2(rp, Request('http://site.local/allowed'), None)
+        rp.allowed.assert_called_once_with('http://site.local/allowed', 'Examplebot')
+
     def assertNotIgnored(self, request, middleware):
         spider = None  # not actually used
         dfd = maybeDeferred(middleware.process_request, request, spider)

pip install -e .[dev]
parse_log_pytest
pytest
pytest pytest-cov pytest-xdist pytest-mock pytest-asyncio
apt-get update apt-get install -y gcc libxml2-dev libxslt1-dev zlib1g-dev libffi-dev libssl-dev
3.9
pytest --no-header -rA --tb=line --color=no -p no:cacheprovider -W ignore::DeprecationWarning
attrs==25.3.0
Automat==24.8.1
cffi==1.17.1
constantly==23.10.4
coverage==7.8.0
cryptography==44.0.2
cssselect==1.3.0
exceptiongroup @ file:///croot/exceptiongroup_1706031385326/work
execnet==2.1.1
hyperlink==21.0.0
idna==3.10
incremental==24.7.2
iniconfig @ file:///home/linux1/recipes/ci/iniconfig_1610983019677/work
jmespath==1.0.1
lxml==5.3.1
packaging @ file:///croot/packaging_1734472117206/work
parsel==1.10.0
pluggy @ file:///croot/pluggy_1733169602837/work
pyasn1==0.6.1
pyasn1_modules==0.4.2
pycparser==2.22
PyDispatcher==2.0.7
pyOpenSSL==25.0.0
pytest @ file:///croot/pytest_1738938843180/work
pytest-asyncio==0.26.0
pytest-cov==6.0.0
pytest-mock==3.14.0
pytest-xdist==3.6.1
queuelib==1.7.0
-e git+https://github.com/scrapy/scrapy.git@a95de71d8ef7de2d5bee6584f0ab26994b8b9d12#egg=Scrapy
service-identity==24.2.0
six==1.17.0
tomli @ file:///opt/conda/conda-bld/tomli_1657175507142/work
Twisted==24.11.0
typing_extensions==4.13.0
w3lib==2.3.1
zope.interface==7.2

name: scrapy
channels:
  - defaults
  - https://repo.anaconda.com/pkgs/main
  - https://repo.anaconda.com/pkgs/r
  - conda-forge
dependencies:
  - _libgcc_mutex=0.1=main
  - _openmp_mutex=5.1=1_gnu
  - ca-certificates=2025.2.25=h06a4308_0
  - exceptiongroup=1.2.0=py39h06a4308_0
  - iniconfig=1.1.1=pyhd3eb1b0_0
  - ld_impl_linux-64=2.40=h12ee557_0
  - libffi=3.4.4=h6a678d5_1
  - libgcc-ng=11.2.0=h1234567_1
  - libgomp=11.2.0=h1234567_1
  - libstdcxx-ng=11.2.0=h1234567_1
  - ncurses=6.4=h6a678d5_0
  - openssl=3.0.16=h5eee18b_0
  - packaging=24.2=py39h06a4308_0
  - pip=25.0=py39h06a4308_0
  - pluggy=1.5.0=py39h06a4308_0
  - pytest=8.3.4=py39h06a4308_0
  - python=3.9.21=he870216_1
  - readline=8.2=h5eee18b_0
  - setuptools=75.8.0=py39h06a4308_0
  - sqlite=3.45.3=h5eee18b_0
  - tk=8.6.14=h39e8969_0
  - tomli=2.0.1=py39h06a4308_0
  - tzdata=2025a=h04d1e81_0
  - wheel=0.45.1=py39h06a4308_0
  - xz=5.6.4=h5eee18b_1
  - zlib=1.2.13=h5eee18b_1
  - pip:
      - attrs==25.3.0
      - automat==24.8.1
      - cffi==1.17.1
      - constantly==23.10.4
      - coverage==7.8.0
      - cryptography==44.0.2
      - cssselect==1.3.0
      - execnet==2.1.1
      - hyperlink==21.0.0
      - idna==3.10
      - incremental==24.7.2
      - jmespath==1.0.1
      - lxml==5.3.1
      - parsel==1.10.0
      - pyasn1==0.6.1
      - pyasn1-modules==0.4.2
      - pycparser==2.22
      - pydispatcher==2.0.7
      - pyopenssl==25.0.0
      - pytest-asyncio==0.26.0
      - pytest-cov==6.0.0
      - pytest-mock==3.14.0
      - pytest-xdist==3.6.1
      - queuelib==1.7.0
      - service-identity==24.2.0
      - six==1.17.0
      - twisted==24.11.0
      - typing-extensions==4.13.0
      - w3lib==2.3.1
      - zope-interface==7.2
prefix: /opt/conda/envs/scrapy

tests/test_downloadermiddleware_robotstxt.py::RobotsTxtMiddlewareTest::test_robotstxt_user_agent_setting
tests/test_downloadermiddleware_robotstxt.py::RobotsTxtMiddlewareTest::test_ignore_robotstxt_request tests/test_downloadermiddleware_robotstxt.py::RobotsTxtMiddlewareTest::test_robotstxt tests/test_downloadermiddleware_robotstxt.py::RobotsTxtMiddlewareTest::test_robotstxt_empty_response tests/test_downloadermiddleware_robotstxt.py::RobotsTxtMiddlewareTest::test_robotstxt_error tests/test_downloadermiddleware_robotstxt.py::RobotsTxtMiddlewareTest::test_robotstxt_garbage tests/test_downloadermiddleware_robotstxt.py::RobotsTxtMiddlewareTest::test_robotstxt_immediate_error tests/test_downloadermiddleware_robotstxt.py::RobotsTxtMiddlewareTest::test_robotstxt_meta tests/test_downloadermiddleware_robotstxt.py::RobotsTxtMiddlewareTest::test_robotstxt_ready_parser tests/test_downloadermiddleware_robotstxt.py::RobotsTxtMiddlewareTest::test_robotstxt_settings
docs/topics/settings.rst docs/topics/downloader-middleware.rst scrapy/settings/default_settings.py scrapy/downloadermiddlewares/robotstxt.py
docs/topics/settings.rst docs/topics/downloader-middleware.rst scrapy/settings/default_settings.py scrapy/downloadermiddlewares/robotstxt.py