Title: Cannot access site with python requests, even after coping headers and cookies from browser
Tags: python;web-scraping;cookies;python-requests;http-headers
Question: I'm trying to access site allegro.pl with requests:
import requests
headers = {
'authority': 'allegro.pl',
'dpr': '1',
'viewport-width': '1920',
'upgrade-insecure-requests': '1',
'user-agent': 'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/+1-803-343-4222 Safari/537.36 OPR/+1-803-343-4222',
'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
'sec-fetch-site': 'none',
'sec-fetch-mode': 'navigate',
'sec-fetch-user': '?1',
'sec-fetch-dest': 'document',
'accept-language': 'en-US,en;q=0.9',
s = requests.Session()
r = s.get('https://allegro.pl')
s.headers = headers
r = s.get('https://allegro.pl/oferta/world-of-warcraft-30-dni-pre-paid-klucz-wow-eu-9538042175')
My first request is to get needed cookies and above method was working even yesterday, but today I'm getting 403 errors both times.
I have checked how request looks in browser, copied headers from browser, tried to copy browser cookies but still nothing.
I think that there must be a problem with requests module, somehow this website can see that I'm using script
Any ideas how to fix my program?


Here is another answer: It looks like the server is using a captcha service to check that JavaScript is enabled and that no ad blockers are running.  You can look at the content of the 403 response and see that message.
import requests
res = requests.get('https://allegro.pl/')
res.content
# returns:
<html&gt;
<head&gt;
<title&gt;allegro.pl</title&gt;
<style&gt;#cmsg{animation: A 1.5s;}@keyframes A{0%{opacity:0;}99%{opacity:0;}100%{opac
ity:1;}}</style&gt;
</head&gt;
<body style=&quot;margin:0&quot;&gt;
<p id=&quot;cmsg&quot;&gt;Please enable JS and disable any ad blocker</p&gt;
<script&gt;var dd={'cid':'AHrlqAAAAAMADKhs3_VXRtQAVxTsrg==','hsh':'77DC0FFBAA0B77570F6B414F8E5BDB','t':'bv','s':29560,'host':'geo.captcha-delivery.com'}
</script&gt;
<script src=&quot;https://ct.captcha-delivery.com/c.js&quot;&gt;</script&gt;
</body&gt;
</html&gt;
You can still scrape it using selenium, there are lots of guides on this.
Comment for this answer: Any chances to enable javascript in python module? Selenium would be too slow :(
Comment for this answer: btw, the same html appeared before, when my script was working, but after getting cookies with first request I was able to access website, so I'm not sure if javascript enabled is mandatory, any ideas how to check it?

{'source': 'StackExchangeDataset', 'subset': 'stackoverflow.com', 'metadata': {'AnswerCount': 1, 'CommentCount': 0, 'ContentLicense': 'CC BY-SA 4.0', 'CreationDate': '2021:00:06/22/21', 'FavoriteCount': None, 'PostTypeId': 1, 'Tags': '<python><web-scraping><cookies><python-requests><http-headers>', 'Title': 'Cannot access site with python requests, even after coping headers and cookies from browser', 'ViewCount': 230, 'id': 68090531, 'user_AboutMe': None, 'user_AccountId': 20135151.0, 'user_DisplayName': 'Kuba -a', 'user_Reputation': 11.0, 'user_Views': 14.0, 'user_id': 16145466.0}}