Error downloading HTTPS webpage
Question: I am downloading some data from some https webpage
<https://www.spar.si/sl_SI/zaposlitev/prosta-delovna-mesta-.html>, so I get
this error due to HTTPS. When I change the webpage manually to HTTP, it
downloads fine. I was looking for similar examples to fix this but I did not
find any. Do you have some idea what to do ?

    
    
    Traceback (most recent call last):
      File "down.py", line 34, in <module>
        soup = BeautifulSoup(urllib.urlopen(url).read(), "html.parser")
      File "g:\python\Lib\urllib.py", line 87, in urlopen
        return opener.open(url)
      File "g:\python\Lib\urllib.py", line 213, in open
        return getattr(self, name)(url)
      File "g:\python\Lib\urllib.py", line 443, in open_https
        h.endheaders(data)
      File "g:\python\Lib\httplib.py", line 1049, in endheaders
        self._send_output(message_body)
      File "g:\python\Lib\httplib.py", line 893, in _send_output
        self.send(msg)
      File "g:\python\Lib\httplib.py", line 855, in send
        self.connect()
      File "g:\python\Lib\httplib.py", line 1274, in connect
        server_hostname=server_hostname)
      File "g:\python\Lib\ssl.py", line 352, in wrap_socket
        _context=self)
      File "g:\python\Lib\ssl.py", line 579, in __init__
        self.do_handshake()
      File "g:\python\Lib\ssl.py", line 808, in do_handshake
        self._sslobj.do_handshake()
    IOError: [Errno socket error] [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:5
    90)
    

This is my programme:

    
    
    #!/usr/bin/python
    # -*- coding: utf-8 -*-
    # encoding=UTF-8  
    #
    # DOWNLOADER
    # To grab the text content of webpages and save it to TinyDB database.
    
    
    import re, time, urllib, tinydb
    from bs4 import BeautifulSoup 
    
    
    start_time = time.time()
    
    
    
    #Open file with urls.
    with open("G:/myVE/vacancies/urls2.csv") as f:
        lines = f.readlines()
    
    #Open file to write HTML to.
    with open("G:/myVE/downloader/urls2_html.txt", 'wb') as g:
    
        #We parse the content of url file to get just urls without the first line and without the text.
        for line in lines[1:len(lines)]:
    
            #Read the url from the file.
            #url = line.split(",")[0]
            url = line
    
            print "test"
    
            #Read the HTML of url
            soup = BeautifulSoup(urllib.urlopen(url).read(), "html.parser")
    
            print url
            #Mark of new HTML in HTML file.
            g.write("\n\nNEW HTML\n\n")
    
            #Write new HTML to file.
            g.write(str(soup))
    
    
    
    
    
    
    
    print "Html saved to html.txt"
    print "--- %s seconds ---" % round((time.time() - start_time),2)
    
    
    
    
    
    
    """
    #We read HTML of the employment webpage that we intend to parse.
    soup = BeautifulSoup(urllib.urlopen('http://www.simplybusiness.co.uk/about-us/careers/jobs/').read(), "html.parser")
    
    
    
    #We write HTML to a file.
    with open("E:/analitika/SURS/tutorial/tutorial/html.txt", 'wb') as f:
       f.write(str(soup)) 
    
    
    
    print "Html saved to html.txt"
    print "--- %s seconds ---" % round((time.time() - start_time),2)
    """
    

Thank you!


Answer: You should use the `requests` library, see <http://docs.python-
requests.org/en/latest/user/advanced/#ssl-cert-verification> as a reference.

**Updated to add**

Now with your url here is an example with the `requests` library.

    
    
    import requests
    
    url = "https://www.spar.si/sl_SI/zaposlitev/prosta-delovna-mesta-.html" 
    r = requests.get(url, verify=True)
    print(r.text)
    

Here is an example with `beautifulsoup` and Python 3.3 which also seems to
work.

    
    
    import urllib
    from bs4 import BeautifulSoup
    
    url = "https://www.spar.si/sl_SI/zaposlitev/prosta-delovna-mesta-.html" 
    soup = BeautifulSoup(urllib.request.urlopen(url).read(), "html.parser")
    print(soup)
    

