[Python]파이썬 크롤링 오류 처리 방법 (ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired)

파이썬 크롤링하는 과정에 SSL 관련 오류가 아래와 같이 발생하는 경우가 있다.

C:\Users\ilike\AppData\Local\Programs\Python\Python39\python.exe C:/python/Workspace/main.py
Traceback (most recent call last):
  File "C:\Users\ilike\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 1342, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "C:\Users\ilike\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "C:\Users\ilike\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "C:\Users\ilike\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "C:\Users\ilike\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1010, in _send_output
    self.send(msg)
  File "C:\Users\ilike\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 950, in send
    self.connect()
  File "C:\Users\ilike\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1424, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "C:\Users\ilike\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "C:\Users\ilike\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 1040, in _create
    self.do_handshake()
  File "C:\Users\ilike\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1122)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\python\Workspace\main.py", line 56, in <module>
    request = urllib.request.urlopen(url)
  File "C:\Users\ilike\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "C:\Users\ilike\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 517, in open
    response = self._open(req, data)
  File "C:\Users\ilike\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 534, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "C:\Users\ilike\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 494, in _call_chain
    result = func(*args)
  File "C:\Users\ilike\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 1385, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "C:\Users\ilike\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 1345, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1122)>

Process finished with exit code 1

오류가 발생한 파이썬 스크립트는 다음과 같다.

url = '크롤링을 타켓팅하는 url'



request = urllib.request.urlopen(url)
html = request.read()
beautifulSoup = BeautifulSoup(html, 'html.parser')
div_list = beautifulSoup.find('div', attrs={'class': 'header'}).find_all('li', attrs={'class': ''})
print(div_list)

파이썬 SSLCertVerificationError 오류 해결법

오류 해결하기 위해서는 ssl 라이브러리를 임포트하여 사용하면 해결할 수 있다.

context = ssl._create_unverified_context()
request = urllib.request.urlopen(url, context=context)

오류를 해결한 스크립트는 다음과 같다.

import ssl
.....
이하생략...

url = '크롤링을 타켓팅하는 url'

context = ssl._create_unverified_context()
request = urllib.request.urlopen(url, context=context)

html = request.read()
beautifulSoup = BeautifulSoup(html, 'html.parser')
div_list = beautifulSoup.find('div', attrs={'class': 'header'}).find_all('li', attrs={'class': ''})
print(div_list)