How to Fix the \'CERTIFICATE_VERIFY_FAILED\' Error While Scraping Wikipedia with Python?

Scraping: How to Resolve CERTIFICATE_VERIFY_FAILED Error for http://en.wikipedia.org

This question addresses the common "CERTIFICATE_VERIFY_FAILED" error encountered while scraping web pages using Python. It's particularly relevant to those attempting web scraping practices with Python as described in the "Web Scraping with Python" book.

Background and Error Description

The issue arises when trying to open a web page using urllib.request.urlopen() method, which results in the following error:

urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1049)>

Solution for macOS Users

To address this issue on macOS, follow these steps:

1. Go to the "Python3.6" folder located in "Macintosh HD > Applications."
2. Double-click on the "Install Certificates.command" file.

This action ensures that your Python installation has the necessary certificates to verify SSL connections properly.

Note for Scrapy Users

The original question also mentions encountering a command not found error when using Scrapy. However, the solution provided in the answer is specific to the "CERTIFICATE_VERIFY_FAILED" error and does not address the Scrapy issue.

The above is the detailed content of How to Fix the \'CERTIFICATE_VERIFY_FAILED\' Error While Scraping Wikipedia with Python?. For more information, please follow other related articles on the PHP Chinese website!