How Can I Efficiently Check for 404 Errors in PHP Before Web Scraping?

Easy Way to Test a URL for 404 in PHP

Finding that your code is running into issues due to URLs returning 404 is a common pain point in web scraping. To resolve this efficiently, implementing a test at the code's start to verify if a URL has a 404 response is essential.

While suggestions like using @fsockopen() may not account for redirects, a more suitable approach is utilizing curl's curl_getinfo() function. Here's how:

// Initialize a cURL handle with the given URL
$handle = curl_init($url);
// Enable return of transfer as a string
curl_setopt($handle, CURLOPT_RETURNTRANSFER, TRUE);
// Get the response (HTML or data linked to the URL)
$response = curl_exec($handle);
// Check for 404 (file not found) response
$httpCode = curl_getinfo($handle, CURLINFO_HTTP_CODE);
// Handle 404
if ($httpCode == 404) {

    /* Perform your custom 404 handling here. */
}
// Close the curl session
curl_close($handle);
// Continue processing with the retrieved $response

By incorporating this code, you can effectively check for 404 responses, allowing your code to skip the problematic URLs and proceed with the available ones.

The above is the detailed content of How Can I Efficiently Check for 404 Errors in PHP Before Web Scraping?. For more information, please follow other related articles on the PHP Chinese website!