How to Handle URL Redirects and Page Moved Errors Using cURL?

How to Retrieve Page Content Using cURL

In the pursuit of extracting content from Google search results, you may encounter issues such as redirects and "page moved" errors when utilizing cURL. Often, these hindrances arise due to encoded query strings.

To effectively retrieve the desired content, consider the following PHP implementation:

<code class="php">/**
 * Get a web file (HTML, XHTML, XML, image, etc.) from a URL.  Return an
 * array containing the HTTP server response header fields and content.
 */
function get_web_page( $url )
{
    $user_agent='Mozilla/5.0 (Windows NT 6.1; rv:8.0) Gecko/20100101 Firefox/8.0';

    $options = array(

        CURLOPT_CUSTOMREQUEST  => "GET",        // set request type post or get
        CURLOPT_POST           => false,        // set to GET
        CURLOPT_USERAGENT      => $user_agent, // set user agent
        CURLOPT_COOKIEFILE     => "cookie.txt", // set cookie file
        CURLOPT_COOKIEJAR      => "cookie.txt", // set cookie jar
        CURLOPT_RETURNTRANSFER => true,     // return web page
        CURLOPT_HEADER         => false,    // don't return headers
        CURLOPT_FOLLOWLOCATION => true,     // follow redirects
        CURLOPT_ENCODING       => "",       // handle all encodings
        CURLOPT_AUTOREFERER    => true,     // set referer on redirect
        CURLOPT_CONNECTTIMEOUT => 120,      // timeout on connect
        CURLOPT_TIMEOUT        => 120,      // timeout on response
        CURLOPT_MAXREDIRS      => 10,       // stop after 10 redirects
    );

    $ch      = curl_init( $url );
    curl_setopt_array( $ch, $options );
    $content = curl_exec( $ch );
    $err     = curl_errno( $ch );
    $errmsg  = curl_error( $ch );
    $header  = curl_getinfo( $ch );
    curl_close( $ch );

    $header['errno']   = $err;
    $header['errmsg']  = $errmsg;
    $header['content'] = $content;
    return $header;
}</code>

Example:

<code class="php">// Read a web page and check for errors:

$result = get_web_page( $url );

if ( $result['errno'] != 0 )
    ... error: bad url, timeout, redirect loop ...

if ( $result['http_code'] != 200 )
    ... error: no page, no permissions, no service ...

$page = $result['content'];</code>

The above is the detailed content of How to Handle URL Redirects and Page Moved Errors Using cURL?. For more information, please follow other related articles on the PHP Chinese website!