PHP Master | Using cURL for Remote Requests

If you’re a Linux user then you’ve probably used cURL. It’s a powerful tool used for everything from sending email to downloading the latest My Little Pony subtitles. In this article I’ll explain how to use the cURL extension in PHP. The extension offers us the functionality as the console utility in the comfortable world of PHP. I’ll discuss sending GET and POST requests, handling login cookies, and FTP functionality. Before we begin, make sure you have the extension (and the libcURL library) installed. It’s not installed by default. In most cases it can be installed using your system’s package manager, but barring that you can find instructions in the PHP manual.

Key Takeaways

• cURL, a powerful tool used for a variety of tasks from sending emails to downloading subtitles, can be used in PHP through an extension offering the same functionality as the console utility.
• cURL can be used to retrieve web pages, log into websites, work with FTP and send multiple requests. For instance, you can simulate logging into a WordPress-powered website by sending a POST request with specific details.
• Troubleshooting cURL requests is simplified with the use of two functions: curl_getinfo() and curl_error(). These functions provide detailed information about the channel and any errors that may occur during the request.
• cURL is an efficient and powerful tool for making remote calls, making it ideal for tasks such as accessing external APIs or crawling. It offers a user-friendly interface and relatively easy execution of requests.

How Does it Work?

All cURL requests follow the same basic pattern:

1. First we initialize the cURL resource (often abbreviated as ch for “cURL handle”) by calling the curl_init() function.
2. Next we set various options, such as the URL, request method, payload data, etc. Options can be set individually with curl_setopt(), or we can pass an array of options to curl_setopt_array().
3. Then we execute the request by calling curl_exec().
4. Finally, we free the resource to clear out memory.

So, the boilerplate code for making a request looks something like this:

<span><span><?php </span></span><span><span>// init the resource
</span></span><span><span>$ch = curl_init();
</span></span><span>
</span><span><span>// set a single option...
</span></span><span><span>curl_setopt($ch, OPTION, $value);
</span></span><span><span>// ... or an array of options
</span></span><span><span>curl_setopt_array($ch, array(
</span></span><span><span>OPTION1 => $value1,
</span></span><span><span>OPTION2 => $value2
</span></span><span><span>));
</span></span><span>
</span><span><span>// execute
</span></span><span><span>$output = curl_exec($ch);
</span></span><span>
</span><span><span>// free
</span></span><span><span>curl_close($ch);</span></span></span>

The only thing that changes for the request is what options are set, which of course depends on what you’re doing with cURL.

Retrieve a Web Page

The most basic example of using cURL that I can think of is simply fetching the contents of a web page. So, let’s fetch the homepage of the BBC as an example.

<span><span><?php </span></span><span><span>curl_setopt_array(
</span></span><span><span>$ch, array(
</span></span><span><span>CURLOPT_URL => 'http://www.bbc.co.uk/',
</span></span><span><span>CURLOPT_RETURNTRANSFER => true
</span></span><span><span>));
</span></span><span>
</span><span><span>$output = curl_exec($ch);
</span></span><span><span>echo $output;</span></span></span>

Check the output in your browser and you should see the BBC website displayed. We’re lucky as the site displays correctly because of its absolute linking to stylesheets and images. The options we just used were:

• CURLOPT_URL – specifies the URL for the request
• CURLOPT_RETURNTRANSFER – when set false, curl_exec() returns true or false depending on the success of the request. When set to true, curl_exec() returns the contents of the response.

Log in to a Website

cURL executed a GET request to retrieve the BBC page, but cURL can also use other methods, such as POST and PUT. For this example, let’s simulate logging into a WordPress-powered website. Logging in is done by sending a POST request to http://example.com/wp-login.php with the following details:

• login – the username
• pwd – the password
• redirect_to – the URL we want to go to after logging in
• testcookie – should be set to 1 (this is just for WordPress)

Of course these parameters are specific to each site. You should always check the input names for yourself, something that can easily be done by viewing the source of an HTML page in your browser.

<span><span><?php </span></span><span><span>// init the resource
</span></span><span><span>$ch = curl_init();
</span></span><span>
</span><span><span>// set a single option...
</span></span><span><span>curl_setopt($ch, OPTION, $value);
</span></span><span><span>// ... or an array of options
</span></span><span><span>curl_setopt_array($ch, array(
</span></span><span><span>OPTION1 => $value1,
</span></span><span><span>OPTION2 => $value2
</span></span><span><span>));
</span></span><span>
</span><span><span>// execute
</span></span><span><span>$output = curl_exec($ch);
</span></span><span>
</span><span><span>// free
</span></span><span><span>curl_close($ch);</span></span></span>

The new options are:

• CURLOPT_POST – set this true if you want to send a POST request
• CURLOPT_POSTFIELDS – the data that will be sent in the body of the request
• CURLOPT_FOLLOWLOCATION – if set true, cURL will follow redirects

Uh oh! If you test the above however you’ll see an error message: “ERROR: Cookies are blocked or not supported by your browser. You must enable cookies to use WordPress.” This is normal, because we need to have cookies enabled for sessions to work. We do this by adding two more options.

<span><span><?php </span></span><span><span>curl_setopt_array(
</span></span><span><span>$ch, array(
</span></span><span><span>CURLOPT_URL => 'http://www.bbc.co.uk/',
</span></span><span><span>CURLOPT_RETURNTRANSFER => true
</span></span><span><span>));
</span></span><span>
</span><span><span>$output = curl_exec($ch);
</span></span><span><span>echo $output;</span></span></span>

The new options are:

• CURLOPT_COOKIESESSION – if set to true, cURL will start a new cookie session and ignore any previous cookies
• CURLOPT_COOKIEJAR – this is the name of the file where cURL should save cookie information. Make sure you have the correct permissions to write to the file!

Now that we’re logged in, we only need to reference the cookie file for subsequent requests.

Working with FTP

Using cURL to download and upload files via FTP is easy as well. Let’s look at downloading a file:

<span><span><?php </span></span><span><span>$postData = array(
</span></span><span><span>'login' => 'acogneau',
</span></span><span><span>'pwd' => 'secretpassword',
</span></span><span><span>'redirect_to' => 'http://example.com',
</span></span><span><span>'testcookie' => '1'
</span></span><span><span>);
</span></span><span>
</span><span><span>curl_setopt_array($ch, array(
</span></span><span><span>CURLOPT_URL => 'http://example.com/wp-login.php',
</span></span><span><span>CURLOPT_RETURNTRANSFER => true,
</span></span><span><span>CURLOPT_POST => true,
</span></span><span><span>CURLOPT_POSTFIELDS => $postData,
</span></span><span><span>CURLOPT_FOLLOWLOCATION => true
</span></span><span><span>));
</span></span><span>
</span><span><span>$output = curl_exec($ch);
</span></span><span><span>echo $output;</span></span></span>

Note that there aren’t many public FTP servers that allow anonymous uploads and downloads for security reasons, so the URL and credentials above are just place-holders. This is almost the same as sending an HTTP request, but only a couple minor differences:

• CURLOPT_URL – the URL of the file, note the use of “ftp://” instead of “http://”
• CURLOT_USERPWD – the login credentials for the FTP server

Uploading a file via FTP is slightly more complex, but still managable. It looks like this:

<span><span><?php </span></span><span><span>curl_setopt_array($ch, array(
</span></span><span><span>CURLOPT_URL => 'http://example.com/wp-login.php',
</span></span><span><span>CURLOPT_RETURNTRANSFER => true,
</span></span><span><span>CURLOPT_POST => true,
</span></span><span><span>CURLOPT_POSTFIELDS => $postData,
</span></span><span><span>CURLOPT_FOLLOWLOCATION => true,
</span></span><span><span>CURLOPT_COOKIESESSION => true,
</span></span><span><span>CUROPT_COOKIEJAR => 'cookie.txt'
</span></span><span><span>));</span></span></span>

The important options here are:

• CURLOPT_UPLOAD – obvious boolean
• CURLOPT_INFILE – a readable stream for the file we want to upload
• CURLOPT_INFILESIZE – the size of the file we want to upload in bytes

Sending Multiple Requests

Imagine we have to perform five requests to retrieve all of the necessary data. Keep in mind that some things will be beyond our control, such as network latency and the response speed of the target servers. It should be obvious then that any delays when issuing five consecutive calls can really add up! One way to mitigate this problem is to issue the requests asynchronously. Asynchronous techniques are more common in the JavaScript and Node.js communities, but briefly instead of waiting for a time-consuming task to complete, we assign the task to a different thread or process and continue to do other things in the meantime. When the task is complete we come back for its result. The important thing is that we haven’t wasted time waiting for a result; we spent it executing other code independently. The approach for performing multiple asynchronous cURL requests is a bit different from before. We start out the same – we initiate each channel and then set the options – but then we initiate a multihandler using curl_multi_init() and add our channels to it with curl_multi_add_handle(). We execute the handlers by looping through them and checking their status. In the end we get a response’s content with curl_multi_getcontent().

<span><span><?php </span></span><span><span>// init the resource
</span></span><span><span>$ch = curl_init();
</span></span><span>
</span><span><span>// set a single option...
</span></span><span><span>curl_setopt($ch, OPTION, $value);
</span></span><span><span>// ... or an array of options
</span></span><span><span>curl_setopt_array($ch, array(
</span></span><span><span>OPTION1 => $value1,
</span></span><span><span>OPTION2 => $value2
</span></span><span><span>));
</span></span><span>
</span><span><span>// execute
</span></span><span><span>$output = curl_exec($ch);
</span></span><span>
</span><span><span>// free
</span></span><span><span>curl_close($ch);</span></span></span>

The above code took around 1,100 ms to execute on my laptop. Performing the requests sequentially without the multi interface it took around 2,000 ms. Imagine what your gain will be if you are sending hundreds of requests! Multiple projects exist that abstract and wrap the multi interface. Discussing them is beyond the scope of the article, but if you’re planning to issue multiple requests asynchronously then I recommend you take a look at them:

• github.com/petewarden/ParallelCurl
• semlabs.co.uk/journal/object-oriented-curl-class-with-multi-threading

Troubleshooting

If you’re using cURL then you are probably performing your requests to third-party servers. You can’t control them and much can go wrong: servers can go offline, directory structures can change, etc. We need an efficient way to find out what’s wrong when something doesn’t work, and luckily cURL offers two functions for this: curl_getinfo() and curl_error() . curl_getinfo() returns an array with all of the information regarding the channel, so if you want to check if everything is all right you can use:

<span><span><?php </span></span><span><span>// init the resource
</span></span><span><span>$ch = curl_init();
</span></span><span>
</span><span><span>// set a single option...
</span></span><span><span>curl_setopt($ch, OPTION, $value);
</span></span><span><span>// ... or an array of options
</span></span><span><span>curl_setopt_array($ch, array(
</span></span><span><span>OPTION1 => $value1,
</span></span><span><span>OPTION2 => $value2
</span></span><span><span>));
</span></span><span>
</span><span><span>// execute
</span></span><span><span>$output = curl_exec($ch);
</span></span><span>
</span><span><span>// free
</span></span><span><span>curl_close($ch);</span></span></span>

If an error pops up, you can check it out with curl_error():

<span><span><?php </span></span><span><span>curl_setopt_array(
</span></span><span><span>$ch, array(
</span></span><span><span>CURLOPT_URL => 'http://www.bbc.co.uk/',
</span></span><span><span>CURLOPT_RETURNTRANSFER => true
</span></span><span><span>));
</span></span><span>
</span><span><span>$output = curl_exec($ch);
</span></span><span><span>echo $output;</span></span></span>

Conclusion

cURL offers a powerful and efficient way to make remote calls, so if you’re ever in need of a crawler or something to access an external API, cURL is a great tool for the job. It provides us an nice interface and a relatively easy way to execute requests. For more information, check out the PHP Manual and the cURL website. See you next time! Comments on this article are closed. Have a question about PHP? Why not ask it on our forums? Image via Fotolia

Frequently Asked Questions (FAQs) about Using cURL for Remote Requests

What is cURL and why is it used in PHP?

cURL, or Client URL, is a library that allows you to make HTTP requests in PHP. It’s used to communicate with different types of servers and to download or upload data. cURL supports various protocols like HTTP, HTTPS, FTP, and more. It’s a powerful tool that can be used to interact with APIs, scrape web pages, or even download files.

How do I install and enable cURL in PHP?

cURL is usually included in most web servers. However, if it’s not enabled, you can do so by modifying your PHP.ini file. Locate the line that says “;extension=curl” and remove the semicolon. If the line doesn’t exist, you can add it at the end of the file. After making changes, save the file and restart your web server.

How do I make a simple cURL request in PHP?

To make a simple cURL request, you first need to initialize cURL with the curl_init() function. Then, set your options using the curl_setopt() function. Finally, execute the request with curl_exec() and close the session with curl_close(). Here’s a basic example:

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "http://example.com");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$output = curl_exec($ch);
curl_close($ch);

How can I handle errors in cURL?

You can handle errors in cURL by using the curl_errno() and curl_error() functions. These functions return the last error number and error message respectively. Here’s an example:

if(curl_errno($ch)) {
echo 'Error:' . curl_error($ch);
}

How do I send a POST request using cURL?

To send a POST request, you need to set the CURLOPT_POST option to true and the CURLOPT_POSTFIELDS option to an array of data you want to send. Here’s an example:

curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, "postvar1=value1&postvar2=value2");

How can I set custom headers for a cURL request?

You can set custom headers by using the CURLOPT_HTTPHEADER option. This option takes an array of headers as its value. Here’s an example:

$headers = array(
'Content-Type: application/json',
'Authorization: Bearer ' . $token
);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);

How do I follow redirects with cURL?

To follow redirects, you need to set the CURLOPT_FOLLOWLOCATION option to true. Here’s how you can do it:

curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);

How can I get the response headers from a cURL request?

To get the response headers, you can set the CURLOPT_HEADER option to true. This will include the headers in the output. Here’s an example:

curl_setopt($ch, CURLOPT_HEADER, true);

How do I send a file using cURL?

To send a file, you can use the CURLOPT_POSTFIELDS option and prefix the file path with an @ symbol. Here’s an example:

curl_setopt($ch, CURLOPT_POSTFIELDS, array('file' => '@/path/to/file.txt'));

How do I use cURL with a proxy?

To use cURL with a proxy, you can set the CURLOPT_PROXY option to the address of the proxy. Here’s how you can do it:

curl_setopt($ch, CURLOPT_PROXY, "http://proxy.example.com:8080");

The above is the detailed content of PHP Master | Using cURL for Remote Requests. For more information, please follow other related articles on the PHP Chinese website!