Detailed explanation of the steps to simulate login and capture data using PHP using Curl

This time I will bring you a detailed explanation of the steps for PHP to use Curl to simulate login and capture data. What are the precautions for PHP to use Curl to simulate login and capture data? The following is a practical case. Get up and take a look. Using PHP's Curl extension library can simulate login and capture some data that can only be viewed after logging in with a user account. The specific implementation process is as follows (personal summary):

1. First, you need to analyze the html source code of the corresponding login page to obtain some necessary information:

(1) The login page Address;

(2) Address of

verification code

; (3) Name and submission method of each field that needs to be submitted in the login form;

(4) The address submitted by the login form;

(5) In addition, you need to know the address where the data to be captured is located.

2. Get cookies and store them (for websites that use cookie files):

$login_url = 'http://www.xxxxx';  //登录页面地址
$cookie_file = dirname(FILE)."/pic.cookie";  //cookie文件存放位置（自定义）
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $login_url);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file);
curl_exec($ch);
curl_close($ch);

3. Get verification codes and store them (for websites that use verification codes):

$verify_url = "http://www.xxxx";   //验证码地址
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $verify_url);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$verify_img = curl_exec($ch);
curl_close($ch);
$fp = fopen("./verify/verifyCode.png",'w');  //把抓取到的图片文件写入本地图片文件保存
fwrite($fp, $verify_img);
fclose($fp);

Note:

Since the verification code cannot be recognized, what I do here is to capture the verification code image and store it in a local file, and then use it in the html in my project It is displayed on the page and allows the user to fill it in. Wait for the user to fill in the account number, password and verification code, and click the submit button before proceeding to the next step.

4. Simulate submission of login form:

$ post_url = 'http://www.xxxx';   //登录表单提交地址
$post = "username=$account&password=$password&seccodeverify=$verifyCode";//表单提交的数据（根据表单字段名和用户输入决定）
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $ post_url);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post);     //提交方式为post
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file);
curl_exec($ch);
curl_close($ch);

5. Capture data:

$data_url = "http://www.xxxx";   //数据所在地址
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $data_url);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,0);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file);
$data = curl_exec($ch);
curl_close($ch);

So far, the page where the data is located has been captured and stored. In

String

variable$data. It should be noted that what is captured is the html source code of a web page, which means that this string not only contains the data you want, but also contains many

html tags

Wait for something you don’t want. So if you want to extract the data you need, you have to analyze the HTML code of the page where the data is stored, and then use string manipulation functions, regular matching and other methods to extract the data you want. The above method is effective for general websites using http protocol. But if you want to simulate logging in to a website that uses https protocol, you need to add the following processing:

1. Skip https verification:

curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, false);

2. Use user agent:

$UserAgent = 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.0.04506; .NET CLR 3.5.21022; .NET CLR 1.0.3705; .NET CLR 1.1.4322)';
curl_setopt($curl, CURLOPT_USERAGENT, $UserAgent);

Note:

If you do not add these processes, the simulated login will not be successful. Using the above program to simulate logging into a website is generally successful, but in fact it still needs to be considered based on the specific circumstances of the simulated login website. For example: some websites have different encodings, so the pages you capture are garbled. In this case, you need to perform encoding conversion, such as:

$data = iconv("gb2312", "utf-8",$data) ;

, convert gbk encoding to utf8 encoding. There are also some websites that have relatively high security requirements, such as online banking, which will put the verification code in an inline frame. In this case, you need to first crawl the page of the inline frame and then extract the address of the verification code from it. Go grab the verification code again. There are also some websites (such as online banking) that submit forms in js code. Before submitting the form, they will also do some processing, such as encryption, etc., so if you submit it directly, you will not be able to log in successfully. You must do it Submit after similar processing, but in this case, if you can know the specific operations performed in the js code, such as encryption, what the encryption algorithm is, you can perform the same processing as it does, and then submit the data, so It can also be successful. However, here comes the key point. If you don’t know what operations it performs at all, for example, it is encrypted, but you don’t know the specific encryption algorithm, then you will not be able to perform the same operation, and you will not be able to successfully simulate it. Logged in. A typical case in this regard is online banking. It uses the online banking control to perform some processing on the password and verification code submitted by the user before submitting the form in the js code. However, we have no idea what operations it performs, so we cannot simulate it. So if you think you can simulate logging into online banking after reading this article, then you are too naive. Can you simulate logging into the bank's website so easily? Of course, if you can crack the online banking controls, that's another matter. Having said that, why do I feel so deeply? Because I have encountered this problem. If I don’t talk about it, I will shed tears if I talk too much. . .

I believe you have mastered the method after reading the case in this article. For more exciting information, please pay attention to other related articles on the php Chinese website!

Recommended reading:

Detailed explanation of the steps for PHP MySQL to process high-concurrency locking transactions


Summary of methods to implement shopping cart settlement


The above is the detailed content of Detailed explanation of the steps to simulate login and capture data using PHP using Curl. For more information, please follow other related articles on the PHP Chinese website!