It is often found that the data to be used is all on the same website, and the data presentation format is the same. For example, there are thousands of products on Taobao or Amazon. If the information is entered manually, the workload will be too much. At this time, we can write a collection program to directly collect and display it. The server supports file_get_contents and curl
First add a text box and submit button to the page. The text box is used to enter the collection page address.
Collection needs to use the regular interception function
function preg_substr($start, $end, $str) // 正则截取函数 { $temp = preg_split($start, $str); $content = preg_split($end, $temp[1]); return $content[0]; }
Collection needs to use the string interception function
function str_substr($start, $end, $str) // 字符串截取函数 { $temp = explode($start, $str, 2); $content = explode($end, $temp[1], 2); return $content[0]; }
There is also a function to save the collected content:
function writelog($str) { @unlink("log.txt"); $open=fopen("log.txt","a" ); fwrite($open,$str); fclose($open); }
Sometimes the collected content is inconsistent with the content we view through the browser, causing us to not be able to find the correct regular expression, here you can Open the saved txt file and find the correct intercepted string in it.
If you need to collect even pictures, you need to use the picture function:
function getImage($url, $filename='', $dirName, $fileType, $type=0) { if($url == ''){return false;} //获取文件原文件名 $defaultFileName = basename($url); //获取文件类型 $suffix = substr(strrchr($url,'.'), 1); if(!in_array($suffix, $fileType)){ return false; } //设置保存后的文件名 $filename = $filename == '' ? time().rand(0,9).'.'.$suffix : $defaultFileName; //获取远程文件资源 if($type){ $ch = curl_init(); $timeout = 5; curl_setopt($ch, CURLOPT_URL, $url); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout); $file = curl_exec($ch); curl_close($ch); }else{ ob_start(); readfile($url); $file = ob_get_contents(); ob_end_clean(); } //设置文件保存路径 $dirName = $dirName.'/'.date('Y', time()).'/'.date('m', time()).'/'.date('d',time()).'/'; if(!file_exists($dirName)){ mkdir($dirName, 0777, true); } //保存文件 $res = fopen($dirName.$filename,'a'); fwrite($res,$file); fclose($res); return $dirName.$filename; }
Add the collection code. Since the collection code is added here to prevent submission, you can directly Above;
Let’s take a product page on Amazon as an example: enter a product link:
Look at the collection results as shown below. Only the content is displayed here. It is relatively simple to add it to the database. There will be time to introduce the collection of automatically entering lower-level links or automatically turning pages.
The above is the detailed content of Use file_get_contents and curl to write collection. For more information, please follow other related articles on the PHP Chinese website!

curl和Pythonrequests都是发送HTTP请求的强大工具。虽然curl是一种命令行工具,可让您直接从终端发送请求,但Python的请求库提供了一种更具编程性的方式来从Python代码中发送请求。将curl转换为Pythonrequestscurl命令的基本语法如下所示:curl[OPTIONS]URL将curl命令转换为Python请求时,我们需要将选项和URL转换为Python代码。这是一个示例curlPOST命令:curl-XPOSThttps://example.com/api

在Linux下更新curl版本,您可以按照以下步骤进行操作:检查当前curl版本:首先,您需要确定当前系统中安装的curl版本。打开终端,并执行以下命令:curl--version该命令将显示当前curl的版本信息。确认可用的curl版本:在更新curl之前,您需要确定可用的最新版本。您可以访问curl的官方网站(curl.haxx.se)或相关的软件源,查找最新版本的curl。下载curl源代码:使用curl或浏览器,下载您选择的curl版本的源代码文件(通常为.tar.gz或.tar.bz2

使用Java的File.length()函数获取文件的大小文件大小是在处理文件操作时很常见的一个需求,Java提供了一个很方便的方法来获取文件的大小,即使用File类的length()方法。本文将介绍如何使用该方法来获取文件的大小,并给出相应的代码示例。首先,我们需要创建一个File对象来表示我们想要获取大小的文件。以下是创建File对象的方法:Filef

PHP8.1发布:引入curl多个请求并发处理近日,PHP官方发布了最新版本的PHP8.1,其中引入了一个重要的特性:curl多个请求并发处理。这个新特性为开发者提供了一个更加高效和灵活的方式来处理多个HTTP请求,极大地提升了性能和用户体验。在以往的版本中,处理多个请求往往需要通过创建多个curl资源,并使用循环来分别发送和接收数据。这种方式虽然能够实现目

php blob转file的方法:1、创建一个php示例文件;2、通过“function blobToFile(blob) {return new File([blob], 'screenshot.png', { type: 'image/jpeg' })}”方法实现Blob转File即可。

从头到尾:如何使用php扩展cURL进行HTTP请求引言:在Web开发中,经常需要与第三方API或其他远程服务器进行通信。而使用cURL进行HTTP请求是一种常见而强大的方式。本文将介绍如何使用php扩展cURL来执行HTTP请求,并提供一些实用的代码示例。一、准备工作首先,确保php已安装cURL扩展。可以在命令行执行php-m|grepcurl查

PHPCurl中如何处理网页的301重定向?在使用PHPCurl发送网络请求时,时常会遇到网页返回的301状态码,表示页面被永久重定向。为了正确处理这种情况,我们需要在Curl请求中添加一些特定的选项和处理逻辑。下面将详细介绍在PHPCurl中如何处理网页的301重定向,并提供具体的代码示例。301重定向处理原理301重定向是指服务器返回了一个30

在linux中,curl是一个非常实用的、用来与服务器之间传输数据的工具,是一个利用URL规则在命令行下工作的文件传输工具;它支持文件的上传和下载,是综合传输工具。curl提供了一大堆非常有用的功能,包括代理访问、用户认证、ftp上传下载、HTTP POST、SSL连接、cookie支持、断点续传等等。


Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

AI Hentai Generator
Generate AI Hentai for free.

Hot Article

Hot Tools

SublimeText3 English version
Recommended: Win version, supports code prompts!

MinGW - Minimalist GNU for Windows
This project is in the process of being migrated to osdn.net/projects/mingw, you can continue to follow us there. MinGW: A native Windows port of the GNU Compiler Collection (GCC), freely distributable import libraries and header files for building native Windows applications; includes extensions to the MSVC runtime to support C99 functionality. All MinGW software can run on 64-bit Windows platforms.

Notepad++7.3.1
Easy-to-use and free code editor

PhpStorm Mac version
The latest (2018.2.1) professional PHP integrated development tool

ZendStudio 13.5.1 Mac
Powerful PHP integrated development environment