Crawling and Searching Entire Domains with Diffbot

This tutorial demonstrates building a SitePoint search engine surpassing WordPress capabilities using Diffbot's structured data extraction. We'll leverage Diffbot's API for crawling and searching, employing a Homestead Improved environment for development.

Key Advantages:

• Diffbot excels at creating custom search engines beyond WordPress's functionality.
• Diffbot's Crawljob efficiently indexes and updates SitePoint's content. It allows customization of spidered URLs, notifications, crawl limits, refresh intervals, and new page processing.
• The Diffbot Search API efficiently searches indexed data, even incomplete datasets, using keywords, date ranges, specific fields, and boolean operators.
• Ideal for large websites or media conglomerates, consolidating content from multiple domains. However, always check website terms of service before crawling.

Implementation:

We'll create a SitePoint search engine in two steps:

1. A Crawljob to index SitePoint.com, automatically updating with new content.
2. A GUI (in a subsequent post) for querying the indexed data via the Search API.

The Diffbot Crawljob:

1. Spiders URLs based on a pattern (seed URL).
2. Processes spidered URLs using a specified API engine (e.g., Article API for SitePoint articles).

Creating a Crawljob (using the Diffbot PHP client):

1. Install the client: composer require swader/diffbot-php-client
2. Create job.php:

include 'vendor/autoload.php';
use Swader\Diffbot\Diffbot;
$diffbot = new Diffbot('my_token'); // Replace 'my_token' with your Diffbot token
$job = $diffbot->crawl('sp_search');
$job
    ->setSeeds(['https://www.sitepoint.com'])
    ->notify('your_email@example.com') // Replace with your email
    ->setMaxToCrawl(1000000)
    ->setMaxToProcess(1000000)
    ->setRepeat(1)
    ->setMaxRounds(0)
    ->setPageProcessPatterns([''])
    ->setOnlyProcessIfNew(1)
    ->setUrlCrawlPatterns(['^http://www.sitepoint.com', '^https://www.sitepoint.com'])
    ->setApi($diffbot->createArticleAPI('crawl')->setMeta(true)->setDiscussion(false));
$job->call();

Running php job.php creates the Crawljob, visible in the Diffbot Crawlbot interface.

Searching with the Search API:

Use the Search API to query the indexed data:

$search = $diffbot->search('author:"Bruno Skvorc"');
$search->setCol('sp_search');
$result = $search->call();

// Display results (example)
echo '<table><thead><tr><td>Title</td><td>Url</td></tr></thead><tbody>';
foreach ($search as $article) {
    echo '<tr><td>' . $article->getTitle() . '</td><td><a href="' . $article->getResolvedPageUrl() . '">Link</a></td></tr>';
}
echo '</tbody></table>';

The Search API supports advanced queries (keywords, date ranges, fields, boolean operators). Meta information is accessible via $search->call(true);. Crawljob status is checked using $diffbot->crawl('sp_search')->call();.

Conclusion:

Diffbot provides a powerful solution for creating custom search engines. While potentially costly for individuals, it offers significant benefits for teams and organizations managing large websites. Remember to respect website terms of service before crawling. The next part will focus on building the search engine's GUI.

Frequently Asked Questions (rephrased and consolidated):

This section answers common questions regarding crawling, indexing, and using Diffbot for large-scale data extraction. The original FAQ section is quite extensive and repetitive; this condensed version maintains the core information.

• Crawling vs. Indexing: Crawling gathers data; indexing organizes it for efficient search.
• How Diffbot Works: Diffbot uses AI and machine learning to extract structured data from web pages.
• Crawling an Entire Domain: Use the Crawlbot API, specifying the domain and parameters.
• Benefits of Diffbot: AI-powered data extraction, easy-to-use API, scalability.
• Search Engine Crawling: Bots scan websites, collecting data for indexing.
• Website Optimization for Crawling: Use clear site structure, SEO-friendly URLs, meta tags, and regular content updates.
• Sitemap's Role: Sitemaps guide crawlers to important pages.
• How Google's Search Engine Works: Crawling, indexing, and algorithm-based result ranking.
• Domain Crawling's Usefulness: SEO analysis, content aggregation, data mining.
• Preventing Page Crawling: Use a robots.txt file to restrict access.

The above is the detailed content of Crawling and Searching Entire Domains with Diffbot. For more information, please follow other related articles on the PHP Chinese website!