Robots.txt is the first file that search engines look at when they visit a website. It is a text file used to specify the scope of crawling of website content by search engines. When a search spider visits a site, it will first check whether robots.txt exists in the root directory of the site. If it exists, it will determine the scope of the visit based on the content in the file.
In the process of website construction, we will have some content that we do not want to be crawled by search engines or do not want it to appear on the Internet, so what should we do? ? How do I tell search engines not to crawl my xx content? This is where robots come in handy.
Robots.txt is the first file that search engines look at when visiting a website. The Robots.txt file tells the spider what files on the server can be viewed.
When a search spider visits a site, it will first check whether robots.txt exists in the root directory of the site. If it exists, the search robot will determine the scope of access based on the contents of the file; if If the file does not exist, all search spiders will be able to access all pages on the website that are not password protected.
Syntax: The simplest robots.txt file uses two rules:
• User-Agent: The robot to which the following rules apply
• Disallow: The web page to be blocked
But we need to pay attention to a few points:
1.robots.txt must be stored in the root directory of the website,
2. Its naming Must be robots.txt, and the file name must be all lowercase.
3.Robots.txt is the first page that search engines visit the website
4.Robots.txt must specify user-agent
robots.txt Misunderstandings
Misunderstanding 1: All files on my website need to be crawled by spiders, so there is no need for me to add the robots.txt file. Anyway, if the file does not exist, all search spiders will be able to access all pages on the website that are not password protected by default.
Whenever a user attempts to access a URL that does not exist, the server will record a 404 error (file cannot be found) in the log. Whenever a search spider looks for a robots.txt file that does not exist, the server will also record a 404 error in the log, so you should add a robots.txt to your website.
Misunderstanding 2: Setting all files in the robots.txt file to be crawled by search spiders can increase the inclusion rate of the website.
Even if the program scripts, style sheets and other files in the website are included by spiders, it will not increase the website's inclusion rate and will only waste server resources. Therefore, you must set it in the robots.txt file not to allow search spiders to index these files.
Specific files that need to be excluded are detailed in the article Tips on Using Robots.txt.
Misunderstanding 3: Search spiders waste server resources when crawling web pages. All search spiders set in the robots.txt file cannot crawl all web pages.
If this is the case, the entire website will not be indexed by search engines.
robots.txt usage tips
1. Whenever a user tries to access a URL that does not exist, the server will record a 404 error (File cannot be found) in the log ). Whenever a search spider looks for a robots.txt file that doesn't exist, the server will also record a 404 error in the log, so you should add a robots.txt to your site.
2. Website administrators must keep spider programs away from certain directories on the server - to ensure server performance. For example: most website servers have programs stored in the "cgi-bin" directory, so it is a good idea to add "Disallow: /cgi-bin" to the robots.txt file to prevent all program files from being indexed by spiders. Can save server resources. Files that do not need to be crawled by spiders in general websites include: background management files, program scripts, attachments, database files, encoding files, style sheet files, template files, navigation pictures and background pictures, etc.
The following is the robots.txt file in VeryCMS:
User-agent: *
Disallow: /admin/ Background management file
Disallow: / require/ Program file
Disallow: /attachment/ Attachment
Disallow: /images/ Picture
Disallow: /data/ Database file
Disallow: / template/ template file
Disallow: /css/ style sheet file
Disallow: /lang/ encoding file
Disallow: /script/ script file
3. If your website has dynamic web pages, and you create static copies of these dynamic web pages to make them easier for search spiders to crawl. Then you need to set up settings in the robots.txt file to prevent dynamic web pages from being indexed by spiders to ensure that these web pages will not be regarded as containing duplicate content.
4. The robots.txt file can also directly include links to the sitemap file. Like this:
Sitemap: http://www.***.com/sitemap.xml
The search engine companies that currently support this include Google, Yahoo, Ask and MSN. Chinese search engine companies are obviously not in this circle. The advantage of this is that the webmaster does not need to go to the webmaster tools or similar webmaster sections of each search engine to submit his own sitemap file. The search engine spider will crawl the robots.txt file and read the content in it. sitemap path, and then crawl the linked web pages.
5. Proper use of the robots.txt file can also avoid errors during access. For example, you can’t let searchers go directly to the shopping cart page. Since there is no reason for the shopping cart to be included, you can set it in the robots.txt file to prevent searchers from entering the shopping cart page directly
The above is the detailed content of What is robots.txt?. For more information, please follow other related articles on the PHP Chinese website!

Many local businesses have recently experienced a significant drop in the number of reviews displayed on their Google Business Profiles. This issue, affecting both Google Search and Google Maps listings, began surfacing on Friday and has prompted wi

Website migrations: A minefield of SEO surprises Website migrations are notoriously challenging, even for seasoned technical SEOs. No matter how meticulous your planning, unexpected issues inevitably arise. Post-migration monitoring, therefore, is

Boosting SEO Revenue: Seven Proven Strategies to Increase Client Payments With marketing budgets tightening and Google reporting strong earnings, selling SEO services effectively is more crucial than ever. This article outlines seven proven methods t

A Google legal representative recently revealed that less than 1% of YouTube views originate from Google search clicks. The Statement: During a court proceeding, Attorney John Schmidtlein, representing Google, stated that "roughly less than 1%

Chrome browser improves Google search speed: use the Speculation Rules API to speed up search results Google announced that Chrome browser users will be able to search faster when using Google search. This improvement is driven by the utilization of the Speculation Rules API, which accelerates the overall search experience by prefetching search results. Google wrote in the Chrome Developer Blog: "Google Search has always used the Speculation Rules API to improve navigation speeds from search results pages to result links, and we have used some API features that can be used to create a new look at the latest update.

Google News is going fully automatic. Starting in March, Google will automatically generate all publication pages, ending manual customization options in Publisher Center. This follows an April 2024 announcement phasing out manual publication additi

SEO: A Love Story for Digital Marketers Many shy away from SEO's complexity and constant evolution. But for others, it's an all-consuming passion, a dynamic puzzle that keeps them engaged. This Valentine's Day, let's explore why SEO transcends algor

Google enhances its Merchant listing structured data guidelines with a new beta feature for member pricing. This update, announced by Google, introduces the validForMemberTier property (alongside the existing priceType property) to more accurately r


Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

Hot Tools

VSCode Windows 64-bit Download
A free and powerful IDE editor launched by Microsoft

Atom editor mac version download
The most popular open source editor

ZendStudio 13.5.1 Mac
Powerful PHP integrated development environment

MinGW - Minimalist GNU for Windows
This project is in the process of being migrated to osdn.net/projects/mingw, you can continue to follow us there. MinGW: A native Windows port of the GNU Compiler Collection (GCC), freely distributable import libraries and header files for building native Windows applications; includes extensions to the MSVC runtime to support C99 functionality. All MinGW software can run on 64-bit Windows platforms.

DVWA
Damn Vulnerable Web App (DVWA) is a PHP/MySQL web application that is very vulnerable. Its main goals are to be an aid for security professionals to test their skills and tools in a legal environment, to help web developers better understand the process of securing web applications, and to help teachers/students teach/learn in a classroom environment Web application security. The goal of DVWA is to practice some of the most common web vulnerabilities through a simple and straightforward interface, with varying degrees of difficulty. Please note that this software
