Building a Meta Tags Scraping API in Under Lines of Code-JS Tutorial-php.cn

Home

Web Front-end

JS Tutorial

Building a Meta Tags Scraping API in Under Lines of Code

DDD

Oct 21, 2024 pm 04:33 PM

Have you ever wondered how messaging apps like Whatsapp or Telegram let you see a preview of a link that you send?

Building a Meta Tags Scraping API in Under Lines of Code

Whatsapp and Telegram url previews

In this post, we'll be building a scraping API with Deno that accepts a URL and retrieves the meta tags for it, so we can get fields like the title, description, image and more from almost any website.

For example:

curl https://metatags.deno.dev/api/meta?url=https://dev.to

will give this result

{
  "last-updated": "2024-10-15 15:10:02 UTC",
  "user-signed-in": "false",
  "head-cached-at": "1719685934",
  "environment": "production",
  "description": "A constructive and inclusive social network for software developers. With you every step of your journey.",
  "keywords": "software development, engineering, rails, javascript, ruby",
  "og:type": "website",
  "og:url": "https://dev.to/",
  "og:title": "DEV Community",
  "og:image": "https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8lvvnvil0m75nw7yi6iz.jpg",
  "og:description": "A constructive and inclusive social network for software developers. With you every step of your journey.",
  "og:site_name": "DEV Community",
  "twitter:site": "@thepracticaldev",
  "twitter:title": "DEV Community",
  "twitter:description": "A constructive and inclusive social network for software developers. With you every step of your journey.",
  "twitter:image:src": "https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8lvvnvil0m75nw7yi6iz.jpg",
  "twitter:card": "summary_large_image",
  "viewport": "width=device-width, initial-scale=1.0, viewport-fit=cover",
  "apple-mobile-web-app-title": "dev.to",
  "application-name": "dev.to",
  "theme-color": "#000000",
  "forem:name": "DEV Community",
  "forem:logo": "https://media.dev.to/cdn-cgi/image/width=512,height=,fit=scale-down,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F8j7kvp660rqzt99zui8e.png",
  "forem:domain": "dev.to",
  "title": "DEV Community"
}

pretty cool, isn't it?

Meta tags and why do we need them

Meta tags are HTML elements that are used to provide additional information about a page to search engines and other clients.
These tags typically include a name or property attribute that defines the type of information, and a content attribute that contains the value of that information. Here’s an example of two meta tags:

<meta name="description" content="The <meta> HTML element represents metadata that cannot be represented by other HTML meta-related elements, like <base>, <link>, <script>, <style> or <title>.">
<meta property="og:image" content="https://developer.mozilla.org/mdn-social-share.cd6c4a5a.png">

The first tag provides a description of the page, while the second is an Open Graph tag that defines an image to display when the page is shared on social media.

One practical application of meta tags is building a bookmark manager. Instead of manually adding the title, description, and image for each bookmark, you can automatically scrape this information from the bookmarked URL using meta tags.

Open Graph

Open Graph is an internet protocol that was originally created by Facebook to standardize the use of metadata within a webpage to represent the content of a page, it helps social networks generate rich link previews.
Read more about it here.

Why Deno?

Deno has secure defaults, meaning it requires explicit permission for file, network, and environment access, reducing the risk of security vulnerabilities.
Deno is built on web standards, uses ES Modules, and aims to use Web Platform APIs (like fetch) over proprietary APIs, making Deno code very similar to the code you'll write in the browser - but still has some spec deviation from the browser.
Deno has built-in TypeScript support, allowing you to write TypeScript code without a build step.
Deno comes with a standard library that includes modules for common tasks like HTTP servers, file system operations, and more.
Deno provides a Linter, Formatter and Test runner, allowing you to use the platform instead of relying on third party packages or tools, making it an all-in-one tool for Javascript development.
Deno provides Deno Deploy which is a scalable platform for serverless JavaScript/Typescript applications that are globally distributed, ensuring minimal latency and maximum uptime.

The API that we're building will consist of two parts, a function for fetching and parsing the meta tags, and an API server which will respond to HTTP requests.

Getting the meta tags

Let's start by going to Deno Deploy and signing in.
After signing in click on "New Playground"
Building a Meta Tags Scraping API in Under Lines of Code
This we'll give us a hello world starting point.
Now we'll add function that is called getMetaTags that accepts a url and uses the Fetch API to get the HTML of the requested URL and passes it on to a package for HTML parsing (deno-dom).
To add deno-dom to our project we can use the jsr package manager:

curl https://metatags.deno.dev/api/meta?url=https://dev.to

Now we'll use the Fetch API to get the HTML as text:

{
  "last-updated": "2024-10-15 15:10:02 UTC",
  "user-signed-in": "false",
  "head-cached-at": "1719685934",
  "environment": "production",
  "description": "A constructive and inclusive social network for software developers. With you every step of your journey.",
  "keywords": "software development, engineering, rails, javascript, ruby",
  "og:type": "website",
  "og:url": "https://dev.to/",
  "og:title": "DEV Community",
  "og:image": "https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8lvvnvil0m75nw7yi6iz.jpg",
  "og:description": "A constructive and inclusive social network for software developers. With you every step of your journey.",
  "og:site_name": "DEV Community",
  "twitter:site": "@thepracticaldev",
  "twitter:title": "DEV Community",
  "twitter:description": "A constructive and inclusive social network for software developers. With you every step of your journey.",
  "twitter:image:src": "https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8lvvnvil0m75nw7yi6iz.jpg",
  "twitter:card": "summary_large_image",
  "viewport": "width=device-width, initial-scale=1.0, viewport-fit=cover",
  "apple-mobile-web-app-title": "dev.to",
  "application-name": "dev.to",
  "theme-color": "#000000",
  "forem:name": "DEV Community",
  "forem:logo": "https://media.dev.to/cdn-cgi/image/width=512,height=,fit=scale-down,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F8j7kvp660rqzt99zui8e.png",
  "forem:domain": "dev.to",
  "title": "DEV Community"
}

After getting the HTML, we can use deno-dom to parse it and then use standard DOM functions like querySelectorAll to get all the meta HTML elements, iterate on them and use getAttribute to get the name, property and content of each one of those tags:

<meta name="description" content="The <meta> HTML element represents metadata that cannot be represented by other HTML meta-related elements, like <base>, <link>, <script>, <style> or <title>.">
<meta property="og:image" content="https://developer.mozilla.org/mdn-social-share.cd6c4a5a.png">

Finally, we'll also query the

element of the page to add it as a field in our API:<br> <pre class="brush:php;toolbar:false">import { DOMParser, Element } from "jsr:@b-fuze/deno-dom"; </pre> <p>It isn't exactly a meta tag, but I think that it is a useful field, so it's going to be part of our API anyway. :)</p> <p>Our final getMetaTags function should look like this:<br> </p> <pre class="brush:php;toolbar:false"> const headers = new Headers(); headers.set("accept", "text/html,application/xhtml+xml,application/xml"); const res = await fetch(url, { headers }); const html = await res.text(); </pre> <h2> The server </h2> <p>For simplicity, I've decided to use Deno's built in http server which is just a simple Deno.serve() call.<br> Thanks to the fact that deno is built on web standards, we can use the built in Response object in the Fetch API to respond to requests.<br> </p> <pre class="brush:php;toolbar:false">curl https://metatags.deno.dev/api/meta?url=https://dev.to </pre> <p>Our server parses the request url, checks if we received a GET request to the /api/meta path, and calls the getMetaTags function we created and then returns the meta tags as the response body.</p> <p>We also add two headers, the first is Content-Type that is needed for the client to know the kind of data they're getting in the response, which in our case is a JSON response.</p> <p>The second header is Access-Control-Allow-Origin that allows our API to accept requests from specific origins, in our case I chose "*" to accept any origin, but you might want to change it to only accept request from your frontend's origin.<br> Note that the CORS headers will only affect requests made by the browser, meaning the browser will block the request according to the origin that is specified in the header, but directly calling the API from a server will still be possible. Read more about CORS here.</p> <p>You can now click on "Save & Deploy"<br> <img src="/static/imghwm/default1.png" data-src="https://img.php.cn/upload/article/000/000/000/172949959089268.jpg?x-oss-process=image/resize,p_40" class="lazy" alt="Building a Meta Tags Scraping API in Under Lines of Code"><br> Then wait for deno deploy to deploy your code to the playground:<br> <img src="/static/imghwm/default1.png" data-src="https://img.php.cn/upload/article/000/000/000/172949959198494.jpg?x-oss-process=image/resize,p_40" class="lazy" alt="Building a Meta Tags Scraping API in Under Lines of Code"><br> The url on the top right is your playground's url, copy it and add /api/meta?url=https://dev.to to see it in action, the url should look something like https://metatags.deno.dev/api/meta?url=https://dev.to<br> You should now see the API responding with dev.to's meta tags!<br> <img src="/static/imghwm/default1.png" data-src="https://img.php.cn/upload/article/000/000/000/172949959294656.jpg?x-oss-process=image/resize,p_40" class="lazy" alt="Building a Meta Tags Scraping API in Under Lines of Code"></p> <h2> Deployment </h2> <p>Using Deno deploy's playground means your code is technically already deployed, it is public and can be accessed by anyone.<br> For a simple API like the one we're building, a single file playground can be enough, but in many cases we'd like to scale our project further, for that you can use Deno deploy's Github export to make a proper code repository for you API, with support for automatic building on new code pushes:<br> <img src="/static/imghwm/default1.png" data-src="https://img.php.cn/upload/article/000/000/000/172949959428755.jpg?x-oss-process=image/resize,p_40" class="lazy" alt="Building a Meta Tags Scraping API in Under Lines of Code"><br> or from the playground's settings:<br> <img src="/static/imghwm/default1.png" data-src="https://img.php.cn/upload/article/000/000/000/172949959544011.jpg?x-oss-process=image/resize,p_40" class="lazy" alt="Building a Meta Tags Scraping API in Under Lines of Code"></p> <h2> Caveats </h2> <p>The scraping method presented in this post only works on websites that have meta tags in the html file that’s returned from the server, meaning server rendered or prerendered sites are more likely to return proper results, single page apps can also work as long as the meta tags are set in build time and not in runtime.</p> <h2> Conclusion </h2> <p>We've demonstrated how quick and simple it is to build and deploy an API with Deno, we've gone over Meta tags, and how we can use the Fetch API, a DOM parser and Deno's built in server to build a Meta tags scraping API in under 40 lines of code.</p> <p>To see the project built in this post you can check out the Deno deploy playground (You'll need to add /api/meta?url=https://dev.to to the url bar on the right to see an example response) or this github repository.</p> <hr> <h2> What Will You Build Next? </h2> <p>I hope this post has inspired you to explore the power of meta tags and Deno! Try building your own version of the API or integrate it into a project like a bookmark manager. </p> <p>Got stuck, have questions, or want to show off what you built? Drop a comment below or connect with me on Twitter/X – I’d love to hear from you! </p> <p>Check out my previous post about building a react state management library in under 40 lines of code here.</p>

The above is the detailed content of Building a Meta Tags Scraping API in Under Lines of Code. For more information, please follow other related articles on the PHP Chinese website!

Statement

The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Python vs. JavaScript: A Comparative Analysis for DevelopersMay 09, 2025 am 12:22 AM

The main difference between Python and JavaScript is the type system and application scenarios. 1. Python uses dynamic types, suitable for scientific computing and data analysis. 2. JavaScript adopts weak types and is widely used in front-end and full-stack development. The two have their own advantages in asynchronous programming and performance optimization, and should be decided according to project requirements when choosing.

Python vs. JavaScript: Choosing the Right Tool for the JobMay 08, 2025 am 12:10 AM

Whether to choose Python or JavaScript depends on the project type: 1) Choose Python for data science and automation tasks; 2) Choose JavaScript for front-end and full-stack development. Python is favored for its powerful library in data processing and automation, while JavaScript is indispensable for its advantages in web interaction and full-stack development.

Python and JavaScript: Understanding the Strengths of EachMay 06, 2025 am 12:15 AM

Python and JavaScript each have their own advantages, and the choice depends on project needs and personal preferences. 1. Python is easy to learn, with concise syntax, suitable for data science and back-end development, but has a slow execution speed. 2. JavaScript is everywhere in front-end development and has strong asynchronous programming capabilities. Node.js makes it suitable for full-stack development, but the syntax may be complex and error-prone.

JavaScript's Core: Is It Built on C or C ?May 05, 2025 am 12:07 AM

JavaScriptisnotbuiltonCorC ;it'saninterpretedlanguagethatrunsonenginesoftenwritteninC .1)JavaScriptwasdesignedasalightweight,interpretedlanguageforwebbrowsers.2)EnginesevolvedfromsimpleinterpreterstoJITcompilers,typicallyinC ,improvingperformance.

JavaScript Applications: From Front-End to Back-EndMay 04, 2025 am 12:12 AM

JavaScript can be used for front-end and back-end development. The front-end enhances the user experience through DOM operations, and the back-end handles server tasks through Node.js. 1. Front-end example: Change the content of the web page text. 2. Backend example: Create a Node.js server.

Python vs. JavaScript: Which Language Should You Learn?May 03, 2025 am 12:10 AM

Choosing Python or JavaScript should be based on career development, learning curve and ecosystem: 1) Career development: Python is suitable for data science and back-end development, while JavaScript is suitable for front-end and full-stack development. 2) Learning curve: Python syntax is concise and suitable for beginners; JavaScript syntax is flexible. 3) Ecosystem: Python has rich scientific computing libraries, and JavaScript has a powerful front-end framework.

JavaScript Frameworks: Powering Modern Web DevelopmentMay 02, 2025 am 12:04 AM

The power of the JavaScript framework lies in simplifying development, improving user experience and application performance. When choosing a framework, consider: 1. Project size and complexity, 2. Team experience, 3. Ecosystem and community support.

The Relationship Between JavaScript, C , and BrowsersMay 01, 2025 am 12:06 AM

Introduction I know you may find it strange, what exactly does JavaScript, C and browser have to do? They seem to be unrelated, but in fact, they play a very important role in modern web development. Today we will discuss the close connection between these three. Through this article, you will learn how JavaScript runs in the browser, the role of C in the browser engine, and how they work together to drive rendering and interaction of web pages. We all know the relationship between JavaScript and browser. JavaScript is the core language of front-end development. It runs directly in the browser, making web pages vivid and interesting. Have you ever wondered why JavaScr

See all articles