Take stock of more than 20 powerful and free data sources that anyone can use to build AI

When we talk about artificial intelligence in business and society today, we are actually referring to machine learning. Machine learning is an application that uses an algorithm (a set of instructions) to become better and better at performing a specific task as it is exposed to more and more data relevant to that task.

These tasks can be anything from answering questions, creating text or images (as apps like ChatGPT or Dall-E can do) to recognizing images (computer vision) or getting a self-driving car from point A to Navigate to location B.

Enterprises that want to train their own machine learning algorithms to automate daily tasks need data sources to support these tasks.

What types of data are there?

Enterprise data is usually divided into two categories - internal data and external data.

• Internal data is data collected by the enterprise organization itself from its operations, which usually includes financial data, customer feedback data, human resources data, operational data, and other data from more sources. Data collected by an organization in the course of monitoring its own operations is called proprietary data and is valuable because it provides information about a specific business.
• External data is data from sources external to the organization, typically collected from third-party data sources listed below. If the data is freely available to anyone, it is called open data.

In addition, data can also be divided into structured, unstructured or semi-structured data.

• Structured data is information that fits nicely and neatly into a table—for example, sales data showing what products a business sells, when, where, and for what price is internally structured data. Alternatively, businesses may choose to analyze historical market data and economic indicators to predict future trends in their markets (structured external data).
• Unstructured data is everything else, such as images, videos, text, and social media content, which can certainly contain valuable insights but is more difficult to analyze. However, AI has proven particularly useful for extracting meaning from unstructured data. For example, image recognition algorithms can tell businesses useful information about customer behavior by analyzing in-store CCTV images (internal unstructured data), and also by analyzing business-related images posted on social media (unstructured external data) to find valuable insights.

Fortunately, data is everywhere. Governments, research institutions, private companies, NGOs all provide data for free for research and even commercial purposes. So here are some of the best sources of free online data available in 2023.

Data Search Engine and Repository

• Google Dataset Search – This is essentially a search engine for Google’s cataloged datasets; use this search engine to find almost anything you might need data.
• AWS Open Data Search - Another data set search engine provided by Amazon's AWS.
• Microsoft Research Open Data - A free, open dataset collected by Microsoft with a primary focus on science.
• UCI Machine Learning Repository - A repository of more than 600 open datasets curated and maintained by the University of California, Irvine, that can be used to train machine learning algorithms.
• Kaggle Datasets – The online data science platform Kaggle also offers a curated catalog of datasets covering everything from university rankings to Google search trends, retail sales, online movie reviews and crime statistics.
• Reddit R/Datasets - Huge datasets submitted by users of the online community site Reddit, covering hundreds of topics.

Datasets for Governments and Intergovernmental Organizations

• Data.Gov - an open data portal provided by the U.S. government, hosting one million data published by government agencies Nearly a quarter of the data is concentrated.
• Data.Census.Gov – If you’re looking specifically for U.S. demographic data, this is a great place to start!
• Data.EU – The European Union’s open data portal, containing data from EU organizations and data from member state governments.
• Data.gov.uk - An open dataset published by UK government agencies.
• World Health Organization Data - Data sets related to global health and well-being.
• World Bank Open Data - Data sets related to economic development, international financial markets, social indicators and environmental issues.

Image Data

• Google Open Images - Millions of images classified and labeled in various ways, used to train many different types of computer vision algorithms .
• ImageNet Open Dataset - Another dataset consisting of labeled images that is free for non-commercial machine learning applications.
• COCO Dataset - The Common Objects in Context (COCO) dataset contains over 200,000 images selected for training object detection and captioning algorithms.

Voice Data

• Mozilla Common Voice - an open recording data set that can be used to train any AI application involving speech.
• Audioset - Another dataset curated by Google, this one focuses on sound and contains hundreds of thousands of 10-second samples broken down into categories such as instruments, vehicles, and vocals.
• Million Song Dataset - Samples and metadata from one million contemporary popular music tracks.

Text Data

• Wikidata - Database download of Wikipedia articles in many different formats.
• Common Crawl - An open data repository scraped from the World Wide Web, best known for training GPU large language models for ChatGPT and other chatbots.

Other and Miscellaneous Datasets

• Amazon Reviews - A database of approximately 35 million Amazon product reviews, including product information and ratings.
• Waymo Open Dataset – Alphabet’s self-driving subsidiary Waymo has disclosed a large amount of data collected through self-driving vehicles, including data from cameras and LiDAR sensors.
• Apolloscape Dataset——More autonomous driving data is provided by Baidu’s open source Apollo platform.

The above is the detailed content of Take stock of more than 20 powerful and free data sources that anyone can use to build AI. For more information, please follow other related articles on the PHP Chinese website!