How to use Google BigQuery for big data calculation and storage in PHP development

As the amount of data continues to increase, traditional database management systems can no longer meet the storage and computing needs of big data. Google BigQuery, as a new type of cloud storage and computing service, is used by more and more enterprises and developers. This article will introduce how to use Google BigQuery for big data calculation and storage in PHP development.

1. What is Google BigQuery

Google BigQuery is a powerful cloud big data analysis service that can efficiently query massive data through SQL statements and store it through Google Cloud Storage. Google BigQuery enables rapid analysis of data without the expense and stress of any server or database management. Google BigQuery supports SQL language and can handle petabyte-level data.

2. What preparations are needed to use Google BigQuery

1. Google Cloud Platform account: Before using Google BigQuery, you need to apply for a Google Cloud Platform account and activate the Google BigQuery service.
2. Google Cloud SDK installation: Using Google BigQuery in local development requires installing the Google Cloud SDK and making the necessary configurations. Installation and configuration can be done via the official website or the command line.
3. API Credentials: Before developing with the Google BigQuery API, you need to obtain Google API credentials. You need to enable the Google BigQuery API on the Google Cloud Console first and create an API credential. Types of credentials include OAuth2.0 client ID, service account key, and API key. Among them, the OAuth2.0 client ID is suitable for use in web applications, the service account key is suitable for use in applications executed on the backend, and the API key is suitable for use in simple HTTP/REST applications.

3. Use Google BigQuery for data query

In PHP development, we can use the Google API client library to use the Google BigQuery API. First, you need to create a project in the Google Cloud Console and enable the BigQuery API service in the project. Then, create a Service Account in Google Cloud Console and obtain the credentials.json credentials file. Finally, download and install the Google API PHP client library.

The following is the code implementation of using Google BigQuery for data query:

<?php
require_once __DIR__ . '/vendor/autoload.php';
putenv('GOOGLE_APPLICATION_CREDENTIALS=' . __DIR__ . '/credentials.json');
$client = new Google_Client();
$client->useApplicationDefaultCredentials();
$client->addScope(Google_Service_Bigquery::BIGQUERY);

// 设置查询选项
$options = [
    'useLegacySql' => false
];

// 查询SQL语句
$sql = 'SELECT count(*) as count FROM `project.dataset.table`';

// 创建BigQuery服务对象
$service = new Google_Service_Bigquery($client);

// 从BigQuery查询数据
$results = $service->jobs->query('project-id', new Google_Service_Bigquery_QueryJobConfiguration([
    'query' => $sql,
    'useLegacySql' => false
]));

// 从结果中获取行数据
if ($rows = $results->getRows()) {
    $count = $rows[0]['f'][0]['v'];
    echo 'count: ' . $count . PHP_EOL;
}

In the above code, we use the Google API PHP client library to create a BigQuery service object. Then, we set the query options, query the data through SQL statements, and obtain row data from the query results. Finally, we can perform data processing on the query results as needed.

4. Use Google BigQuery for data storage

In Google BigQuery, we can store data into the data set in various ways, including batch insertion, streaming data insertion, table import, etc. Below we will take batch insertion as an example to introduce how to use Google BigQuery for data storage.

1. Create datasets and tables

Before using Google BigQuery for data storage, you need to create a dataset and table in the Google Cloud Console. By creating datasets and tables, we can specify the data type and structure for the data we want to store.

2. Install the PHP extension of Google Cloud BigQuery

Using Google BigQuery for data storage requires the installation of the PHP extension of Google Cloud BigQuery, which can be installed on the official website or the command line. After the installation is complete, you need to add the Google Cloud BigQuery extension to the PHP configuration file.

3. Writing a PHP program

The following is an example of a PHP program using Google BigQuery for data storage:

<?php
require_once __DIR__ . '/vendor/autoload.php';
putenv('GOOGLE_APPLICATION_CREDENTIALS=' . __DIR__ . '/credentials.json');
$client = new Google_Client();
$client->useApplicationDefaultCredentials();
$client->addScope(Google_Service_Bigquery::BIGQUERY);

// 数据集和表的名称
$datasetName = 'project.dataset';
$tableName = 'table';

// 插入数据
$service = new Google_Service_Bigquery($client);
$rows = [
    ['column1' => 'value1', 'column2' => 123],
    ['column1' => 'value2', 'column3' => 'value3']
];
$service->tabledata->insertAll($projectId, $datasetName, $tableName, new Google_Service_Bigquery_TableDataInsertAllRequest([
    'rows' => $rows
]));

In the above example, we use Google API The PHP client library creates a BigQuery service object, specifies the name of the data set and table to be stored, and inserts data into the data table through the tabledata->insertAll() method. Among them, $rows is the row data to be inserted, and each row data is an associative array (the key is the column name and the value is the column value).

Summary

As a new type of cloud storage and computing service, Google BigQuery provides developers with powerful big data analysis capabilities. In PHP development, we can use the Google BigQuery API through the Google API PHP client library to achieve efficient query and storage of data. I hope this article will help you use Google BigQuery in PHP development.

The above is the detailed content of How to use Google BigQuery for big data calculation and storage in PHP development. For more information, please follow other related articles on the PHP Chinese website!