Tencent Cloud releases its self-developed big data high-performance computing engine Meson, with performance increased by up to 6 times

On July 4th, Tencent Cloud released its new self-developed big data high-performance computing engine Meson.

Meson can significantly provide better computing performance and save more computing resources for big data tasks in AI and other scenarios through the integrated application of software and hardware acceleration and intelligent technology.

For example, in the data lake scenario, Meson can help speed up single data query and analysis by 6 times. In the WeChat Reading "AI Book Questioning" project, Meson helped big data tasks save 90% of resource consumption.

Currently, Meson has been launched in the three major business lines of Tencent Cloud data lake, search analysis service, and cloud data warehouse. As a unified computing acceleration base, Meson provides acceleration services for enterprise big data business.

In the field of big data, data storage and computing are crucial core links. Improving the performance of computing and storage has always been the core task of big data technology.

Tencent Cloud big data high-performance computing engine Meson has made acceleration optimizations in many aspects:

• A series of software acceleration projects such as vectorized calculations at the software layer and JIT Compilation
• GPU and FPGA acceleration at the hardware layer
• Based on intelligence Intelligent technologies such as query optimization and intelligent load prediction allow big data analysis to automatically adjust processing strategies based on past experience, making data processing more efficient.

The processing and analysis of massive data are the first to experience the performance improvements brought by Meson.

In terms of data lakes that store massive amounts of data, Tencent Cloud has implemented vectorized computing capabilities that are fully compatible with Spark based on Meson, which can achieve higher performance when processing large-scale data.

In the high-frequency demand for search analysis, Tencent Cloud Elasticsearch Service (ES for short) combined with the Meson solution has become the world's first ES service to support GPU acceleration, significantly improving the efficiency of ES vector generation and vector retrieval.

For the enterprise-level data warehouse TCHouse, Meson can greatly improve the efficiency and quality of massive data analysis by using query optimization technology, vectorized computing mode and AI intelligent resource scheduling and other technical means.

So far, Tencent Cloud Big Data has continued to build domestic big data platforms for more than 1,000 medium and large customers, covering energy, manufacturing, power and other fields, such as State Grid, COMAC, Sany Heavy Industry, Shaanxi Construction Engineering, etc. Top customers in various industries.

The above is the detailed content of Tencent Cloud releases its self-developed big data high-performance computing engine Meson, with performance increased by up to 6 times. For more information, please follow other related articles on the PHP Chinese website!