


With the advent of the Industry 4.0 era, the production environment has become more efficient with the introduction of digitization and automation. At the same time, people are beginning to pay attention to the potential value of the massive data brought by smart devices, but how to efficiently store the data generated by smart devices and how to better analyze the massive data has become a problem. Traditional database models and storage methods can no longer meet these needs. Therefore, the time series database emerged as the times require, aiming to achieve efficient data storage and query, and help better explore the potential value of data
Faced with such a situation, Tsinghua University in 2015 Started the development of IoTDB. On September 23, 2020, Apache IoTDB graduated and became an Apache Top-Level Project. It is currently the only Apache Foundation top-level project initiated by Chinese universities and is also the only open source project in the field of IoT data management under the Apache Foundation. In October 2021, the Apache IoTDB core team founded Tianmou Technology and continues to operate IoTDB to help industrial users solve the problems of data "storage, search, and use".
Regarding the core technology developed by Apache IoTDB, several participants collaborated to publish a review paper, which elaborated on the design of IoTDB in detail and completely. The article takes an industrial company that needs to manage tens of thousands of excavators as an example and describes the requirements: "The data is first packaged into the device, and then sent to the server through the 5G mobile network. In the server, the data is written into the time series database, For OLTP queries. Finally, data scientists can load data from the database into the big data platform for complex analysis and prediction, that is, OLAP tasks."
- Paper address: https://dl.acm.org/doi/abs/10.1145/3589775
- ##Project address: https://github.com/ apache/iotdb
The key points of the paper include the following parts:
1. Design of data model: Organization of time series at the logical level and storage in physical schema;
2. TsFile file Format: Self-developed columnar storage file format, which also meets the efficiency of writing, querying, etc.;
3. IoTDB engine : Mainly includes storage engines, query engines, etc.;
Distributed solutions refer to decomposing a task or problem into multiple subtasks , and allocate these subtasks to multiple computers or nodes for processing. This solution improves system reliability, scalability and performance. By distributing tasks to multiple computers, the load on a single computer can be reduced and the concurrent processing capabilities of the system can be improved. At the same time, distributed solutions can also enhance the fault tolerance of the system through redundant backup and failover. Even if a node fails, the system can still continue to run. In today's big data and cloud computing environment, distributed solutions have become a common architectural pattern and are widely used in various fields, such as distributed databases, distributed storage systems, distributed computing platforms, etc.
For the following content, we will explain these key parts in more detail
Detailed interpretation
Requires data model design
As shown in the figure below, we use The tree structure is used to meet high-intensity write operations, and can effectively handle the common delayed data arrival problem in IoT scenarios
In the tree structure, each leaf node represents A sensor, each sensor has a corresponding device. As shown in the bottom two levels in the figure, the same applies to the upward levels
(2) The logical structure has been explained in the previous article, Now we will look at the implementation of the physical structure, which mainly includes two parts: time series (Time series) and sequence family (Series family). The figure below shows that each time series consists of two attributes: time and value. The time series is located through the complete path from the root node to the leaf node. The above figure shows the concept of sequence cluster. A sequence cluster may contain multiple devices, and their data will be stored together in TsFile (a file structure, which will be explained later)
What needs to be rewritten is: 2. TsFile file format design
TsFile is Apache IoTDB's self-developed columnar storage file format. Its structure is shown in the figure below:
When designing TsFile, the research team mainly focused on solving the following problems:
- Save space and compress data as much as possible
- Reduce the number of files
- Will query the time series together in the physical location Close to
- Reduce disk fragmentation
- Efficient access
Mainly provided The solution is:
- Column storage: eliminates null values, saves disk usage; data access locality
- time Sequence encoding: Taking advantage of the unique characteristics of time series in IoT scenarios
- Frequency domain encoding: Frequency domain analysis of time series is widely performed in signal processing
- Specific structure analysis: Page (Page) is the basic storage unit. Chunk contains multiple Pages. The pages in a chunk belong to the same time series and are variable in size; Chunk Group contains multiple Chunks, multiple pages in a group. Chunk belongs to one or more series of devices written within the same period of time. They are placed in continuous disk space because they are often queried together; Block is in memory, and the written block group is first in memory. Buffering is performed in TsFile, and when the memory reaches the threshold, all block groups are flushed to TsFile; the index (FileIndex) records information at the end of the file for data access.
The content that needs to be rewritten is: 3. IoTDB engine
In this part, researchers mainly focus on delayed arrival, efficient query processing and the design of SQL-like queries in IoT scenarios. The structure of the IoTDB engine is shown in the following figure:
In the figure, we can see that the storage engine part is mainly used to process the writing of TsFile, Read and manage. In this part, automatic delay separation technology is adopted (as shown in the figure below)
In most cases, when the time range in TsFile Lazy data separation is recommended when there is no overlap. However, for situations where most data is unordered, lazy data separation is not recommended
After rewriting: Another important component is the query engine, which is responsible for converting SQL Queries are converted into operators that can be executed in the database. At the same time, in order to adapt to industrial IoT scenarios, Apache IoTDB has also designed a rich time series data query function
The content that needs to be rewritten is: 4. Decentralized solution
TsFile can be distributed on HDFS and operated by Spark. In addition, it also provides native solutions for better handling of data distribution and query processing, including partition replication, NB-Raft replication and dynamic read consistency
Comparison results
In the paper, we compare TsFile and IoTDB, which are state-of-the-art file formats and timing databases widely used in industry. Through the following figure, we demonstrate the advantages of Apache IoTDB in many aspects
The above two figures show TsFile’s advantages in write throughput, read time cost and synchronization performance. This is mainly due to TsFile’s IoT-aware structure design, which avoids storing redundant information such as deviceId. Although there is no obvious advantage in disk usage of TsFile, this is because a more granular index is built, resulting in more space taken up. However, this sacrifice can lead to extraordinary query time improvements, as we can see a clear advantage in read time cost
in the above chart It can be clearly seen that IoTDB exhibits better performance in almost all tests, including higher write throughput and lower write latency
In the above experiments, we observed that IoTDB showed better performance when the query data size was larger. Especially in large-scale data aggregation, the advantages of IoTDB are particularly obvious
Summary
This paper introduces a new method called Apache IoTDB A new time series data management system that adopts an open architecture and is specifically designed to support real-time query and big data analysis for IoT applications. The system includes a new time series file format called TsFile, which uses column storage to store times and values to avoid null values and achieve effective compression. Based on TsFile, the IoTDB engine uses an LSM tree-like strategy to handle high-intensity writes and can handle the common delayed data arrival problem in IoT scenarios. Rich scalable query functions and pre-computed statistical information in TsFile enable IoTDB to efficiently handle OLTP and OLAP tasks
IoTDB has become better able to cope with industrial IoT scenarios A new type of database, which is the result of the above technology
The above is the detailed content of Apache IoTDB: an innovative database that solves storage, query and usage problems in industrial IoT scenarios. For more information, please follow other related articles on the PHP Chinese website!

ai合并图层的快捷键是“Ctrl+Shift+E”,它的作用是把目前所有处在显示状态的图层合并,在隐藏状态的图层则不作变动。也可以选中要合并的图层,在菜单栏中依次点击“窗口”-“路径查找器”,点击“合并”按钮。

ai橡皮擦擦不掉东西是因为AI是矢量图软件,用橡皮擦不能擦位图的,其解决办法就是用蒙板工具以及钢笔勾好路径再建立蒙板即可实现擦掉东西。

虽然谷歌早在2020年,就在自家的数据中心上部署了当时最强的AI芯片——TPU v4。但直到今年的4月4日,谷歌才首次公布了这台AI超算的技术细节。论文地址:https://arxiv.org/abs/2304.01433相比于TPU v3,TPU v4的性能要高出2.1倍,而在整合4096个芯片之后,超算的性能更是提升了10倍。另外,谷歌还声称,自家芯片要比英伟达A100更快、更节能。与A100对打,速度快1.7倍论文中,谷歌表示,对于规模相当的系统,TPU v4可以提供比英伟达A100强1.

ai可以转成psd格式。转换方法:1、打开Adobe Illustrator软件,依次点击顶部菜单栏的“文件”-“打开”,选择所需的ai文件;2、点击右侧功能面板中的“图层”,点击三杠图标,在弹出的选项中选择“释放到图层(顺序)”;3、依次点击顶部菜单栏的“文件”-“导出”-“导出为”;4、在弹出的“导出”对话框中,将“保存类型”设置为“PSD格式”,点击“导出”即可;

Yann LeCun 这个观点的确有些大胆。 「从现在起 5 年内,没有哪个头脑正常的人会使用自回归模型。」最近,图灵奖得主 Yann LeCun 给一场辩论做了个特别的开场。而他口中的自回归,正是当前爆红的 GPT 家族模型所依赖的学习范式。当然,被 Yann LeCun 指出问题的不只是自回归模型。在他看来,当前整个的机器学习领域都面临巨大挑战。这场辩论的主题为「Do large language models need sensory grounding for meaning and u

ai顶部属性栏不见了的解决办法:1、开启Ai新建画布,进入绘图页面;2、在Ai顶部菜单栏中点击“窗口”;3、在系统弹出的窗口菜单页面中点击“控制”,然后开启“控制”窗口即可显示出属性栏。

ai移动不了东西的解决办法:1、打开ai软件,打开空白文档;2、选择矩形工具,在文档中绘制矩形;3、点击选择工具,移动文档中的矩形;4、点击图层按钮,弹出图层面板对话框,解锁图层;5、点击选择工具,移动矩形即可。

引入密集强化学习,用 AI 验证 AI。 自动驾驶汽车 (AV) 技术的快速发展,使得我们正处于交通革命的风口浪尖,其规模是自一个世纪前汽车问世以来从未见过的。自动驾驶技术具有显着提高交通安全性、机动性和可持续性的潜力,因此引起了工业界、政府机构、专业组织和学术机构的共同关注。过去 20 年里,自动驾驶汽车的发展取得了长足的进步,尤其是随着深度学习的出现更是如此。到 2015 年,开始有公司宣布他们将在 2020 之前量产 AV。不过到目前为止,并且没有 level 4 级别的 AV 可以在市场


Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

AI Hentai Generator
Generate AI Hentai for free.

Hot Article

Hot Tools

SAP NetWeaver Server Adapter for Eclipse
Integrate Eclipse with SAP NetWeaver application server.

MinGW - Minimalist GNU for Windows
This project is in the process of being migrated to osdn.net/projects/mingw, you can continue to follow us there. MinGW: A native Windows port of the GNU Compiler Collection (GCC), freely distributable import libraries and header files for building native Windows applications; includes extensions to the MSVC runtime to support C99 functionality. All MinGW software can run on 64-bit Windows platforms.

VSCode Windows 64-bit Download
A free and powerful IDE editor launched by Microsoft

MantisBT
Mantis is an easy-to-deploy web-based defect tracking tool designed to aid in product defect tracking. It requires PHP, MySQL and a web server. Check out our demo and hosting services.

mPDF
mPDF is a PHP library that can generate PDF files from UTF-8 encoded HTML. The original author, Ian Back, wrote mPDF to output PDF files "on the fly" from his website and handle different languages. It is slower than original scripts like HTML2FPDF and produces larger files when using Unicode fonts, but supports CSS styles etc. and has a lot of enhancements. Supports almost all languages, including RTL (Arabic and Hebrew) and CJK (Chinese, Japanese and Korean). Supports nested block-level elements (such as P, DIV),
