DeepSeek Open Source Week Day 5: 'Power Thruster' Fire-Flyer File System

DeepSeek open source project adds another weapon! Today, a new parallel file system 3FS was released to enable DeepSeek data access.

3FS (Fire-Flyer File System) makes full use of the high bandwidth of modern SSD and RDMA networks to achieve aggregation read throughput of up to 6.6 TiB/s in a 180-node cluster, and a throughput of 3.66 TiB/min in the GraySort benchmark test of a 25-node cluster. The peak throughput of KVCache search for a single client node exceeds 40 GiB/s.

The system adopts a separate architecture, supports strong consistency semantics, and can effectively support training data preprocessing, data set loading, checkpoint saving/overloading, embedded vector search, and KVCache search in inference in V3/R1.

The DeepSeek team said that 3FS aims to provide efficient and reliable storage and access solutions for its data-intensive tasks, thereby accelerating the AI ​​research and development process.

The above is the detailed content of DeepSeek Open Source Week Day 5: 'Power Thruster' Fire-Flyer File System. For more information, please follow other related articles on the PHP Chinese website!