FATE 2.0 released: realizing interconnection of heterogeneous federated learning systems

• FATE 2.0Comprehensive upgrade to promote large-scale application of private computing federated learning

FATE open source platform announced the release of FATE 2.0 version, as the world's leading industrial-grade open source framework for federated learning. This update realizes the interconnection between federated heterogeneous systems and continues to enhance the interconnection capabilities of the privacy computing platform. This progress further promotes the development of large-scale applications of federated learning and privacy computing.

##FATE 2.0Designed with comprehensive interoperability as the design concept , using open source methods to transform the four levels of application layer, scheduling, communication, and heterogeneous computing (algorithms), realizes the integration of systems and systems, systems and algorithms, and algorithms and algorithms. The ability of heterogeneous interoperability.

##The design of FATE 2.0

is compatible with the "Financial Industrial Privacy Computing InteroperabilityAPITechnical Document》##[3] and other industry standards, before release, FATE 2.0 has completed interconnection and interoperability verification with multiple heterogeneous privacy computing platforms. Recently Beijing Financial Technology Industry Alliance mentioned in a document released that “the research team joined forces with the FATE open source community and leading technology companies to complete a five-party cross-platform, cross- The interoperability and joint debugging of algorithms have verified the feasibility and security of interface documents in supporting the interoperability of multi-party heterogeneous platforms".

Visit the following URL to get version 2.0 of FATE:

##https:/ /www.php.cn/link/99113167f3b816bdeb56ff1af6cec7af

##FATE 2.0

• Highlight OverviewApplication Layer Interconnection:
Building
• StandardCan Extended federated DSL, supporting application layer interconnection, unifiedDSLAdapting to multiple heterogeneous privacy computing platforms Task description
  • Scheduling layerInterconnection: Decoupling system modules from multiple levels toBuild an open and standardized interconnection and interoperability scheduling platform, Support task scheduling between multiple heterogeneous privacy computing platforms

  
  

  • Transmission Layer cross-site interconnection: Build open cross-site interconnection communication components, support multiple transmission modes and multiple communication protocols, can adapt to data transmission between multiple heterogeneous privacy computing platforms, enhance transmission efficiency and System stability

  
  

  • ##Federal heterogeneous computing interconnection: Build Distributed and plain ciphertextTensor/DataFrame, DecouplingHE、MPC and other security protocols and federated algorithm protocols, facilitate the interconnection of federated heterogeneous computing engines

  
  

  ## Core algorithm migration and expansion, algorithm development experience and performance are significantly enhanced: using distributed, plain-ciphertext
  • Tensor/Dataframe programming model to realize core algorithm migration and expansion; core algorithm performance Improvement: PSI Privacy protection intersection algorithm performance improvement 1.8 times, vertical federation SSHE-LR Algorithm performance is improved 4.3 times, vertical federated neural network algorithm performance is improved 143 times, etc.

  

  FATE 2.0

  Schematic diagram of the overall interconnection architecture

  
  

  • FATE 2.0 Feature List

  FATE-Client 2.0: Building a scalable federated DSL,Support application layer interconnection

  1.

  Introducing a new scalable and standardized federated DSL IR, namely federated modeling Process DSL standardized middle layer representation

  2. Supports compiling python client federated modeling process code into DSL IR

  3. DSL IR Protocol extension enhancement: Support multi-party asymmetric scheduling

  4. Support FATE's standardized federated DSL IR and other protocol conversion, such asMutual conversion of Beijing Financial Technology Industry Alliance Interoperability BFIA protocol

  5. Complete the migration of Flow Cli and Flow SDK functions

  
  

  FATE-Flow 2.0: Building an open and standardized interconnection scheduling platform

  1. Adapt to the scalable and standardized FATE 2.0 federated DSL IR

  2. Build an interconnection scheduling layer framework and support other protocols through adapters, such as "Privacy Computing Interconnection" The control layer interface involved in the "Interoperability API Technical Document".

  3. Optimize process scheduling, the scheduling logic is decoupled and customizable, and priority scheduling is added

  4. Optimize algorithm component scheduling, support container-level algorithm loading, and improve support for cross-platform heterogeneous scenarios

  5. Optimization Multi-version algorithm component registration supports registration of component operating modes

  6. Federated DSL IR extension enhancement: supports multi-party asymmetric scheduling

  7. Optimize client authentication logic and support permission management of multiple clients

  8. Optimize RESTful The interface makes the input parameter fields and types, return fields and status codes clearer

  9． Added OFX (Open Flow Exchange) module: encapsulates the scheduling client, allowing cross-platform scheduling

  10.Supports the new communication engine OSX, while working with FATE Flow All engines in 1.x remain compatible

  11. The system layer and algorithm layer are decoupled, and the system configuration is moved from the FATE repository to the Flow repository

  12. Released the FATE Flow package in PyPI, and added a service-level CLI for service management

  13. Complete 1.x main function migration

  
  

  OSX (Open Site Exchange ) 1.0: Building open cross-site interconnection and communication components

  1. ReferenceBeijing Finance The "Financial Industry Privacy Computing Interoperability API Technical Documentation" released by the Technology Industry Alliance implements the Interconnection transmission interface. The transmission interface is compatible with FATE 1.X version and FATE 2.X version
  2. Support grpc synchronous transmission and streaming, support TLS secure transmission protocol, compatible with FATE 1.X rollsite component
  3. Support Http 1.X protocol Transmission, supports TLS secure transmission protocol
  4. Supports message queue mode transmission, used to replace rabbitmq and pulsar components in FATE1.X
  5. Support eggroll and spark computing engines
  6. support networking as Exchange components and support FATE 1.X and FATE 2.X access
  7. Compare The rollsite component improves the logic of exception handling during transmission and provides more accurate log output for quickly locating exceptions
  8. The routing configuration is basically the same as the original rollsite, reducing the difficulty of transplantation
  9. Support http interface to modify the routing table and provide simple permission verification
  10. Improving the network connection management logic, reducing the risk of connection leakage and improving transmission efficiency
  11. Use different ports to handle access requests inside and outside the cluster, so that different security policies can be adopted for different ports

  
  

  ##FATE-Arch 2.0: Build a unified and standardized API to facilitate the interconnection of federated heterogeneous computing engines

  1. Context: Introducing "Context" to manage developer-friendly APIs, such as "distributed computing", "federated learning", "encryption" Algorithm", "Tensor Operation", "Metric" and "Input and Output Management"
  2. Tensor: Introducing Tensor data structure to handle local and distributed matrix operations , supports built-in heterogeneous acceleration; PHETensor abstraction layer optimization, using a variety of underlying PHE implementations through standard interfaces, free switching
  3. DataFrame: Introducing "DataFrame" Two-dimensional tabular data structure, used for data input and output and basic feature engineering. The new data block manager supports column multi-type management and feature anonymous logic; new statistics, comparison, indexing, data binning and transformation, etc. 30 operator interfaces
  4. Reconstruct Federation: Provide a unified federationcommunicationinterface, including unified serialization/deserialization control and a more friendly API
  5. Config: Provide unified configuration settings for FATE, including security configuration, system configuration and algorithm configuration
  6. Reconstruct "logger"： Customize logging details according to different usage methods and needs
  7. Launcher: A simplified federated program execution tool, especially suitable for stand-alone operation And local debugging
  8. Protocol layer: Support SSHE (hybridsecuritymulti-party computation and homomorphic encryption protocol), ECDH, secure aggregation protocol
  9. Integration of Deepspeed: supports training scheduling of distributed GPU clusters through Eggroll
  10. Experimental integration of Crypten: supports SMPC, more protocols will be added in the future and functions

  
  

  ##FATE-Component 2.0: Build standardized algorithms Components, adapted to different scheduling engines

  1. Introducing the component toolbox: encapsulating the machine learning module into a standard executable program
  2. Provide clear API through spec and loader to facilitate internal expansion and integration with external systems
  3. Input and output: further decouple FATE-Flow and provide a standardized black box calling process
  4. Component definition: supports type-based definition, automatically checks component parameters, supports multiple data and model input and output types, and multiple inputs

  
  

  FATE-ML 2.0: Core algorithm migration and extension,Algorithm development experience and performance are significantly enhanced

  1. Adopt distributed, clear and ciphertextTensor/Dataframe programming model to realize core algorithm migration and expansion:
  2. Data preprocessing: Add DataFrame Transformer, complete Reader, PSI, Union and DataSplit migration
  3. Feature engineering: Complete migration of HeteroFederatedBinning, HeteroFeatureSelection, DataStatistics, Sampling, FeatureScale and Pearson Correlation
  4. Federated training algorithm migration: including HeteroSecureBoost, HomoNN, HeteroCoordinatedLogisticRegressio, HeteroCoordinatedLinearRegression, SSHE-LogisticRegression and SSHE-LinearRegression
  5. New federated training algorithm Protocol: SSHE-HeteroNN

  based on MPC and homomorphic encryption hybrid protocol, FedPASS-HeteroNN

  # based on FedPASS protocol
  1. ##Performance is significantly improved
  2. PSIPrivacy protection intersection: tested on a data set of 100 million ids, and the intersection result is 100 million, the performance is 1.8 times that of FATE-1.11
  3. Vertical federated binning algorithm: one hundred thousand rows * thirty-dimensional features in guest, one hundred thousand rows * three hundred-dimensional features in host Tested on data, the performance is 1.5 times that of FATE-1.11
  4. Vertical federated SSHE-LR algorithm: 100,000 rows*30-dimensional features in guest, 100,000 rows*300-dimensional features in host Tested on feature data, the performance is 4.3 times that of FATE-1.11
  5. Longitudinal federated LR algorithm with coordinator: 100,000 rows*30-dimensional features in guest, 100,000 rows in host *Tested on data with three hundred dimensional features, the performance is 1.2 times that of FATE-1.11
  6. Longitudinal federated neural network (based on FedPass protocol): One hundred thousand rows in guest * Thirty dimensional features , tested on host 100,000 rows*300-dimensional feature data, the performance is basically consistent with the plain text, and the performance is 143 times that of FATE-1.11

  
  

  Eggroll 3.0: Comprehensive enhancements to system performance, availability and reliability

  ##1. JVM

  Enhancement

  Core component reconstruction: cluster-manager and node-manager components are fully rebuilt using Java language to ensure uniformity and improve performance
  Transmission component modification: remove the rollsite transmission component and replace it with a more efficient osx component
  Process management improvement: implement more advanced process management logic, display To reduce the risk of process leakage
  Data storage logic enhancement: data storage mechanism optimization, improve performance and reliability
  Concurrency control improvement: upgrade the original There is concurrency control logic in the component to improve performance
  Visual component: New visual component is added to facilitate monitoring of calculation information
  Log improvement: log System enhancement, output is more accurate, helping to detect abnormalities quickly
  2. Python

  Upgrade

  • roll_pair and egg_pair refactoring: support caller-controlled serialization and partitioning methods , Serialization is safe by the caller Unified party management
  • Automatic cleaning of intermediate tables: solves the problem of automatic cleaning of intermediate tables between federation and calculation without requiring additional operations on the caller
  • Unified configuration control: Introducing a flexible configuration system that supports direct transfer, configuration files and environment variables to meet diverse needs
  • Client PyPI installation: Eggroll 3.0.0 supports client through PyPI for easy installation
  • Log configuration optimization: The caller can customize the log format as needed
  • Code structure adjustment: The code base is streamlined and structured The logic is clearer and a lot of redundant code is removed

  
  

  
  

  • ##Gather the power of open source to help the development of the privacy computing industry

  Cross-industry and cross-institutional data integration in finance, telecommunications, medical, government affairs, advertising and marketing, wisdom Many scenes such as cities have a wide range of needs. Privacy computing has become a powerful tool for breaking down data barriers between industries, and interconnection is the whetstone for giving full play to this powerful tool. FATE 2.0 provides an open source framework to achieve interconnection and interoperability, solving a major pain point in the industry. Most privacy computing platforms can achieve the purpose of interacting and integrating with heterogeneous systems by implementing open interoperability interfaces.

  The launch of FATE 2.0

  provides strong support for the realization of interconnection between heterogeneous platforms, and continuous iteration shows commitment to continuous improvement of technology. It is not only about data privacy protection, but also about the development of the entire industry. In this process, privacy computing industry users and technology partners have more opportunities to participate. Through the joint efforts of the community, we can better address the challenges of data security and privacy protection, and lay a solid foundation for building a more secure and reliable digital society. The release of FATE 2.0 is a new chapter of industry cooperation and win-win. We look forward to more innovators and practitioners joining in to jointly promote the vigorous development of privacy computing technology.

  [1] National Data Bureau: The National Data Bureau and other departments have issued the "Data Notice of "Three-Year Action Plan (2024-2026)" of Elements Lv Ailin of the Academy of Information and Communications Technology and others: Progress and Trends in my country’s Data Element Market Cultivation

  [3] "Privacy in the Financial Industry Computing Interoperability API Technical Documentation v1.0 has been hosted in the FATE community warehouse: https://github.com/FederatedAI/InterOp

The above is the detailed content of FATE 2.0 released: realizing interconnection of heterogeneous federated learning systems. For more information, please follow other related articles on the PHP Chinese website!