search
HomeJavajavaTutorialIntroduction to Java multi-threaded batch data import method

This article brings you an introduction to the method of Java multi-threaded batch data import. It has certain reference value. Friends in need can refer to it. I hope it will be helpful to you.

Foreword: When encountering a large amount of data import, in order to increase the processing speed, you can choose to use multi-threading to batch process these processes. Common scenarios include:

  1. Importing large files into the database (this file is not necessarily a standard CSV importable file or requires certain processing in memory)
  2. Data synchronization (pulling data from a third-party interface and processing it and then writing it into your own database)

The above scenarios have one thing in common. The scenario of importing this type of data is simply to Data is moved from one data source to another data source, which must be divided into two steps

  1. Data reading: read data from the data source to the memory
  2. Data writing: Write the data in the memory to another data source. There may be data processing

And the speed of data reading is generally It is much faster than the data writing speed, that is, is fast to read and is slow to write.

Design ideas

Since the characteristics of the scenario are fast reading and slow writing, if multi-thread processing is used, it is recommended to data writingPartially transformed to multi-threading. And Data reading can be transformed into batch reading of data. To put it simply, there are two key points:

  1. Batch reading of data
  2. Multiple threads to write data

Example

Multiple threads The simplest solution for batch processing is to use a thread pool for processing. The following will use a service that simulates batch reading and writing, and a multi-threaded write call to this service as an example to show how to import multi-threaded batch data.

Simulation Service

import java.util.concurrent.atomic.AtomicLong;

/**
 * 数据批量写入用的模拟服务
 *
 * @author RJH
 * create at 2019-04-01
 */
public class MockService {
    /**
     * 可读取总数
     */
    private long canReadTotal;

    /**
     * 写入总数
     */
    private AtomicLong writeTotal=new AtomicLong(0);

    /**
     * 写入休眠时间(单位:毫秒)
     */
    private final long sleepTime;

    /**
     * 构造方法
     *
     * @param canReadTotal
     * @param sleepTime
     */
    public MockService(long canReadTotal, long sleepTime) {
        this.canReadTotal = canReadTotal;
        this.sleepTime = sleepTime;
    }

    /**
     * 批量读取数据接口
     *
     * @param num
     * @return
     */
    public synchronized long readData(int num) {
        long readNum;
        if (canReadTotal >= num) {
            canReadTotal -= num;
            readNum = num;
        } else {
            readNum = canReadTotal;
            canReadTotal = 0;
        }
        //System.out.println("read data size:" + readNum);
        return readNum;
    }

    /**
     * 写入数据接口
     */
    public void writeData() {
        try {
            // 休眠一定时间模拟写入速度慢
            Thread.sleep(sleepTime);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        // 写入总数自增
        System.out.println("thread:" + Thread.currentThread() + " write data:" + writeTotal.incrementAndGet());
    }

    /**
     * 获取写入的总数
     *
     * @return
     */
    public long getWriteTotal() {
        return writeTotal.get();
    }

}

Batch Data Processor

import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

/**
 * 基于线程池的多线程批量写入处理器
 * @author RJH
 * create at 2019-04-01
 */
public class SimpleBatchHandler {

    private ExecutorService executorService;

    private MockService service;
    /**
     * 每次批量读取的数据量
     */
    private int batch;
    /**
     * 线程个数
     */
    private int threadNum;

    public SimpleBatchHandler(MockService service, int batch,int threadNum) {
        this.service = service;
        this.batch = batch;
        //使用固定数目的线程池
        this.executorService = Executors.newFixedThreadPool(threadNum);
    }

    /**
     * 开始处理
     */
    public void startHandle() {
        // 开始处理的时间
        long startTime = System.currentTimeMillis();
        System.out.println("start handle time:" + startTime);
        long readData;
        while ((readData = service.readData(batch)) != 0) {// 批量读取数据,知道读取不到数据才停止
            for (long i = 0; i  service.writeData());
            }
        }
        // 关闭线程池
        executorService.shutdown();
        while (!executorService.isTerminated()) {//等待线程池中的线程执行完

        }
        // 结束时间
        long endTime = System.currentTimeMillis();
        System.out.println("end handle time:" + endTime);
        // 总耗时
        System.out.println("total handle time:" + (endTime - startTime) + "ms");
        // 写入总数
        System.out.println("total write num:" + service.getWriteTotal());
    }

}

Test Class

/**
 * SimpleBatchHandler的测试类
 * @author RJH
 * create at 2019-04-01
 */
public class SimpleBatchHandlerTest {

    public static void main(String[] args) {
        // 总数
        long total=100000;
        // 休眠时间
        long sleepTime=100;
        // 每次拉取的数量
        int batch=100;
        // 线程个数
        int threadNum=16;
        MockService mockService=new MockService(total,sleepTime);
        SimpleBatchHandler handler=new SimpleBatchHandler(mockService,batch,threadNum);
        handler.startHandle();
    }
}

Running Results

start handle time:1554298681755
thread:Thread[pool-1-thread-2,5,main] write data:1
thread:Thread[pool-1-thread-1,5,main] write data:2
...省略部分输出
thread:Thread[pool-1-thread-4,5,main] write data:100000
end handle time:1554299330202
total handle time:648447ms
total write num:100000

Analysis

The execution time in single-threaded case should be total*sleepTime, that is, 10000000ms, while the execution time after transformation to multi-threading is 648447ms.

【Related recommendations: Java Video Tutorial

The above is the detailed content of Introduction to Java multi-threaded batch data import method. For more information, please follow other related articles on the PHP Chinese website!

Statement
This article is reproduced at:segmentfault. If there is any infringement, please contact admin@php.cn delete
Are there any emerging technologies that threaten or enhance Java's platform independence?Are there any emerging technologies that threaten or enhance Java's platform independence?Apr 24, 2025 am 12:11 AM

Emerging technologies pose both threats and enhancements to Java's platform independence. 1) Cloud computing and containerization technologies such as Docker enhance Java's platform independence, but need to be optimized to adapt to different cloud environments. 2) WebAssembly compiles Java code through GraalVM, extending its platform independence, but it needs to compete with other languages ​​for performance.

What are the different implementations of the JVM, and do they all provide the same level of platform independence?What are the different implementations of the JVM, and do they all provide the same level of platform independence?Apr 24, 2025 am 12:10 AM

Different JVM implementations can provide platform independence, but their performance is slightly different. 1. OracleHotSpot and OpenJDKJVM perform similarly in platform independence, but OpenJDK may require additional configuration. 2. IBMJ9JVM performs optimization on specific operating systems. 3. GraalVM supports multiple languages ​​and requires additional configuration. 4. AzulZingJVM requires specific platform adjustments.

How does platform independence reduce development costs and time?How does platform independence reduce development costs and time?Apr 24, 2025 am 12:08 AM

Platform independence reduces development costs and shortens development time by running the same set of code on multiple operating systems. Specifically, it is manifested as: 1. Reduce development time, only one set of code is required; 2. Reduce maintenance costs and unify the testing process; 3. Quick iteration and team collaboration to simplify the deployment process.

How does Java's platform independence facilitate code reuse?How does Java's platform independence facilitate code reuse?Apr 24, 2025 am 12:05 AM

Java'splatformindependencefacilitatescodereusebyallowingbytecodetorunonanyplatformwithaJVM.1)Developerscanwritecodeonceforconsistentbehavioracrossplatforms.2)Maintenanceisreducedascodedoesn'tneedrewriting.3)Librariesandframeworkscanbesharedacrossproj

How do you troubleshoot platform-specific issues in a Java application?How do you troubleshoot platform-specific issues in a Java application?Apr 24, 2025 am 12:04 AM

To solve platform-specific problems in Java applications, you can take the following steps: 1. Use Java's System class to view system properties to understand the running environment. 2. Use the File class or java.nio.file package to process file paths. 3. Load the local library according to operating system conditions. 4. Use VisualVM or JProfiler to optimize cross-platform performance. 5. Ensure that the test environment is consistent with the production environment through Docker containerization. 6. Use GitHubActions to perform automated testing on multiple platforms. These methods help to effectively solve platform-specific problems in Java applications.

How does the class loader subsystem in the JVM contribute to platform independence?How does the class loader subsystem in the JVM contribute to platform independence?Apr 23, 2025 am 12:14 AM

The class loader ensures the consistency and compatibility of Java programs on different platforms through unified class file format, dynamic loading, parent delegation model and platform-independent bytecode, and achieves platform independence.

Does the Java compiler produce platform-specific code? Explain.Does the Java compiler produce platform-specific code? Explain.Apr 23, 2025 am 12:09 AM

The code generated by the Java compiler is platform-independent, but the code that is ultimately executed is platform-specific. 1. Java source code is compiled into platform-independent bytecode. 2. The JVM converts bytecode into machine code for a specific platform, ensuring cross-platform operation but performance may be different.

How does the JVM handle multithreading on different operating systems?How does the JVM handle multithreading on different operating systems?Apr 23, 2025 am 12:07 AM

Multithreading is important in modern programming because it can improve program responsiveness and resource utilization and handle complex concurrent tasks. JVM ensures the consistency and efficiency of multithreads on different operating systems through thread mapping, scheduling mechanism and synchronization lock mechanism.

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

WebStorm Mac version

WebStorm Mac version

Useful JavaScript development tools

ZendStudio 13.5.1 Mac

ZendStudio 13.5.1 Mac

Powerful PHP integrated development environment

SecLists

SecLists

SecLists is the ultimate security tester's companion. It is a collection of various types of lists that are frequently used during security assessments, all in one place. SecLists helps make security testing more efficient and productive by conveniently providing all the lists a security tester might need. List types include usernames, passwords, URLs, fuzzing payloads, sensitive data patterns, web shells, and more. The tester can simply pull this repository onto a new test machine and he will have access to every type of list he needs.

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)