java如何高效读写10G以上大文件

Question

有一份10G以上大文本文件，需要替换里面的一些文本信息（每一行都有），如何高效读并替换掉生成新的文件

Answer 1

1. Split into multiple files first

2. Multiple threads operate multiple files to avoid two threads operating the same file

3. Read files line by line and write new files line by line

4. Merge all files

1,4 Just use linux commands~

Answer 2

File file = new File(filepath);
BufferedInputStream fis = new BufferedInputStream(new FileInputStream(file));
BufferedReader reader = new BufferedReader(new InputStreamReader(fis,"utf-8"),510241024);
String line = "";
while((line = reader.readLine()) != null){

//进行替换操作和其他业务    

}

Answer 3

In order to improve performance, you may need mapped IO. For details, please refer to:

1. Why use Memory Mapped File or MappedByteBuffer in Java

2. java large file read and write operations, java nio's MappedByteBuffer, efficient file/memory mapping

3. A simple comparison of the performance of java.io and java.nio

Answer 4

If it is a simple text replacement, just use the sed command of Linux.

If it is a more complex text replacement, see below:

1. http://stackoverflow.com/ques...

2. http://www.baeldung.com/java-...

Answer 5

用spark分析、
lines=sc.textFile("your_file");
filterlines=lines.filter(your_filter_function)
filterlines.xxx()