How to use Java 8's Stream to implement list deduplication?

1. The distinct() method of Stream

distinct() is a method provided by Stream in Java 8. It returns a stream composed of different elements in the stream. distinct() uses hashCode() and eqauls() methods to get distinct elements.

Therefore, the class that needs to be deduplicated must implement the hashCode() and equals() methods. In other words, we can achieve certain special needs by overriding the customized hashCode() and equals() methods.

distinct() method is declared as follows:

Stream<T> distinct();

1.1 For deduplication of String list

Because the String class has overridden the equals() and hashCode() methods, so You can successfully remove duplicates.

@Test
public void listDistinctByStreamDistinct() {
  // 1. 对于 String 列表去重
  List<String> stringList = new ArrayList<String>() {{
    add("A");
    add("A");
    add("B");
    add("B");
    add("C");
  }};
  out.print("去重前：");
  for (String s : stringList) {
    out.print(s);
  }
  out.println();
  stringList = stringList.stream().distinct().collect(Collectors.toList());
  out.print("去重后：");
  for (String s : stringList) {
    out.print(s);
  }
  out.println();

}

The results are as follows:

Before deduplication: AABBC
After deduplication: ABC

1.2 Deduplication of entity class lists

Note: In the code, we use the @Data annotation of the Lombok plug-in, which can automatically override the equals() and hashCode() methods.

/**
* 定义一个实体类
*/  
@Data
public class Student {
  private String stuNo;
  private String name;
}

@Test
public void listDistinctByStreamDistinct() throws JsonProcessingException {
    ObjectMapper objectMapper = new ObjectMapper();
    // 1. 对于 Student 列表去重
    List<Student> studentList = getStudentList();
    out.print("去重前：");
    out.println(objectMapper.writeValueAsString(studentList));
    studentList = studentList.stream().distinct().collect(Collectors.toList());
    out.print("去重后：");
    out.println(objectMapper.writeValueAsString(studentList));
  }

The results are as follows:

Before deduplication: [{"stuNo":"001","name":"Tom"},{"stuNo":"002" ,"name":"Mike"},{"stuNo":"001","name":"Tom"}]
After deduplication: [{"stuNo":"001","name":" Tom"},{"stuNo":"002","name":"Mike"}]

2. Remove duplicates based on a certain attribute of the Object in List

2.1 Create a new list

@Test
  public void distinctByProperty1() throws JsonProcessingException {
    // 这里第一种方法我们通过新创建一个只有不同元素列表来实现根据对象某个属性去重
    ObjectMapper objectMapper = new ObjectMapper();
    List<Student> studentList = getStudentList();
    out.print("去重前        :");
    out.println(objectMapper.writeValueAsString(studentList));
    studentList = studentList.stream().distinct().collect(Collectors.toList());
    out.print("distinct去重后:");
    out.println(objectMapper.writeValueAsString(studentList));
    // 这里我们引入了两个静态方法，以及通过 TreeSet<> 来达到获取不同元素的效果
    // 1. import static java.util.stream.Collectors.collectingAndThen;
    // 2. import static java.util.stream.Collectors.toCollection;
    studentList = studentList.stream().collect(
      collectingAndThen(
        toCollection(() -> new TreeSet<>(Comparator.comparing(Student::getName))), ArrayList::new)
    );
    out.print("根据名字去重后 :");
    out.println(objectMapper.writeValueAsString(studentList));
  }

The result is as follows:

Before deduplication :[{"stuNo":"001","name":"Tom"},{" stuNo":"001","name":"Tom"},{"stuNo":"003","name":"Tom"}]
After removing duplicates: [{"stuNo":"001 ","name":"Tom"},{"stuNo":"003","name":"Tom"}]
After removing duplicates based on the name: [{"stuNo":"001","name ":"Tom"}]

2.2 Through the filter() method

We first create a method as a parameter of Stream.filter(), and its return type is Predicate. The principle is To determine whether an element can be added to the Set, the code is as follows:

private static <T> Predicate<T> distinctByKey(Function<? super T, ?> keyExtractor) {
    Set<Object> seen = ConcurrentHashMap.newKeySet();
    return t -> seen.add(keyExtractor.apply(t));
}

Use as follows:

@Test
  public void distinctByProperty2() throws JsonProcessingException {
    // 这里第二种方法我们通过过滤来实现根据对象某个属性去重
    ObjectMapper objectMapper = new ObjectMapper();
    List<Student> studentList = getStudentList();
    out.print("去重前        :");
    out.println(objectMapper.writeValueAsString(studentList));
    studentList = studentList.stream().distinct().collect(Collectors.toList());
    out.print("distinct去重后:");
    out.println(objectMapper.writeValueAsString(studentList));
    // 这里我们将 distinctByKey() 方法作为 filter() 的参数，过滤掉那些不能加入到 set 的元素
    studentList = studentList.stream().filter(distinctByKey(Student::getName)).collect(Collectors.toList());
    out.print("根据名字去重后 :");
    out.println(objectMapper.writeValueAsString(studentList));
  }

The result is as follows:

Before deduplication :[{" stuNo":"001","name":"Tom"},{"stuNo":"001","name":"Tom"},{"stuNo":"003","name":"Tom" }]
After distinct deduplication: [{"stuNo":"001","name":"Tom"},{"stuNo":"003","name":"Tom"}]
After removing duplicates based on names: [{"stuNo":"001","name":"Tom"}]

The above is the detailed content of How to use Java 8's Stream to implement list deduplication?. For more information, please follow other related articles on the PHP Chinese website!