How to use the HyperLogLog data type in Redis

1. Principle of HyperLogLog

Redis HyperLogLog uses a probability algorithm, the HyperLogLog algorithm, to estimate the cardinality. Using a set of hash functions and a bit array of length m, HyperLogLog is able to estimate the number of unique elements in a set.

In the HyperLogLog algorithm, each element is hashed, and after converting the hash value into binary, each element is scored according to the number of 1's in the binary string prefix. For example, if the hash value of an element is 01110100011, then the number of 1's in the prefix is ​​3, so in the HyperLogLog algorithm, the score of this element is 3.

After counting the scores of all elements, take the reciprocal of each score (1 / 2^n), then add these reciprocals and take the reciprocal, and you will get a cardinality estimate, which is HyperLogLog The estimation results of the algorithm.

The HyperLogLog algorithm trades off the size of the length m of the bit array, compromising the memory occupied by the data structure and the accuracy of the estimated value (i.e., the estimated error), and obtains the result between the space occupied by the data and the smaller degree of error. perfect balance.

In short, the core idea of ​​the HyperLogLog algorithm is based on hash functions and bit operations. By converting the hash value into a bit stream and counting the number of leading 0s, it can quickly estimate the unique value in a large data set. quantity. Using the hyperloglog algorithm, we are able to quickly identify duplicate web pages in very large datasets.

2. Usage steps:

Redis HyperLogLog is a data structure that can be used to estimate the number of elements in a collection. It can maintain massive amounts of data by using very little memory. It is more accurate than conventional estimation algorithms and very fast when processing large amounts of data.

A simple example, we can use HyperLogLog to calculate the number of independent IPs visiting the website. Specifically, you can follow the following steps:

• First create a HyperLogLog data structure: PFADD hll:unique_ips 127.0.0.1

• Add the ip for each access to the unique_ips data structure: PFADD hll:unique_ips 192.168.1.1

• Get an approximation of the number of elements in the calculated collection: PFCOUNT hll:unique_ips

• ##You can pass multiple HyperLogLog structures (such as by day or hour) to get a more accurate count.

It should be noted that although HyperLogLog can save a lot of memory, it is an estimation algorithm and the error range is not completely accurate. You should pay attention to its scope of application when using it in practice.

3. Example of using page views to implement request IP deduplication

4.Using Jedis client

　1. Add dependencies, Introduce jedis dependency:

<dependency>
    <groupId>redis.clients</groupId>
    <artifactId>jedis</artifactId>
    <version>3.6.0</version>
</dependency>

2. Create a Jedis object:

Jedis jedis = new Jedis("localhost");

3. Add elements to the HyperLogLog data structure:

jedis.pfadd("hll:unique_ips", "127.0.0.1");

4. Get the number of elements in the collection Approximate value:

Long count = jedis.pfcount("hll:unique_ips");
System.out.println(count);

5. A more accurate count can be obtained by merging multiple HyperLogLog structures. In Jedis, you can use the

PFMERGE command to merge the HyperLogLog data structure:

jedis.pfmerge("hll:unique_ips", "hll:unique_ips1", "hll:unique_ips2", "hll:unique_ips3");

5. Redission uses dependencies

　1. Create a RedissonClient object

Config config = new Config();
config.useSingleServer().setAddress("redis://localhost:6379");
RedissonClient redisson = Redisson.create(config);

　2 .Create RHyperLogLog object

RHyperLogLog<String> uniqueIps = redisson.getHyperLogLog("hll:unique_ips");

　3.Add elements

uniqueIps.add("127.0.0.1");

　4.Get approximate quantity

long approximateCount = uniqueIps.count();
System.out.println(approximateCount);

　5.Merge multiple HyperLogLog objects

RHyperLogLog<String> uniqueIps1 = redisson.getHyperLogLog("hll:unique_ips1");
RHyperLogLog<String> uniqueIps2 = redisson.getHyperLogLog("hll:unique_ips2");
uniqueIps.mergeWith(uniqueIps1, uniqueIps2);

6 .What features and methods does HyperLogLog provide?

Features:

• The accuracy is low, but it takes up very little memory.

• Supports inserting new elements without double counting.

• Provides instructions to optimize memory usage and counting accuracy. For example, PFADD, PFCOUNT, PFMERGE and other instructions.

• Be able to estimate the number of different elements in a data set, that is, the cardinality of the set.

• Supports merging operations on multiple HyperLogLog objects to obtain an approximation of the total cardinality of these collections.

Commonly used methods in HyperLogLog:

• PFADD key element [element ...]: Add one or more elements to the HyperLogLog structure.

• PFCOUNT key [key ...]: Get the cardinality estimate of one or more HyperLogLog structures.

• PFMERGE destkey sourcekey [sourcekey ...]: Merge one or more HyperLogLog structures into a target structure.

• PFSELFTEST [numtests]: Test HyperLogLog valuation performance and accuracy (only for Redis4.0 version)

It should be noted that, Although HyperLogLog can save a lot of memory, it is still an estimation algorithm, the error range is not completely accurate, and it has a certain computational cost. Depending on the actual application, you need to consider whether to use HyperLogLog or other data structures to estimate the number of elements.

7. Summary of usage scenarios:

The main function of Redis using HyperLogLog is to perform deduplication counting in the case of large data streams (view, IP, city).

Specifically, the following are some scenarios where Redis HyperLogLog is used for deduplication and counting:

• Count Page Views - In web applications, HyperLogLog can be used to count how many unique visitors there are for each page. Use HyperLogLog technology to calculate the average number of visits to this page across different time periods.

• HyperLogLog has significant utility in analyzing the number of users in big data collections. A probability-based data structure is particularly effective when dealing with data sets such as unique user IDs. HyperLogLog only saves a limited number of hash values ​​after hashing and is able to deduce the size of the data set.

• Count advertising clicks - For advertising analysis on a website or application, HyperLogLog can be used to capture the number of effective clicks, that is, the number of distinct or unique clicks.

The above is the detailed content of How to use the HyperLogLog data type in Redis. For more information, please follow other related articles on the PHP Chinese website!