What is Huffman encoding? How to implement Huffman encoding and decoding in php

What is Huffman coding? Huffman coding is a data compression algorithm. The core of our commonly used zip compression is Huffman encoding, and in HTTP/, Huffman encoding is used to compress HTTP headers. In this article, I will share with you the implementation method of Huffman encoding and decoding in php.

1. Huffman encoding

Word count

The first step of Huffman encoding is to count the number of occurrences of each character in the document, PHP The built-in function count_chars() can do this:

$input = file_get_contents('input.txt');
$stat = count_chars($input, 1);

Constructing the Huffman tree

Next, the Huffman tree is constructed based on the statistical results. The construction method is described in detail in Wikipedia. Here is a simple version written in PHP:

$huffmanTree = [];foreach ($stat as $char => $count) {
    $huffmanTree[] = [        
    'k' => chr($char),        
    'v' => $count,        
    'left' => null,        
    'right' => null,
    ];
}// 构造树的层级关系，思想见wiki：https://zh.wikipedia.org/wiki/%E9%9C%8D%E5%A4%AB%E6%9B%BC%E7%BC%96%E7%A0%81$size = count($huffmanTree);for ($i = 0; $i !== $size - 1; $i++) {
    uasort($huffmanTree, function ($a, $b) {        
    if ($a['v'] === $b['v'])
     {            
     return 0;
        }        
        return $a[&#39;v&#39;] < $b[&#39;v&#39;] ? -1 : 1;
    });
    $a = array_shift($huffmanTree);
    $b = array_shift($huffmanTree);
    $huffmanTree[] = [        
    &#39;v&#39; => $a[&#39;v&#39;] + $b[&#39;v&#39;],        
    &#39;left&#39; => $b,        
    &#39;right&#39; => $a,
    ];
}
$root = current($huffmanTree);

After calculation, $root will point to the root node of the Huffman tree

Generate a coding dictionary based on the Huffman tree

Yes With the Huffman tree, you can generate a dictionary for encoding:

function buildDict($elem, $code = '', &$dict) {    
if (isset($elem['k']))
 {
        $dict[$elem['k']] = $code;
    } else {
        buildDict($elem['left'], $code.'0', $dict);
        buildDict($elem['right'], $code.'1', $dict);
    }
}
$dict = [];
buildDict($root, '', $dict);

Write file

Use the dictionary to encode the file content and write it to the file. There are several things to pay attention to when writing Huffman encoding to a file:

After writing the encoding dictionary and encoding content to the file together, it is impossible to distinguish their boundaries, so they need to be written at the beginning of the file. Number of bytes occupied

The fwrite() function provided by PHP can write 8-bit (one byte) or an integer multiple of 8 bits at a time. However, in Huffman encoding, a character may be represented by only 1-bit, and PHP does not support the operation of writing only 1-bit to the file. Therefore, we need to splice the encoding ourselves, and only write the file after every 8-bit is obtained.

Write every time 8-bit is obtained

Similar to the second item, the final file size must be an integer multiple of 8-bit. So if the size of the entire encoding is 8001-bit, 7 0s must be added at the end

$dictString = serialize($dict);// 写入字典和编码各自占用的字节数
$header = pack('VV', strlen($dictString), strlen($input));
fwrite($outFile, $header);// 写入字典本身
fwrite($outFile, $dictString);// 写入编码的内容$buffer = '';
$i = 0;while (isset($input[$i])) {
    $buffer .= $dict[$input[$i]];    
    while (isset($buffer[7])) {
        $char = bindec(substr($buffer, 0, 8));
        fwrite($outFile, chr($char));
        $buffer = substr($buffer, 8);
    }
    $i++;
}// 末尾的内容如果没有凑齐 8-bit，需要自行补齐
if (!empty($buffer))
 {
    $char = bindec(str_pad($buffer, 8, '0'));
    fwrite($outFile, chr($char));
}
fclose($outFile);

2. Decoding of Huffman encoding

Decoding of Huffman encoding Relatively simple: first read the encoding dictionary, and then decode the original characters according to the dictionary.

There is a problem that needs to be noted during the decoding process: since we have added several 0-bits at the end of the file during the encoding process, if these 0-bits happen to be the encoding of a certain character in the dictionary, This will cause incorrect decoding.

So during the decoding process, when the number of decoded characters reaches the document length, decoding will stop.

<?php
$content = file_get_contents(&#39;a.out&#39;);// 读出字典长度和编码内容长度
$header = unpack(&#39;VdictLen/VcontentLen&#39;, $content);
$dict = unserialize(substr($content, 8, $header[&#39;dictLen&#39;]));
$dict = array_flip($dict);
$bin = substr($content, 8 + $header[&#39;dictLen&#39;]);
$output = &#39;&#39;;
$key = &#39;&#39;;
$decodedLen = 0;
$i = 0;
while (isset($bin[$i]) && $decodedLen !== $header[&#39;contentLen&#39;]) {
    $bits = decbin(ord($bin[$i]));
    $bits = str_pad($bits, 8, &#39;0&#39;, STR_PAD_LEFT);    
    for ($j = 0; $j !== 8; $j++) {        // 每拼接上 1-bit，就去与字典比对是否能解码出字符
        $key .= $bits[$j];        
        if (isset($dict[$key]))
         {
            $output .= $dict[$key];
            $key = &#39;&#39;;
            $decodedLen++;            
            if ($decodedLen === $header[&#39;contentLen&#39;])
             {             
                break;
            }
        }
    }
    $i++;
}echo $output;

3. Test

We save the HTML code of the Huffman coding Wiki page locally and conduct the Huffman coding test. The test results are:

coding Before: 418,504 bytes

After encoding: 280,127 bytes

The space is saved by 33%. If the original text has a lot of repeated content, the space saved by Huffman encoding can reach more than 50%.

In addition to the text content, we try to Huffman encode a binary file, such as the f.lux installation program. The test results are as follows:

Before encoding: 770,384 bytes

Encoding After: 773,076 bytes

After encoding, it takes up more space. On the one hand, it is because we do not do additional processing when storing the dictionary, which takes up a lot of space. On the other hand, in binary files, the probability of each character appearing is relatively even, and the advantages of Huffman coding cannot be used.

Related recommendations:

php encodes, decodes and parses URL parameters

The above is the detailed content of What is Huffman encoding? How to implement Huffman encoding and decoding in php. For more information, please follow other related articles on the PHP Chinese website!