How to implement Nginx request compression

1. Introduction

Request compression is to compress the server's results through Nginx and then send them to the client to reduce network transmission pressure and improve transmission efficiency.

The two common request methods are: gzip and brotli (Google), which are equivalent to brotli and will be more efficient. Details will be explained later.

Requesting compression is divided into: dynamic compression and static compression. Dynamic compression will cause the sendfile inside Nginx to fail. For some unchanged content, static compression can be used to improve request efficiency.

Used to compress the request results, both the client and the server need to support the compression protocol. The server compresses the results and the client decompresses the data. The compression will occupy some performance efficiency on the server side. This loss is based on It depends on the level of compression. The higher the level, the greater the loss. Can reduce network transmission pressure.

Compression only takes effect for proxy requests.

2. Request compression process

The execution process is: the client sends a request to the server, and after nginx receives the request, it will send it to the upstream server When sending a request, a network channel will be created between Nginx and the upstream server, and data will be transmitted between them. If the compression operation is turned on here, Nginx will compress the result data and return the data to the client. When the browser receives a request from nginx, it will first process the request header and find that there is a compression protocol. Then it will determine whether the current browser supports the protocol. If it supports it, it will decompress the data and display the data to the user.

After turning on compression, the user will be insensitive, which can reduce the transmission pressure. However, it is not recommended to compress pictures and videos because the resulting size does not change much. Gzip is compressed for network transmission and requires client support. , the server side also needs to support compressing the transmitted data and making the transmitted data smaller. We can set it. Of course, the higher the compression, the longer the decompression and compression time will be, and the greater the pressure on the server side.

3. Gzip compression

3.1 Introduction to gzip

Gzip is the abbreviation of GNUzip, which was first used for file compression in UNIX systems. Gzip encoding over HTTP protocol is a technology used to improve the performance of web applications. The web server and client (browser) must both support Gzip. Currently, mainstream browsers: Chrome, Firefox, etc. all support this protocol. Common servers: Apache, Nginx, and IIS also support Gzip.

Gzip’s compression ratio is about 3-10 times (plain text), which can greatly save the server’s network bandwidth. In actual applications, not all files are compressed, usually only static files (js\css\html) are compressed. Files such as JPEG are not compressed well enough with Gzip, and compression consumes CPU resources.

So how does Gzip compress? Simply put, Gzip compression finds similar strings in a text file and temporarily replaces them to make the entire file smaller. This form of compression is ideal for the Web, since HTML and CSS files often contain large amounts of repeated strings such as spaces, tags, etc.

3.2 Use of gzip

The gzip module is built-in in Nginx, so there is no need to add other defaults, just configure and install Nginx.

Use scope: http, server, location

Nginx is configured as follows

gzip on ;

• Whether to enable compression.

• Default value: gzip off is turned off by default

gzip_buffers 32 4k | 16 8k

• Compression Buffer size.

• Default value: gzip_buffers 32 4k | 16 8k

gzip_comp_level 1 ;

• Compression level 1-9, the larger the number, the higher the compression ratio.

• The smaller the compression speed, the faster the decompression speed, and the smaller the compression ratio

• Default value: gzip_comp_leve 1

gzip_http_version 1.1

• The smallest http version using gzip.

gzip_min_length

• Sets the minimum length of a response that will be gzip compressed. The length is determined only by the "Content-Length" response header field.

gzip_proxied Multiple selection

• For the upstream server to return different heads to decide whether to compress, there are two common ones: off Any

• Generally, if the cache is enabled, it has already been compressed, so some details can be avoided.

• off is not restricted.

• When acting as a reverse proxy, compress the header information returned by the upstream server.

• expired - Enable compression, if the header header contains "Expires" header information

• no-cache - Enable compression, if the header header contains Contains "Cache-Control:no-cache" header information

• no-store - Enables compression if the header contains "Cache-Control:no-store" header information

• private - Enable compression, if the header contains the "Cache-Control:private" header information

• no_last_modified - Enable compression, if the header does not Contains "Last-Modified" header information

• ##no_etag - Enables compression if the header does not contain "ETag" header information

• auth - Enable compression, if the header contains the "Authorization" header

• any - compression is enabled unconditionally.

gzip_vary on ;

• Add a header to adapt to old browsers Vary: Accept-Encoding

gzip_types:

• Which mime types of files are compressed.

gzip_disable:

• Which browsers are prohibited from using gzip.

• It is recommended not to configure

• Default value: gzip_disable 'msie6MSIE [4-6]\.MSIE 6.0'

location / {
  
  gzip off ;  # 开启gzip压缩
  gzip_buffers 32 4k ; # 设置缓冲区大小
  gzip_comp_level 5; # 设置压缩等级 1-9 
  gzip_disable 'msie6MSIE [4-6]\.MSIE 6.0';  # 禁止哪些浏览器不使用压缩
  gzip_http_version 1.1; # 设置压缩所需要的最低的http版本。 
  gzip_min_length 20 ; # 设置响应的数据最小限制，在这个限制之后再回进行压缩
  gzip_vary on ; # 增加一个header ，适用于老的浏览器 Vary:Accept-Encoding 
  gzip_proxied any; # 无条件启动压缩
  # 哪些mime类型的文件进行压缩 
  #gzip_types text/plain application/x-javascript text/css application/xml; 
  gzip_types
    text/xml application/xml application/atom+xml application/rss+xml application/xhtml+xml image/svg+xml
    text/javascript application/javascript application/x-javascript
    text/x-json application/json application/x-web-app-manifest+json
    text/css text/plain text/x-component
    font/opentype application/x-font-ttf application/vnd.ms-fontobject
    image/x-icon;
}

3.3 gzip request

When gzip is started, our request goes to the nginx server, and nginx has already produced response heard for us, but the data is still there It was not generated, and he didn’t know how big the specific data was, because nginx is an asynchronous response request. He did it step by step. He first prepared the header and then requested the content to be compressed. Finally, he merged and compressed the two pieces of content, and finally sent it. , that is, because of asynchronousness, he does not know the specific size.

He returns the request header first and then reads the data slowly.

----------------------------------响应体-------------------------------------------------
Connection: keep-alive
Content-Encoding: gzip # 结果启动了gzip压缩 
Content-Type: application/json # 响应结果 
Date: Mon, 13 Feb 2023 09:13:19 GMT
Keep-Alive: timeout=65
Server: nginx/1.20.2
Transfer-Encoding: chunked # 传输的格式这个对应的就是length，这个是他会发送一个一个的包，当最后一个包是0表示传输结束
Vary: Accept-Encoding
 
 
 
------------------------------------请求头-----------------------------------------------
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate
Accept-Language: zh-CN,zh;q=0.9,en;q=0.8
Cache-Control: max-age=0
Connection: keep-alive
Host: 192.168.101.128
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36

Note: The above is dynamic compression, which means that all requests will undergo compression once. This has a fatal disadvantage that it does not support sendfile. Sendfile is a zero copy of data and will not load the data into nginx, but through The network interface transmits data, but once gzip dynamic compression is turned on, sendfile will become invalid. Static compression can be used.

When using multi-layer Nginx, it is recommended to enable Gzip on the Nginx that is connected to the server. This will support gzip, and gzip will be enabled on any server.

3.4 Static compression

#Why use static compression?

First of all, dynamic compression cannot use sendfile, while static compression perfectly solves this problem.

Static compression is a complementary or extended function. It can compress the requested resource files into a compressed package in advance.

First of all, static resources must be more efficient than dynamic resources. After compression, the size of the disk can be reduced, and network channel resources can also be saved.

What is static compression

Static compression is not suitable for reverse proxy and is only suitable for resource servers. He can pass the compressed files to the client. Static compression is in the resource There will be a resource file under the path and a compressed package with the corresponding name. And Nginx will give priority to finding compressed packages and pass the data directly through sendfile.

Nginx sends the compressed file over the network, and then when the browser receives the compressed package file sent by Nginx, it decompresses it. He will send the compressed package interface to the client before sending it to the client.

3. Configuration

The http_gzip_static_module module allows sending precompressed files with the "" file extension, .gz instead of regular files. This module is not built by default, it should be enabled using the --with-http_gzip_static_module configuration parameter. This module is not in the precompiled package and needs to be added manually. The function of this module is to decompress the compressed package.

with is an internal module and add is an external module.

Use when installing nginx for the first time

./configure --prefix==/usr/local/nginx --with-hhtp_gzip_static_module

make && make install

Already After installing nginx, use the command

./configure --prefinx==/usr/local/nginx --with-http_gzip_static_module

make when upgrading the nginx client. Do not make install otherwise it will be overwritten. The previous file

Copy the nginx program of objs to /usr/local/nginx/sbin·, please note that you need to back up the original nginx program.

Syntax: gzip_static on | off | always;

Function: Whether to turn on the static compression function.

Parameter value:

• on: Turn on static compression and check whether the browser supports it. If it does not support it, static compression will not be performed.

• off: Turn off static compression

• always: Whether to use static compression, regardless of whether the browser supports static compression function.

• This will cause a problem. If the client does not support it, it cannot be decompressed. If there are no uncompressed files on the disk, 404 will be reported by default. You can cooperate with ngx_http_gunzip_module. Module usage.

• Default value: gzip_static on

• Applicable to: http, server, location

使用的需要将本地的资源文件进行压缩 ，压缩成 xxx.gz的文件

cd /usr/local/nginx/html 
gzip *

注意：开启之后默认就会先访问 .gz 的文件了，如果不支持 静态压缩则会访问 正常文件，如果没有正常的文件只有 .gz 那么就会报错 。

4. nginx_http_gunzip_module 模块

这个模块是配合 gzip_static always时使用的 ，因为 当配置程 always 时，所有的请求会都进行找压缩文件，在文件存不存在或者说浏览器支不支持 静态压缩，nginx都会将静态压缩文件返回给浏览器，如果浏览器不支持的话会导致文件解不开，也就是 404 。

这个模块它没有在预编译的包里，需要手动添加，这个模块的作用就是把静态的压缩包解压开，他会在发送给客户端之前将压缩包接口在发送给客户端，它相当于是一层拦截器，它的作用就是可以把源文件进行压缩，我们可以把源文件进行删除了，节省磁盘空间，但是一般会用到。

注意 ： with 是内部 、 add 是外部的

安装命令 ：

./configure --prefix=/usr/local/nginx --add-module=/tools/nginx-sticky --with-http_gzip_static_module --with-http_gunzip_module

make

如果是替换的话，则将这个文件中的这个文件 cp 到 nginx的安装目录下的 sbin

移动命令：

 cp /usr/local/nginx/sbin/nginx /usr/local/nginx/sbin/nginx.old2
 
 cd /tools/nginx/objs 
 mv nginx /usr/local/nginx/sbin/

这里没有gzip，但也没有 context-length 因为nginx需要把文件解压缩，他也不知道具体文件有多大，这个 gzip_static 适用场景 ：在作为cdn服务器或者 cdn上游服务器文件存储服务器时，我们就可以把本地原始文件删了，只展示zip 包，把解压缩的压力丢弃给客户端 ， 同时可以把本地压缩等级，提高不是 gzip的压缩等级 ，经常高配访问的一些页面 css js ，也可以通过 static压缩。

静态压缩 响应结果会有 Context-Encoding : gzip 、Conten-Length：392 有展示 context-length 表示他开启了静态压缩，预压缩直接返回zip包 没有源文件 ，1.速度快，2节省服务器资源。

四、Brotli

Brotli 是 Google 推出的开源压缩算法，通过变种的 LZ77 算法、Huffman 编码以及二阶文本建模等方式进行数据压缩，与其他压缩算法相比，它有着更高的压缩效率，性能也比我们目前常见的 Gzip 高17-25%，可以帮我们更高效的压缩网页中的各类文件大小及脚本，从而提高加载速度，提升网页浏览体验。需要说明的是 Brotli 压缩只在 https 下生效，因为 在 http 请求中 request header 里的 Accept-Encoding: gzip, deflate 是没有 br 的。

Brotli 如此高的压缩比率，得益于其使用一个预定义的字典，该字典包含超过 13000 个来自文本和 HTML 文档的大型语料库的常用字符串，预定义的算法可以提升较小文件的压缩密度，而压缩与解压缩速度则大致不变。

Brotli 凭借它优异的压缩性能迅速占领了市场，从下图可以看到，除了 IE 和 Opera Mini 之外，几乎所有的主流浏览器都已支持 Brotli 算法，因此处于资源占用的考虑，比如说流量，建议启用：

Brotil 规范由 Google 员工 Jyrki Alakuijala 和 Zoltan Szabadka 于 2013-2016开发，并伴随着规范的俩个作者Evgenii Kuliuchniko 和 Lode Vandevenne共同开发的参考实现，后者之前开发了谷歌的zopfli在2013年重新实现了收缩 /gzip 压缩格式。与zopfli不同，后者是对现有数据格式规范的重新实现，Brotli 是一种新的数据格式，并允许作者进一步提高压缩比。

4.1 Brotli 概述

Brotli 的编码器提供了 12 个质量级别 （从 0 到 11）。它们是压缩速度换取压缩效率的压缩模式：更高质量的几倍速度较慢，但会产生更好的压缩比。

一个 Brotli 压缩文件由 元块（meta-blocks）集合组成。每个元块最多可容纳 16 MiB，由俩部分组成：一个 数据部分（data part），它存储 LZ77 压缩的放入快，以及一个 标题（header），每个块的压缩遵循经典的 ①LZ77 压缩方案并由 ②计算具有良好的LZ77解析和计算 LZ 短语的简洁编码这俩个主要阶段组成。

它效率高是因为内置了 n多个字典，包含都是一些常见的文件文件 css 、js 等等一些标签，如果我们将这些标签归类生成一个字典之后，我们就可以按照序号去解压这个文件了。

并且它在 Nginx 中话是可以和 Gzip 共存，开启了Brotli 不会导致 Gzip失效，如果浏览器支持brotli 那么优先使用 Brotli ，不支持在使用 Gzip。

4.2 Brotli 的安装

--add-dynamic-module表示动态的引入模块在配置文件中还需要单独加入 load_module path/xxx

官网

• https://github.com/google/ngx_brotli

• https://codeload.github.com/google/brotli/tar.gz/refs/tags/v1.0.9

下载 俩个项目

解压缩

• brotli-1.0.9 是 brotli 算法模块，需要先解压。

• tar -zxvf brotli-1.0.9.tar.gz

• ngx_brotli-1.0.0rc 是nginx的 brotli的模块。这模块里需要引入 brotli 算法模块

• tar -zxvf ngx_brotli-1.0.0rc.tar.gz

接下来讲 brotli 的内容全部 复制到 ngx_brotli 的 deps/brotli/目录

• cd /tools/brotli-1.0.9

• cp -r * /tools/ngx_brotli-1.0.0rc/deps/brotli/

模块化编译 ：

./configure --with-compat --add-dynamic-module=/tools/ngx_brotli-1.0.0rc --prefix=/usr/local/nginx/

或

--add-dynamic-module=brotli目录

加载所有的压缩模块

./configure --with-compat --add-dynamic-module=/tools/ngx_brotli-1.0.0rc --prefix=/usr/local/nginx/ --add-module=/tools/nginx-sticky --with-http_gzip_static_module --with-http_gunzip_module

make && make install

下载的两个模块 拷贝到 /usr/local/nginx/modules/

首先在 /usr/local/nginx创建一个modules文件夹 mkdir modules

mv ngx_http_brotli_filter_module.so ngx_http_brotli_static_module.so /usr/local/nginx/modules/

最后将新编译的 nginx 启动程序复制到 /usr/local/nginx/sbin下 做好之前程序复制。

• cp /usr/local/nginx/sbin/nginx /usr/local/nginx/sbin/nginx.oid3

• mv /tools/nginx-12.0/objs/nginx /usr/local/nginx/sbin/

在配置文件中动态加载模块

load_module "/usr/local/nginx/modules/ngx_http_brotli_filter_module.so";
load_module "/usr/local/nginx/modules/ngx_http_brotli_static_module.so";

4.3 配置选项

具体的配置选项可以查看GitHub： https://github.com/google/ngx_brotli

brotli的配置可以参考Gzip的配置，几乎一致。

brotli

• 语法：brotli on | off

• 默认值 ：brotli off

• 适用于： http、server、location、if

• 作用：开启或者禁用brotli 压缩功能。

brotli_static

• 语法：brotli_static on | off | always

• 默认值：brotli_static off

• 适用于：http、server、location

• 作用：brotli 也是支持预先压缩文件，启用或禁用检查是否存在带扩展名的预压缩文件.br 。使用该always值，在所有情况下都使用预压缩文件，而不检查客户端是否支持它。

brotli_types

• 语法：brotli_types [...]

• 默认值：brotli_types text/html

• 适用于： http、server、location

• 作用：指定哪些资源类型需要进行压缩操作，特殊值*匹配任何 MIME类型。

brotli_buffers

• 语法：brotli_buffers

• 默认值： 32 4k | 16 8 k

• 适用于： http、server、location

• 作用：设置压缩缓冲区大小，最新版本以及弃用了 。

brotli_comp_level

• 语法：brotli_comp_level

• 默认值 ：6'

• 适用于 ： http、server、location

• 作用 ： 设置即时压缩 Brotli 质量（压缩）level。0可接受的值在从到 的范围内11。

brotli_window

• 语法：brotli_window

• 默认值 ： 512k

• 适用于 ：http、server、 location

• 作用：设置 Brotli 窗口size。可以比作是一个桌子，将要压缩的文件同时放在这个桌子上，这个桌子上可以放多少文件的大小，这个值也不越大越好，他比较占内存，值增加建议是2的倍数，可接受的值为1k, 2k, 4k, 8k, 16k, 32k, 64k, 128k, 256k, 512k, 1m, 2m,和4m。8m 16m

brotli_min_length

• 语法：brotli_min_length

• 默认值：20

• 适用于： http、server、location

• 作用：指定进行压缩的文件最小的长度，如果小于这个值则不压缩。

#加载动态模块 
load_module "/usr/local/nginx/modules/ngx_http_brotli_filter_module.so";
load_module "/usr/local/nginx/modules/ngx_http_brotli_static_module.so";
worker_processes  1;
events {
    worker_connections  1024;
}
 
http {
    server {
    listen       80;
    server_name  localhost;
        location  / {
          #brotli配置 
          brotli on;  # 开启 压缩
        brotli_static on; # 是否开启预先压缩，开启之后就会 .br的压缩包
          brotli_comp_level 6; # 压缩等级
          brotli_buffers 16 8k; # 缓冲区大小 ，已经启用 
          brotli_min_length 20; # 压缩时文件最小限制 
          # 对哪些mime.types类型进行压缩
          brotli_types text/plain text/css text/javascript application/javascript text/xml application/xml application/xml+rss application/json image/jpeg image/gif image/png;
    
          }
  }
}

4.4 brotli 测试

因为默认 brotli 是必现 https 请求才能使用的，因为 http的请求 请求头的 Accept-Encoding 是没有 br的，所以服务器是无法知道客户端可以使用的。

测试方案：

使用 linux 的 curl 命令 ：

curl -H 'Accept-Encding : br' -I 192.168.101.128/index.html

[root@localhost sbin]# curl -H Accept-Encoding:br  -I http://192.168.101.128/static_page.html
HTTP/1.1 200 OK
Server: nginx/1.20.2
Date: Fri, 17 Feb 2023 08:11:05 GMT
Content-Type: text/html
Last-Modified: Fri, 17 Feb 2023 03:28:14 GMT
Connection: keep-alive
Keep-Alive: timeout=65
Vary: Accept-Encoding
ETag: W/"63eef44e-31"
Content-Encoding: br
[root@localhost sbin]# curl  -I http://192.168.101.128/static_page.html
HTTP/1.1 200 OK
Server: nginx/1.20.2
Date: Fri, 17 Feb 2023 08:11:54 GMT
Content-Type: text/html
Last-Modified: Fri, 17 Feb 2023 03:28:14 GMT
Connection: keep-alive
Keep-Alive: timeout=65
Vary: Accept-Encoding
ETag: W/"63eef44e-31"

The above is the detailed content of How to implement Nginx request compression. For more information, please follow other related articles on the PHP Chinese website!