Redis high availability practice

0×01 序文

Redis は、ANSI C 言語で書かれたオープンソースのログタイプの Key-Value データベースで、ネットワークをサポートし、メモリベースで永続化でき、複数の言語で API を提供します。

現在、インターネット ビジネス データはより速い速度で増加しており、データの種類はますます豊富になっており、データ処理の速度と機能に対する要求が高まっています。 Redis は、開発者に破壊的なエクスペリエンスをもたらすオープンソースのインメモリ非リレーショナル データベースです。 Redis は、最初から最後まで高いパフォーマンスを念頭に置いて設計されており、現在利用できる最速の NoSQL データベースです。

高パフォーマンスを考慮する一方で、高可用性も重要な考慮事項です。インターネットは、24 時間 365 日中断のないサービスと、障害発生時の最速のフェールオーバーを提供するため、企業にもたらす損失は最小限に抑えられます。

それでは、実際のアプリケーションにおける高可用性アーキテクチャとは何でしょうか?それぞれのアーキテクチャの長所と短所は何ですか?どうやって選べばいいのでしょうか？ベストプラクティスにはどのようなものがありますか?

0×02センチネル原理

Redis の高可用性ソリューションを説明する前に、まず Redis Sentinel の原則 を見てみましょう。

1. Sentinel クラスターは、指定された構成ファイルを通じてマスターを検出し、マスターの開始時にマスターを監視します。 info 情報をマスターに送信して、このサーバーの下にあるすべてのスレーブ サーバーを取得します。
2. Sentinel クラスターは、他の Sentinel にその存在を通知するために、コマンド接続を通じて監視対象のマスター サーバーとスレーブ サーバーに hello 情報を送信します (1 秒に 1 回)。この情報には、Sentinel 自体の IP、ポート、ID などが含まれます。
3. Sentinel クラスターは、同じマスター サーバーを監視している他の Sentinel を検出するために、サブスクリプション接続を通じて他の Sentinel から送信された hello 情報を受信します。hello を送受信するマスター/スレーブ サーバーがすでに存在するため、クラスターは通信のために相互にコマンド接続を作成します。 Sentinel と Sentinel の間に接続が作成されます。
4. Sentinel クラスターは、ping コマンドを使用してインスタンスの状態を検出します。指定された時間 (ミリ秒後) 以内に応答がない場合、または誤った応答が返された場合、インスタンスはオフラインであると判断されます。
5. フェイルオーバーのアクティブ/スタンバイ切り替えがトリガーされると、フェイルオーバーはすぐには続行されません。また、フェイルオーバーを実行する前に、Sentinel 内の大半の Sentinel の承認が必要になります。指定されたクォーラム Sentinel の承認を取得すると、ODOWN 状態に入ります。たとえば、5 つの Sentinel 間で 2 つのクォーラムが構成されている場合、2 つの Sentinel がマスターが停止していると判断したときにフェイルオーバーが実行されます。
6. Sentinel は、マスターとして選択されたスレーブに SLAVEOF NO ONE コマンドを送信します。スレーブを選択する条件は、Sentinel が優先順位に従ってスレーブを最初に並べ替えることです。優先順位が同じ場合は、レプリケーションのサブスクリプトを確認して、マスターからより多くのレプリケーション データを受信した方が最初にランク付けされます。優先度とインデックスが同じ場合は、プロセス ID の小さい方が選択されます。
7. Sentinel が承認されると、障害が発生したマスターの最新の構成バージョン番号 (config-epoch) が取得され、フェイルオーバーの実行が完了すると、このバージョン番号が最新の構成に使用され、ブロードキャスト Sentinel や他の Sentinel を通じて他の人に通知されます。対応するマスターの構成を更新します。

1 ～ 3 は自動検出メカニズムです:

• 監視対象のマスターにinfoコマンドを10秒ごとに送信し、その返信に基づいて現在のマスター情報を取得します。
• Sentinel を含むすべての Redis サーバーに 1 秒の頻度で PING コマンドを送信し、応答を通じてサーバーがオンラインかどうかを判断します。
• 現在の Sentinel マスター情報メッセージを、監視対象のすべてのマスター サーバーとスレーブ サーバーに 2 秒の頻度で送信します。

4 は検出メカニズム、5 と 6 はフェイルオーバー メカニズム、7 は更新構成メカニズムです。 [1]

0×03 Redis 高可用性アーキテクチャ

After explaining the principle of Redis Sentinel, let’s explain the commonly used Redis high availability architecture.

• Redis Sentinel Cluster + Intranet DNS + Custom Script
• Redis Sentinel Cluster + VIP + Custom Script
• Encapsulate the client to directly connect to the Redis Sentinel port
  • JedisSentinelPool, suitable for Java
  • PHP is self-packaged based on phpredis
• Redis Sentinel Cluster + Keepalived/Haproxy
• Redis M/S + Keepalived
• Redis Cluster
• Twemproxy
• Codis

Next, we will explain it one by one with pictures and text.

3.1 Redis Sentinel Cluster + Intranet DNS + Custom Script

The picture above is a solution that has been applied in the online environment. The bottom layer is the Redis Sentinel cluster, which acts as an agent for the Redis master and slave. The Web side connects to the intranet DNS to provide services. Intranet DNS is allocated according to certain rules, such as xxxx.redis.cache/queue.port.xxx.xxx. The first segment indicates the business abbreviation, and the second segment indicates that this is the Redis intranet Domain name, the third segment represents the Redis type, cache represents the cache, queue represents the queue, the fourth segment represents the Redis port, and the fifth and sixth segments represent the main intranet domain name.

When the master node fails, such as machine failure, Redis node failure or network unreachability, the Sentinel cluster will call the client-reconfig-script configured script to modify the intranet domain name of the corresponding port. The intranet domain name of the corresponding port points to the new Redis master node.

Advantages:

• Second-level switching, complete the entire switching operation within 10s
• Customized scripts, controllable architecture
• Transparent to the application, the front-end does not have to worry about changes in the back-end

Disadvantages:

• The maintenance cost is slightly high, it is recommended to invest in more than 3 machines for the Redis Sentinel cluster
• Depends on DNS, there is resolution delay
• Sentinel mode service will be unavailable for a short period of time
• This solution cannot be used when accessing services through the external network

3.2 Redis Sentinel Cluster + VIP + Custom Script

This plan is slightly different from the previous one. The first solution uses the intranet DNS, and the second solution replaces the intranet DNS with a virtual IP. The bottom layer is the Redis Sentinel cluster, which acts as an agent for Redis master and slave, and the Web side provides services through VIP. When deploying Redis master-slave, you need to bind the virtual IP to the current Redis master node. When the master node fails, such as machine failure, Redis node failure, or network unreachability, the Sentinel cluster will call the client-reconfig-script configured script to drift the VIP to the new master node.

Advantages:

• Second-level switching, complete the entire switching operation within 5s
• Customized scripts, controllable architecture
• Transparent to the application, the front-end does not have to worry about changes in the back-end

Disadvantages:

• The maintenance cost is slightly high, it is recommended to invest in more than 3 machines for the Redis Sentinel cluster
• Using VIP increases maintenance costs and risks IP confusion
• Sentinel mode service will be unavailable for a short period of time

3.3 Encapsulating the client to connect directly to the Redis Sentinel port

Some businesses can only access Redis through the external network. Neither of the above two solutions are available, so this solution was derived. Web uses the client to connect to a certain port of a machine in one of the Redis Sentinel clusters, and then obtains the current master node through this port, and then connects to the real Redis master node to perform corresponding salesman operations. It is important to note that both the Redis Sentinel port and the Redis master node require open access. If the front-end business uses Java, JedisSentinelPool can be reused; if the front-end business uses PHP, secondary encapsulation can be done based on phpredis.

Advantages:

• Service detects faults promptly
• DBA maintenance cost is low

Disadvantages:

• Depends on client support Sentinel
• Sentinel server and Redis nodes require open access
• Intrusive to the application

3.4 Redis Sentinel Cluster + Keepalived/Haproxy

The bottom layer is the Redis Sentinel cluster, which acts as an agent for Redis master and slave, and the Web side provides services through VIP. When the master node fails, such as machine failure, Redis node failure, or network unreachability, switching between Redis is guaranteed by the internal mechanism of Redis Sentinel, and VIP switching is guaranteed by Keepalived.

Advantages:

• Switch in seconds
• Transparent to application

Disadvantages:

• High maintenance cost
• Having split brain
• Sentinel mode service will be unavailable for a short period of time

3.5 Redis M/S + Keepalived

This solution does not use Redis Sentinel. This solution uses native master-slave and Keepalived. VIP switching is guaranteed by Keepalived. Switching between Redis master-slave requires custom scripts.

Advantages:

• Switch in seconds
• Transparent to application
• Simple deployment and low maintenance cost

Disadvantages:

• Requires script to implement switching function
• Having split brain

3.6 Redis Cluster

From: http://intro2libsys.com/focused-redis-topics/day-one/intro-redis-cluster

Redis 3.0.0 was officially released on April 2, 2015, more than two years ago. Redis cluster adopts P2P mode and is non-centralized. Divide the key into 16384 slots, and each instance is responsible for a part of the slots. The client requests the corresponding data. If the instance slot does not have corresponding data, the instance will be forwarded to the corresponding instance. In addition, the Redis cluster synchronizes node information through the Gossip protocol.

Advantages:

• Components are all-in-box, easy to deploy and save machine resources
• Performance is better than proxy mode
• Automatic failover and data available during slot migration
• Official native cluster solution, guaranteed updates and support

Disadvantages:

• The architecture is relatively new and there are few best practices
• Multi-key operation support is limited (the driver can save the country through curves)
• In order to improve performance, the client needs to cache routing table information
• Node discovery and reshard operations are not automated enough

3.7 Twemproxy

From: http://engineering.bloomreach.com/the-evolution-of-fault-tolerant-redis-cluster

Multiple isomorphic Twemproxy (same configuration) works at the same time, accepts client requests, and forwards them to the corresponding Redis according to the hash algorithm.

Twemproxy solution is relatively mature. Our team has used this solution for a long time, but the effect is not very satisfactory. On the one hand, the positioning problem is more difficult, and on the other hand, its support for automatically eliminating nodes is not very friendly.

Advantages:

• Easy to develop and almost transparent to the application
• Long history and mature solutions

Disadvantages:

• Proxy affects performance
• LVS and Twemproxy will have node performance bottlenecks
• Redis expansion is very troublesome
• Twitter has abandoned the use of this solution internally, and the new architecture is not open source

3.8 Codis

From: https://github.com/CodisLabs/codis

Codis is an open source product by Wandoujia and involves many components. Among them, ZooKeeper stores routing tables and proxy node metadata, and distributes Codis-Config commands; Codis-Config is an integrated management tool with a Web interface for use; Codis-Proxy is A stateless proxy compatible with the Redis protocol; Codis-Redis is a secondary development based on Redis 2.8 version, adding slot support to facilitate data migration.

Advantages:

• Easy to develop and almost transparent to the application
• Performance is better than Twemproxy
• Has a graphical interface, easy expansion, and convenient operation and maintenance

Disadvantages:

• Proxy still affects performance
• Too many components, requiring a lot of machine resources
• The Redis code has been modified, resulting in the inability to synchronize with the official, and the follow-up of new features is slow
• The development team is preparing to promote reborndb based on Redis transformation

0×04 Best Practices

The so-called best practices are practices that are most suitable for specific scenarios.

We mainly recommend the following plans:

• Redis Sentinel cluster + intranet DNS + custom script
• Redis Sentinel Cluster + VIP + Custom Script

The following are the best practices summarized during actual combat:

• Redis Sentinel cluster is recommended to use >= 5 machines
• Different large businesses can use a Redis Sentinel cluster to proxy all ports under the business
• Divide the Redis port range according to different businesses
• Custom scripts are recommended to be implemented in Python for easy expansion
• Custom scripts need to pay attention to determine the current Sentinel role
• Pass in the parameters of the custom script:
• Custom scripts require remote ssh to operate the machine. It is recommended to use the paramiko library to avoid repeatedly establishing SSH connections and consuming time
• To accelerate SSH connection, it is recommended to turn off the following two parameters
  • UseDNS no
  • GSSAPIAuthentication no
• If you receive an alert via WeChat or email, it is recommended to fork a process to avoid blocking the main process
• Automatic switching and failover, it is recommended that all operations be completed within 15s

0×05 Summary

This event shared the necessity of Redis high availability, the Sentinel principle, the common architecture of Redis high availability and the best practices summarized in the actual combat process. I hope it will be helpful to readers. If you need follow-up communication, you can add me WeChat (Wentasy), or send an email to: dbarobinwen@gmail.com

Attached PPT download: https://github.com/dbarobin/slides

Video Playback: Best Practices for Redis High Availability Architecture

0×06 Thanks

Thank you to Tingyun and Operation and Maintenance Gang for their careful organization, and thank you to everyone who came to participate in this event despite the heavy rain. This sharing was videotaped by the IT guru, and I would like to thank the IT guru for his technical support.

0×07 Reference

[1] jyzhou (2016-06-12). Redis replication, Sentinel construction and principle explanation. Retrieved from http://www.cnblogs.com/zhoujinyi/p/5570024.html.

The above is the detailed content of Redis high availability practice. For more information, please follow other related articles on the PHP Chinese website!