찾다
데이터 베이스MySQL 튜토리얼Twemproxy, a Redis proxy from Twitter

While a big number of users use large farms of Redis nodes, from the point of view of the project itself currently Redis is a mostly single-instance business. I've big plans about going distributed with the project, to the extent that I'm

While a big number of users use large farms of Redis nodes, from the point of view of the project itself currently Redis is a mostly single-instance business.

I've big plans about going distributed with the project, to the extent that I'm no longer evaluating any threaded version of Redis: for me from the point of view of Redis a core is like a computer, so that scaling multi core or on a cluster of computers is the same conceptually. Multiple instances is a share-nothing architecture. Everything makes sense AS LONG AS we have a *credible way* to shard :-)

This is why Redis Cluster will be the main focus of 2013 for Redis, and finally, now that Redis 2.6 is out and is showing to be pretty stable and mature, it is the right moment to focus on Redis Cluster, Redis Sentinel, and other long awaited improvements in the area of replication (partial resynchronization).

However the reality is that Redis Cluster is not yet production ready and requires months of work. Still our users already need to shard data on multiple instances in order to distribute the load, and especially in order to use many computers to get a big amount of RAM ready for data.

The sole option so far was client side sharding. Client side sharding has advantages as there are no intermediate layers between clients and nodes, nor routing of request, so it is a very scalable setup (linearly scalable, basically). However to implement it reliably requires some tuning, a way to take clients configuration in sync, and the availability of a solid client with consistent hashing support or some other partitioning algorithm.

Apparently there is a big news in the landscape, and has something to do with Twitter, where one of the biggest Redis farms deployed happen to serve timelines to users. So it comes as no surprise that the project I'm talking about in this blog post comes from the Twitter Open Source division.

Twemproxy
---

Twemproxy is a fast single-threaded proxy supporting the Memcached ASCII protocol and more recently the Redis protocol:

https://github.com/twitter/twemproxy

It is written entirely in C and is licensed under the Apache 2.0 License.
The project works on Linux and AFAIK can't be compiled on OSX because it relies on the epoll API.

I did my tests using my Ubuntu 12.04 desktop.

But well, I'm still not saying anything useful. What twemproxy does actually? (Note: I'll focus on the Redis part, but the project is also able to do the same things for memcached as well).

1) It works as a proxy between your clients and many Redis instances.
2) It is able to automatically shard data among the configured Redis instances.
3) It supports consistent hashing with different strategies and hashing functions.

What's awesome about Twemproxy is that it can be configured both to disable nodes on failure, and retry after some time, or to stick to the specified keys -> servers map. This means that it is suitable both for sharding a Redis data set when Redis is used as a data store (disabling the node ejection), and when Redis is using as a cache, enabling node-ejection for cheap (as in simple, not as in bad quality) high availability.

The bottom line here is: if you enable node-ejection your data may end into other nodes when a node fails, so there is no guarantee about consistency. On the other side if you disable node-ejection you need to have a per-instance high availability setup, for example using automatic failover via Redis Sentinel.

Installation
---

Before diving more inside the project features, I've good news, it is trivial to build on Linux. Well, not as trivial as Redis, but… you just need to follow those simple steps:

apt-get install automake
apt-get install libtool
git clone git://github.com/twitter/twemproxy.git
cd twemproxy
autoreconf -fvi
./configure --enable-debug=log
make
src/nutcracker -h

It is pretty trivial to configure as well, and there is sufficient documentation in the project github page to have a smooth first experience. For instance I used the following configuration:

redis1:
listen: 0.0.0.0:9999
redis: true
hash: fnv1a_64
distribution: ketama
auto_eject_hosts: true
timeout: 400
server_retry_timeout: 2000
server_failure_limit: 1
servers:
- 127.0.0.1:6379:1
- 127.0.0.1:6380:1
- 127.0.0.1:6381:1
- 127.0.0.1:6382:1

redis2:
listen: 0.0.0.0:10000
redis: true
hash: fnv1a_64
distribution: ketama
auto_eject_hosts: false
timeout: 400
servers:
- 127.0.0.1:6379:1
- 127.0.0.1:6380:1
- 127.0.0.1:6381:1
- 127.0.0.1:6382:1

Basically the first cluster is configured with node ejection, and the second as a static map among the configured instances.

What is great is that you can have multiple setups at the same time possibly involving the same hosts. However for production I find more appropriate to use multiple instances to use multiple cores.

Single point of failure?
---

Another very interesting thing is that, actually, using this setup does not mean you have a single point of failure, since you can run multiple instances of twemproxy and let your client connect to the first available.

Basically what you are doing with twemproxy is to separate the sharding logic from your client. At this point a basic client will do the trick, sharding will be handled by the proxy.

It is a straightforward but safe approach to partitioning IMHO.

Currently that Redis Cluster is not available, I would say, it is the way to go for most users that want a cluster of Redis instances today. But read about the limitations before to get too excited ;)

Limitations
---

I think that twemproxy do it right, not supporting multiple keys commands nor transactions. Currently is AFAIK even more strict than Redis Cluster that instead allows MULTI/EXEC blocks if all the commands are about the same key.

But IMHO it's the way to go, distribute the subset you can distribute efficiently, and pose this as a design challenge early to the user, instead to invest a big amount of resources into "just works" implementations that try to aggregate data from multiple instances, but that will hardly be fast enough once you start to have serious loads because of too big constant times to move data around.

However there is some support for commands with multiple keys. MGET and DEL are handled correctly. Interestingly MGET will split the request among different servers and will return the reply as a single entity. This is pretty cool even if I don't get the right performance numbers with this feature (see later).

Anyway the fact that multi-key commands and transactions are not supported it means that twemproxy is not for everybody, exactly like Redis Cluster itself. Especially since apparently EVAL is not supported (I think they should support it! It's trivial, EVAL is designed to work in a proxy like that because key names are explicit).

Things that could be improved
---

Error reporting is not always stellar. Sending a non supported command closes the connection. Similarly sending just a "GET" from redis-cli does not report any error about bad number of arguments but hangs the connection forever.

However other errors from the server are passed to the client correctly:

redis metal:10000> get list
(error) WRONGTYPE Operation against a key holding the wrong kind of value

Another thing that I would love to see is support for automatic failover. There are many alternatives:

1) twemproxy is already able to monitor instance errors, count the number of errors, and eject the node when enough errors are detected. Well it is a shame it is not able to take slave nodes as alternatives, and instead of eject nodes use the alternate nodes just after sending a SLAVE OF NOONE command. This would turn it into an HA solution as well.

2) Or alternatively, I would love if it could be able to work in tandem with Redis Sentinel, checking the Sentinel configuration regularly to upgrade the servers table if a failover happened.

3) Another alternative is to provide a way to hot-configure twemproxy so that on fail overs Sentinel could switch the configuration of the proxy ASAP.

There are many alternatives, but basically, some support for HA could be great.

Performances
---

This Thing Is Fast. Really fast, it is almost as fast as talking directly with Redis. I would say you lose 20% of performances at worst.

My only issue with performances is that IMHO MGET could use some improvement when the command is distributed among instances.

After all if the proxy has similar latency between it and all the Redis instances (very likely), if the MGETs are sent at the same time, likely the replies will reach the proxy about at the same time. So I expected to see almost the same numbers with an MGET as I see when I run the MGET against a single instance, but I get only 50% of the operations per second. Maybe it's the time to reconstruct the reply, I'm not sure.

Conclusions
---

It is a great project, and since Redis Cluster is yet not here, I strongly suggest Redis users to give it a try.

Personally I'm going to link it in some visible place in the Redis project site. I think the Twitter guys here provided some real value to Redis itself with their project, so…

Kudos! Comments
성명
본 글의 내용은 네티즌들의 자발적인 기여로 작성되었으며, 저작권은 원저작자에게 있습니다. 본 사이트는 이에 상응하는 법적 책임을 지지 않습니다. 표절이나 침해가 의심되는 콘텐츠를 발견한 경우 admin@php.cn으로 문의하세요.
MySQL의 라이센스는 다른 데이터베이스 시스템과 어떻게 비교됩니까?MySQL의 라이센스는 다른 데이터베이스 시스템과 어떻게 비교됩니까?Apr 25, 2025 am 12:26 AM

MySQL은 GPL 라이센스를 사용합니다. 1) GPL 라이센스는 MySQL의 무료 사용, 수정 및 분포를 허용하지만 수정 된 분포는 GPL을 준수해야합니다. 2) 상업용 라이센스는 공개 수정을 피할 수 있으며 기밀이 필요한 상업용 응용 프로그램에 적합합니다.

MyISAM을 통해 언제 innodb를 선택 하시겠습니까?MyISAM을 통해 언제 innodb를 선택 하시겠습니까?Apr 25, 2025 am 12:22 AM

MyISAM 대신 InnoDB를 선택할 때의 상황에는 다음이 포함됩니다. 1) 거래 지원, 2) 높은 동시성 환경, 3) 높은 데이터 일관성; 반대로, MyISAM을 선택할 때의 상황에는 다음이 포함됩니다. 1) 주로 읽기 작업, 2) 거래 지원이 필요하지 않습니다. InnoDB는 전자 상거래 플랫폼과 같은 높은 데이터 일관성 및 트랜잭션 처리가 필요한 응용 프로그램에 적합하지만 MyISAM은 블로그 시스템과 같은 읽기 집약적 및 트랜잭션이없는 애플리케이션에 적합합니다.

MySQL에서 외국 키의 목적을 설명하십시오.MySQL에서 외국 키의 목적을 설명하십시오.Apr 25, 2025 am 12:17 AM

MySQL에서 외국 키의 기능은 테이블 간의 관계를 설정하고 데이터의 일관성과 무결성을 보장하는 것입니다. 외국 키는 참조 무결성 검사 및 계단식 작업을 통해 데이터의 효과를 유지합니다. 성능 최적화에주의를 기울이고 사용할 때 일반적인 오류를 피하십시오.

MySQL의 다른 유형의 인덱스는 무엇입니까?MySQL의 다른 유형의 인덱스는 무엇입니까?Apr 25, 2025 am 12:12 AM

MySQL에는 B-Tree Index, Hash Index, Full-Text Index 및 공간 인덱스의 네 가지 주요 인덱스 유형이 있습니다. 1.B- 트리 색인은 범위 쿼리, 정렬 및 그룹화에 적합하며 직원 테이블의 이름 열에서 생성에 적합합니다. 2. HASH 인덱스는 동등한 쿼리에 적합하며 메모리 저장 엔진의 HASH_Table 테이블의 ID 열에서 생성에 적합합니다. 3. 전체 텍스트 색인은 기사 테이블의 내용 열에서 생성에 적합한 텍스트 검색에 사용됩니다. 4. 공간 지수는 지리 공간 쿼리에 사용되며 위치 테이블의 Geom 열에서 생성에 적합합니다.

MySQL에서 인덱스를 어떻게 생성합니까?MySQL에서 인덱스를 어떻게 생성합니까?Apr 25, 2025 am 12:06 AM

toreateanindexinmysql, usethecreateindexstatement.1) forasinglecolumn, "createindexidx_lastnameonemployees (lastname);"2) foracompositeIndex를 사용하고 "createDexIdx_nameonemployees (forstName, FirstName);"3)을 사용하십시오

MySQL은 sqlite와 어떻게 다릅니 까?MySQL은 sqlite와 어떻게 다릅니 까?Apr 24, 2025 am 12:12 AM

MySQL과 Sqlite의 주요 차이점은 설계 개념 및 사용 시나리오입니다. 1. MySQL은 대규모 응용 프로그램 및 엔터프라이즈 수준의 솔루션에 적합하며 고성능 및 동시성을 지원합니다. 2. SQLITE는 모바일 애플리케이션 및 데스크탑 소프트웨어에 적합하며 가볍고 내부질이 쉽습니다.

MySQL의 색인이란 무엇이며 성능을 어떻게 향상 시키는가?MySQL의 색인이란 무엇이며 성능을 어떻게 향상 시키는가?Apr 24, 2025 am 12:09 AM

MySQL의 인덱스는 데이터 검색 속도를 높이는 데 사용되는 데이터베이스 테이블에서 하나 이상의 열의 주문 구조입니다. 1) 인덱스는 스캔 한 데이터의 양을 줄임으로써 쿼리 속도를 향상시킵니다. 2) B-Tree Index는 균형 잡힌 트리 구조를 사용하여 범위 쿼리 및 정렬에 적합합니다. 3) CreateIndex 문을 사용하여 CreateIndexIdx_customer_idonorders (customer_id)와 같은 인덱스를 작성하십시오. 4) Composite Indexes는 CreateIndexIdx_customer_orderOders (Customer_id, Order_Date)와 같은 다중 열 쿼리를 최적화 할 수 있습니다. 5) 설명을 사용하여 쿼리 계획을 분석하고 피하십시오

MySQL에서 트랜잭션을 사용하여 데이터 일관성을 보장하는 방법을 설명하십시오.MySQL에서 트랜잭션을 사용하여 데이터 일관성을 보장하는 방법을 설명하십시오.Apr 24, 2025 am 12:09 AM

MySQL에서 트랜잭션을 사용하면 데이터 일관성이 보장됩니다. 1) STARTTRANSACTION을 통해 트랜잭션을 시작한 다음 SQL 작업을 실행하고 커밋 또는 롤백으로 제출하십시오. 2) SavePoint를 사용하여 부분 롤백을 허용하는 저장 지점을 설정하십시오. 3) 성능 최적화 제안에는 트랜잭션 시간 단축, 대규모 쿼리 방지 및 격리 수준을 합리적으로 사용하는 것이 포함됩니다.

See all articles

핫 AI 도구

Undresser.AI Undress

Undresser.AI Undress

사실적인 누드 사진을 만들기 위한 AI 기반 앱

AI Clothes Remover

AI Clothes Remover

사진에서 옷을 제거하는 온라인 AI 도구입니다.

Undress AI Tool

Undress AI Tool

무료로 이미지를 벗다

Clothoff.io

Clothoff.io

AI 옷 제거제

Video Face Swap

Video Face Swap

완전히 무료인 AI 얼굴 교환 도구를 사용하여 모든 비디오의 얼굴을 쉽게 바꾸세요!

뜨거운 도구

PhpStorm 맥 버전

PhpStorm 맥 버전

최신(2018.2.1) 전문 PHP 통합 개발 도구

메모장++7.3.1

메모장++7.3.1

사용하기 쉬운 무료 코드 편집기

SublimeText3 Linux 새 버전

SublimeText3 Linux 새 버전

SublimeText3 Linux 최신 버전

mPDF

mPDF

mPDF는 UTF-8로 인코딩된 HTML에서 PDF 파일을 생성할 수 있는 PHP 라이브러리입니다. 원저자인 Ian Back은 자신의 웹 사이트에서 "즉시" PDF 파일을 출력하고 다양한 언어를 처리하기 위해 mPDF를 작성했습니다. HTML2FPDF와 같은 원본 스크립트보다 유니코드 글꼴을 사용할 때 속도가 느리고 더 큰 파일을 생성하지만 CSS 스타일 등을 지원하고 많은 개선 사항이 있습니다. RTL(아랍어, 히브리어), CJK(중국어, 일본어, 한국어)를 포함한 거의 모든 언어를 지원합니다. 중첩된 블록 수준 요소(예: P, DIV)를 지원합니다.

Eclipse용 SAP NetWeaver 서버 어댑터

Eclipse용 SAP NetWeaver 서버 어댑터

Eclipse를 SAP NetWeaver 애플리케이션 서버와 통합합니다.