Slave memory leak and trigger oom-killer_MySQL

Slave memory leak and trigger oom-killer_MySQL

Bug Description
We have this problem:
we have set innodb_buffer_pool =80GB on both master and slave , master offer usually workload,
but slave with nothing workload except these slave threads, But with the memory consumption is increasing , 
after a few days , it uses about 120GB of memory(RES) and sometime the machine starts swapping out. 
and in the /var/log/message(loged :Out of memory: Kill process 43817 (mysqld) score 847 or sacrifice 
childKilled process 43817, UID 500, (mysqld) total-vm:126723628kB, anon-rss:125497616kB, file-rss:3388kB)
We have the exact version of PerconaServer on the master (with production workload) and it doesn't 
have this problem
This problem is the same for a few recent versions of Percona server, not just the last one.
we do not user MyISAM tables and just use Innodb Tables;TokuDB is not installed.
Other Information:
this one just OOM last night T-T:
xiean@(none) 11:02> \s--------------mysql Ver 14.14 Distrib 5.6.23-72.1, for Linux (x86_64) 
using EditLine wrapper
Connection id: 252482Current database:Current user: xiean@localhostSSL: Not in useCurrent pager: 
stdoutUsing outfile: ''Using delimiter: ;Server version: 5.6.23-72.1-log Source distributionProtocol 
version: 10Connection: Localhost via UNIX socketServer characterset: utf8Db characterset: 
utf8Client characterset: utf8Conn. characterset: utf8UNIX socket: /data/mysql/3307/var/run/mysql.sockUptime:
 10 hours 32 min 9 sec
Threads: 6 Questions: 5888709 Slow queries: 312 Opens: 200 Flush tables: 1 Open tables: 193 Queries per 
second avg: 155.256--------------
CentOS release 6.5 (Final)Linux xxxxxxx 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 
UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
my.cnf-----------------------------------------------------[mysqld]
skip-external-lockingskip-name-resolvedefault-storage-engine = InnoDBcharacter-set-client-handshake = 
FALSEcharacter-set-server = utf8collation-server = utf8_unicode_ciinit_connect='set names utf8'
connect_timeout = 20back_log = 600log_bin_trust_function_creators = 1
binlog_format = rowmax_binlog_size = 512Mbinlog_cache_size = 2Mexpire-logs-days = 7
slave-net-timeout = 20log_slave_updates = 1relay_log_recovery = 1sync_master_info = 1sync_relay_log_info = 
1master_info_repository = TABLErelay_log_info_repository = TABLE
slow_query_log = 1long_query_time = 1
max_connections = 2000max_user_connections = 1000max_connect_errors = 100000key_buffer_size = 
64Mmax_allowed_packet = 512Mtable_open_cache = 6144table_definition_cache = 4096sort_buffer_size = 
512Kread_buffer_size = 512Kread_rnd_buffer_size = 512kjoin_buffer_size = 512Ktmp_table_size = 
64Mmax_heap_table_size = 128Mquery_cache_type = 0query_cache_size = 0bulk_insert_buffer_size = 
32Mthread_cache_size = 64thread_stack = 256K
innodb_data_file_path = ibdata1:1024M:autoextendinnodb_buffer_pool_size = 80Ginnodb_buffer_pool_instances = 
8innodb_additional_mem_pool_size = 16Minnodb_use_sys_malloc = 0innodb_log_file_size = 
8196Minnodb_log_buffer_size = 64Minnodb_log_files_in_group = 3innodb_lock_wait_timeout = 
30innodb_spin_wait_delay = 96innodb_thread_concurrency = 0innodb_max_dirty_pages_pct = 
75innodb_support_xa = 0innodb_flush_log_at_trx_commit = 2innodb_flush_method = O_DIRECTinnodb_read_io_threads = 
16innodb_write_io_threads = 16innodb_io_capacity = 2000innodb_flush_neighbors = 0innodb_page_size = 
4Kinnodb_log_block_size = 4096innodb_file_per_table = 1innodb_file_format = Barracudainnodb_file_format_max = 
Barracudainnodb_strict_mode = 1innodb_purge_threads = 2innodb_purge_batch_size = 32innodb_old_blocks_pct = 
37innodb_old_blocks_time = 1000innodb_change_buffering = allinnodb_stats_on_metadata = off
sync_binlog = 0

innodb_buffer_pool_dump_at_shutdown = 1innodb_buffer_pool_load_at_startup = 1
show engine innodb status\G
=====================================2016-03-22 11:08:07 7f8496af9700 INNODB MONITOR OUTPUT====================
=================Per second averages calculated from the last 3 seconds-----------------BACKGROUND THREAD------
-----------srv_master_thread loops: 35735 srv_active, 0 srv_shutdown, 2389 srv_idlesrv_master_thread log flush
 and writes: 38124----------SEMAPHORES----------OS WAIT ARRAY INFO: reservation count 27319OS WAIT ARRAY INFO: 
 signal count 694283Mutex spin waits 13779381, rounds 15836207, OS waits 12843RW-shared spins 3653866, rounds
  16984184, OS waits 11004RW-excl spins 562597, rounds 1648229, OS waits 2980Spin rounds per wait: 1.15 mutex,
  4.65 RW-shared, 2.93 RW-excl
....................................................
----------------------BUFFER POOL AND MEMORY----------------------
Total memory allocated 107356939316; in additional pool allocated 8441088Total memory allocated by 
read views 496Internal hash tables (constant factor + variable factor)    
Adaptive hash index 2118609544 (1359985288 + 758624256)    
Page hash 42500456 (buffer pool 0 only)    
Dictionary cache 340576340 (339997712 + 578628)    
File system 867904 (812272 + 55632)    
Lock system 849995224 (849992296 + 2928)    
Recovery system 0 (0 + 0)Dictionary memory allocated 578628Buffer pool size 20971512Buffer pool size, 
bytes 85899313152Free buffers 793287Database pages 19993014Old database pages 
7380256Modified db pages 857786Pending reads 0Pending writes: LRU 0, flush list 0, 
single page 0Pages made young 1500483, not young 7919383.00 youngs/s, 3.67 non-youngs/sPages read 19777369, 
created 365977, written 17055406.33 reads/s, 9.00 creates/s, 255.25 writes/sBuffer pool hit rate 1000 / 1000, 
young-making rate 0 / 1000 not 0 / 1000Pages read ahead 0.00/s, evicted without access 0.00/s, 
Random read ahead 0.00/sLRU len: 19993014, unzip_LRU len: 0I/O sum[36104]:cur[1320], unzip sum[0]:cur[0]
----------------------INDIVIDUAL BUFFER POOL INFO-------------------------
BUFFER POOL 0Buffer pool size 2621439Buffer pool size, bytes 10737414144Free buffers 1024Database
 pages 2597173Old database pages 958702Modified db pages 110539Pending reads 0Pending writes: LRU 0, 
 flush list 0, single page 0Pages made young 497074, not young 3051540.67 youngs/s, 3.67 non-youngs/sPages 
 read 2603703, created 46719, written 2711371.00 reads/s, 0.33 creates/s, 17.33 writes/sBuffer pool hit 
 rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000Pages read ahead 0.00/s, evicted without
  access 0.00/s, Random read ahead 0.00/sLRU len: 2597173, unzip_LRU len: 0I/O sum[4513]:cur[165], 
  unzip sum[0]:cur[0]---BUFFER POOL 1Buffer pool size 2621439Buffer pool size, bytes 10737414144Free 
  buffers 1024Database pages 2597316Old database pages 958754Modified db pages 115355Pending reads 
  0Pending writes: LRU 0, flush list 0, single page 0Pages made young 496041, not young 2381371.00 youngs/s, 
  0.00 non-youngs/sPages read 2593888, created 50425, written 2643540.33 reads/s, 3.00 creates/s, 29.66
   writes/sBuffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000Pages read ahead 0.00/s, 
   evicted without access 0.00/s, Random read ahead 0.00/sLRU len: 2597316, unzip_LRU len: 0I/O 
   sum[4513]:cur[165], unzip sum[0]:cur[0]---BUFFER POOL 2Buffer pool size 2621439Buffer pool size, 
   bytes 10737414144Free buffers 1024Database pages 2597259Old database pages 958733Modified db pages 
   115748Pending reads 0Pending writes: LRU 0, flush list 0, single page 0Pages made young 496770, 
   not young 2486471.00 youngs/s, 0.00 non-youngs/sPages read 2595651, created 51694, 
   written 2406831.00 reads/s, 0.00 creates/s, 35.99 writes/sBuffer pool hit rate 1000 / 1000, 
   young-making rate 0 / 1000 not 0 / 1000Pages read ahead 0.00/s, evicted without access 0.00/s, 
   Random read ahead 0.00/sLRU len: 2597259, unzip_LRU len: 0I/O sum[4513]:cur[165], 
   unzip sum[0]:cur[0]---BUFFER POOL 3Buffer pool size 2621439Buffer pool size, bytes 
   10737414144Free buffers 117082Database pages 2481220Old database pages 915938Modified db pages 
   112864Pending reads 0Pending writes: LRU 0, flush list 0, single page 0Pages made young 2305, 
   not young 00.00 youngs/s, 0.00 non-youngs/sPages read 2429753, created 51467, 
   written 2489670.00 reads/s, 0.00 creates/s, 40.99 writes/sBuffer pool hit rate 1000 / 1000, 
   young-making rate 0 / 1000 not 0 / 1000Pages read ahead 0.00/s, evicted without access 0.00/s, 
   Random read ahead 0.00/sLRU len: 2481220, unzip_LRU len: 0I/O sum[4513]:cur[165], 
   unzip sum[0]:cur[0]---BUFFER POOL 4Buffer pool size 2621439Buffer pool size, 
   bytes 10737414144Free buffers 207591Database pages 2390638Old database pages 
   882501Modified db pages 103991Pending reads 0Pending writes: LRU 0, flush list 0, single page 
   0Pages made young 2163, not young 00.33 youngs/s, 0.00 non-youngs/sPages read 2345049, created 45589, 
   written 2144341.00 reads/s, 1.00 creates/s, 32.99 writes/sBuffer pool hit rate 1000 / 1000, young-making 
   rate 0 / 1000 not 0 / 1000Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/sLRU
    len: 2390638, unzip_LRU len: 0I/O sum[4513]:cur[165], unzip sum[0]:cur[0]---BUFFER POOL 5Buffer pool size 
    2621439Buffer pool size, bytes 10737414144Free buffers 202652Database pages 2395634Old database pages 
    884332Modified db pages 99353Pending reads 0Pending writes: LRU 0, flush list 0, single page 0Pages made 
    young 2061, not young 00.00 youngs/s, 0.00 non-youngs/sPages read 2354114, created 41520, 
    written 1473470.67 reads/s, 3.67 creates/s, 30.99 writes/sBuffer pool hit rate 1000 / 1000, young-making 
rate 0 / 1000 not 0 / 1000Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/sLRU 
len: 2395634, unzip_LRU len: 0I/O sum[4513]:cur[165], unzip sum[0]:cur[0]---BUFFER POOL 6Buffer pool size 
2621439Buffer pool size, bytes 10737414144Free buffers 209285Database pages 2389036Old database pages 
881910Modified db pages 96899Pending reads 0Pending writes: LRU 0, flush list 0, single page 0Pages 
made young 1913, not young 00.00 youngs/s, 0.00 non-youngs/sPages read 2349930, created 39106, written 
1501741.33 reads/s, 0.67 creates/s, 38.65 writes/sBuffer pool hit rate 1000 / 1000, young-making rate 0 / 
1000 not 0 / 1000Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/sLRU 
len: 2389036, unzip_LRU len: 0I/O sum[4513]:cur[165], unzip sum[0]:cur[0]---BUFFER POOL 7Buffer pool 
size 2621439Buffer pool size, bytes 10737414144Free buffers 53605Database pages 2544738Old database pages 
939386Modified db pages 103037Pending reads 0Pending writes: LRU 0, flush list 0, single page 0Pages made 
young 2156, not young 00.00 youngs/s, 0.00 non-youngs/sPages read 2505281, created 39457, written 1684441.00 
reads/s, 0.33 creates/s, 28.66 writes/sBuffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 
1000Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/sLRU len: 2544738, unzip_LRU 
len: 0I/O sum[4513]:cur[165], unzip sum[0]:cur[0]
--------------ROW OPERATIONS--------------
0 queries inside InnoDB, 
0 queries in queue1 read views open inside InnoDB0 RW transactions 
active inside InnoDB0 RO transactions active inside InnoDB0 out of 1000 descriptors 
used---OLDEST VIEW---Normal read viewRead view low limit trx n:o 4402678824Read view 
up limit trx id 4402678824Read view low limit trx id 4402678824Read view inpidually stored trx ids:
-----------------Main thread process no. 70096, id 140207583508224, 
state: sleepingNumber of rows inserted 1347486, updated 6484099, deleted 505223, 
read 4665086631106.63 inserts/s, 1017.66 updates/s, 0.00 deletes/s, 1682192.60 reads/s
----------------------------END OF INNODB MONITOR OUTPUT============================

The above is the content of Slave memory leak and trigger oom-killer_MySQL. For more related content, please pay attention to the PHP Chinese website (www.php.cn)!