What are the techniques for memory allocation and usage statistics of Redis?

Specifically:

If Google’s TC_MALLOC library exists in the system, use the tc_malloc family of functions instead of the original malloc family of functions.

If the current system is a Mac system, use the memory allocation function.

In other cases, an additional fixed-length field is allocated at the beginning of each allocated space to record the allocated space size.

The source codes are in config.h and zmalloc.c respectively:

/* config.h */

#if defined(USE_TCMALLOC)

#include

　#if TC_VERSION_MAJOR >= 1 && TC_VERSION_MINOR >= 6

　#define HAVE_MALLOC_SIZE 1

　#define redis_malloc_size(p) tc_malloc_size(p)

　 #endif

　 #elif defined(__APPLE__)

　 #include

　 #define HAVE_MALLOC_SIZE 1

　 #define redis_malloc_size(p) malloc_size(p )

　#endif

　/* zmalloc.c */

　#ifdef HAVE_MALLOC_SIZE

　#define PREFIX_SIZE (0)

　# else

　#if defined(__sun)

　#define PREFIX_SIZE (sizeof(long long))

　#else

　#define PREFIX_SIZE (sizeof(size_t ))

　#endif

　#endif

Because tc_malloc and the malloc function family under the Mac platform provide functions for calculating the size of allocated space (tc_malloc_size and malloc_size respectively) , so there is no need to allocate a separate space to record the size. For Linux and Sun platforms, the size of the allocated space must be recorded. For linux, use sizeof(size_t) fixed-length field record; for sun os, use sizeof(long long) fixed-length field record. That is the PREFIX_SIZE macro in the source code above.

Re-express this sentence: The purpose of this record is to calculate how much memory is occupied by the current process. In zmalloc.c, there is such a static variable:

static size_t used_memory = 0;

It records the total amount of memory currently occupied by the process. Whenever memory is allocated or released, this variable must be updated. Because when allocating memory, you can clearly know how much memory to allocate. But when releasing memory, (for platforms that do not provide the malloc_size function) you cannot know how much space is released just by pointing to the pointer to the memory to be released. In this case, the fixed-length field specified by PREFIX_SIZE comes into play, and the size of the space can be obtained from the information recorded in it. The zmalloc function is as follows (removing irrelevant code):

void *zmalloc(size_t size) {

void *ptr = malloc(size PREFIX_SIZE);

if (!ptr) zmalloc_oom(size);

　*((size_t*)ptr) = size;

　update_zmalloc_stat_alloc(size PREFIX_SIZE,size);

　return (char*)ptr PREFIX_SIZE;

 . When running on a Mac system or using tc_malloc, the value of PREFIX_SIZE is zero. Next, the first size_t bytes of the memory block pointed to by the pointer ptr will be used to record the allocated size of the memory block. The last thing returned is the pointer beyond the record area. The zfree function is similar (removing irrelevant code):

void zfree(void *ptr) {

void *realptr;

size_t oldsize;

if ( ptr == NULL) return;

realptr = (char*)ptr-PREFIX_SIZE;

oldsize = *((size_t*)realptr);

update_zmalloc_stat_free(oldsize PREFIX_SIZE );

free(realptr);

#endif

}

First move the pointer forward PREFIX_SIZE, and then take out the space saved when allocating space length. Finally, free the entire space.

The two macros update_zmalloc_stat_alloc(__n,__size) and update_zmalloc_stat_free(__n) are responsible for updating the used_memory variable when allocating or releasing memory. Defining it as a macro is mainly due to efficiency considerations. Restore it to a function, as follows:

void update_zmalloc_stat_alloc(__n,__size)

　{

　do {

　size_t _n = (__n );

　size_t _stat_slot = (__size < ZMALLOC_MAX_ALLOC_STAT) ? __size : ZMALLOC_MAX_ALLOC_STAT; ​if (_n&(sizeof(long)-1)) _n = sizeof(long)-(_n&(sizeof(long)-1)); ​if (zmalloc_thread_safe) { ​pthread_mutex_lock(&used_memory_mutex); ​used_memory = _n; ​zmalloc_allocations[_stat_slot]; ​pthread_mutex_unlock(&used_memory_mutex); } else { ​used_memory = _n; ​zmalloc_allocations[_stat_slot]; } } while(0) } void update_zmalloc_stat_free(__n) { do { size_t _n = (__n); ​if (_n&(sizeof(long)-1)) _n = sizeof(long)-(_n&(sizeof(long)-1)); ​if (zmalloc_thread_safe) { ​pthread_mutex_lock(&used_memory_mutex); ​used_memory -= _n; ​pthread_mutex_unlock(&used_memory_mutex); } else { ​used_memory -= _n; } } while(0) } In addition to updating the used_memory variable in the code, there are several other things to pay attention to: First, the low-order bits of _n are rounded up, and finally _n becomes a multiple of sizeof(long). For example, for a 32-bit system, sizeof(long) == 100 (binary). After _n is rounded up, the two low-order bits become is 0. If there are multiple threads in the process, locks must be locked when updating variables. There is another statistic to be updated in the zmalloc function: zmalloc_allocations[]. In zmalloc.c, zmalloc_allocations is defined as follows: size_t zmalloc_allocations[ZMALLOC_MAX_ALLOC_STAT 1]; Its function is to count the number of requests for different sizes of space when the program allocates memory. The statistical space ranges from 1 byte to 256 bytes, and the space larger than 256 bytes is counted as 256. Statistical results are returned by calling the zmalloc_allocations_for_size function: size_t zmalloc_allocations_for_size(size_t size) { if (size > ZMALLOC_MAX_ALLOC_STAT) return 0;

Return zmalloc_allocations[size];

}

Another memory usage statistics is returned by calling the zmalloc_used_memory function:

Size_t zmalloc_used_memory(void) {

size_t um;

if (zmalloc_thread_safe) pthread_mutex_lock(&used_memory_mutex);

um = used_memory;

　if (zmalloc_thread_safe) pthread_mutex_unlock(&used_memory_mutex);

return um;

　}

In addition, zmalloc.c also implements the zmalloc_get_rss function for different systems. In linux In the system, system statistics of memory usage are obtained by reading the /proc/$pid/stat file.

The above is the detailed content of What are the techniques for memory allocation and usage statistics of Redis?. For more information, please follow other related articles on the PHP Chinese website!