Analysis of the reasons why PHP cannot recover after overload_PHP tutorial

Recently, PHP machines have been frequently overloaded and can no longer provide services. As long as a request is sent, the PHP process responsible for processing the request will occupy 100% of the CPU. The original load balancing strategy is to reduce the weight of a certain machine once a connection timeout occurs in the PHP request of the machine, and the probability of requests sent to the machine will be reduced. Although there is a certain lag effect, it should eventually be able to reduce the pressure and finally restore the service. , but this strategy suddenly failed recently. After this situation occurs, any request that cannot be sent to php-fpm will be cpu100%, even if the request is an empty php file. So I guessed it might be caused by eaccelerator.

The request_terminate_timeout of our Php-fpm is set to 5s, so as long as a request is executed for more than 5s, the execution process will be killed by php-fpm. There were a large number of 5s timeouts before and after the problem occurred. The initial guess may be Because of the shared memory of eaccelerator, the shared memory was written incorrectly when the child process was killed, causing errors in all requests. However, this did not explain the problem that new files would also be stuck, so I looked at the code of eacceleraotr and found the following Code

[cpp]

#define spinlock_try_lock(rw) asm volatile("lock ; decl %0" :"=m" ((rw)->lock) : : "memory")

#define _spinlock_unlock(rw) asm volatile("lock ; incl %0" :"=m" ((rw)->lock) : : "memory")

static int mm_do_lock(mm_mutex* lock, int kind)

{

while (1) {

spinlock_try_lock(lock);

if (lock->lock == 0) {

lock->pid = getpid();

lock->locked = 1;

return 1;

} }

_spinlock_unlock(lock);

sched_yield();

}  

return 1;

}

static int mm_do_unlock(mm_mutex* lock) {

if (lock->locked && (lock->pid == getpid())) {

lock->pid = 0;

lock->locked = 0;

_spinlock_unlock(lock);

}

return 1;

}

[cpp]

The mm_mutex points to shared memory, which means that eac uses shared memory as an inter-process lock, and uses the spinlock method, so everything can be explained. Imagine the following situation. After a process gets the lock, it is killed by php-fpm. It does not unlock. In this way, all php-fpm child processes cannot get the lock, so everyone is in this while(1) Stuck in the loop. Now that we have a conjecture, how can we confirm it? The original idea was to read the shared memory directly, but it turned out that php was IPC_PRIVATE, so there was no way to read it. So I had to wait until something went wrong online and go to gdb to check the memory. Today I finally have conclusive evidence

[html]

(gdb) p *mm->lock

$8 = {lock = 4294966693, pid = 21775, locked = 1}

You can see here that the memory has been obtained by the process with process number 21775, but the fact is that this process has been killed a long time ago.

The problem has been confirmed, then let’s look back at the conditions under which this problem occurred

1. The request execution time is very long, so long that it will be killed by php-fpm

2. When the process is killed, php is requiring the file, and eac gets the lock

As you can see from here, there are some specific situations that will amplify this probability

1. The request_terminate_timeout time is very short

2. Use the auoload method, or require the file in the execution logic, because if all files are loaded before the request starts, then unless the require file alone has timed out, it should not be killed when the file is required. But there is also an ugly way to avoid this problem using the same autload method, which is to judge in the autload function. If the execution time is too long, just exit instead of require

Personally, I think the best way to solve this problem is to set the request_terminate_timeout time long enough, such as 30s, 300s, and place all timeout judgments on the application layer. This problem cannot be handled through php-fpm, php-fpm Facts should only be used as a last resort insurance, insurance that has to be used. In addition, there is a timeout setting max_execution_time in php, but this timeout is cpu time in cgi mode, so it has little effect

http://www.bkjia.com/PHPjc/477815.htmlwww.bkjia.comtruehttp: //www.bkjia.com/PHPjc/477815.htmlTechArticleRecently, PHP machines have been frequently overloaded and can no longer provide services. As long as there is a request, we will be responsible for processing it. The requested PHP process is using 100% of the CPU. Original load balancing strategy...