java - Stress test tomcat, unable to understand its working mechanism?

Question

Premise:
1. The company conducted a stress test on a machine deployed with http service. On the first day, the test was conducted for one night (10h) (200 concurrency at a time), and the throughput was about 56 %about. The next day (tomcat was not restarted during this period), a night (10h) test (200 concurrency at a time) was also conducted, but the throughput dropped to about 16%. The environment is the same, why is the gap so big?
2. When I stress tested 1,000 concurrency on the first day, the number of threads was about 1,000. However, when I stress tested 1,000 concurrency on the second day (I did not restart tomcat during this period), the number of threads dropped to 700. Why did the number of threads drop to 700 before? It is directly proportional to the number of threads, but not later?

For the above two premises, does Tomcat have any strategies, or does the jdbc connection pool or redis connection pool cause the above phenomenon (using the G1 recycling mechanism)?

Answer 1

This situation is complicated. Its complexity depends on the running environment of your project and which other services it depends on. If your stress test environment is complex (that is, many people are running their own things on your server), it is foreseeable that the stress test results will be unstable.

When encountering a drop in throughput, first determine where the bottleneck is:

1. Are the local resources tight? Native resources mainly include CPU, memory, network bandwidth, and disk throughput. These all require observation and investigation.

2. Whether dependent services are under strain, such as whether the processing time of databases and external interfaces is too long.

3. If none of these can clearly locate the problem, then enter the program debugging stage: during each request processing process, record the duration of each step and find out which step the bottleneck is. This granularity will be very fine and it will need to be modified repeatedly. Log, run and observe repeatedly, but you will definitely find the problem.

Don’t make unfounded random guesses as soon as you encounter a problem. “Divergent thinking” won’t help at this time. What you need to do is to follow the problem and analyze it rigorously.