Why your FastAPI (or Flask) App performs poorly with high loads

First of all, apologies for the title bait ?, but I figured this issue out last night and I am still under the effects of the dopamine rush. I just have to share this.

This text is intended for entry-level developers or Data scientists (not senior Python software engineers) and I will write this as a narrative, or in other words the chronological sequence of events as they happened, instead of a "technical paper (structured in problem, solution, discussion). I like this approach because it shows how things happen in real life.

Initial Considerations

These tests were done on GCP Cloud Run using a single processor, and 512M RAM machine, and we used Locust, an incredible tool (for Python, LoL).

Also, if you are already having performance issues on single requests on Postman, I strongly suggest you take a look at this repo dedicated to increase FastAPI performance from kisspeter and this one from LoadForge.

First Test Round

Using a single request in Postman, after Cloud Run started, I was getting around 400ms response time. Not the best, but totally within an acceptable range.

Our load test is quite simple: reads, writes and deletes in one table ( or GETs, POSTs and DELETEs to the API endpoints). 75% reads, 20% writes, 5% deletes. We run it with 100 concurrent users for 10 min.

At the end we got a 2s average response time, but the most disturbing part is that the avg time was still increasing when the test ended, so it is very likely the number would still grow more before ( and if ) it stabilizes.

I tried to run it locally on my machine, but to my surprise, the response time in Postman was 14ms only. However, when running the load test for 500 concurrent users, the problem appeared again ? ...

By the end of the test, the response time was about 1.6s and still increasing, but some glitch happened, and the 95th percentile sky rocketed (and ruined the graph =( ). Here are the stats:

Now, why does a server that responds with 14ms suddenly go up to 1.6 seconds with only 500 concurrent users?

My machine is a core i7, 6 cores, 2.6GHz, 16Gb RAM, SSD. It should not happen.

What gave me a good hint was my processor and memory logs... They were extremely low!

This probably means my server is not using all the resources from my machine. And guess what? It was not. Let me present to you a concept the vast majority of developers forget when deploying FastAPI or Flask applications to prod: the process worker.

As per getorchestra.io:

Understanding Server Workers

Server workers are essentially processes that run your application code. Each worker can handle one request at a time. If you have multiple workers, you can process multiple requests simultaneously, enhancing the throughput of your application.

Why Server Workers are Important

• Concurrency: They allow concurrent handling of requests, leading to better utilization of server resources and faster response times.
• Isolation: Each worker is an independent process. If one worker fails, it doesn't affect the others, ensuring better stability.
• Scalability: Adjusting the number of workers can easily scale your application to handle varying loads.

In practice, all you need to do is add the optional --workers param to your server initialization line. The calculation of how many workers you need depends a lot on the server you are running your application and the behavior of your application: especially when it comes to memory consumption.

After doing it, I got much better results locally for 16 workers, converging to 90ms (for 500 concurrent users) after 10 min:

Final Test Round

After configuring the microservices with the appropriate number of workers (I used 4 for my single processor Cloud Run instance), my results were incredibly better in GCP:

The final value converges to 300ms at the end of the test in the GCP server, which is at least acceptable. ?

The above is the detailed content of Why your FastAPI (or Flask) App performs poorly with high loads. For more information, please follow other related articles on the PHP Chinese website!