search
HomePHP FrameworkLaravelDetailed explanation on using Lazy Collections to improve the performance of Laravel Excel reading (easily supports millions of data)

A new type of collection has been added in Laravel 6: Lazy Collections. They are great if you need to process very large data sets (thousands or millions of rows) without running into memory constraints.

Recommended: laravel tutorial

My most recent task was to refactor Excel export in a project at work. The problem is that the export can no longer be created because the dataset is too large for Laravel to handle. The database query returned approximately 300,000 results! The application times out or consistently runs out of memory.

A naive approach is to increase the timeout or memory limit and hope that the next time something goes wrong, someone else will handle it. But that's not how I work. I don't like Band-Aids. I like concrete, long-term solutions.

Laravel Excel extension pack is already quite flexible. It does a great job of reducing database load by using "chunks" when using FromQuery-concerns. However, our export still struggles with large data sets.

My colleagues and I discussed whether we should completely rewrite this feature: push the export to a queue and send a notification to the user when the export is complete. However, this feature is just a minor thing in this app. It doesn't make sense to us to add so much overhead just for a simple export.

Later that day, I had a little "Eureka" moment because I remembered that Laravel had LazyCollections.

I rewrote the export: it now uses FromCollection-concern instead of FromQuery. The only change I had to make to the collection() method was to replace the get() method at the end of the query builder chain with a cursor().

Below is a simplified version of our export functionality. The Request object is passed through the constructor so we can adapt the query based on what the user selects in the UI.

<?php 
namespace App\Exports;
use App\User;
use Maatwebsite\Excel\Concerns\Exportable;
use Maatwebsite\Excel\Concerns\FromCollection;
use Illuminate\Http\Request;
class UsersExport implements FromCollection
{
    use Exportable;
    protected Request $request;
    public function __construct(Request $request)
    {
        $this->request = $request;
    }
    public function collection()
    {
        return User::query()
            ->when($this->request->get(&#39;include_subscribed&#39;), function ($q) {
                return $q->where(&#39;is_subscribed&#39;, true);
            })
            ->cursor(); // ← 重要的一点
    }
}

I believe you are experiencing memory issues in your project. You increased the memory limit and hopefully that solved the problem (I've done this countless times myself).

If it were in a Laravel project, I hope, I could have you revisit that code and rewrite it using LazyCollections.

Fixing this issue was very interesting, so I did a little benchmark: our export can now easily export millions of rows without hitting memory limits. so cool!

The above is the detailed content of Detailed explanation on using Lazy Collections to improve the performance of Laravel Excel reading (easily supports millions of data). For more information, please follow other related articles on the PHP Chinese website!

Statement
This article is reproduced at:learnku. If there is any infringement, please contact admin@php.cn delete
Tech Troubles: Ensuring Equitable Access to Tools and Resources for Distributed Team MembersTech Troubles: Ensuring Equitable Access to Tools and Resources for Distributed Team MembersApr 29, 2025 am 12:40 AM

Methods to ensure that distributed team members have fair access to tools and resources include: 1) using low-bandwidth alternatives, such as asynchronous video or text updates, to solve connection problems; 2) setting up core overlapping working hours and providing flexible working hours to manage time zone differences; 3) adapt to different cultural needs through translation functions and cultural awareness training. These strategies help create an inclusive and efficient remote working environment.

Instant Messaging Must-Haves: Fostering Real-Time Communication in Remote SettingsInstant Messaging Must-Haves: Fostering Real-Time Communication in Remote SettingsApr 29, 2025 am 12:38 AM

Forenhancingremotecollaboration,aninstantmessagingtoolmusthave:1)reliabilityforconsistentmessagedelivery,2)anintuitiveuserinterfaceforeasynavigation,3)real-timenotificationstostayupdated,4)seamlessfilesharingforefficientdocumentexchange,5)integration

Have you ever faced any challenges while working in distributed teams?Have you ever faced any challenges while working in distributed teams?Apr 29, 2025 am 12:35 AM

Thebiggestchallengeofmanagingdistributedteamsiscommunication.Toaddressthis,usetoolslikeSlack,Zoom,andGitHub;setclearexpectations;fostertrustandautonomy;implementasynchronousworkpatterns;andintegratetaskmanagementwithcommunicationplatformsforefficient

What are the security improvements in the new Laravel version?What are the security improvements in the new Laravel version?Apr 29, 2025 am 12:17 AM

Laravel's latest version has significantly improved security, including: 1. Enhanced CSRF protection, through a more robust token verification mechanism; 2. Improved SQL injection protection, through an enhanced query construction method; 3. Better session encryption to ensure user data security; 4. Improved authentication system, supporting finer granular user authentication and multi-factor authentication (MFA).

Time Zone Tango: Navigating Scheduling Conflicts in a Global WorkforceTime Zone Tango: Navigating Scheduling Conflicts in a Global WorkforceApr 29, 2025 am 12:13 AM

Tonavigateschedulingconflictsinaglobalworkforce,usetechnology,empathy,andstrategicplanning:1)EmploytoolslikeWorldTimeBuddyorCalendlyforscheduling;2)Rotatemeetingtimestoensurefairness;3)Establishcorehoursforoverlap;4)Beculturallysensitiveandflexiblewi

Full-Stack Development with Laravel: Managing APIs and Frontend LogicFull-Stack Development with Laravel: Managing APIs and Frontend LogicApr 28, 2025 am 12:22 AM

In Laravel full-stack development, effective methods for managing APIs and front-end logic include: 1) using RESTful controllers and resource routing management APIs; 2) processing front-end logic through Blade templates and Vue.js or React; 3) optimizing performance through API versioning and paging; 4) maintaining the separation of back-end and front-end logic to ensure maintainability and scalability.

Lost in Translation: Cultural Nuances and Misunderstandings in Distributed TeamsLost in Translation: Cultural Nuances and Misunderstandings in Distributed TeamsApr 28, 2025 am 12:22 AM

Totackleculturalintricaciesindistributedteams,fosteranenvironmentcelebratingdifferences,bemindfulofcommunication,andusetoolsforclarity.1)Implementculturalexchangesessionstosharestoriesandtraditions.2)Adjustcommunicationmethodstosuitculturalpreference

Measuring Connection: Analytics and Insights for Remote Communication EffectivenessMeasuring Connection: Analytics and Insights for Remote Communication EffectivenessApr 28, 2025 am 12:16 AM

Toassesstheeffectivenessofremotecommunication,focuson:1)Engagementmetricslikemessagefrequencyandresponsetime,2)Sentimentanalysistogaugeemotionaltone,3)Meetingeffectivenessthroughattendanceandactionitems,and4)Networkanalysistounderstandcommunicationpa

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

SublimeText3 Linux new version

SublimeText3 Linux new version

SublimeText3 Linux latest version

SecLists

SecLists

SecLists is the ultimate security tester's companion. It is a collection of various types of lists that are frequently used during security assessments, all in one place. SecLists helps make security testing more efficient and productive by conveniently providing all the lists a security tester might need. List types include usernames, passwords, URLs, fuzzing payloads, sensitive data patterns, web shells, and more. The tester can simply pull this repository onto a new test machine and he will have access to every type of list he needs.

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

VSCode Windows 64-bit Download

VSCode Windows 64-bit Download

A free and powerful IDE editor launched by Microsoft

PhpStorm Mac version

PhpStorm Mac version

The latest (2018.2.1) professional PHP integrated development tool