I want to do some kind of multi-threading in a PHP8/Symfony6 environment and it works like this:
I'm making a principle request that serves about a million rows
I want to create multiple parallel processes to process these rows at the same time. My request is made via getResult() which serves the doctrine object. Is there a way to create a thread with an array of objects as parameter. Then create threads whenever my "queue" is filled with data?
I found some information about "pthreads" or "popen" but I'm not sure if it's still relevant since the topic is a bit old
P粉2112735352023-09-15 10:29:17
In this case, one of the things Doctrine does is hydrate a class with the data content, so all those millions of records are converted to classes and hold them in memory, so a couple of suggestions: p>
public function paginate(int $start = 0, int $offset = 10): Doctrine\ORM\Tools\Pagination\Paginator { $qb = $this->createQueryBuilder('student'); $qb ->where($qb->expr()->isNull('student.schoolRef')) ->setFirstResult($start) ->setMaxResults($offset); return new Paginator($qb, true); } src/Service.php $start = 0; $offset = 100; do { $i = 0; $students = $this->studentRepository->paginate($start, $offset); foreach($students as $student) { //do stuff ++$i; } $this->entityManager->flush(); $this->entityManager->clear(); } while ($i == $offset);
<?php $sql = <<<SQL SELECT * FROM user u SQL; $statement = $connection->prepare($sql); $statement->execute(); while ($user = $statement->fetch()) { // print_r($user); }
You can also use the fetchAll
method to fetch all users at once, but be aware that you may run out of memory if your query returns too many users. Using the while
loop above, you are only fetching one user into memory at a time, so memory performance is better.
<?php
// ...
// all users are in memory here
$users = $connection->fetchAll($sql);
foreach ($users as $user) {
// print_r($user);
}
As someone commented, you can use