I am trying to loop through records to build a csv data feed, and I want to be memory efficient. So I select items one at a time from the database, and mark the item as sent once it is processed, so that I can select one at a time until my select returns no more items. I use a scope that left joins my item to the table where I mark each item as sent.
I cannot figure out why this code continues to use more memory over and over. Even though I select one thing at a time and continually use the same variable to hold the active record object. I have tried even setting the variable to null before performing another find, and nothing I do causes the memory allocation to go down.
do {
$current_mem = memory_get_usage();
$this->currentItem = MyModel::model()->scopeIsActive()->scopeThatDoesLeftJoin()->find(['limit' => 1]);
$this->processItem(); //outputs some from $this->currentItem columns to a csv file and market the item as sent
$mem_delta = memory_get_usage() - $current_mem;
echo "Item {$this->numItemsProcessed}: {$this->currentItem->id} - mem usage {$current_mem} delta {$mem_delta}\n";
} while (!is_null($this->currentItem));
Is there an appropriate way to loop through a dataset with active record in a memory efficient fashion? Does anyone know why this happens?
Each time I run this code, I get the memory allocation error in a different part of the yii base code, presumably because each row is a slightly different size, and so pulling different rows allocates different amounts of memory, and it is generally not the active record that finally hits the memory limit.
Thanks for the link. I know this can be done without active record, but at this point I am curious about the memory issue and trying to find out why there is such a leak.
Is it just a known issue that active record models never release their memory after instantiation?
This is not an issue, but memory required by using Active Record (if you investigate Active Record content, you will find that it uses many data structures).
I’ve narrowed it down to a simple loop, to show that this is absolutely a CActiveRecord memory leak:
<?php
class TestMemoryCommand extends CConsoleCommand
{
protected $currentRecord;
protected $numRowsProcessed = 0;
protected $maxRowsToProcess = 50000;
public function actionTestMemory()
{
do {
$memoryUsage = memory_get_usage();
$this->currentRecord = null; // running through a bunch of stuff causes memory probs. I'm trying to trigger garbage collection to fire
gc_collect_cycles();
$this->currentRecord = YourModel::model()->find();
$memDelta = memory_get_usage() - $memoryUsage;
echo "Row {$this->numRowsProcessed}: mem usage {$memoryUsage} delta {$memDelta}\n";
$this->numRowsProcessed++;
} while ($this->numRowsProcessed < $this->maxRowsToProcess);
}
}
If you run the above code, (substitute YourModel for any CActiveRecord model, obviously) you will see that the memory usage continues to climb no matter what.
As I am digging around, I have noticed that calling Yii::log and Yii::trace allocate memory that is never released. I am still digging in to see if there could be more to this - could so much logging happen that over time it becomes problematic, or is something else happening as well?
In any event, the culprit is CActiveRecord.find() and the methods that it calls, at least in Yii 1.14. The hunt continues.
I got it. The trick is to set autoDump to true and tell the logger not to hold on to things forever. Adding the following to the init function solved my problem.
