Do you really need to access 120 million rows of uncached, unprocessed data on demand???
If you are doing some kind of statistical analysis on this information wouldn’t it be better to process stuff at 3am via cron and store results in an intermediate table for retrieval?
Ok, Yii doesn’t pull all the data at once, but if I save the data into an array this array will allocate all my RAM.
So is there any possibility to work with all 120 million rows in a loop without saving them into an arrays first…
My Yii-App is started with a cronjob every day at 4am. Than it has to walk through all 120 million lines and to save things… This works but at the moment I write all rows into an array and walk through the array. The array is the problem in my case… It’s not good to have a multidimensional array with 120 million rows…
Should I use an iterator/cursor to walk through my rows in the database? Will this solve my problem and is there something like this?
I have found this:
Iterator support
It sounds like the same problem as I have… Should I give this a try or exists any better solution?
I want to process maybe about 100 rows, than select the next 100 rows and process them and so on…
I think this should be possible with this iterator feature, query that query builder generates should look like this:
SELECT * FROM logger LIMIT 0, 100;
SELECT * FROM logger LIMIT 101, 200;
…
So I wouldn’t have any problem. If this feature doesn’t do this correctly, I will make my own feature that makes this and release it under an open source license.
At this point I have a question: I have read that Yii 2 will have iterator support integrated into active record… What does Yii 2 in the background if I make a
foreach (Logger::find() as $logentry)
and Logger::find() will find about 120 million rows?
Does the iterator load everything into an array in the background and end in a "master crash"?
do not use queryAll when there are too much data . it will load all data into ram . use stream/cursor (do you know the “cursor” in sql , if don’t hear about that please google it ) , the CDbDataReader support it(mimic way) .