Memory usage when inserting records

Hi,

in my application I have one instance where I need to inset lots of new records. I noticed that it doesn’t take very long until I reach the memory limit

For testing this, I’ve used my group record which only has id and name columns.




for($i=1000;$i<3000;$i++) {

	echo "$i: ".memory_get_usage(true)."\n<br />";

	$g = new Group();

	$g->attributes = array('name' => $i);

	$g->save();


}



turns out I can save less than 400 records before I reach the memory limit of 16MB.

Every new record needs about 33kb of new memory without any every being released.

Doing unset($g) also doesn’t help

Does anyone have an idea of what to do here?

How about defining YII_DEBUG to be false in the entry script?

The logged messages perhaps consume most of the memory.

I just tried that and it only gave a minor improvement

even when I do only




for($i=1000;$i<3000;$i++) {

  echo "$i: ".memory_get_usage(true)."\n<br />";

  $g = new Group();

  unset($g);

}



it only does it about 900 times.

So just creating the object seems to take up memory which is not being freed up

What does the result of your echo statements look like? I just did a test, but couldn’t see the problem. My table has 13 columns. When YII_DEBUG is true, it takes about 500 bytes for each insertion. When YII_DEBUG is false, each new insertion doesn’t increase memory consumption. That means, the incremental memory consumption is caused by the logging. Also, from the code point of view, internally there is no caching or buffering for insertion at the Yii layer. I suspect the memory accumulation occurs at the PDO layer which we have no control on how it manages the memory.

Hi quiang

sorry that the response took a while.

I did some investigating. At first I created a brand new application and everything was fine.

Then I went back to my application and really the only difference was that my Group model has a behavior attached to it.




public function behaviors() {

  return array(

   'AutoTimestampBehavior' => array('class' => 'AutoTimestampBehavior'),

  );

}



when commenting this out everything is fine. The memory leak seems to be related to attaching behaviors.

Even when I comment out everything in the behavior the memory still keeps increasing.

When I remove the behavior everything is fine and the memory usage stays at around 3.5MB

I looked into it some more and I think it is PHP’s circular reference memory leak since the Behavior has an owner property that links back to the model

I don’t really know the details about this. Is there any way how one could manually do some unset() to fix this?

That makes sense. Could you please try the following to see if it helps:

Modify CModel.php by adding a destructor in which you call $this->detachBehaviors().

That doesn’t work. For whatever reason __destruct is not called the script is shutting down (due to the fatal error)

According to the manual __destruct is not actually called until all references to that object are removed.

Unless you have a better idea I’ll just have to call detachBehaviors manually before unsetting the object. Then it works just fine.

I think it could be done by implementing a finalize method in CComponent for deterministic cleanup of resources (like in C#).

/Tommy

I think I’ve found something interesting. The magic method chain (i.e. CActiveRecord, CComponent) does not get called if I unset() the object:




unset($g);



but it is if I unset() some existing or fictive property




unset($g->not_defined_dummy);



Edit:

Thus, no memory leak in the above example if $g is replaced with $g->something.

/Tommy

not exactly sure what you mean. Could you elaborate that?

Thus, no memory leak in the above example if $g is replaced with $g->something.

I just tried that and I can’t confirm that.

Whenever there is a behavior attached there will be a memory leak unless you call $model->detachBehaviors() before the variable is being unset or loses scope.

Sorry, I forgot I’d commented out behaviors() from the model.

I’m confused about the semantics of unset() vs __unset() and about the exact reason for __destruct() not being called (I’m still learning PHP).

I guess it’s not too wise to place detachBehaviors in the models (or parents) __unset().

/Tommy

As far as I know, magic __unset method only applies to non-existent class properties.

What if you use $class=null instead of unset?

That doesn’t help for the memory leak. The __unset was brought up by me, digging into the framework, trying to understand what’s going on. I’m probably on the wrong track.

To conclude what this thread is about: AR objects created in a loop with behaviors automatically attached, leaks memory. An explicit call to detachBehaviors() before unsetting the AR object itself will take care of that, but it seems like there is no easy way to unhook the behavior automatically since __destruct() won’t be called until all references to the AR object are released (the behavior holds at least one reference to the object). The question is: who will call detachBehaviors()?

/Tommy

I highly doubt that there is a way to handle this automatically. The best option is to upgrade to PHP 5.3 since that allegedly fixes the circular references memory leak (didn’t try it yet).

If PHP 5.3 is not an option it gets more difficult and you’ll have to think about where in your code you might instantiate many objects and remember to call detachBehaviors() before they lose scope. I don’t think there is any other way

@MarcS

Thanks for pointing to this interesting topic. This is definitely something one should be aware when fighting strange memory issues of an application.

Googling for "PHP circular references" gave me another very nice article by Derick Rethans explaining the details:

http://derickrethans.nl/circular_references.php

I agree to your opinion that there’s no automatic solution. Except preventing circular references in the first place. Which seems not to be an option for behaviors.

This kept me thinking. One workaround would be to change Cbehavior and CComponent like this:


// In CBehavior:

    public function attach($owner)

    {

        //$this->_owner=$owner;

        foreach($this->events() as $event=>$handler)

            $owner->attachEventHandler($event,array($this,$handler));

    }


     public function attachOwner($owner)

    {

        $this->_owner=$owner;

    }


     public function detachOwner($owner)

    {

        unset($this->_owner);

    }




// In CComponent

    public function __call($name,$parameters)

    {

        if($this->_m!==null)

        {

            foreach($this->_m as $object)

            {

                if($object->enabled && method_exists($object,$name)) {

                    $object->attachOwner($this);

                    $retval= call_user_func_array(array($object,$name),$parameters);

                    $object->detachOwner();

                    return $retval;

                }

            }

        }

        throw new CException(Yii::t('yii','{class} does not have a method named "{name}".',

            array('{class}'=>get_class($this), '{name}'=>$name)));

    }



Edit:

This solution misses some logic to also make event handlers in the behavior work that access $this->owner.