Allowing Search Engine Bots


One of my controller does not allow unauthenticated users any CRUD operation.

However i want to allow ‘view’ operation to search engine bots like google, yahoo, bing and similar bots - so that my pages get indexed by the search engines.

How do i allow bots in ?

thanks for replying :)

allow google to crawl password protected pages

hope this helps

no it didn’t help.

i needed a YII specific answer more specifically w.r.t access rules and probable methods to allow specific user agents the view rights from within the controller - not just usual replies and general stuff like robots.txt etc which google spits out.

Unfortunately there is no discussion on these lines in the YII forum - not that any has been indexed by google at least.

thanks for replying anyways

look at the accessRules method of your controller. Each rule can be bound with ips or expression Search for a white list of google bot ips and try to work around that. I have not implemented such a solution so cannot go into details but I believe this should be a good start

Uhm … what speaks against allowing all users (logged in or not) to access the view action of said controller?

This is the sensible option. If you let Google crawl the pages then they’re going to be available to everyone via Google’s cache anyway. You should either grant everyone read access or you should not allow the pages to be crawled at all.

thanks for your replies,

further probe lead me to First Click Free Policy advocated by google for such protected content.

Not sure how it would be implemented in YII though.

thanks again