I have application made based on Yii2 advanced app template. One part of the frontent app is small shop with about 6K url-s. Big part of the log files now is filled by search engine crawlers. I wanted to know if there is anyone out there who has implementing log filtering for example based on user-agent? Or what kind of approach would be feasible here. The aim is that under normal processing I would have info level messages for users stored in the log file. But not for the googlebot or bingbot or petalbot or …
I was thinking that maybe during bootstrap process disable log completely if user is one of the search engine crawlers. Maybe somebody can offer different / better way?
I use modsecurity, which would not only help to filter them out of your logs, it can block or drop their connection if they are bad actors, or just a nuisance wasting band width. It comes with standard rules, plus you can add custom rules based on partial text matches. I have one rule for Yandex which was slowing my site down.
SecRule REQUEST_HEADERS:User-Agent “@contains YandexBot” “id:300,phase:1,drop,nolog,status:403,msg:‘Crawler Attempt’”
If it is just a case of not having them fill up the logs, the ‘nolog’ command will prevent it, you can leave out the “drop” or “block” commands and the access will occur normally, just without being logged.
