crawl links

Yii 1.1 General Discussion
1

hi,

I have a table of links, which I must check links content (a table content has many rows and growing faster each day), I need a consultant, I should crawl links (e.g every 12 hours), what should I do in this problem?

I think about run a cronjob every 12 hours and get links which I should check at this time for crawling and adding to queue to crawl links each one or multi at the same time in each queued job, is it the good idea?

and one another question, I want to retry link in a queue if it is not done at a specific situation which for example my content not available on that link, I want to retry not done jobs e.g 4 times and between each try wait 10 minutes for retrying. is it possible doing this with the yii2 queue and can anyone give me help about ttr and what is the meaning of maximum time, does it means what I need? :mellow:

2

Yii2 queue is what you need. Better than fixed cronjobs.

3

I have seen you posted this on Yii1.1. Are you using this version? If so my solution is not valid for you.

4

exeuse me, it was mistake, im using yii2, can you plz describe a little how to do this with retry option on yii2 queue, how to set time between each try on a queue? some one tell me about new task if failed occure in crawl.