screen-scraper stops execution if run for long time
Hi all,
Thanks in advance.
I ve to scrap properties in "UK for sale" from some sites using postcode.
My postcode db has 1.7 million records.
Our php script calls screen-scraper to fetch the data.
But the problem is screen-scrpaer stops execution,when it scraps data for around 10,000 postcodes.
I ve increased php execution time to 10000,input time to 60 and memery size to 500M in my php.ini file
Also i ve changed setBuffersize in remote_scraping_session to 64000.
Can any one help me in solving this issue.
I would like to know one more thing, how can i access my mysqldb from screen-scraper directly.If possible pls give me some sample code
Cheers :D ,
Thanks in advance,
Arun
screen-scraper stops execution if run for long time
Hi,
Regarding screen-scraper stopping execution, this FAQ might help: here. It's possible that screen-scraper is running out of memory, so you'll want to be especially careful about that. You might check the error.log and wrapper.log inside screen-scraper's log folder for any other clues.
On your question about accessing your database directly, you might try this FAQ: database, as well as this tutorial: here.
Kind regards,
Todd Wilson