Would installing multiple instances help?
Hi,
I am trying to scrape multiple sites via the web console. Each scraping session scrapes a different site. The scraping sessions would stop (each at different time) even though it has not completely scraped the whole site. There are no error messages. But if I run the sessions individually, it would scrape the whole site correctly. Should I try installing multiple instances of Screen-Scraper on one machine to have it run properly?
Thank you in advance for your help!
You can have multiple
You can have multiple instances on one machine, but I'm not sure you'd need to. What version of screen-scraper are you running? Can you tell if maybe you're running out of memory or if your CPU is being overtaxed?
Its the latest stable version
Its the latest stable version 4.5.41a. We run it on a low end system which is dedicated to scraping: 1.2GHz VIA with 1GB DDR2 ram and 80GB storage on Win Xp.
Memory runs to less than 70-80%. Maximum concurrent scraping sessions is set at 5 but we are testing with only 2-4 scraping sessions. The scraping sessions would display "scraping session finished" but it did not do a complete scrape of the site. And this happens only when we run more than one scraping sessions. So I was wondering would running multiple instances help?
If it were an out of memory
If it were an out of memory issue, the logs would indicate that. Sometimes multiple instances do help, and instructions for that are here: http://community.screen-scraper.com/faq#81n156