Can't Use More than 1024MB of memory
I am running some fairly large scrapes and I am running into outofmemory errors quite frequently. I was just wondering if there was any way to set the memory allocation higher than 1024MB. I am in the process of making my scrapes more efficient by using some tips from http://blog.screen-scraper.com/2008/07/07/large-data/ . Any reason why I can't use more memory to help alleviate this problem?
Well, technically you can go
Well, technically you can go higher, but we've found that going higher doesn't seem to offer much further performance, and on 32 bit machines it may actually introduce some instability. On my Win7 64bit, I've been playing with it though, and I am not seeing the instability issues.
That said, I've done a lot of scraping, and most of the time you don't need more memory that 1G. In the case that you have hundreds of scrapes, you may want more, but it's more advantageous to have multiple screen-scraper servers running so if one goes down, the others are unaffected.
Thanks for the reply Jason!
Thanks for the reply Jason!
andrewbkillen, Keep in mind,
andrewbkillen,
Keep in mind, too, that you're allowed any number of installs on a single machine with your license. So, go ahead and push it and report back if you run into any issues.
-Scott