Running out of Memory
First off, GREAT PRODUCT!! I'm about at the point of needing to upgrade to the professional version, but I have been very impressed thus far.
My issue - I am getting the following error in my log:
Java Exception: class com.ibm.bsf.BSFException Target method exception(java.lang.OutOfMemoryError) message is: nullstack tracejava.lang.OutOfMemoryError
I've increased the "Maximum memory allocation in megabytes" in the settings, but am still running into issues. Is there a way reduce the amount of memory the program takes when running? Admittedly, I'm scraping large amounts of data but before I go buy more RAM, I'd like to see if there are any options.
Running out of Memory
Hi,
Based on the error message you're getting ("Object required 'dataRecord'") it appears as though you simply need to change the time your script gets run. I can't see all of the log, but it looks as though you're invoking your script "After pattern is applied" instead of "After each pattern application". If this is the case then the 'dataRecord' object won't be in scope at that moment. It might help to look over the "Variable scope" section on this page http//www.screen-scraper.com/support/docs/using_scripts.php , as well as this FAQ http//www.screen-scraper.com/support/faq/faq.php#ScopeScriptError .
Kind regards,
Todd
Variable issue
Thanks for the tips. I now remember why I was using session variables in the first place: I was having trouble with the "dataRecord" script. Looking at my log, I'm extracting "YEARBUILT" into a normal variable. Here's the log:
Storing this value in a session variable.
COUNTY=Clackamas
YEARBUILT=1997
Storing this value in a session variable.
KITCHENFEATURES= BI-MICO, COOK-IS, DISHWAS, DISPOSL, GASAPPL, INST-HW, PANTRY, PLB-ICE, FS-REFR
Storing this value in a session variable.
STORIES= 2
Immediately after running this extractor patter/sub-extractor pattern combo, I run a script called "Show YEARBUILT". This is the results in the log and the corresponding code:
LOG:
_______________________Show YEARBUILT________________________________
Home Details Page: An error occurred while processing the script: Show YEARBUILT
Home Details Page: The error message was: Scripting engine failure
Microsoft VBScript runtime error: 1:0
Object required: 'dataRecord'
(scode=0x800a01a8 wcode=0x0)
CODE:
session.log "_______________________Show YEARBUILT________________________________"
session.log dataRecord.Get ( "YEARBUILT" )
I also tried just showing the variable when I write to a file, which doesn't cause an error but shows only a null value:
LOG:
Processing script: "Write home data to file"
_____________________________________________________________________
null
CODE:
session.log "_____________________________________________________________________"
session.log ( DataRecord.Get ( "YEARBUILT" ) )
I've tried troubleshooting why I'm having an issue with datarecords, but am coming up short. Any thoughts? Thanks so much for your help.
Running out of Memory
Hi,
That's great to hear that screen-scraper is working out for you. Regarding the memory issue--how are you running your scraping sessions? If you're running them from within the workbench (GUI) it's likely that you'll run out of memory pretty quickly as the log will fill up. As an alternative, you ought to consider running screen-scraper from the command line (http//www.screen-scraper.com/support/docs/invoking_screenscraper_from_the_command_line.php).
Also, in terms of general optimization, ideally you should minimize the amount of information you're saving in session variables, which can have a big impact on memory usage. For example, if you're storing the scraped data sets in session variables you might consider instead simply writing them out to a file as the information gets scraped.
Hopefully that helps resolve the issue. If not, please feel free to post a reply.
Kind regards,
Todd Wilson