Screen Scraper Hangs or Suddenly Closes
I recently purchased a new computer and am having issues with Screen Scraper Basic Edition v2.6 hanging (i.e. “Not Respondingâ€) or suddenly closing while running my scripts. Any help you can give me will definitely be appreciated!
The scripts run fine on my old machine using Screen Scraper basic edition v2.5 except that they eventually stop due to an Out of Memory error (hence the new, more powerful computer purchase). My goal is to fix these errors, run the script successfully from start to finish for the dataset I want to capture, and then automate the whole thing by upgrading to the professional version and using ADO to write straight to a database. Here’s what I have tried:
1) Install v2.5 on the new machine and import the scripts using the XML feature.
2) Uninstall the virus checker software and run the Screen Scraper. Originally, MacAfee was stopping my script and asking if it was ok to proceed.
3) Instead of using the XML import feature, I tried copying the ‘db’ folder from the old version to the new version.
4) Once Screen Scraper fails, I would immediately rerun it to see if it failed at the same spot in an effort to see if certain code or data was causing an issue. Upon doing so, the script would fail at a different point in the process.
5) Run Screen Scraper alone without any other programs running.
6) Add Screen Scraper to the exceptions list in Windows Firewall
7) Run different scripts that pull data from entirely different websites to see if one of the scripts was bad.
8 ) I downloaded and installed the latest Java executable from Sun.
None of these approaches uncovered the issue and I’m running out of ideas to troubleshoot. The only other two things I can think of to try is install Screen Scraper on a 3rd machine and see if it has the same issues as the new one or uninstall and try the 30-day trial of the professional version to see if that is somehow different.
I have been suspecting that the issue is with the Java executable file and vaguely remember upgrading the version on my original machine (with SS working). I did get a clue this morning when I experienced another crash and opened the log files that are sent to Microsoft as part of the “Send Error Report†process. Here’s what they said:
Error log file in the Microsoft temp folder:
Server=watson.microsoft.com
UI LCID=1033
Flags=1672016
Brand=WINDOWS
TitleName=javaw.exe
DigPidRegPath=HKLM\Software\Microsoft\Windows NT\CurrentVersion\DigitalProductId
ErrorText=The program is not responding.
HeaderText=You chose to end the nonresponsive program, javaw.exe.
EventLogSource=Application Hang
Stage1URL=
Stage1URL=/StageOne/javaw_exe/0_0_0_0/hungapp/0_0_0_0/00000000.htm
Stage2URL=
Stage2URL=/dw/stagetwo.asp?szAppName=javaw.exe&szAppVer=0.0.0.0&szModName=hungapp&szModVer=0.0.0.0&offset=00000000
DataFiles=C:\DOCUME~1\JEFFFO~1\LOCALS~1\Temp\WER6167.dir00\javaw.exe.mdmp|C:\DOCUME~1\JEFFFO~1\LOCALS~1\Temp\WER6167.dir00\appcompat.txt
Heap=C:\DOCUME~1\JEFFFO~1\LOCALS~1\Temp\WER6167.dir00\javaw.exe.hdmp
ErrorSubPath=javaw.exe\0.0.0.0\hungapp\0.0.0.0\00000000
DirectoryDelete=C:\DOCUME~1\JEFFFO~1\LOCALS~1\Temp\WER6167.dir00
THESE ARE THE TWO FILES SENT TO MS:
C:\DOCUME~1\JEFFFO~1\LOCALS~1\Temp\WER6167.dir00\appcompat.txt
This is the “appcompat.txt†file:
Screen Scraper Hangs or Suddenly Closes
Hi,
Thanks for the posting. First, regarding transferring your scraping sessions from one instance of screen-scraper to the other, rather than copying your database, I would instead recommend simply exporting them from one and importing them into the other. Instructions on doing that can be found here http//www.screen-scraper.com/support/docs/importing_and_exporting_objects.php .
As far as the OutOfMemoryError goes, hopefully you wanted a new computer regardless, but it could be that it wasn't necessary ) Please take a look at this FAQ for reducing screen-scraper's memory requirements http//www.screen-scraper.com/support/faq/faq.php#Optimizing .
That's fine that you downloaded Sun's JVM, but screen-scraper actually uses its own (found in the "jre" folder).
At this point, assuming you still have your exported XML files, I would recommend doing a clean un-install of screen-scraper on your new computer (including deleting the "screen-scraper basic edition" folder, if it doesn't get deleted in the un-install). After that, re-install screen-scraper version 2.6, import your scraping sessions, then try running them.
If that still fails, would you mind posting the contents of your "error.log" file as well? It's located in screen-scraper's "log" folder.
Kind regards,
Todd Wilson