wrapper logfile: JVM appears hung
Hi,
It seems to me I am not solving problems but moving them to another place in my scrape application. Now I often have a certain error in the wrapper.log file:
JVM appears hung: Timed out waiting for signal from JVM.
JVM did not exit on request, terminated
Screen-scraper is written in Java and so is my code in the scripts, but what did I do wrong to have this problem with the Java Virtual Machine ?
My scrape application is running all day long now more than a week. The ColdFusion code gets website addresses out of a database and starts scraping sessions for everyone of them. The scraping of 1 such site is no problem, but the problem is going on. Frequently an error causes the scrapings to stop until I restart the server. I try to solve all causes, but then new ones start to appear. This is the cause of all my posts on this really very useful forum. :-)
Thanks in advance,
Tamara Vos
wrapper logfile: JVM appears hung
Tamara,
This happens generally when you're maxing out the memory resources on the machine where screen-scraper is running. wrapper.exe sends a signal, multiple times if needed, to the JVM but due to limited resources the JVM is unable to respond in time. This would be evident by OutOfMemoryError entries in your error.log file.
You have a few options to alleviate this problem and they all involve allocating more memory resources to screen-scraper (and subsequently the underlying JVM it's running on).
1. Increase the amount of memory allocated to screen-scraper. This can be done through the settings available within the workbench.
2. Increase the physical memory of the machine(s) screen-scraper is running on (i.e. add more RAM). Then, also increase the memory allocated to screen-scraper.
3. Distribute your scraping sessions over multiple machines.
Given the number of scraping sessions you're running it's understandable that screen-scraper and the JVM are choking at times.
-Scott