Screen Scrapper gets hang when clicked on Proxy Server
Screen Scrapper gets hang when clicked on Proxy Server.
I can see my java.exe usage goes to 50% and hence my screen scraper stops responding.
normally this java.exe take 0% of CPU usage.
This will sometimes happen when screen-scraper is processing large binary files. In most cases you can alleviate this problem by checking "Don't log binary files". However, even when this is checked, screen-scraper may still inadvertently process binary files when a particular request has a content type that indicates that it is something other than a binary file (mis-labelled by the server) or is a content-type not recognized by screen-scraper.
To help us track down cases such as this, would you mind sifting through your proxy logs where this is happening and try to identify which item in the log might be causing this? One thing you can look for is any transaction with a Content-Length of +100000 in the response.
If you're not able to sift through your log because screen-scraper hangs, would you mind either posting the site you are proxying here or email it to me? My email is my first name followed by "w" at our domain.
Web Data Extraction
Proxying large binary files
diptirmaya,
This will sometimes happen when screen-scraper is processing large binary files. In most cases you can alleviate this problem by checking "Don't log binary files". However, even when this is checked, screen-scraper may still inadvertently process binary files when a particular request has a content type that indicates that it is something other than a binary file (mis-labelled by the server) or is a content-type not recognized by screen-scraper.
To help us track down cases such as this, would you mind sifting through your proxy logs where this is happening and try to identify which item in the log might be causing this? One thing you can look for is any transaction with a Content-Length of +100000 in the response.
If you're not able to sift through your log because screen-scraper hangs, would you mind either posting the site you are proxying here or email it to me? My email is my first name followed by "w" at our domain.
Thanks,
Scott