Home » screen-scraper FAQ

When using screen-scraper's proxy server my browser hangs or I don't see anything in my proxy session. How do I fix this?

Unfortunately, screen-scraper's proxy server isn't perfect, and, on occasion, you'll encounter sites that it has difficulty with. Frequently the issue can be resolved by using a different web browser, such as Firefox or Opera.

Depending on your operating system, instead of designating "localhost" in your web browser, you may need to enter "127.0.0.1" or the IP address of your computer.

If you normally connect to the Internet through a proxy server (outside of screen-scraper), you'll need to configure screen-scraper to use that proxy server. This can be done in the "Settings" window (click on the wrench icon), under the "External Proxy Server" section.

If changing your browser doesn't help, it's still possible that you can proxy the site enough that you can create scrapeable files from the requests. It simply needs to be done in a more piecemeal fashion. If you need to resort to this, try the following for each page you need to scrape:

  1. Without using the proxy server, in your web browser go to the page containing the link or form that points to the page you want to scrape.
  2. Start up screen-scraper's proxy server and configure your browser to use the proxy.
  3. Click the link or submit the form that links to the page you want to scrape.
  4. screen-scraper's proxy may hang, but if you click on the HTTP transaction under the "Progress" tab in screen-scraper you may see at least the request portion of the transaction. If this is the case then you can stop the proxy and create a scrapeable file from the HTTP transaction that was recorded.

Note also that you typically only need to proxy forms that use POST requests. Scrapeable files corresponding to normal links and forms that use the GET method can be created by simply copying the URL from your web browser.

3rd Party Options

Alternatively, if the screen-scraper proxy freezes entirely and does not record any of the transaction you can access the HTTP header information within your browser by utilizing one of the following.

scraper on 07/03/2008 at 12:32 pm