screen-scraper support for licensed users

Questions and answers regarding the use of screen-scraper. Only licensed Professional and Enterprise Edition users can post; anyone can read. Licensed users please contact support with your registered email address for access. This forum is monitored closely by screen-scraper staff. Posts are generally responded to in one business day.

Logging

According to this page, http://community.screen-scraper.com/documentation/api/session/logging/log each logging method is associated with where screen scraper is being invoked from. Is there a setting or something in a config file to change this default behavior? For example I want to log to the Log tab and log to a file from the workbench.

Setting for VPN in SS

Hello,

I'm trying to figure out if there is a way set individual scrapes to connect via a VPN? Seems like the proxy settings in the Advanced tab will require a port to work, something vpn providers does not supply?

Update available message, but cannot update.

Hi. I have SS Professional version 6.0.23a on mac osx Mountain Lion. Since last week whenever I start the workbench I get a message like this:

An update is available, but can only be installed when there are no command line instances running. Please stop any running command line instances, then check again for updates.

However there is not any command line instance running.
Is there really an update? How can I solve this?

thanks,
Boga.

Proxy connections are getting blocked from web sites

I've noticed that we are getting blocked on several sites consistently since upgrading to the past few versions of screen scraper. Zillow, Trulia and a few others seem to be able to detect the IP address even though we have proxies in use.

Search results: Resolved URL: http://www.trulia.com/validate.php?tst=h&display=for%20sale&search=01420&type=&min_price=min&max_price=max&min_num_beds=0&min_num_baths=0&min_size=0
Search results: The request timed out while requesting the URL: http://www.trulia.com/validate.php

Using proxy server: null:-1 An internal error occurred

This may be related to the other NULL pointer issue I logged earlier.
Running the proxy pool scripts. Cleared the proxy pool files with
echo > filename.txt
Did this for each of the files so I could verify they were getting populated with new values.
Ran the proxy session and got the "Using proxy server: null:-1 An internal error occurred" error right away:

Starting scraper.
Running scraping session: R
Processing scripts before scraping session begins.
Processing script: "reset_proxy_file"
2013-04-11_22:45:28 Setting the APPEND_TO_FILE session variable to false

Unexpected end of ZLIB input stream

I've noticed several of our scraper session are throwing this error.
Any idea as to what is going on?
Any way to correct for this and keep the session going?

An EOFException occurred when decompressing the server response: Unexpected end of ZLIB input stream
java.io.EOFException: Unexpected end of ZLIB input stream
at java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:223)
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:141)
at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:90)
at java.io.FilterInputStream.read(FilterInputStream.java:90)

Keep getting a Null Pointer Exception error. error message was: NullPointerException (line 60): proxyServerPool .filter

Screen Scraper version 6.0.25a
Error keep recurring: NullPointerException (line 60): proxyServerPool .filter
New bug in the 6.0.25a release?

Script output:

Validated proxy 53 of 55. This proxy server will be removed: 93.126.43.244:3128 (the following I/O error was encountered: The host did not accept the connection within timeout of 9000 ms)
Validated proxy 54 of 55. This proxy server will be removed: 31.3.230.18:3128 (the following I/O error was encountered: The host did not accept the connection within timeout of 9000 ms)

Is there a method to capture log events with an external jar?

I have a jar in the ext directory and I would like to have it monitor events from the log. For example if "session.logError" is called I'd like to have the external library handle a specific task, maybe send an email.

I could go into the screen-scraper session and add some code which triggers the external library, however I have several hundred sessions that I would have to update and it would be much easier to hook into the log events if that's possible.

Thanks, Jeremy

LogError message gets formatted into html

String msg = "<h2>Options:</h2><div><ul><li>item1</li><li>item2</li></ul></div>";
when calling session.logError(msg);, the above message gets formatted in the workbench log tab.
It used to display the log as a regular string:
<h2>Options:</h2><div><ul><li>item1</li><li>item2</li></ul></div>

But now it displays it like this:

Options:

  • item1
  • item2

This is only for session.logError, the other log levels display correctly. This started happening on version 6.0.24a. Is this a bug?

Error when running thru RemoteScrapingSession

When I run my scrape thru the screen-scraper development environment it works just fine,
But when I run thru my vb.net program I get the following error:
An input/output error occurred while connecting to 'http://portal.qlmortgageservices.com/guru/'. The message was Software caused connection abort: recv failed.

Thanks
Bart