HTTP Error - Invalid query

Hi,

Aside from a lack of better error handling (which I've just added in), I believe this is actually functioning as designed. I'll explain a bit.

Given a URL with embedded session variables, such as

http//www.foo.com/default.asp?bar=~#BAR#~

screen-scraper will resolve those session variables, but will not attempt to URL-encode them. In the case above, if the session variable "BAR" held the value "this and that", screen-scraper would resolve the URL to

http//www.foo.com/default.asp?bar=this and that

This is an invalid URL since space characters are disallowed. It would need to be encoded instead like so

http//www.foo.com/default.asp?bar=this+and+that

Now, you might ask, why screen-scraper doesn't just encode the values automatically. In answer, consider the following URL

http//www.~#MY_DOMAIN#~.com/

Or even

~#URL#~

In both of these cases, if screen-scraper were to URL encode the values of the embedded session variables it would potentially result in an invalid URL.

Getting back to the original question, I think there are a few good options to resolve the problem

1. Remove the GET parameters from the URL, and put them instead under the "Parameters" tab (as proposed by fnirt). Here screen-scraper will safely URL-encode any of the values automatically.
2. Leave the URL as is, but encode any values in a script "Before the file is scraped" (as proposed by Scott). That way the session variables will already be encoded when the URL gets generated.

Hopefully this clarifies things. Just let us know if we can provide further detail.

Kind regards,

Todd

jclerie,

The error you're experiencing is related to a previous issue with how screen-scraper handled certain characters in a URI (in this case it's probably the pipe character between "DPPO" & "Dental"). This issue is resolved in a later release of screen-scraper. I was able to scrape the URL in question using 3.0.66a but received the error message when trying it in the 3.0 basic edition.

Only the professional edition has the option of upgrading to alpha versions. If you are running the professional edition you can upgrade by:

1. Within the workbench, click on the wrench icon to open the settings window.
2. Uncheck the box next to where it says "Allow upgrading to unstable versions".
3. Close the settings window.
4. Click on the Options menu and select "Check for updates".
5. Follow the instructions presented to complete the upgrade.

Now, you should be able to scrape the URL without the previous error.

If you're not running the professional edition then you'll want to manually URL encode the URL in a script before using it in a scrapeable file. You have two ways you can do this.

1. If I'm right that the character that's causing the error is the pipe "|" then do a replace all on that one character like this.

url = url.replaceAll( "\\|", "%7C" );

2. If you want to be more thorough and take care of any bad characters you come across apply the Java method [url=http://java.sun.com/j2se/1.4.2/docs/api/java/net/URLEncoder.html]URLEncoder[/url] to the URL and it will handle all of the characters that would cause that error.

Please let us know how this goes.

Thanks,
Scott

jclerie,

I don't have any suggestions off hand. There are a few too many variables to consider. Could you provide the URL in question or perhaps more information from the proxy log?

What version of screen-scraper are you using?

Thanks,
Scott