An error occurred while preparing to issue the HTTP request: null
Guess I'm in for the weekly Null award at Screen-scraper...
I'm working on a new session and I ran into a suspicions error that I just can't seem to get around. I'm sure that the solution is right there in my face but I cannot see it.
Following normal procedure I have a Search result page where a Datarecord extractor pattern grabs HTML pieces that include the links to the detail pages. A sub-extractor pattern extracts the URLs from the datarecord and save them into a session variable "URL". Each search result page contains six links and after "each application pattern" I call a script that completes the URL to the details page and scrapes the details page. The first application works normally like:
----
URL=correctURLending
PROPERTY_TYPE=Apartment
Storing this value in a session variable.
Search results: Processing scripts after a pattern application.
Processing script: "Coe - fix details URL"
Processing script: "Coe - scrape details page"
Scraping file: "Details"
Details: Processing scripts before a file is scraped.
Details: Preliminary URL: ~#URL#~
Details: Using strict mode.
Details: Resolved URL: http://www.correcturl.com
Details: Sending request.
Details: Extracting data for pattern "Broker"
---
The mysterious error appears on the second link and all else following that one:
--
URL=correctURLending
PROPERTY_TYPE=Apartment
Storing this value in a session variable.
Search results: Processing scripts after a pattern application.
Processing script: "Co - fix details URL"
Processing script: "Co - scrape details page"
Scraping file: "Details"
Details: Processing scripts before a file is scraped.
Details: Preliminary URL: ~#URL#~
Details: Using strict mode.
Details: An error occurred while preparing to issue the HTTP request: null
---
And then it stops.
Any clues anyone?
Best,
Johan
Similar problem
Hi
I just had a similar problem, with some characters, that SS does not understand. I solved it by using the Java class URI and called the method toACSII(), which returns a international valid URL(URI).
Maybe you could implement something standard that will do this for future use?
Hans
Session variables in the URL
Session variables in the URL field can be tricky. Assuming that you're absolutely sure that the variable isn't 'null' (ensuring that it's got a checkmark on the 'Save to session variable?' box in the extractor pattern... use a "session.breakpoint();" in a script, just before this second scrapableFile happens), then make sure that the variable doesn't contain any special characters that normally are not allowed in a URL. This includes "<" or ">", or a slew of others.
If you can find the actual value of ~#URL#~, could you post it here?
Tim
The error turned out to be a
The error turned out to be a special char problem which was a bit hard to detect given the error message. Solved now via utf-8 parsing and a few replacements.
/Johan