Screen Scraper not processing HTTP Response

Hi,
Something very strange is happening. I have a scrapeable file that looks like it's ignoring the response from the web server. I'm trying to scrape the following url; http://www.pittsburghlegaljournal.org/subscribe/pn_sheriffsale.php.

The log file shows the following;

Starting scraper.
Running scraping session: Allegheny PA Sheriff Sales
Processing scripts before scraping session begins.
Scraping file: "Allegheny PA Sheriff Sale Scraper"
Allegheny PA Sheriff Sale Scraper: Preliminary URL: http://www.pittsburghlegaljournal.org/subscribe/pn_sheriffsale.php
Allegheny PA Sheriff Sale Scraper: Using strict mode.
Allegheny PA Sheriff Sale Scraper: Resolved URL: http://www.pittsburghlegaljournal.org/subscribe/pn_sheriffsale.php
Allegheny PA Sheriff Sale Scraper: Sending request.

And it just sits there. The workbench doesn't hang, but it also doesn't timeout.

One test I've done is run the scraper while running Ethereal to catch the traffic coming to and leaving my machine. The request does get sent to the site and the web server returns the page but screen-scraper doesn't seem to pick it up. I've also tried running screen-scraper as a server and then calling the scrapping session from C# with no success.

I've also created another scraping session that "walks" the links to get to that page. So it starts at the home page, "clicks" the link for the section that my page is in, then "clicks" the link to get to that page. The first two pages (home and the first click) work fine but the same strangeness happens when it "clicks" the link for http://www.pittsburghlegaljournal.org/subscribe/pn_sheriffsale.php

Any help would be great.

I spent a good little bit of

I spent a good little bit of time with one of the other developers, and found out that it's a problem with the HTML tidier.

It seems to be getting stuck in an infinite loop while trying to tidy. The tidier is not our own, and is due to be replaced by a better tidier, one of these days.

It seems that if you turn off HTML tidying for that scrapeableFile, it should work without further troubles.

Thanks

That fixed the issue. Seems to be working now. Thanks for the quick response!