Redirect is Ruining Me
I am trying to scrape a store locator but after the URL is resolved, I am redirected to a different page that doesn't contain the data that I need. Any insights to why this is happening would be greatly appreciated!
Log output:
Starting scraper.
Running scraping session: Sealy
Processing scripts before scraping session begins.
Processing script: "Sealy Read Terms"
Scraping file: "Sealy Search Results"
Sealy Search Results: Preliminary URL: http://sealy.com/ShopEasily/SearchResults.aspx
Sealy Search Results: Using strict mode.
Sealy Search Results: Resolved URL: http://sealy.com/ShopEasily/SearchResults.aspx?zip=95071&dist=5&brand=All
Sealy Search Results: Sending request.
Sealy Search Results: Redirecting to: http://sealy.com/ShopEasily/shopEasily.aspx
Sealy Search Results: Processing scripts before all pattern applications.
Sealy Search Results: Extracting data for pattern "Store Details"
Sealy Search Results: The pattern did not find any matches.
Sealy Search Results: Processing scripts after all pattern applications.
Sealy Search Results: Warning! No matches were made by any of the extractor patterns associated with this scrapeable file.
Scraping file: "Sealy Search Results"
Sealy Search Results: Preliminary URL: http://sealy.com/ShopEasily/SearchResults.aspx
Sealy Search Results: Using strict mode.
Sealy Search Results: Resolved URL: http://sealy.com/ShopEasily/SearchResults.aspx?zip=95071&dist=5&brand=All
Sealy Search Results: Sending request.
Sealy Search Results: Redirecting to: http://sealy.com/ShopEasily/shopEasily.aspx
Sealy Search Results: Processing scripts before all pattern applications.
Sealy Search Results: Extracting data for pattern "Store Details"
Sealy Search Results: The pattern did not find any matches.
Sealy Search Results: Processing scripts after all pattern applications.
Sealy Search Results: Warning! No matches were made by any of the extractor patterns associated with this scrapeable file.
Scraping file: "Sealy Search Results"
Sealy Search Results: Preliminary URL: http://sealy.com/ShopEasily/SearchResults.aspx
Sealy Search Results: Using strict mode.
Sealy Search Results: Resolved URL: http://sealy.com/ShopEasily/SearchResults.aspx?zip=95071&dist=5&brand=All
Sealy Search Results: Sending request.
Sealy Search Results: Redirecting to: http://sealy.com/ShopEasily/shopEasily.aspx
Sealy Search Results: Processing scripts before all pattern applications.
Sealy Search Results: Extracting data for pattern "Store Details"
Sealy Search Results: The pattern did not find any matches.
Sealy Search Results: Processing scripts after all pattern applications.
Sealy Search Results: Warning! No matches were made by any of the extractor patterns associated with this scrapeable file.
Scraping file: "File from Sealy"
File from Sealy: Preliminary URL: http://sealy.com/ShopEasily/SearchResults.aspx
File from Sealy: Using strict mode.
File from Sealy: Resolved URL: http://sealy.com/ShopEasily/SearchResults.aspx?zip=95071&dist=5&brand=All
File from Sealy: Sending request.
File from Sealy: Redirecting to: http://sealy.com/ShopEasily/shopEasily.aspx
Processing scripts after scraping session has ended.
Scraping session "Sealy" finished.
Redirect is Ruining Me
misterchou,
The solution may take a while to sort through for me, so instead I'm going to give you a rod and reel and ask if you'll take a trip to the lake for a day of fishing.
Have a read through my blog entry [url=http://blog.screen-scraper.com/2008/06/04/scraping-aspnet-sites/]here [/url]and let us know if it helps or if you have any questions.
Thanks,
Scott