.aspx?itemID=
I am grateful for any and all help I can get!
I can't seem to figure out how to set the scraper to automatically crawl through pages.
Page structure: *some*site . com/pagetitle.aspx?itemID=(# goes here)
In this specific case - there are several thousand pages - how can I program to scrape the range?
-mike
page iteration with aspx
Hi Mike,
So most aspx pages are a pain. You'll need to proxy this really well to see what parameters are set (post,get) and then you'll need to very carefully build the navigation. If _VIEWSTATES and other similar variables are used you'll need to make sure to use the viewstate from the page you are currently on as a parameter for the page you are requesting. Occasionally you'll get lucky and a site won't use viewstates on their results pages, but instead only on the product/details pages. If this is the case the job becomes a little easier.
If they show 20 products a page and there are 1000 pages you'll need to find out the pattern of what happens in a next page request. Duplicate this in screen-scraper exactly and you should be able to get from page to page. If you can see how many total pages there are then you could build a loop to take you from page 1 to 1000. Or you can search our suggestions found here:
http://community.screen-scraper.com/script_repository
They are labled as "next page..."
best of luck. keep us posted if you run into more problems.
scraper