Scraping NEXT pages
I am trying to scrape the "Next Page" results from http://www.walmart.com/search/search-ng.do?ic=48_0&search_constraint=394.... I have tried numerous ways of getting this done but nothing in the tutorials seem to work. The way they have their pages setup is in order to get to the next page you have to select a number (1-?). There are 5 numbers on every page.
The term "ic"=start of number of items shown. Ex. 48_0 = 48 items shown, 0=starting point. I have searched ALL over for a next page link, but there is none! Someone PLEASE help!
scraping next pages
What i usually do for page iteration is use the screen-scraper proxy to grab the first page then the next page. Then look at the parameters and figure out which ones are variables that control which page you are on. I Then I use a script to manually set those variables and scrape the search results page. The next thing to do is figure out a way to determine how many pages total there are, either by it saying something like page 2 of 50, or 523 results returned, and you know there are say 10 per page. Then you can write a loop that keeps scraping the page and incrementing the page value. For example lets say my page variable for the GET parameter is saved in PAGE, and i have an extractor pattern in my scrapeable file to grab the total page number and save it in a variable MAX_PAGE, i could then use this script
session.setVariable("MAX_PAGE","0");
for (int page = 1; page < Integer.parseInt(session.getVariable("MAX_PAGE")); page++)
{
session.setVariable("PAGE", page);
session.scrapeFile("Search results");
}
I manually set the MAX_PAGE at the beginning but note that this will be overwritten the first time Search results is scraped with my extractor pattern. This also has the advantage of being much more memory friendly than scraping a next page link as that results with more and more scrapeable files on the stack.