maximum number of scripts allowed on the stack was reached
Hello all,
I'm trying to scrape a site that has around 600 pages of search results with 10 rows of data per page. For each row of data I'm following the URL and grabbing the results.
The problem is that I'm encountering this error:
Scraping Session: ERROR--halting the scraping session because the maximum number of scripts allowed on the stack was reached. Current number on the stack is: 50.
To workaround this, I have created the following script based on the recommendations in the blog post (http://blog.screen-scraper.com/2008/07/07/large-data/) but I'm still getting the errors.
Here is the script I'm using:
pages = Integer.parseInt(session.getVariable("TOTALPAGES"));
page = Integer.parseInt(session.getVariable("PAGE"));
// Clear session variable so it doesn’t linger
//session.setVariable("TOTALPAGES", null);
// Loop through pages
for (i=0; i<=pages; i++)
{
// Since the page list appears twice, use only a number larger than that just used
if (i>page)
{
session.setVariable("PAGE", i);
session.log("+++Scraping page #" + i);
session.scrapeFile( "Search Results" );
}
else
{
session.log("+++Already have page #" + i + " so not scraping");
}
}
The workflow is:
1) "Search Results" scraping session runs which submits the first search GET request
2) Extractor Pattern "Extract Summary" scrapes multiple product IDs and runs a script after each pattern match to scrape the detail page
3) Extractor Pattern "Next Page" grabs the PAGE and TOTALPAGES session variables then runs the script listed above
4) The script checks that PAGE is less than TOTALPAGES, iterates PAGE by one then calls the "Search Results" scraping session (i.e. returns to (1)) to obtain the next set of results
Can anyone point out what I'm doing wrong? I've been hanging my head against a wall with this for days!
Thanks,
Dean
This scrape is being run on
This scrape is being run on an extractor pattern on the search results scrapeable file, right? You need to make sure the script only runs on page 1.
page = Integer.parseInt(session.getVariable("PAGE"));
if (page==1)
{
pages = Integer.parseInt(session.getVariable("TOTALPAGES"));
// Clear session variable so it doesn’t linger
//session.setVariable("TOTALPAGES", null);
// Loop through pages
for (i=2; i<=pages; i++)
{
// Since the page list appears twice, use only a number larger than that just used
if (i>page)
{
session.setVariable("PAGE", i);
session.log("+++Scraping page #" + i);
session.scrapeFile( "Search Results" );
}
else
{
session.log("+++Already have page #" + i + " so not scraping");
}
}
}