Loop through products
Rather than constructing a rather elaborate scraping session that submits a search (with lots of different search terms), works through the results pages and the calls each of the details pages, I was planning to just call the details pages with a call to the sequential product ID.
I realise this is a little lazy but it is bit quicker and will do what I need at the moment.
The problem I'm having is running the loop. I've set up a session variable 'i' which is passed to the query string for the details page - this works well and I get my results. However, when I run the session, calling the script "before the scraping session starts" I start getting results, good, but when I stop the session (I don't want all of the results yet) I get a very large number of log entries "Stopping the scraping session."
The basic loop script is
{
session.setVariable( "i", i );
session.scrapeFile( "mySite - Details" );
}
the "mySite - Details" scrape file calls another script to write the data out to a text file "After each pattern application"
Any suggestions?
Loop through products
I can't think of a way to get around having all of those log entries. They appear because screen-scraper finishes your for loop. If you want to test a few entries without all of the stopping scraping sesion messages you could just temporarily modify your loop parameters.