beginner question : limit the number of requests
Hi,
is there a way to limit the number of requests screen-scraper does at once?
For example: request 5 pages, stop 10 seconds, request another 5 pages. Or better make 5 and 10 random numbers.
You guessed it: this is in order to mimic normal browsing of a site so you don't get banned and the sites server doesn't get overloaded.
Thanks in advance,
Titzu
You could use....
I'm guessing you found it but once you have pause for the random number then you can call the scraping file through the code after the pause so:
sutil.pause(milli););
session.scrapeFile(
If you want to scrape 5 files and then pause you can also run a counter across all of the scrapes. So each time you scrape a new file then increment the counter and then write the counter value to the session.
int counter = session.getVariable("counter");
counter++;
int counter = session.setVariable("counter", counter);
Once you hit the total number of files then reset the counter back to 0. Shouldn't be too hard to randomise the total number of files you scrape each time between 5 and 10 again reusing the code that you provided.
You'd want to use sutil.pause
You'd want to use sutil.pause for that. You can see in the api here: http://community.screen-scraper.com/API/pause
I also have a script I use to make a random pause.
// Pauses scraping session each time script is run.
// Random interval of 4 to 12 seconds
Random generator = new Random();
seconds = generator.nextInt(8) + 4;
milli = seconds * 1000;
session.log("+++Pausing for " + seconds + " seconds");
sutil.pause(milli);