multiple instances of a refreshing URL
hi
i am a newbie at scripts. i want to scrape a site that refreshes itself every 30 seconds. more specifically there is one variable that changes every 30 seconds and i want to perform a data analysis on the values of that variable. what i have done so far is setup the proxy server and downloaded the data for the day but i dont know how to scrape the multiple pages by just coding once. the number of instances of the pages is very high to the order of 500, so i cant do the pages individually.
plzz help
you will want to call the
you will want to call the scrapeable file from a script in a loop. For example to scrape the same page 100 times every 30 seconds you could do the following (Java)
for(int i = 0; i < 100; i++)
{
session.scrapeFile("name of scrapeable file");
session.pause(30000);
}
Check out the screen-scraper API for all of the screen-scraper specific methods.