Automatically repeating scrape session
Hi,
I have a scraping session set-up which looks through a list of 200 numbers in a csv file, scrapes a URL appended with these numbers and catches three data fields. These data fields can change very frequently so I'd like to be able to start the scraping session again after it reaches the last number in the list. The only way I've found to do it is to copy the range of numbers into the csv file over and over again, however, if possible I'd like it to do it using a script which runs after the end of the scrpaing session to restart it, but I'm not familiar with programming outside of simple VBA stuff. Any suggestions?
Thanks
jt
Well, you *could* have a
Well, you *could* have a script run at the end of the scraping session for the purpose of relaunching the scrape, but then you could get into memory problems.
(The quick rundown on the Why is because if you run a scraping session at the end of scraping session, then the first session is not leaving memory completely until after the second session finishes. But if the second session calls a third, and the third a fourth, then you're going to wind up with a lot of memory useage. Then imagine if you had the thing on auto repeat, calling itself over and over again!)
The best way to repeat a session like that is to make a "control" script, which will run as the very first script on the session, before any of your scrapeableFiles. I've written a script that (hopefully) has no bugs in it: http://community.screen-scraper.com/node/1126 . Just run it before everything else, and make sure you give it a max number of times to run, and/or a quitting time.
Careful with it, since if you run this in your work bench with no maximum number of runs, it'll just keep going and going and it'll never actually stop when you tell it to abort, until the quitting time variable finally makes it stop!
Let me know how it goes, or if you need any help.
Tim
Bug found in script, and suggestions...
Hello, I just posted a comment on the script you refer to, as I did already find a minor bug and a usage issue with it, in that it could not be run as a control script, as is, as it needed to use the runnableScrapingSession object in order to launch the scraping session.
--J
True, but if you were to
True, but if you were to launch a scraping session whose first action is to execute this script, then your troubles are taken care of. You just have to set up the scrapeableFiles to not run in sequence, or else after the script is finished, you'll wind up having your scrapeableFiles run again after you expected them to stop.