Invoking screen-scraper
Thank you for this opportunity. During the last three
weeks, I have used screen-scraper to continuosly gather
price quotes for a particular stock that I may invest in.
I am using a VB.Net executable called from Windows Task
Scheduler at 6:00 a.m. (VB.Net Source Code attached.)
The VB executable runs the inializing scrpt in my
screen-scraper's 'QuotesFromScottrade' Scraping Session.
(Javascript attached.) Shortly after screen-scraper
starts, the VB executable quits. Screen-scraper quits
at 2 p.m.
I have been running this daily for most days starting on
Friday, February 5, 2010. Here's my result summaries:
Scraped Good Usable Quotes No Quotes Received
-------- --------------------- ---------------------
02-05-10 2 hrs 7 mins 36 secs 5 hrs 52 mins 25 secs
02-06-10 48 mins 47 secs 7 hrs 11 mins 14 secs
02-07-10 1 hrs 14 mins 20 secs 6 hrs 45 mins 41 secs
02-08-10 1 hrs 11 mins 21 secs 6 hrs 48 mins 40 secs
02-09-10 5 hrs 6 mins 27 secs 2 hrs 53 mins 34 secs
02-10-10 4 hrs 55 mins 55 secs 3 hrs 4 mins 6 secs
02-11-10 5 hrs 9 mins 29 secs 2 hrs 50 mins 32 secs
02-14-10 1 hrs 12 mins 23 secs 6 hrs 47 mins 38 secs
02-15-10 6 hrs 20 mins 15 secs 1 hrs 39 mins 46 secs
02-16-10 5 hrs 19 mins 47 secs 2 hrs 40 mins 14 secs
02-17-10 3 hrs 47 mins 33 secs 4 hrs 12 mins 28 secs
02-18-10 6 hrs 13 mins 39 secs 1 hrs 46 mins 22 secs
02-19-10 2 hrs 44 mins 7 secs 5 hrs 15 mins 54 secs
02-20-10 6 hrs 42 mins 13 secs 1 hrs 17 mins 48 secs
02-21-10 6 hrs 53 mins 37 secs 1 hrs 6 mins 24 secs
02-22-10 4 hrs 38 mins 37 secs 3 hrs 21 mins 24 secs
02-25-10 6 hrs 23 mins 23 secs 1 hrs 36 mins 38 secs
02-26-10 4 hrs 28 mins 7 secs 3 hrs 31 mins 54 secs
02-27-10 5 hrs 27 mins 49 secs 2 hrs 32 mins 12 secs
02-28-10 1 hrs 48 mins 58 secs 6 hrs 11 mins 3 secs
03-01-10 43 mins 19 secs 7 hrs 16 mins 42 secs
Please note the large amount of time where no quotes were
received. How can I eliminate this? Thank you, in advance,
for considering this!
Marty
| Attachment | Size |
|---|---|
| VB.Net Source Code.txt | 238 bytes |
| JavaScript.txt | 390 bytes |
Update
Scott,
After I entered this question yesterday, and before I went to bed, I thought about something that happended earlier. I had tried invoking screen-scraper through a batch file.
So not expecting a response in enough time to implement any suggestions before today's scraping start, I entered a batch file, GetQuotes.bat, in my screen-scraper basic edition folder. It follows:
CHDIR "C:\Downloads\ScreenScraper\screen-scraper basic edition"
screen-scraper -s QuotesFromScottrade
Next, I changed my Scheduled Task from running the VB.Net executable to running this batch file.
It started just fine at 6:00 a.m., and ended right on time at 2:00 p.m. During the entire day, the DOS window remained open. Also, in the Processes tab of Windows Task Manager, it showed two entrees: 1) cmd.exe (Windows Command Processor), the good old DOS window, and 2) our friend screen-scraper. At 2:00 p.m. the DOS window closed, and both these Processes dropped from the Task Manager.
I immediately ran my Summary repport. Here's the good part:
Good Usable Quotes: 7 hrs 20 mins 51 secs
No Quotes Received: 39 mins 10 secs
----------------------
Total: 8 hrs 0 mins 1 secs
I did scan through the data, and where there were zero records in a given second, there didn't appear to be any seconds next to each other with a zero. Also, I need to eliminate repeating quotes where the price doesn't change at the very beginning, and at the very end of the day.
Will work on adding that to my report. But first I need to get the day's Opening, High, Low, and Closing amounts. So will add that scrape, and write to disk file next. Will start this after dinner.
Again, thanks a million for all your help. Things are looking good right now!
Marty
Marty, You'll want to capture
Marty,
You'll want to capture screen-scraper's log in order to troubleshoot this. I don't know what the syntax would be when executing a program from the shell like you are but here is what it would look like as a batch file.
jre\bin\java.exe -Xmx512M -jar screen-scraper.jar -s QuotesFromScottrade > log\mylog.log
Now, keep in mind, the example above would write all logs to a single file. If you were to run this for more than a day, your log file may end up being very large. So, I recommend copying the log each day & deleting the old one so screen-scraper writes to a new log each time it runs.
I would recommend creating a script in your scraping session that simply writes the current time out to the log. Then call that script at the scraping session level both before and after the scrape runs.
Do this even though your other script is writing date information to ensure you're capturing start and stop times as your scraping session begins and end.
After you have collected your logs, you'll want to review them and look for any errors.
-Scott