read an Extractor pattern , check for a condition , then pass control to a second Extractor Pattern ?
I am using the screen-scaper Basic Edition V4.5
My project uses a site that requests a stock ticker symbol that brings up a screen having a list of dividend info then a list of earnings info on one srapeble file. Both lists are sorted by date. I have 2 extractor patterns ( 1 for div, 1for earnings). Each patern has a script that writes the data to a .cvs file "After each pattern application" if it meets my selection criteria. When the entire file is scraped I read a new ticker from a .cvs file.
My problem is that it can take a very long time to run, depending on how many tickers i have in my .cvs input file.
I need to only scrape data that has been newly added as exampled here :
http://community.screen-scraper.com/script_repository/Check_Post_Freshness
but because I don't have the professional edition, I can not use session.execute(my script) method. Is there any work around I could use to read a pattern , check for a condition , then pass control to the second Extractor Pattern ?
I guess I have a long processing time because if there are 50 records in each list, it seems I have to scrape all 100 records for each ticker ( multiplied by the number of tickers in my input file) when all I really need are maybe the 3 or 4 "fresh" records.
I have programming experience but am new to Java so using screen-scraper has been a bit of a long learning curve, but I am really enamored with the concept of web data extraction and think screen-scraper is a very cool tool. I thank you for making the Basic Edition available. I just can't see myself upgrading to the professional version for the limited number of times I will likely have use for it.
Any feedback greatly appreciated.