Checking for lazy scrape completion
Hi,
I'm new to screen-scraper and using lazyscrape from java to check pricing information and write it back into a database on demand. One page can involve up to 10 remote sessions (one session for each distributor), most of which finish in less than a second.
However, for larger shopping carts, I need to check whether each session has finished prior to allowing a user to access the price comparison page in case the scraper hasn't completed. Ideally, if all sessions have finished, i would display the page, otherwise I would display a progress bar based on how many had completed.
Any help would be greatly appreciated! Also, is the lazyscrape queuing based on jms? Thanks!
Regards,
John
Checking for lazy scrape completion
Hi John,
Thanks for the posting. If you're doing a lazy scrape the scraping process will occur in its own tread, which cuts off communication from your application. That is, once you invoke a lazy scrape there's no way for you to check on it to see when it finishes.
As an alternative, you might consider not doing lazy scrapes, and instead creating threads in your own application. You can then do your own checking and handling of the scraping processes, which will run in parallel.
Kind regards,
Todd Wilson