RunnableScrapingSession
Is there a way to know when a RunnableScrapingSession is finished? I try searching everywhere and couldn't find anything on it. The reason for this is I want to be able to spawn new RunnableScrapingSession objects once x amount has been completed. Would there be another way of handling this? I want to run around 50 simultaneous RunnableScrapingSession's but not sure if screen scraper can handle it.
Thanks
Alright thanks, I'll try running all of them and see how it goes. The main session spawning the RunnableScrapingSession should be invoke via cmd right?
I would suggest a command
I would suggest a command line or, if you have a licensed edition, server mode.
I have some servers that run
I have some servers that run 100 scrapes simultaneously, but that's going to be affected by the complexity of your scrapes.
If you are setting doLazyScrape to TRUE, you can't track progress of the scrapes, but otherwise you should be able to. If you're running several in tandem, though, you are likely doing the lazy scrapes. In that case, I would just have a script at the end of the scrape that somehow informs me its done. Such a script would just write to a CSV when they are done or something like that.
Error in Log
Screen Scraper enterprise x64
I managed to run all 50 scrapes concurrently but had some problems.
1)
Driver is the session that spawns the runnable.
Runnable is the RunnableSession
This was in the error log:
Resetting the following scraping session: Driver
java.lang.RuntimeException
at com.screenscraper.scraper.ScrapingSession.reset(ScrapingSession.java:4938)
at com.screenscraper.scraper.ScrapingSessionManager.scrapingSessionFinished(ScrapingSessionManager.java:1134)
at com.screenscraper.scraper.Scraper.scrape(Scraper.java:463)
at com.screenscraper.scraper.Scraper.run(Scraper.java:126)
Resetting the following scraping session: Runnable
java.lang.RuntimeException
at com.screenscraper.scraper.ScrapingSession.reset(ScrapingSession.java:4938)
at com.screenscraper.scraper.ScrapingSessionManager.scrapingSessionFinished(ScrapingSessionManager.java:1134)
at com.screenscraper.scraper.Scraper.scrape(Scraper.java:463)
at com.screenscraper.scraper.Scraper.run(Scraper.java:126)
Resetting the following scraping session: Runnable
it repeats here
I ran the scrapes from the web and it completed with no errors in the column "Error?".
2) Memory usage peaks at 88%. Is this normal?
I set 1024mb for the memory in settings.
I have 4gb of total ram
3)
I also have this in the stdout log when I start the server:
Initializing screen-scraper server. Please wait...
Ready.
The server is in non-interactive mode. Run the "stop server" script to stop it.
Warning: You're not using a Java Development Kit (SDK) environment so JSP pages can not be compiled.
Starting service LiteWebServer Base Module (lws-base/3.0.3) (with LiteWebServer JSP Module (lws-jsp/1.1.2))