Missing Log Files

An important note: Move the logfile copies to a different directory, right?

With our latest debugging/ops sessions one of our guys noticed that clearing the official "log" directory made things work better. Our assumption is that screen-scraper watches the directory and ALL logfiles in it, not just the ones screen-scraper knows about. So if you write a script to copy the logfiles for archiving, don't keep them in the original directory. Is this correct?

Hi,

In your properties file there should be a "MaxScrapeableSessionsToLoad" property, which determines the number of log files that will get retained over time. This also determines how many sessions you'll see at any given time under the "Run/Running" tab of the web interface. When this threshold is reached, any old records for scraping sessions that have run are removed, as well as their corresponding log files. If you want to keep these files, there are two possible approaches:

- Set this to a high number. I wouldn't recommend this, especially if you may have a lot of them.
- At the end of each scraping session run a script that copies its corresponding log file to an archive folder. This is the approach we typically take.

For the second option you'll want to make use of the session.getLogFileName() method, and you might also find the Apache FileUtils package useful (http://commons.apache.org/io/api-1.4/org/apache/commons/io/FileUtils.html).

Just let us know if we can provide any other details, but hopefully that gets you where you want to go.

Kind regards,

Todd Wilson