SS 3.0: Tidy error log file is being used by another process

Todd:

GOT IT -- four perfect runs in a row.

I change some of the FileWriter code, principally leaving the file open and flushing, closing at end.

I'll update with details if I can.

Thanks for your help -- this has been a terribly vexing problem for some time, and now appears to be solved.

-- Roy

Hi Todd:

Sorry this is so confusing. Let me clarify things.

1. I'm running on Windows XP SP2, on top of a dual CPU platform (Tyan MPX with 2x AMD MP 2000+ CPUs).

2. I'm generally invoking the program from the command line, with driver scripts inside SS running the session and reading and writing files.

3. SS is sometimes open, sometimes not, while the program is running.

4. The program processes 12,334 input files, which are html detail pages I've downloaded previously (and separately) from Sotheby's web site in order to analyze them offline. The program writes data to a tab-delimited output file, and (now) also emit error messages at the occur to an error file. I capture the session output by redirecting the output at the DOS level to a file tt.txt. The input files are about 40KB each, the data output file runs 12,334 lines of output and is abou 11.5MB in size, and the session log captured to tt.txt is about 69MB and runs about 1.1 million lines of output. The run typically takes 33 minutes (2,000 seconds).

5. Sometimes programs are running in the background, sometimes not. I've tested both ways with no obvious differences.

6. The first source of error cropped up when I found that output records were being skipped (output not written). This was traced to some sort of locking problem with tidy.log, as noted earlier.

7. I also noted at the time that there were some missing records that weren't showing up in erro.log. That is, I would find that my output file would be, say, five records short, but there would only be three error records relating to tidy.log being busy in your error.log file. (Also noted earlier).

8. You kindly modified SS to stop using tidy.log late last week. I haven't gotten any tidy.log errors since then. I haven't had any errors in your ..\log\error.log file at all in the past couple of days, though I have made complete runs of the program at least five times and many other runs as well.

9. Notwithstanding this improvement (not having tidy.log locking), I still have missing records. There is no error in ..\log\error.log, to be sure, but I've found them by counting the lines of output in my output file. There is no pattern to the missing output, running two to four records at most. A couple of times I've gotten perfect output (no missing records), and have saved the output.

10. The record locking/output file busy problem hasn't gone away, as now it appears to be in one of my files. Here is the error message from the session data, where my program tries to open my own error logging file, but fails:

The statement it's trying to execute and failing at is this one, I think:

zzerrout = new FileWriter ( "zzerrout.txt", true ); // append mode

I added a statement to null it out before using it, but still didn't affect the missing output lines:

Filewriter zzerrout = null;

11. This has been a long-standing problem, going back at least two or three years. At that time I was scraping a site with 5000+ records per session, and we always had one or two MIA files. I had to move on before this problem was resolved.

But now with a completely different scraping program I'm still getting MIA records.

12. My workarond for this situation is 1) abandon tidy and rewrite the pattens to operate on the native html, and 2) make multiple runs to patch together a complete output file.

* * *

Hope these comments are helpful to you, Todd.

-- Roy

Todd:

Yes, I am running things from the command line, and piping the output to a text file. I have to to search through the output for warnings or errors, since there is no way in the error.log to identify the file being scraped. The piped session log is over a million lines of output. The warnings and errors can be found, obviously, but error.log itself does not identify the file where the error occurred.

-- Roy

Hi, Todd:

Then if it continues to put out messages to error.log, then it would be helpful if it were to identify which file it is scraping. Otherwise, you have to capture the session log and read through that to find the error -- which sort of makes the error.log pointless.

-- Roy

Todd:

Got it -- thanks.

Quick test -- doesn't create tidy.log, but does create error.log Error.log is empty, but there are still two missing output records in a reduced 68-file set I've been trying to reduce to zero. Earlier I noted a two-record difference between what was missing and the lines in error.log, so it's evidently still here. I'll get back to you about this discrepancy.

-- Roy

Todd:

Where do I get the pre-release version for upgrade? "No updates" from my side, and nothing in your blog about it.

On the tidy.log situation, it's never identified any of the files it's tagged by name, so it's never been possible to go back and match error with source file. So on that basis alone it's been pretty lame. And it's a version from 2000, so perhaps it's time to upgrade that module.

FWIW, I'm running these data extractions on a dual CPU system (Tyan S2466N-4M, AMD MP2000+ CPUs). This may have some effect with program execution, even if they're supposedly thread-safe.

-- Roy

Todd:

Just ran test on 196 records using:

session.pause(1000);

Still got one missing record, one error in error.log for tidy.log.

Later, ran on full file set, now ran almost four hours, got 68 missing records:

So the pause isn't going to be the answer, I'm afraid.

-- Roy

Todd:

Isn't there some way to test if the tidy.log is free before it scrapes? I will test your suggestion. although you realize of course that adding a second to each scrape adds 12,334/3600 = 3.4 hours to the run, rather than the 36 minutes it is taking at the moment.

-- Roy