Associating Log Files with scrapes
Do the log file name mean something? Are they a version of a timestamp? I'm trying to associate a log file with a particular scrape that happened. I'll be passing some sort of identifier to the scrapes from now on, so I can search on that parameter in the logs, but for past logs and general knowledge -- what do the log file names (after the scrape name) mean?
Associating Log Files with scrapes
That ROCKS! Thanks a lot! Again with the loving of the Screen-Scraper!
R
Associating Log Files with scrapes
Hi,
We've just implemented this feature, which will be available in the next alpha version of screen-scraper. You'll be able to get access to the log file name by referencing the session variable "SS_LOG_FILE_NAME". If you'd like to try it out before the next alpha version is available, feel free to drop me an email. My address is my first name at screen-scraper.com.
Kind regards,
Todd
Associating Log Files with scrapes
Hi,
Thanks for giving a bit more detail on that. I can see how it would be useful. We don't have it implemented at present, but it wouldn't be terribly hard. I'll add it to our list. If it's quick and easy we may even be able to slip it in to the next version.
Best,
Todd
Associating Log Files with scrapes
Connecting the log file and the scrape could be useful in that we have thousands of scrapes logs, and if a scrape fails, we have to manually sort through the files to find the corresponding log file.
If the scrape were aware of the logfile name, we could pass that back to the requesting application. If there were unexpected or problematic results, the application could refer us to the appropriate log file.
We're loving screen-scraper and it's really working out well for us. But now that we're using it intensively we're moving into the performance, reporting, and statistics phase of the implementation. Performance is great, but following a problem (usually on the scraped site) through the chain of events can be tricky -- especially on a production (i.e. remote and locked down) box. If we knew the logfile name, we could request it via SOAP and get our debug info that way.
It's really less about the filename and more about debugging.
Associating Log Files with scrapes
Hi,
To generate that timestamp we use the following Java method:
http://java.sun.com/j2se/1.4.2/docs/api/java/lang/System.html#currentTimeMillis()
You can read the documentation for a detailed description of how it's generated.
Unfortunately, you don't have access to the file name within a screen-scraper script. I suppose that's a possibility, though. Could you give a bit more detail on why this would be useful to you?
Kind regards,
Todd Wilson
Associating Log Files with scrapes
Can you tell us what format that is? Is it epoch?
Also, I'm guessing it's not, but I'll try: Is that logfile filename available while the scrape is happening, so the scrape could tell me what it's logfile filename is?
Associating Log Files with scrapes
Hi,
You're correct. The number you see in the name of the log is simply a time stamp. We use a time stamp simply to ensure that each name is unique, and to facilitate sorting the file names.
Kind regards,
Todd Wilson