Recording Page Size
We're trying to keep track of the amount of data we're pulling off a site (per scrape), and I was wondering if there were any techniques for keeping a log of and/or reporting the size of each page screen-scraper receives...
I know how to do it by adding a script to each and every scrapeable file, but that's obviously not a preferred solution. I've got almost a thousand scrapeable files in hundreds of scrapes.. So that wouldn't be ideal.
Or.. Does the scrape itself know? Is there something in session that might keep track of that? I suppose I could "scrape" each page as a whole and add them to a session variable or something and then add up the bytes in that at the end of the scrape (I have a summary script at the end of each scrape) but that's going to blow up the size of the logfiles as well as cause some memory headaches..
I've been giving thought to using a proxy and having the proxy keep track of everything, but I was wondering if there were an easier solution?
Recording Page Size
fnirt,
We have a list of suggestions on the following FAQ for overall optimization. Perhaps you may find something here that helps.
http://www.screen-scraper.com/support/faq/faq.php#Optimizing
-Scott