The system is too busy to handle the file download at this time
Hi Support:
This screening scraping session has been working for the last few months but I just started receiving the following message in the log:
An error occurred whil attempting to download the file at: <a href="http://cris.XXX.org/cris/InternetViewImages.aspx?imageID=29464&tn=PropImages" title="http://cris.XXX.org/cris/InternetViewImages.aspx?imageID=29464&tn=PropImages">http://cris.XXX.org/cris/InternetViewImages.aspx?imageID=29464&tn=PropIm...</a>. The system is too busy to handle the file download at this time.
My script is as follows:
session.downloadFile( "http://cris.XXX.org/cris/InternetViewImages.aspx?imageID=" + session.getVariable("PHOTO_ID") + "&tn=PropImages", "C:/Documents and Settings/My Name/Desktop/XXX/XXX/XXX Photos/" + session.getVariable("APN") + ".jpg", 5, true );
session.setVariable("PHOTO_ID","");
session.setVariable("PHOTO_ID","");
The portion of the log associated with this problem is as follows:
04 Status for XXX: Applying extractor pattern: PHOTO_ID
04 Status for XXX: Extracting data for pattern "PHOTO_ID"
04 Status for XXX: The following data elements were found:
PHOTO_ID--DataRecord 0:
PHOTO_ID=29464
Storing this value in a session variable.
04 Status for XXX: Processing scripts after a file is scraped.
Processing script: "08 Download Photo"
LAHD: Downloading <a href="http://cris.XXXorg/cris/InternetViewImages.aspx?imageID=29464&tn=PropImages" title="http://cris.XXXorg/cris/InternetViewImages.aspx?imageID=29464&tn=PropImages">http://cris.XXXorg/cris/InternetViewImages.aspx?imageID=29464&tn=PropImages</a> in its own thread.
LAHD: An error occurred whil attempting to download the file at: <a href="http://cris.XXX.org/cris/InternetViewImages.aspx?imageID=29464&tn=PropImages" title="http://cris.XXX.org/cris/InternetViewImages.aspx?imageID=29464&tn=PropImages">http://cris.XXX.org/cris/InternetViewImages.aspx?imageID=29464&tn=PropIm...</a>. The system is too busy to handle the file download at this time.
04 Status for XXX: Extracting data for pattern "PHOTO_ID"
04 Status for XXX: The following data elements were found:
PHOTO_ID--DataRecord 0:
PHOTO_ID=29464
Storing this value in a session variable.
04 Status for XXX: Processing scripts after a file is scraped.
Processing script: "08 Download Photo"
LAHD: Downloading <a href="http://cris.XXXorg/cris/InternetViewImages.aspx?imageID=29464&tn=PropImages" title="http://cris.XXXorg/cris/InternetViewImages.aspx?imageID=29464&tn=PropImages">http://cris.XXXorg/cris/InternetViewImages.aspx?imageID=29464&tn=PropImages</a> in its own thread.
LAHD: An error occurred whil attempting to download the file at: <a href="http://cris.XXX.org/cris/InternetViewImages.aspx?imageID=29464&tn=PropImages" title="http://cris.XXX.org/cris/InternetViewImages.aspx?imageID=29464&tn=PropImages">http://cris.XXX.org/cris/InternetViewImages.aspx?imageID=29464&tn=PropIm...</a>. The system is too busy to handle the file download at this time.
I copied and pasted the following in my URL:
http://cris.XXX.org/cris/InternetViewImages.aspx?imageID=29464&tn=PropImages
which directed me to the photo.
I know this sounds stupid but I noticed that the word "while" was misspelled in the following error message:
An error occurred while attempting to download the file at:
Thanks for your help.
Adrian, I haven't seen this
Adrian,
I haven't seen this error before. On what OS are you running the scrape, and what version of screen-scraper is it?
OS and Version
Hi Jason,
The OS is:
Windows XP
Professional
Version 2002
Service Pack 3
The version of SS is 5.0.
What do you think is going on?
In your download script, you
In your download script, you have set doLazyScrape to true. My theory is that you're using screen-scraper professional edition, and bumping up against the 5 thread maximum in that edition. If you change that last parameter to false, you should no longer see that error.
Adrian, You could try setting
Adrian,
You could try setting the last parameter of downloadFile to "false". Setting it to true, as it currently is, will cause each download request to run its own thread. There are a limited number of downloads that can run concurrently. To find out how many, call getMaxConcurrentFileDownloads() like so.
session.log("max number of threads: " + session.getMaxConcurrentFileDownloads());If you have that number of downloads taking place and screen-scraper requests one more, it will attempt the 11th download 10 times pausing for 3 seconds between each attempt. After the 10th attempt it throws that error.
By setting the last parameter to false, you are allowing the downloads to share processor threads. This may make downloading a bit slower, but it will better maximize the available resources.
You could also strategically place pauses before downloads begin to give them time to finish.
-Scott
Thanks!!!
It worked !!! YOU ARE A GENIUS !!!