Cannot download images when running remote scrape session from external asp.net program code
Please help! I created a scraping session to scrape/download images from amazon.com. This works perfectly inside screenscraper, but when I try to do the scraping remotely from an external program written in asp.net no images are downloaded. What am I doing wrong? Here is my code:
Private Function RunScrapingSession(ByVal product As String, ByVal productid As Integer) As Boolean
Dim remoteScrapingSession As RemoteScrapingSession
Try
remoteScrapingSession = New RemoteScrapingSession("Amazon Images")
Catch ex As Exception
Return False
End Try
remoteScrapingSession.SetVariable("KEYWORD", product)
remoteScrapingSession.SetVariable("PID", productid)
remoteScrapingSession.SetVariable("n", 0)
Try
remoteScrapingSession.Scrape()
remoteScrapingSession.Disconnect()
Catch ex As Exception
remoteScrapingSession.Disconnect()
Return False
End Try
Return True
End Function
Dim remoteScrapingSession As RemoteScrapingSession
Try
remoteScrapingSession = New RemoteScrapingSession("Amazon Images")
Catch ex As Exception
Return False
End Try
remoteScrapingSession.SetVariable("KEYWORD", product)
remoteScrapingSession.SetVariable("PID", productid)
remoteScrapingSession.SetVariable("n", 0)
Try
remoteScrapingSession.Scrape()
remoteScrapingSession.Disconnect()
Catch ex As Exception
remoteScrapingSession.Disconnect()
Return False
End Try
Return True
End Function
Here is part of the code that I wrote in screen scraper that handles the image capture:
count = 0;
count = session.getVariable("n");
img = "";
img = session.getVariable("PRODUCT_IMAGE");
filename = session.getVariable("PID") + "_" + count;
session.log("Image captured: " + img);
try
{
session.downloadFile( img, "C:/Inetpub/wwwroot/DataScraper/WebDataScraper/WebDataScraper/images/" + filename + ".jpg", 5, false );
session.setVariable("n", count + 1);
}
catch( Exception e )
{
session.log("Image download failed: " + img);
session.setVariable("n", count);
}
count = session.getVariable("n");
img = "";
img = session.getVariable("PRODUCT_IMAGE");
filename = session.getVariable("PID") + "_" + count;
session.log("Image captured: " + img);
try
{
session.downloadFile( img, "C:/Inetpub/wwwroot/DataScraper/WebDataScraper/WebDataScraper/images/" + filename + ".jpg", 5, false );
session.setVariable("n", count + 1);
}
catch( Exception e )
{
session.log("Image download failed: " + img);
session.setVariable("n", count);
}
Close out of screen scraper
Ok I spoke with someone from screenscraper and he told me that it is best to close out of screen scraper when you are trying to use an external program to run a remote scraping session. I tried this and it worked! Thanks!