Issues with v.7 Professional and C# RemoteScrapingSession
I'm having several issues with a remote scraping session, using C Sharp and Version 7.0 (and I tried the alpha version 7.0.4a).
1) In v 7.0 I was not able to get back any datasets. They were always being returned as NULL (i.e. remoteScrapingSession.GetVariable("PRODUCTS")). I used the shopping.sss & shopping.cs files to confirm and was able to consistently get the same NULL results as my own code. I updated to v 7.0.4a and now all values are being returned as "OBJ#java.lang.Object#END_OBJ#null" whether the variable exists or doesn't exist (tested with both my own code and the shopping expample). After updating to 7.0.4a I looked in the \misc\dotNET folder in case there was a new DLL that might be handing the variables differently (there's a mention of this in the Alpha Log). Do any other changes need to occur to the C# Dlls or the windows service during update? I also noticed that only string session variables are being returned from v 7.0, I have variables that are set bool's & ints that also returns as NULL from remoteScrapingSession (can't test in 7.0.4a since they all come back as Java OBJ), but this could be related to the DataSet not returning.
2) I'm having 2 issues with reusing a Scraping Session. I assume this is possible, but I'm not sure. In the below code example, I want to scrape a page, do stuff in .NET - Logging, Database Action, Email, etc., and then scrape the next page using the same active session. The reason for this is so that I can share cookies & session variables since the site I am scraping is a portal with a login. A script within Screen-Scraper is being executed "before file is scraped" that reads the cookies and if they doesn't exist, logs in before scraping the page. I've verified within Screen-Scraper that if I call the Load Scrape several times (via a script) within the same session using similar code as below it doesn't need to log-in multiple times. But using Fiddler, I can see on the 2nd iteration the login page being loaded. With the issue mentioned in #1, I noticed that non-string session variables are accessible, is it possible the session.getCookies() is destroyed or not available on a subsequent Scrape() call? I could save the details of the session cookie to a variable and then rebuild it on every Scrape() call but that seems like work that shouldn't be necessary, and possibly bad if the web site has deterministic data in it such as a timestamp in the cookie that became stale.
2b) The second and more important issue is that only the first iteration of the below code will hit the Console.WriteLine(). On the 2nd iteration remoteScrapingSession.Scrape() hangs and won't continue. I've verified using Fiddler that the Screen-Scraper is making the 2nd call with the new CategoryId and the HTML is being returned, but the below code never continues and just hangs waiting for a response from Scrape().
var categoryIds = new List<string> {"1", "2", "3"};
foreach (var categoryId in categoryIds)
{
remoteScrapingSession.SetVariable("CategoryId", categoryId);
remoteScrapingSession.Scrape();
var productDataSet = remoteScrapingSession.GetVariable("PRODUCTS"); //This is never hit on the 2nd loop
var categoryIdReturned = remoteScrapingSession.GetVariable("CategoryId");
Console.WriteLine(categoryIdReturned);
//DO MORE C# CODE STUFF
}
remoteScrapingSession.Disconnect();
If it helps, I am seeing the following in the logs in both 7.0 & 7.0.4a:
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
at sun.nio.cs.StreamDecoder.read(Unknown Source)
at java.io.InputStreamReader.read(Unknown Source)
at java.io.BufferedReader.fill(Unknown Source)
at java.io.BufferedReader.readLine(Unknown Source)
at java.io.BufferedReader.readLine(Unknown Source)
at com.screenscraper.scraper.RemoteScrapingSessionHandler.run(RemoteScrapingSessionHandler.java:191)
Thanks,
Ross
Thanks Jason, I'll give that
Thanks Jason, I'll give that a shot, however based off your response Otherwise each loop will wait for the scrape to complete before starting the next. which is what I am looking to accomplish. I need .Scrape() to complete so that I can perform actions based off the returned Session variables. It sounds like you think I'm attempting multiple simultaneous asynchronous scrapes, and that isn't the case, since the
For...Loopwon't continue until .Scrape() is complete, the variables extracted and I do stuff.The issue is that it hangs on the 2nd .Scrape() and never completes (normally takes under 10 seconds to run the first scrape, the longest I waited on the 2nd scrape was 3 minutes, but in all cases the 2nd scrape never completes, I used Fiddler and that shows that HTML from the 2nd scrape was returned within the 10 second window mentioned, it's just that
RemoteScrappingSession.Scrape()doesn't return the results so that the remaining code can resume). It's possible that this was the issue fixed in the Alpha's 7.0.1a Fixed an issue where RemoteScrapingSessions would not stop properly in Professional edition. or 7.0.2a Added session.setAutoCloseAfterScrapeEnds (Closeable closeable). I'd be concerned with memory issues if .Scrape() wasn't being closed after a few thousand iterations.Thanks,
Ross
If you want to run multiple
If you want to run multiple copies of the scrape at the same time, you need to setDoLazyScrape to true. Otherwise each loop will wait for the scrape to complete before starting the next.
Question 1 is solved. Looks
Question 1 is solved. Looks like I missed the part about "Automatically Save the data set generated by this extractor pattern in a session variable"
I still need a hand with the 2nd issue and re-using an open Scrape() connection.
Thanks,
Ross