VB.NET program not getting data returned

I created the Shopping Site scraping sesison in tutorial 2. Then I wrote a VB.NET program to use it. The program is basically the VB.NET code provided with the documentation. It does have the reference to the RemoteScrapingSession dll and it has an Imports Screenscraper at the top of the file. The computer is running Windows 7 and IE8. I started the screen scraper service using the Windows 7 task manager.

The program runs. I left the script that creates the dvds.txt file in there and I see it extracts the data and writes it to the dvds.txt file. The screen-scraper log folder contains a text file that is a log for the scraping of the Shopping Site scraping session. There is a lot of data there and it looks OK to me. But it returns a null string, not a populated DataSet.

---------------------
The error.txt file in the log folder contains this:
An IOException occurred. The message was: Connection reset
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(Unknown Source)
at sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
at sun.nio.cs.StreamDecoder.read(Unknown Source)
at java.io.InputStreamReader.read(Unknown Source)
at java.io.BufferedReader.fill(Unknown Source)
at java.io.BufferedReader.readLine(Unknown Source)
at java.io.BufferedReader.readLine(Unknown Source)
at com.screenscraper.scraper.RemoteScrapingSessionHandler.run(RemoteScrapingSessionHandler.java:179)
An IOException occurred. The message was: Connection reset
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(Unknown Source)
at sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
at sun.nio.cs.StreamDecoder.read(Unknown Source)
at java.io.InputStreamReader.read(Unknown Source)
at java.io.BufferedReader.fill(Unknown Source)
at java.io.BufferedReader.readLine(Unknown Source)
at java.io.BufferedReader.readLine(Unknown Source)
at com.screenscraper.scraper.RemoteScrapingSessionHandler.run(RemoteScrapingSessionHandler.java:179)

-----------------------
The stdout.txt file contains this:
Initializing screen-scraper server. Please wait...
Ready.
The server is in non-interactive mode. Run the "stop server" script to stop it.
Warning: You're not using a Java Development Kit (SDK) environment so JSP pages can not be compiled.
Starting service LiteWebServer Base Module (lws-base/3.0.3) (with LiteWebServer JSP Module (lws-jsp/1.1.2))

----------------------------

The program method:
'Scrape ShoppingSite.
Private Sub Scrape_Shopping_Site()
'Generate a RemoteScrapingSession object, which
'acts as the driver, or interface, to screen-scraper.
'If you're running screen-scraper on a different computer
'than the one the VB file resides on you'll want to
'modify and use the top version instead of the bottom one.
'RemoteScrapingSession remoteScrapingSession = new RemoteScrapingSession("Shopping Site", "192.168.0.5", 8778 );
Dim RemoteScrapingSession1 As RemoteScrapingSession = New RemoteScrapingSession("Shopping Site")

'Set the variables.
'Remember that these two top variables correspond to the POST
'parameters we use for the "Login" scrapeable file.
RemoteScrapingSession1.SetVariable("EMAIL_ADDRESS", "[email protected]")
RemoteScrapingSession1.SetVariable("PASSWORD", "testing")
'screen-scraper will use this parameter to search the products.
RemoteScrapingSession1.SetVariable("SEARCH", "dvd")
'We start screen-scraper at page 1 of the search results.
'Note that we could have also done this in an "initialize"
'script within screen-scraper, which is common.
RemoteScrapingSession1.SetVariable("PAGE", "1")

'Tell the session to scrape. This method call might take
'a little while since it will need to wait for screen-scraper
'to fully extract the data before it returns.
RemoteScrapingSession1.Scrape()

'Get the data set that was stored by screen-scraper in a
'session variable. This data set corresponds to the "PRODUCTS"
'extractor pattern found under the "Details page" scrapeable
'file.
'Dim products As DataSet = RemoteScrapingSession1.GetVariable("PRODUCTS")
Dim products As System.String = RemoteScrapingSession1.GetVariable("PRODUCTS")

'Iterate through each of the data records screen-scraper
'extracted, outputting each of them to the browser.
'Dim i As Integer
'For i = 0 To products.NumDataRecords - 1
' Dim product As DataRecord
' product = products(i)
' Console.WriteLine("=======================================")
' Console.WriteLine("Product #" + i.ToString)
' Console.WriteLine("Title: " + product.Item("TITLE"))
' Console.WriteLine("Model: " + product.Item("MODEL"))
' Console.WriteLine("Shipping Weight: " + product.Item("SHIPPING_WEIGHT"))
' Console.WriteLine("Manufactured By: " + product.Item("MANUFACTURED_BY"))
' Console.WriteLine("=======================================")
'Next

'Be sure to disconnect from the server.
RemoteScrapingSession1.Disconnect()
End Sub

--------------------

The problem is that RemoteScrapingSession1.GetVariable("PRODUCTS") returns a NULL string, not a DataSet populated with the data it extracted. I know the program communicated with the screen-scraper service because the log shows it ran and it created a dvds.txt file that does contain the data that should be passed to the program. So everything went OK until it was time to pass the DataSet to the .NET program.

Do I need to add something to the "Write data to a file" script? Should a script attached to a scrapeable file be BUILDING a DataSet with a particular varibale name? Then that varibale is somehow made available to the .NET program?

Do you know why it doesn't return a DataSet?

My guess is that the dataSet

My guess is that the dataSet isn't set as a session variable. Your extractor pattern needs to be named 'PRODUCTS' (it is case sensitive), and you need to Advanced tab of that extractor, and tick 'Automatically save the data set generated by this extractor as a session variable'.

Save the data set as a session variable

Thanks Jason, clicking the checkbox to save the data set generated by this extractor as a session variable solved the problem. The PRODUCTS variable was successfully passed to the program!

Would you explain something to me? "PRODUCTS" is the identifier for an extractor pattern within the scrapeable file named "Details Page". So why is "PRODUCTS" also a DataSet? What makes an identifier or a variable contain a DataSet?

It's mostly the way the data

It's mostly the way the data is structured. If you think of a dataSet is a table, a dataRecord is a row, and a token is a column in the row. It doesn't matter if the table has the same name as a column as they are in different locations.

Now, if you set both as session variables that won't work as the session variable is a different place in the data structure.