Only getting half the dataset
I'm using Tutorial 3 and would like to write out the PRODUCTID, MODEL and PRICE.
So I identified the extractor pattern on the Search Results page "PDL" (and checked the Automatically save the data set ... box) ~ found that if I didn't do this would get the "java.null..." problem.
I also identified the extractor pattern on the Details page "PDL" and checked the "Automatically ..." box.
Tried both append & join (coundn't find any documentation on this feature) seperately.
Here is my code
' Create a RemoteScrapingSession object.
Dim objRemoteSession
Set objRemoteSession = Server.CreateObject("Screenscraper.RemoteScrapingSession")
Call objRemoteSession.Initialize("Shopping Site") Call objRemoteSession.SetVariable( "SEARCH", "dvd" )
Call objRemoteSession.SetVariable( "PAGE", "1" ) ' Check for errors.
If objRemoteSession.isError Then
Response.Write( "Error: " & objRemoteSession.GetErrorMessage & "
" )
Else
' Tell the scraping session to scrape.
Call objRemoteSession.Scrape
For j = 0 To objRemoteSession.GetNumDataRecordsForDataSet("PDL") - 1
Response.Write(objRemoteSession.GetDataSetValue("PDL", j, "PRODUCTID") & " ")
Response.Write(objRemoteSession.GetDataSetValue("PDL", j, "MODEL") & " ")
Response.Write(objRemoteSession.GetDataSetValue("PDL", j, "PRICE") & "
")
Next
Call objRemoteSession.Disconnect
?>
Dim objRemoteSession
Set objRemoteSession = Server.CreateObject("Screenscraper.RemoteScrapingSession")
'
Generate a new scraping session.Call objRemoteSession.Initialize("Shopping Site") Call objRemoteSession.SetVariable( "SEARCH", "dvd" )
Call objRemoteSession.SetVariable( "PAGE", "1" ) ' Check for errors.
If objRemoteSession.isError Then
Response.Write( "Error: " & objRemoteSession.GetErrorMessage & "
" )
Else
' Tell the scraping session to scrape.
Call objRemoteSession.Scrape
Call objRemoteSession
.StoreVariable ("PDL")For j = 0 To objRemoteSession.GetNumDataRecordsForDataSet("PDL") - 1
Response.Write(objRemoteSession.GetDataSetValue("PDL", j, "PRODUCTID") & " ")
Response.Write(objRemoteSession.GetDataSetValue("PDL", j, "MODEL") & " ")
Response.Write(objRemoteSession.GetDataSetValue("PDL", j, "PRICE") & "
")
Next
End
If'
Disconnect from the server.Call objRemoteSession.Disconnect
?>
Here are the results
8
34
20
5
16
12
11
15
13
14
Only the PRODUCTID is written.
What am I doing wrong. Any suggestions?
RF
Only getting half the dataset
So I decided to make some changes to my VBSCRIPT (ASP) and instead of PDL I'm now just looking for the DETAILS dataset. I have made sure that the automatically save ... is checked, disabled breakpoint script (saved everything). Remember it works in GUI. Here is the code:
<%
' Create a RemoteScrapingSession object.
Dim objRemoteSession
Set objRemoteSession = Server.CreateObject("Screenscraper.RemoteScrapingSession")
' Generate a new scraping session.
Call objRemoteSession.Initialize("Shopping Site")
Call objRemoteSession.SetVariable( "SEARCH", "dvd" )
Call objRemoteSession.SetVariable( "PAGE", "1" )
' Check for errors.
If objRemoteSession.isError Then
Response.Write( "Error: " & objRemoteSession.GetErrorMessage & "
" )
Else
' Tell the scraping session to scrape.
Call objRemoteSession.Scrape
Call objRemoteSession.StoreVariable ("DETAILS")
For j = 0 To objRemoteSession.GetNumDataRecordsForDataSet("DETAILS") - 1 [color=red]Fails here (see below)[/color]
Response.Write(objRemoteSession.GetDataSetValue("DETAILS", j, "URL") & " ")
Response.Write(objRemoteSession.GetDataSetValue("DETAILS", j, "MODEL") & " ")
Response.Write(objRemoteSession.GetDataSetValue("DETAILS", j, "PRICE") & "
")
Next
End If
' Disconnect from the server.
Call objRemoteSession.Disconnect
%>
Java Exception error '80004005'
java.lang.NullPointerException
/screenscraper/shopsite.asp, line 21
This looks like the same problem as the post titled: COM Problems from JimBoy
Only getting half the dataset
But only if I split it into 2 datasets - one called PRODUCTS - this has just the productID and another called DETAILS this has everything else
Only getting half the dataset
I was wrong - I see all of the data in the dataset
Only getting half the dataset
Interesting. When the pop-up window displays I see the data for the current record. When I let it run for a few breakpoints then check the data set it only has the current record with a sequence id of 0.
The first thing I checked was whether I had checked "automatically save data records to the data set" and I had. I would really like a complete Dataset to be able to pass onto my asp app for processing.
Thoughts?
Only getting half the dataset
RF,
I am guessing you tried this already, but make sure to run the scraping session from the GUI and check to make sure that there is something being scraped for those values. The best way to do this is to add the following script after the pattern that scrapes this data:
session.breakpoint();
This method will pop up a window which will have the values within the data set and any other values currently in scope.
You are correct that you need to check the box which automatically saves data records to the data set. The append option always appends to the end of the data set. Without going too much into detail the join option sometimes overwrites what is in the data set, so rarely is this the option you would want.
If you check your scraping session and those values are being populated but not being written out, then get back to me and we will look a little deeper into the problem.
Brent
[email protected]