basic basic next page navigation i used to have down pat
Hello -
I lost my SS code to a hard drive crash, trying to relearn here...
I'm scraping Google.
(1) Here's my scrape page URL
http://www.google.com/search?q=%22~#GOOGLESEARCH#~%22&num=100&hl=en&lr=&...
(2) Initializing script contains:
Call RunnableScrapingSession.SetVariable( "GOOGLESEARCH", "implantable+sensor" )
Call RunnableScrapingSession.SetVariable( "RECORDSTART", "0" )
(3) Next page extractor pattern:
"nowrap" class="b"> Log shows RECORDSTART is correctly being changed to 100:<br />
GOOGLERECORDNUMBER='100'<br />
GOOGLERESULTURL=http://grants.nih.gov/grants/guide/rfa-files/RFA-EB-03-001.html<br />
GOOGLERESULTTITLE=NIH Guide: OPERATION OF SENSORS IN VIVO<br />
GoogleMay1: Processing scripts after a pattern application.<br />
Processing script: )
GoogleMay1: Processing scripts after all pattern applications.
GoogleMay1: Processing scripts before all pattern applications.
GoogleMay1: Extracting data for pattern "Nextpage"
GoogleMay1: The following data elements were found:
Nextpage--DataRecord 0:
[color=red]RECORDSTART=100 [/color]
GoogleMay1: Processing scripts after a pattern application.
Processing script: "GoogleNextPage"
Scraping file: "GoogleMay1"
GoogleMay1: Preliminary URL: http://www.google.com/search?q=%22~#GOOGLESEARCH#~%22&num=100&hl=en&lr=&...
GoogleMay1: Resolved URL: http://www.google.com/search?q=%22implantable+sensor%22&num=100&hl=en&lr=&start=[color=red]0[/color]&sa=N
GoogleMay1: Sending request.
GoogleMay1: Processing scripts before all pattern applications.
GoogleMay1: Extracting data for pattern "OldPattern1"
GoogleMay1: The following data elements were found:
OldPattern1--DataRecord 0:
GOOGLERECORDNUMBER='1'
GOOGLERESULTURL=http://www.engj.ulst.ac.uk/nibec/thinfilm/pptpres/shortcource98/index.htm
GOOGLERESULTTITLE=A Review of Implantable Sensor Technology. Dr. J. Mc Laughlin ...
(5) But the 'resolved URL' shows it still as 0, as shown in red above. It seems the original setting in the script cannot be overwritten by a new setting of the variable via an extractor pattern? Or am I creating a DataRecord (of value 100) instead of a Variable (of value 100)?
Also, for common sites like Google and Yahoo, would be great if you could include ready-to-go sample Sessions for these puppies... for us newbie non-programmer folks.
I'd be happy to send the entire Session or screen captures if necessary...
Thanks!
Thanks, Todd, that was the ticket
:D
I tried that before but apparently not when I had all other code correct and had deleted it in my iterations.
Yes, I found google etc. changes periodically, and I can see where instead of thanking you for the start, people might pan you for not maintaining them.
Movin To Montana Soon is an old Frank Zappa tune... very good.
thanks -
basic basic next page navigation i used to have down pat
Hi,
I hope you enjoy Montana :) I believe the fix is a simple one, in this case. For your extractor pattern token "~@RECORDSTART@~" you need to tell screen-scraper to save the value in a session variable. Do this by double-clicking the token, then checking the box labeled "Save in session variable?" It will then overwrite the original "0" value with your new value.
As to your suggestion on sample sessions, we used to do more of that, but ran into trouble when sites started changing, which resulted in broken scraping sessions. As an alternative, you can now get samples in our tutorials (our third tutorial is fairly similar to this) or from our ScrapbookFinds site: here.
Kind regards,
Todd Wilson