Next page scrape of search results
Would someone be so kind to take a moment to help me out. I have had several all night sessions to figure this out but have been unable to move my scrape to the next page. The line of HTML the next link is located in is below. I know that this is probably more than is needed but I didn't want to accidentally leave something out. I don't know much java but have been learning during the last month. I tried the variation given in the forms and APIs on this site and haven't had luck. Also tired to trigger the (shortcut: tap the 'N' key twice) with a keypress/keyrelease script but that didn't work but maybe I wasn't setting it up to run at the right time. Anyway any help would be greatly appreciated.
Ok lets see.
I ran the script and got a void null error on the NEXT_PAGE_URL variable so I assingned it as null before the session starts with a session.setVariable( "NEXT_PAGE_URL", "null" ). But I guess I need to assing (that spelling doesn't look right)the whole link to the "NEXT_PAGE_URL" in order to process it? OK back to the tutorial #2 I think. Funny thing about the tutorial, it's really helpful but when I downloaded the completed scrape for #2 and followed along with the video right about at 35 minutes when you click on the apply to last scrape in the next link and the data set comes up with page 2 and keyword dvd the scrape that I downloaded says no data available. I had it figured out a couple weeks ago and I think it was the incorrect link in the url properties field for the scrape. Since that is the spot in the tutorial that demonstrates setting up a link for the next page it's been driving me insane.
oo, that's kind of ugly :)
oo, that's kind of ugly :)
We've had troubles before when there are any HTML "entities" in the URL... for instance, I see this in the 'next page' link:
And then proceed to try to hit the next page like normal.
So sorry for the delayed response. I've been busy busy with in-office jobs for clients the last few weeks! I'll try to stay on top of the forums more. Let me know if that does the trick for you... usually the error message in the log is really cryptic when this is the problem, so it's not very obvious.
Tim