Please Explain Log Errors
Can someone please explain the use of scrapeableFile.addHTTPParameter? I think it is what I need to solve my problem but the log tells me this:
Data Records: An error occurred while processing the script: Set _dlc
Data Records: The error message was: Attempt to invoke method: addHTTPParameter() on undefined variable or class name: SiteResults : at Line: 3.
This is my script:
SiteResults.addHTTPParameter( new com.screenscraper.common.HTTPParameter(
"_dlc", "RESULT_COUNT", 1, com.screenscraper.common.HTTPParameter.TYPE_GET ));
This code is in a script file called Set_dlc and I am calling it in the scripts tab of the SiteResults scrapeable set to run 'Before File is Scraped'
The session seems to run as I do see the data in the log, but the error message leads me to believe that I may not have something set up quite right.
Thanks in advance.
Please Explain Log Errors
You'll want to make sure that you are setting ~@RESULT_COUNT@~ as a session variable in the "Edit Token" box. Then in order to use that value as a parameter on the next page, you just replace the vaule with the same token, but replace the "@" with "#". Example:
?_dlc=~#RESULT_COUNT#~&Does that help?
Getting Closer ....
It is actually embedded in the Next link:
Next Page
The Next part works great, but how in the world do I set the RESULT_COUNT variable? I used tutorial 3 to get the Next link part working, but I'm just not understanding.
Please Explain Log Errors
That shouldn't be hard to set, and you shouldn't need use addHTTPParameter(). The question is can you find what the value should be anywhere? Sometimes you'll have a list a categories on a page that you're looping through, and if that's the case, the record count could be available thereon.
Another possibility is that you can get to the first page of the category via a different link. On that first page, you can scrape the record count, and use that for each next page of the category.
Once you find where you can scrape the record count from, you just need to set it as the session variable, and it will be available.
Please Explain Log Errors
I suppose I didn't explain what I am trying to do very well. Sorry. Here's another go:
The URL I am scraping looks like this:
http://www.example.com/~#CATEGORY#~/tx/texarkana/~#PAGE#~/?_dlc=~#RESULT_COUNT#~&_dis=0
I am looping through a text file to replace the CATEGORY each time the PAGE (Next) loop finishes. That is working well.
The problem is that the site relies on the _dlc GET parameter being there (RESULT_COUNT). All it is is the number of records found each time the CATEGORY changes. So I am really just trying to update it every time the CATEGORY updates. This parameter has to be there or the site chokes and I get nothing.
I've twiddled with the Parameters tab on your advice, but I guess I really don't understand how it works as it is not updating.
I'll take whatever advice you can give me. If I can get this fixed, I'm good to go!
Thanks for your help.
Please Explain Log Errors
The documentation for that feature can be found [url=http://screen-scraper.com/support/docs/api_documentation.php#addHTTPParameter]here[/url].
That's not one I use too often. Most of the time, the tab "parameters" is the preferred way to go. Each time you request the file, the POST parameters will be the same, just the values differ.
The times that I have used addHTTPParameter are when there's a form with a long list of values that I need to set. Sometimes it's easier to scrape each key/value pair, save them in session variables, and use a script to set them manually with addHTTPParameter.
If that doesn't help, then maybe a better description of what you're aiming for will help me.