Adding dynamic POST parameters
I'm working on a site that has a lot of dynamic POST data which changes for each request. I scrape each key/value pair and save them in session variables but ran into problems when I try to post them via the addHTTPParameter method. I tried to add the data via a "list variable" that groups the parameters in a comma separated list for each ( "key", "value", "sequence",) but the method does not like it. Also, I need to initiate the sequence with number 7.
Any hint on how to group a list that is possible to post via the addHTTPParameter method?
Best,
I've had to do this many
I've had to do this many times. Here's a sample script that I use:
// Script to dynamically create POST parameters for next page
// Function to actually create the parameter
makeParam(name, value, index)
{
scrapeableFile.addHTTPParameter(new com.screenscraper.common.HTTPParameter(name,value,index,com.screenscraper.common.HTTPParameter.TYPE_POST));
}
// Dataset that contains each paramter as a dataRecord
params = session.getVariable("Next page parameters");
// Clear so large values don't linger in memory
session.setVariable("Next page parameters", null);
session.log("+++There are " + params.getNumDataRecords() + " parameters");
// Loop through each paramter
{
for (i=0; i
param = params.getDataRecord(i);
key = param.get("NAME");
// The the key is "eventtarget", I need to set the value from a seperate location
if (key.equals("__EVENTTARGET"))
value = session.getVariable("PAGE");
else
value = param.get("VALUE")!=null ? param.get("VALUE") : "";
session.log("Making param: " + key + "=" + value);
makeParam(key, value, i+1);
}
You see this script expects a session variable with all of the parameters, and loops through them. You need to run this on your scrapeable file before the file file is scraped. You can easily edit line #29 to move the default start point.
Thanks Jason, the script
Thanks Jason, the script looks great!
However, something is producing angry red lines in the log when I try to run the scrips. I get "...bsh.EvalError (line 16): params .getNumDataRecords ( ) -- Error in method invocation: No args method getNumDataRecords() not found" which I think is due to that I handle the "Next page parameters" dataset wrong. I have tried various way to feed the pig but so far no suceess. What would be the content of the "Next page parameters" dataset? My extractor pattern looks as simple as this:
And the extractor pattern
input type='hidden' name='~@NAME@~' value='~@VALUE@~'
Your extractor pattern looks
Your extractor pattern looks right. You need to make sure that the extractor pattern is named "Next page parameters", and you go to the advanced tab of that extractor pattern and tick "Automatically save the data set generated by this extractor pattern in a session variable."
And then it works like a
And then it works like a charm!
Many thanks Jason,
Johan
Episode 2
Hoping not to make this into a soap opera, but how do you manage to pass those parameters to a "Next search results" page? As even more parameters are introduced on the search result page I need to add those and transfer the "value and name" parameters from the first search result page.
Just make sure not to clear
Just make sure not to clear the variable "Next page parameters" (or scrape it again), and run the script on the next page too.
I did just that - not clear
I did just that - not clear the variable - and it worked fine. Phu, this beast is finally down..
Again, thanks a lot.