Problem with post parameters
Hi,
Sorry I have posted quite a few questions in the last week or so. I am having a problem with a site I am trying to scrape. There are two parameters.
Key Value
callCount 1 page=/direct/product/car.htm?_flowId=motor-flow
_flowExecutionKey Long Parameter(click to edit)
I don't have to change the callCount parameter
But I do need to alter the _flowExecutionKey
This looks like this when I click it
e2s1
httpSessionId=0000x05QtZyaf9MvkzOtss1Wpti:10qr9lql8
scriptSessionId=7D5639DB092D5BEE65178280C26DC9B8352
c0-scriptName=VehicleSearchDelegate
c0-methodName=findVehicleByReg
c0-id=0
c0-param0=string:03D45168
batchId=0
So I alter it like this
~#SERVER_NUM#~
httpSessionId=0000~#SESSION_ID#~:10qr9lql8
scriptSessionId=~#ENGINE_SESSION_ID#~
c0-scriptName=VehicleSearchDelegate
c0-methodName=findVehicleByReg
c0-id=0
c0-param0=string:03D2346
batchId=0
The Request is mean't to look like this
callCount=1
page=/direct/product/car.htm?_flowId=motor-flow&_flowExecutionKey=e2s1
httpSessionId=0000x05QtZyaf9MvkzOtss1Wpti:10qr9lql8
scriptSessionId=7D5639DB092D5BEE65178280C26DC9B8352
c0-scriptName=VehicleSearchDelegate
c0-methodName=findVehicleByReg
c0-id=0
c0-param0=string:03D45168
batchId=0
But instead it looks like this :
callCount=1%0Apage%3D%2Fdirect%2Fproduct%2Fcar.htm%3F_flowId%3Dmotor-flow&_flowExecutionKey=e1s1%0AhttpSessionId%3D0000g8iZBBLrC2mMh1VNydyg7Mc%3A10qr9lql8%0AscriptSessionId%3D9C27CCFA3F3455C4CFEDB774369CC9D8168%0Ac0-scriptName%3DVehicleSearchDelegate%0Ac0-methodName%3DfindVehicleByReg%0Ac0-id%3D0%0Ac0-param0%3Dstring%3A03D2346%0AbatchId%3D0%0A
This is adding a %0A for a new line
Adds a %3D for an equals sign
And adds %2F for a backslash.
If I add then as separate parameters it will add a & sign.
Any help would be really appreciated regarding this as I have been trying to solve it for quite a while now.
I have been looking at one of the posts and it is because there are illegal characters and they being encoded. The cookie is created from javascipt so I am guessing I will have to create the cookie manually. Is there an example of how to create one. The API only has three variables. I am looking to create a cookie as shown below. Any simple pointers of how to create a cookie would be great. I don't know how to get the same headers and the same data. Is there an working example or anything.
Thanks in advance,
Seamus McMahon
Thanks Scott
Sorry for the late reply, I got sidetracked. That worked perfectly. Thanks Scott
Seamus, It looks like you may
Seamus,
It looks like you may need to use scrapeableFile.setRequestEntity instead of setting those values under the parameters tab.
So, clear out your parameters tab and create a script that you will call from your scrapeable file before the file is scraped.
Your script should look something like this.
Run your scrape and look under the Last Request tab. Your POST payload should look something like this.
page=/direct/product/car.htm?_flowId=motor-flow&_flowExecutionKey=e2s1
httpSessionId=0000x05QtZyaf9MvkzOtss1Wpti:10qr9lql8
scriptSessionId=7D5639DB092D5BEE65178280C26DC9B8352
c0-scriptName=VehicleSearchDelegate
c0-methodName=findVehicleByReg
c0-id=0
c0-param0=string:03D2346
batchId=0
I have a hunch that two strings in the POST payload may need to be extracted and passed in like SESSION_ID. These look like they could be volatile.
03D2346
Dealing with cookies that are set by Javascript can get kind of hairy. A large majority of the time this is unnecessary, so I would do it as a last resort. Once you have discovered how to reproduce the cookie's value being generated by Javascript you will call session.setCookie.
I hope this helps.
-Scott