The "session" POST variable changes for each search
I recorded search results, created a scraping session, but the "session" POST variable changes for each search. For example, the "9046" will not work for the next search and since there are many users on the site the next number is not predictabe. I tried removing the "session" row but it did not work.
template hc_home_worth_modular 1 GET
step transmit_form_data 2 GET
app oma 3 GET
state NC 4 GET
origin 5 GET
fulladdress ~#ADDRESS#~ 6 POST
city ~#CITY#~ 7 POST
state ca 8 POST
zipcode ~#ZIP#~ 9 POST
ownername 10 POST
custid 11 POST
redirect 66574.9 12 POST
session 9046 13 POST
productcode 0 14 POST
Thanks for your help.
Another problem...
At this point I'm not sure if it is a changing session variable or an incorrect character. The address and city must have a "+" in place of a empty space, so I created a script to fix the address and city but when the charater is posted in the URL it is replaced with a "%2B". The log is as follows:
Processing script: "Fix Address"
33961+Maple+Canyon+Rd
Processing script: "Fix City"
Burbank
Processing script: "Scrape Site"
Scraping file: "Site"
Site: Processing scripts before a file is scraped.
Site: Preliminary URL: http://www.website.com/modular/index.cfm
Site: Using strict mode.
Site: POST data: fulladdress=33961%2BMaple%2BCanyon%2BRd&city=Burbank&state=ca&zipcode=90000&ownername=&custid=&redirect=66574.9&session=9046&productcode=0
I have seen two situations in
I have seen two situations in cases like this:
1) you have to scrape a new 'session' variable on each and every scrapeableFile. this way, you've always got the current value. I would recommend that each scrapeableFile should scrape the 'session' as a different variable name, such as 'session1' and 'session2', etc. The reason is complicated to explain in brief, but it will save you a headache, I'm sure of it.
--and, if that does look right--
2) If there is no 'session' value on each scrapeableFile that you can scrape, then I have seen it once or twice where every time you click on a link on such a website, you'll see not one, but two entries in your proxy server. The first one is the one you want, but it only redirects to a new page. The redirect has the content, but you should make your scrapeableFile out of the first one.
So, to recap-- the most likely scenario is to scrape a new 'session' value on every page.