Home » Forums » screen-scraper » screen-scraper support for licensed users

Can't find webrefid key value

Hi Support,

I can't figure out how the "webrefid" key value is generated. I had a similar problem in the past with another website but I was able to figure it out by looking at the log in the proxy session for a redirect but in this one I can't find any clue in the log. The screen scraping is configured in the following steps:

Step 1

http://www.smartlaw.org/index.cfm

as you suggested in another post I set the parameters as follows:

fuseaction___ConfirmSelection___1___POST
parmDiv___BodyContent___2___POST
parmClick___0___3___POST
AttyID___0___4___POST
parmPanel___LLTTER___5___GET

I'm able to scrape the following:

ReferralAttorney" value="~@REFERRAL_ATTORNEY@~" />
CFTOKEN=~@CFTOKEN@~;
CFID=~@CFID@~;

Step 2

http://www.smartlaw.org/index.cfm

the parameters are set as follows:

fuseaction___ConfirmSelection ___1___POST
parmDiv___BodyContent___2___POST
parmClick___0___3___POST
AttyID___0___4___POST
parmPanel___LLTTER___5___GET
Fuseaction___FormReview___6___POST
Language___7___POST
Region___8___POST
PanelCD___LLTTER___9___POST
Waived___WAIVED___10___POST
ReferralAttorney___~#REFERRAL_ATTORNEY#~___11___POST
Message___0___12___POST
FirstName___Z 13___POST
MiddleName___14___POST
LastName___Z___15___POST
Address1___Z___16___POST
City___Z___17___POST
State___Z___18___POST
Zip___Z___19___POST
DayPhone___Z___20___POST
EveningPhone___21___POST
[email protected]___22___POST

Step 3

http://www.smartlaw.org/index.cfm

the parameters are set as follows:

fuseaction___ConfirmSelection ___1___POST
parmDiv___BodyContent___2___POST
parmClick___0___3___POST
AttyID___0___4___POST
parmPanel___LLTTER___5___GET
Fuseaction___FormReview___6___POST
Language___7___POST
Region___8___POST
PanelCD___LLTTER___9___POST
Waived___WAIVED___10___POST
ReferralAttorney___~#REFERRAL_ATTORNEY#~___11___POST
Message___0___12___POST
FirstName___Z 13___POST
MiddleName___14___POST
LastName___Z___15___POST
Address1___Z___16___POST
City___Z___17___POST
State___Z___18___POST
Zip___Z___19___POST
DayPhone___Z___20___POST
EveningPhone___21___POST
[email protected]___22___POST
submitform___Finish___23___POST

Step 4

http://www.smartlaw.org/index.cfm

This is where I'm having trouble. I can't figure out how the "webrefid" key value is generated. Otherwise the parameters are set as follows:

fuseaction___ReferralResults___1___GET
webrefid____36513____2____GET
Message____0____3____GET
CFID____~#CFID#~____4____GET
CFTOKEN____~#CFTOKEN#~____5____GET

Thanks for your help.

‹ 500 null / JRun Servlet Error Referrer ›
Adrianjay on 06/12/2010 at 6:08 pm

Adrian, It looks like they're

Adrian,

It looks like they're doing things a bit different for these two additional pages. Because I'm not able to proxy the site using screen-scraper (which we're looking into) I, again, used Charles Proxy. Doing so revealed that the two additional pages are using ajax to pass the form data to the server (note the "Form" tab in Charles).

The data is a very nice and neat string of key/value pairs. Not what you'd expect from ajax. That makes it look like the data belongs under the parameters tab. Instead, however, you'll want to clear out your parameters tab and pass the data using


scrapeableFile.setRequestEntity(String requestEntity);

1st request:

scrapeableFile.setRequestEntity("Fuseaction=FormReview&Language=&Region=&PanelCD=LLTTER&Waived=WAIVED&ReferralAttorney=962&Message=0&FirstName=z&MiddleName=&LastName=z&Address1=z&City=z&State=z&Zip=z&DayPhone=z&EveningPhone=&EmailAddress=Z%40AOL.COM");

2nd request:

scrapeableFile.setRequestEntity("Fuseaction=ProcessReferral&Language=&Region=&PanelCD=LLTTER&Waived=WAIVED&ReferralAttorney=+++++962&Message=0&FirstName=z&MiddleName=&LastName=z&Address1=z&City=z&State=z&Zip=z&DayPhone=z&EveningPhone=&EmailAddress=Z%40AOL.COM&submitform=Finish");

In the HTML of the last response of the 2nd request you will find where you can scrape the value of webrefid.

Hope this helps.

-Scott

 swilsonmc on 06/14/2010 at 3:54 pm

Passing a Saved Variable

Hi Scott,

How do I pass a saved variable in?:


scrapeableFile.setRequestEntity(String requestEntity);

For example,in Step 1


http://www.smartlaw.org/index.cfm

I save the following as a session variable:


ReferralAttorney" value="~@REFERRAL_ATTORNEY@~" />

But I don't know how to replace the the save variable: REFERRAL_ATTORNEY for the "962" in the following script:


scrapeableFile.setRequestEntity("Fuseaction=FormReview&Language=&Region=&PanelCD=LLTTER&Waived=WAIVED&ReferralAttorney=962&Message=0&FirstName=z&MiddleName=&LastName=z&Address1=z&City=z&State=z&Zip=z&DayPhone=z&EveningPhone=&EmailAddress=Z%40AOL.COM");

Thanks for your help.

 Adrianjay on 06/16/2010 at 1:41 pm

Adrian, One

Adrian,

One way:

scrapeableFile.setRequestEntity("Fuseaction=FormReview&Language=&Region=&PanelCD=LLTTER&Waived=WAIVED&ReferralAttorney=" + session.getVariable("REFERRAL_ATTORNEY") + "&Message=0&FirstName=z&MiddleName=&LastName=z&Address1=z&City=z&State=z&Zip=z&DayPhone=z&EveningPhone=&EmailAddress=Z%40AOL.COM");

Another way (expanded):


// Internally we tend to name our variables based on this convention:
// Session variables / DataRecord values: ALL_CAPS (spaces as underscores)
// Local variables: camelCase (no spaces)

panelCD = session.getVariable("LLTTER");
session.setVariable("LLTTER",null);

waved = dataRecord.get("WAIVED");

referralAttorney = session.getVariable("REFERRAL_ATTORNEY");

requestEntity = "Fuseaction=FormReview&Language=&Region=";
requestEntity += "&PanelCD=" + panelCD;
requestEntity += "&Waived=" + waved;
requestEntity += "&ReferralAttorney=" + referralAttorney;
requestEntity += "&Message=0&FirstName=z&MiddleName=&LastName=z&Address1=z&City=z&State=z&Zip=z&DayPhone=z&EveningPhone=&EmailAddress=Z%40AOL.COM";

session.log("requestEntity: " + requestEntity);

scrapeableFile.setRequestEntity(requestEntity);

This has been a lesson in string concatenation.

-Scott

 swilsonmc on 06/17/2010 at 4:59 pm