How to interpret cookies and headers and how to use them?

Gary,

It's always best to take things one step at a time. By this I mean, focus on getting from the first page to the next page and no farther until it works.

I set up a scrape using this as the url of the first scrapeable file. No get or postparameters. Just the url.

https://committing.efanniemae.com/eCommitting/eCommitting

The second item in my proxy transaction list that was not a css or javascript file had the same url but with the following post parameters.


actionResource=LoginSubmit
originalSessionId=FbdCLy7BqKQnwBVVFtG8GzDy2dJV2DLBQdhb97lyxppsLtqjnbTc%21-1920279331%211275366514%211265826657688
submittedPage=Login
userID=[ommitted]
password=[ommitted]


Because the internet is by design a "stateless" environment web developers use certain techniques to help understand the current "state" of each user. The two most common techniques are by using cookies and/or get/post data. Because each session must be unique the corresponding cookie and/or get/post data will be unique.

If you try to use session data from a previous visit the server will likely kick you back to the login page because it does not have that value stored as a currently active session.

Most often, screen-scraper will automatically handle the passing of session cookies. The one exception is when a cookie is being set in the browser using Javascript (this is very rare, but you would look for "document.cookie" in some Javascript). For get/post session data it is necessary to scrape the corresponding value(s) from the previous page and pass them as session variables to the next page.

After you run your scrape, view the last response of the first scrapeable file. Search for the word "session". Note the hidden form field "originalSessionId". Create an extractor pattern for the value of that hidden field, let's call the token, "originalSessionId" and set it as a session variable.

Now, move to your second scrapeable file and click on the parameters tab. Delete the long wacky value of the originalSessionId parameter and replace it with ~#originalSessionId#~. Using the tilde-pound like this will reference the value of a session variable by that same name.

Finally, run your scraping session. When finished, view the Last Response of the second scrapeable file. Click on the Display Response in Browser button. If all went well you should see something other than the login screen.

Just to reiterate an earlier point, click on the Last Request tab of your second scrapeable file. Note the JSESSIONID cookie automatically set, ready to be passed to the server. Also view the Last Response of the first scrapeable file and you'll see screen-scraper happily acknowledging the presence of the JSESSIONID cookie handed to it from the server.

For pages farther on down the trough you'll do something very similar. It's always a good idea to continually re-scrape the value of your session parameter even if (in theory) it should not change during your session. It may be that with the other site that keeps kicking you off you just need to be sure you're passing a session value for each request.

The warning you're seeing in IE about "running scripts or ActiveX controls" could just be and formerly woefully neglectful browser now being overly anxious about protecting you from innocuous behavior by the site. Try tuning down (or off) your phishing filter or better yet use Firefox and find yourself forgetting that Internet Explorer was ever conceived of.

We've developed screen-scraper to automatically handle many of the goings-on over HTTP so you won't have to. By default, try trusting that it will and only intervene if it is proving that is can't.

-Scott