Home » Forums » screen-scraper » screen-scraper public support

Problem logging on to site - I don't see any POST data

Hi,

I've run through the tutorials and everything worked just fine. This product seems to be perfect for what I want to achieve but now I'm trying to scrape a 'real' site I'm running into problems logging in.

The site in question is http://www.lambaplc.com/ and I'm able to login with my credentials whilst Proxy is running but I don't see any POST data. I've also run the HTTP headers add-on for firefox but couldn't see anything useful.

All i see is the http://www.lambaplc.com/login.jsp and then once i enter credentials, there are some GET responses with my username and my password returned as some hashed value such as CDAFF6EFE6F0E5F3E3F58D - I can't see where my username / password is posted to prior to this.

I'm not overly technical (as you may have guessed) but any pointers would be greatly appreciated.

Thanks in advance

Scott

‹ delay for website load time Error When Writing to MySQL Database ›
scottf on 01/30/2012 at 12:42 pm

I see the Request - GET

I see the Request - GET http://lambaplc.com/login.jsp?request=login_validate&userid=EDITEDOUT&userpassword=670563797C6A7B6D796F0B&rememberme=false&widgetName=kam&widgetCategory=login HTTP/1.1

So am I right in saying that the hash of my user password in the url above is being returned from the function in QVS_II.src? Presumably its also using a session id or something as well? Happy to supply a UID and PWD if needs be by PM but obviously don't want to put it on a public forum.

Any pointers really appreciated

 scottf on 02/01/2012 at 12:30 pm

scottf,There is one other

scottf,

There is one other function in that same script you'll need to emulate. Search for the userPassword() function. This function calls the one Jason mentioned earlier.

So, on the login page extract the value of "key" from a hidden form field. Then, call a script containing the following.

**Be sure to set your script's language to JavaScript**

/* Original functions found at
 http://www.lambaplc.com/js/QVS_II.src
*/

function encrypt(inputString, start) {
  var hex = ["0","1","2","3","4","5","6","7","8","9","A","B","C","D","E","F"];
  var outputString = "";
  var xor = start;
  var value;
  var character;
  for(var i = 0; i < inputString.length; i++) {
    character = String.fromCharCode(inputString.charCodeAt(i) ^ xor++);
    value = character.charCodeAt(0);
    outputString += hex[Math.floor(value / 16)] + hex[value % 16];
    if(xor > 255) xor = start;
  }
  return outputString;
}

/*
Modified function userPassword()
*/

var hex = "0123456789ABCDEF";
var key = session.getv("key");
userpassword = escape(encrypt(session.getv("userid"), (hex.indexOf(key.substr(6,1)) * 16) + hex.indexOf(key.substr(7,1))));

session.setv("userpassword",userpassword);

session.log("userpassword: " + session.getv("userpassword"));

session.scrapeFile("submit");

And, as the last line suggests, call the login scrapeable file where you pass it your userid (as you typed it) and the userpassword after it's been encrypted.

Be sure you use the transaction from your proxy session so that this last scrapeable file contains the four other GET parameters (request=login_validate, rememberme=false, widgetName=kam, widgetCategory=login).

-Scott

 swilsonmc on 02/03/2012 at 11:51 pm

IT WORKS!! Scott / Jason -

IT WORKS!!

Scott / Jason - thank you both so much for taking the time to help me

 scottf on 02/06/2012 at 9:54 am

Also, if you look in the file

Also, if you look in the file at http://www.lambaplc.com/js/QVS_II.src, you can see a JavaScript function like this:

function encrypt(inputString, start) {
  var hex = ["0","1","2","3","4","5","6","7","8","9","A","B","C","D","E","F"];
  var outputString = "";
  var xor = start;
  var value;
  var character;
  for(var i = 0; i < inputString.length; i++) {
    character = String.fromCharCode(inputString.charCodeAt(i) ^ xor++);
    value = character.charCodeAt(0);
    outputString += hex[Math.floor(value / 16)] + hex[value % 16];
    if(xor > 255) xor = start;
  }
  return outputString;
}

You will need to emulate what that is doing to your values when you pass them in.

 jason on 01/31/2012 at 2:25 pm

Thanks for the swift reply...

Thanks for the swift reply... I'll take a look at QVS_II.src and let you know how I get on

 scottf on 02/01/2012 at 10:06 am

Without an account to test

Without an account to test with, I can't see exactly what they are doing. You should try clearing your cache/open sessions/cookies, and then proxy it. Do no filter the less useful results as sometimes they trick the proxy. Once done, use the find button and search for your username. It has to be in one if the requests.

 jason on 01/30/2012 at 5:15 pm