Can't see where login info is being sent
Using the basic edition, I'm trying to scrape mint.com (a personal finance site) and I can't figure out where the login information is transmitted.
I've been through the tutorials, and have successfully scraped other sites needing logins, finding the login information in the POST data, and creating the scrapeable page such that the Parameters tab has the user/password automatically filled out.
But mint.com has me stymied. Going to the login page via the proxy I get these files in the proxy log:
mint.com/login.event
mint.com/..../library.css
mint.com/..../reset.css
mint.com/..../library.js
mint.com/..../login.js
...And after login, these additional files appear:
mint.com/summary.event
mint.com/..../menu.js
mint.com/..../popup.xevent?query...
mint.com/..../summary.js
mint.com/..../popup.xevent?query...
None of them have POST information listed in the Request tab of the HTTP Transactions section of the proxy. The initial "login.event" page seems the most promising, but when I generate the scrapeable page, no parameters are listed.
I checked the pages recorded after the login ("Display Response in Browser"), and it's clear they're the post-login pages, so the info must be sent before then.
And I'm not super-knowledgeable about html, but I assume looking at the .css and .js files isn't what I'm after either.
Any ideas?
Since it's obvious that I'm
Since it's obvious that I'm the one taking care of the forums (as evidenced by the many threads I'm replying to), I wanted to apologize that I haven't gotten to your issue a little sooner. I have read it, but I can't seem to figure it out yet. Honestly, I haven't had enough time to sit down and tackle it yet.
I'll probably get to it this afternoon!!
Sorry for the long wait.
Tim
Ooookay. This is my
Ooookay. This is my diagnosis:
You are right-- there is no record of user data being passed back and forth while logging in.
However, you're missing a page... There is a transaction called "https://www.mint.com/loginUserSubmit.xevent". You can only see it when you uncheck the "Don't log binary files" checkbox on the "General" tab of the proxy server. I can't remember if you have control over this option in basic edition. I've been trying to start my copy of basic edition for the last 10 minutes, but my VMware is acting up.
The side effect of this is that you'll be logging all js, css, and image files along the way. But if you get rid of those miscellaneous files, you'll have the true path of the login.
You're lucky, though.. I was afraid it was going to be some nasty onslaught of javascript-set secure cookies that were being sent. Had that been the case, you would have had to recreate the cookie-making process.
In case you have basic edition, and if it can't allow you to change the option I told you about, here is the way it works:
Assuming that you're in your browser at https://wwws.mint.com/login.event , I turned on my proxy and got the following results:
https://wwws.mint.com/loginUserSubmit.xevent
Parameters tab:
validation: [{"fnc":"isEmail","evt":"BLUR","fld":"username","msg":"<div class='ajax' id='user-email-valid'>Enter an email like &#8216;[email protected]’<\/div>"}]
username: ~#your_email#~
password: ~#password#~
task: L
nextPage: [leave it blank]
You can probably leave those fields hard-coded, except for the username/password.
Next was:
https://wwws.mint.com/summary.event
No parameters
This one will redirect you to https://wwws.mint.com/account.event , but you can let screen-scraper worry about that.
The only other concern is cookie setting. Since the magical missing scrapeableFile (the "loginUserSubmit.xevent" one) is actually binary data, there's nothing to actually scrape from it. It's not readable at all. You just get some initial HTML & BODY tags (etc, etc) and then binary data that you can't process. Simply visiting it might be good enough to complete the login, though.
Give it a shot. I haven't seen anything quite like this, so I'm interested in the outcome.
Tim
Tim, thanks for the post.
Tim, thanks for the post. This is the public forum and I understand it means no guarantee for help, so any response at all is very much appreciated.
Anyway, I believe the basic edition does allow allowing unchecking the "Don't log binary files," so I'll give your advice a go, and post the results here.