How many scrapeable files do you have to make?
I am attempting to scrape a Fannie Mae website. They have a login page, and you must ne logged-in before you can access other pages. The scenario is that you go to the Home page, the Login page, then step through a few other pages until you come to the page that you are interested in downloading. I set up scrapeable files for the Home and Login pages, and another one for the fina page that I want to download. When I run the scraping session, it appears to be going to the correct pages -- Home page, then using the correct login words to login. It then goes to the page that I want to download, and processes it. I have a script in that page that is supposed to download te page. It does download a page, but the page is one that is sent by Fannie Mae that says "System Error" and "Our system is currently experiencing technical difficulties.". However, when I login manually, there is no problem -- I can go to the page.
I am wondering what the difference is between getting to the page with screen-scraper and manually? Do I have to set up a scrapeable file for every page between the login page and the page that I want to download? This is the log:
Starting scraper.
Running scraping session: FM Peoples Bank
Processing scripts before scraping session begins.
Scraping file: "Home"
Home: Preliminary URL: https://www.ecommitone.com/webapp/action/DispatcherAction
Home: Using strict mode.
Home: Resolved URL: https://www.ecommitone.com/webapp/action/DispatcherAction
Home: Sending request.
Scraping file: "Login"
Login: Preliminary URL: https://www.ecommitone.com/webapp/action/Login
Login: Using strict mode.
Login: POST data: userId=l0200knl&password=later9&lenderId=218910009
Login: Resolved URL: https://www.ecommitone.com/webapp/action/Login
Login: Sending request.
Login: Redirecting to: https://www.ecommitone.com/webapp/action/ViewCommitmentSummary
Scraping file: "Print"
Print: Processing scripts before a file is scraped.
Print: Preliminary URL: https://www.ecommitone.com/webapp/action/BrowsePricesResultsPrint
Print: Using strict mode.
Print: Resolved URL: https://www.ecommitone.com/webapp/action/BrowsePricesResultsPrint
Print: Sending request.
Print: Processing scripts after a file is scraped.
Processing script: "DownloadFile"
Wrote file from: https://www.ecommitone.com/webapp/action/BrowsePricesResultsPrint to file: C:\screen-scraper\FMPeoples.html
Processing scripts after scraping session has ended.
Scraping session "FM Peoples Bank" finished.
There has to be some
There has to be some difference between the HTTP requests from screen-scraper vs those from your browser. I would look at the POST/GET parameters first, then maybe if there are uncommon headers in the POST data, or a cookie required.
problem with this website's security certificate
Perhaps the problem is that when I run through the web pages when I am using the proxy session, IE8 displays "There is a problem with this website's security certificate". I can click a button to continue. In the progress tab, I see a few transactions labeled "error". This only happens when the proxy session is active.
I checked the web for solutions. One suggestion was to adjsut IE8 options: Go to options, security, trusted sites and add the web site as a trusted site. I also set the security to medium low and then to low. None of that worked.
How do you use screen-scraper to scrape websites with the security certificate problem?
Most of the time those errors
Most of the time those errors you see in the proxy have the correct HTTP request, but the error was in the response. In those cases you can make your Scrapeable File with the correct response and screen-scraper won't respond with an error. In some cases that won't work, and you have to use another proxy. I use Charles proxy sometimes, and it works well.
Create a scrapeable file for every page that contains parameters
I got it to work. My goal was to download the Print Page, which you get to by stepping through some other pages. I created scrapeable files for the Login page and the Print page. The problem is that there was an intermediate page between the Login page and the Print page that contained parameters, and I didn't have a scrapeable fle for that page. I created a scrapeable file for that page, and what do you know, the Print page got the correct results! I'm a happy dude.
This helps answer the question for this post "How many scrapeable files do you need". Apparently you need to create a scrapeable file for every page that contains parameters, so that those parameters can be passed to the next page as listed in the proxy session. Please tell me more about this if there is anything to add.
------------------
Also regarding your advice above: "you can make your Scrapeable File with the correct response". Not exactly sure what you mean by that. Would you explain that briefly, maybe with an short example of what you mean?
Thanks much!