Received a status code of: 400
I'm trying to access the calendar of this page: https://www.airbnb.com/rooms/1371522. The calendar is at: https://www.airbnb.com/rooms/calendar_tab_inner2/1371522?cal_month=8&cal_year=2013¤cy=USD when you capture it with the proxy. I'm not sure what I'm missing, but the server needs something that I'm not provide before it will return a valid page.
Thanks,
Jeremy
The "addHTTPHeader" method is not available in this edition
Hi Scott,
The AddHTTPHeader was a call I started with, but I received this error and decided to open this topic. Here's the message I receive when using it in the Pro Edition 6.0 on Windows Ultimate.
ERROR--Airbnb: An error occurred while processing the script: Airbnb - Set Ref
Airbnb: The error message was: Exception (line 9): Availability Page: The "addHTTPHeader" method is not available in this edition of screen-scraper.-- Method Invocation scrapeableFile.addHTTPHeader
Not sure which calendar you
Not sure which calendar you seek. The check in/check out dates? If I go to https://www.airbnb.com/rooms/calendar_tab_inner2/1371522?cal_month=8&cal_year=2013¤cy=USD in a browser, I get nothing.
That's the problem
Hi Jason, you're correct, check-in/out. The empty page is the problem. If you capture the main page with the screen-scraper proxy, you'll see that the calendar page is: https://www.airbnb.com/rooms/calendar_tab_inner2/1371522?cal_month=8&cal_year=2013¤cy=USD, however when I run it back the page is empty. Is there an additional call that I need to make before requesting this page? For example I tried to set the referrer but that didn't work.
Jeremy, I got this working.
Jeremy,
I got this working. Remember the key is to replicate the request from the proxy as close as you can in your scraping session.
So, to get it working I added the following script which gets called prior to requesting the URL.
https://www.airbnb.com/rooms/calendar_tab_inner2/1371522
session.setCookie("www.airbnb.com", "bev", "1376419567_wwiu65gZX%2Ftpk0dp");
session.setCookie("www.airbnb.com", "_user_attributes", "%7B%22curr%22%3A%22USD%22%2C%22guest_exchange%22%3A1%2C%22ref%22%3A%22100%22%7D");
session.setCookie("www.airbnb.com", "_csrf_token", "XzauT0Tq3uvBzMOinxZejR4mpFNVboW2Vz-XrCAVvrM");
session.setCookie("www.airbnb.com", "flags", "301998080");
session.setCookie("www.airbnb.com", "_airbed_session_id", "890dafb009a14a55b9c8575f2106037d");
scrapeableFile.addHTTPHeader("X-CSRF-Token", "XzauT0Tq3uvBzMOinxZejR4mpFNVboW2Vz-XrCAVvrM");
scrapeableFile.addHTTPHeader("X-Requested-With", "XMLHttpRequest");
Now, it worked for me because I added the script in a relatively timely manner after proxying the site. The same values above may not work for you because they may be tied to my session. So, in order for them to work for you, you will need to track down the source of many of the values. The source values can originate in any of the previous transactions.
If you are not able to find the origin of a particular cookie you may find that the site will automatically set the value of the cookie simply by you requesting the right file. We've seen cookies set by css, jpg, gif, etc. So, by simply requesting the right URL the site will set the appropriate cookie automatically. The trick is to find which URL sets which cookie.
I hope this is helpful.
-Scott