The site I'm trying to scrape is telling me that cookies need to be enabled. How to I fix this?
There are a few possible causes for this issue:
- The URL or POST parameters are incorrect. You'll want to ensure that the URL and POST parameters you're sending to the remote site are exactly what the site is expecting. For example, if you inadvertently leave off a session ID from the URL or a POST parameter, the remote server could report that cookies are disabled.
- screen-scraper isn't handling cookies as the remote server is expecting. If this is the case, you'll need to modify the cookie settings under the "Advanced" tab for your scraping session (the "Cookie policy" drop-down list and "Use HTTP strict mode" checkbox). The most common fix is to set the "Cookie policy" to "Compatibility" and to check the "Use HTTP strict mode" checkbox.
- The server is setting cookies via JavaScript. screen-scraper doesn't execute any of the JavaScript in an HTML page, so if the site is setting cookies via JavaScript, you'll need to set the manually via a script within screeen-scraper. This is probably the trickiest one to debug, and may require examining the HTML and .js files for the site to determine where and how cookies are being set. Once you can determine how that's being done, you can use screen-scraper's session.setCookie method to set the cookie manually.