session.scrape clarification?
I am scraping a site that has multiple days, and multiple pages per day. I am successfully scraping through a single day, but would like to scrape one other day.
I have a session variable that decides which day to run, and I check to see if we're done with the current day using an interpreted java script.
My question is, if I were to change the session variable that chooses the day, and then perform session.scrape (after which the currently running session ends immediately) will it inherit the session variables from the session I ran .scrape from? (will I start scraping the next day by changing that session variable, assuming I am doing everything else correctly?)
Thanks,
-Dave
session.scrape clarification?
Excellent! Don't hesitate to write again if we can help with anything else.
Kind regards,
Todd
that's what I did :D
Thanks for the response.
I ended up doing exactly as you suggested. It took me a while to figure out the Java behind it (I'm a VB man myself). But it's working without a hitch.
Thanks for the suggestion,
-Dave
session.scrape clarification?
Hi Dave,
Typically you would handle logic in between scraping sessions in an external program. If you'd like to do it within screen-scraper, though, my recommendation would be to include some kind of looping logic such that the scraping session simply goes back and starts at the first scrapeable file once it's done with the rest. If you try calling the "scrape" method again it likely won't have the same session variables as before.
Best,
Todd Wilson