Set all variables as saved in session

Hi,

If you're just saving individual extracted data items from tokens it's perfectly fine to save them in session variables. It's when you start saving entire data sets in session variables that you can run into trouble (under the "Advanced" tab for the extractor pattern).

On your second suggestion, you can actually do that now--try creating a script with this line in it:

session.breakpoint();

Then call it from wherever you'd like. It will show you a list of the current session variables, among other things.

Kind regards,

Todd

[quote="todd"]Hi,

Thanks much for your suggestions. I can see what you mean on your first suggestion, though, amongst the many hundreds of sites we've scraped, I don't know that that need has ever come up. Usually you want to be careful about saving too much information in session variables, lest the memory allocated to screen-scraper get overloaded. Perhaps it would make more sense to do this on an extractor pattern level? Maybe a button that would automatically do that for all tokens in an extractor pattern?
[/quote]

Yes, that's what I was thinking. I want to have everything in an extractor pattern automatically marked as a saved variable.

[quote]
The general procedure is that you want to write extracted data out to a file or database right when it's extracted, so as to optimize the performance of the scrape.
[/quote]

OK, to clarify here, in the walkthrough, with your "Hello world" example in tutorial 1, the data was written to file right after extraction, but you still need to save the session variable.

I guess my question is, how can I avoid using session variables. I don't know based on what I've read, but I'd like to know.

[quote]
You second suggestion (copying objects) has actually been made a number of times. The seeds of it are in the very latest pre-release, so keep an eye out for it soon.
[/quote]

Cool, that would make things a lot easier.

DK