Workbench Suggestions
These are just a few suggestions, the ones I find myself wanting as I develop away at a scraping session.
Execute Scrape on Cached Data - Instead of having to constantly run a scraping session against a live website while developing in workbench, run the scrape against previously captured data / last response data. Right now I'm developing a scrape where I must put in sleep()'s of 1 minute between each requests if I'm to comply with a site's TOS, so I'm constantly having to wait. This kind of function would readily increase the pace of development.
A syntax checker (lint-type) for interpreted java in workbench, I'm constantly starting a scrape session in workbench and having it fail once I hit a certain script. I've been copying and pasting over to netbeans to do this, but it would be very convenient to have it built-in. Even better if there were a stacktrace output window (dumped to log tab maybe?) of the error if there is one.
Collapsible / Shrinkable Extractor Pattern textarea's The Extractor Pattern textarea (Scraper -> Extractor Patterns tab -> Pattern text box) and in the Sub-Extractor tab too, its about 12 lines in my workbench and to be honest I don't need them, I have a lot of one liners out of necessity. My suggestion would be to make them collapsible like code folding, a [+] to expand the extractor pattern and a [-] to collapse the extractor pattern, maybe leaving the top line visible as a label.
ie. [+] - Username:~@USERNAME@~... [Sequence NUM]
Or maybe just have the textarea's adjustable in vertical height.
I would also like to say that screen-scraper is a great product, keep up the good work.
Hi, These are great
Hi,
These are great suggestions. Here's my response:
- We're actually already doing this in some of our internal projects, and have plans to build it into ss. It might still be a bit down the road, though.
- A syntax checker would be great. We've tried to approximate that to some degree with syntax highlighting, but a full-blown syntax checker would be a step beyond that.
- I also agree that the extractor pattern text areas can be a bit too big. Just yesterday we implemented a feature that will automatically resize the pane containing them (watch for it in the next alpha), but that doesn't help with the text areas themselves.
I'll add the second two to our list. Thanks for passing them along. Watch for them in future versions.
Todd