Importing a URL list to create scrapeable files

Hi,

We actually do quite a bit of what you're describing, only using more manual methods, like those I referred to previously. The difficulty is that sites vary so dramatically in how they present data and such. If what you're proposing is simply a way to interface with Excel, that could certainly be done, but that's the easy part ) The harder part is extracting the desired values from the page, saving them out, and potentially performing logic using the data. I'm unaware of any solution that automates this process much more than we do. That certainly doesn't mean that they don't exist, of course. For the time being, what you see is what you get, as far as our offering goes. I'll certainly give you suggestions a bit more thought, though. It's very possible we could be missing out on a big market simply because our tool doesn't automate enough of the process.

Kind regards,

Todd

Just one other thought--if you're grabbing those URL's from a separate web page, another option would be to scrape each URL individually, and set them each as session variables. This would be a similar approach to using the text file, but would be more dynamic given that you'd be able to draw the URL's directly from the source, rather than having to copy and paste them into the text file.

Best,

Todd

.

thanks todd.

I did not want to bother you with specifics, but maybe some greater detail shall be more useful as an example for people having a similar request in the future.

so: here is a URL list, a real case example:

http://hamptoninn.hilton.com/en/hp/hotels/index.jhtml?ctyhocn=YWGCAHX
http://www.hilton.com/en/hi/hotels/index.jhtml?ctyhocn=YWGWIHF
http://www.hilton.com/en/hi/hotels/index.jhtml?ctyhocn=STJHITW
http://hiltongardeninn.hilton.com/en/gi/hotels/index.jhtml?ctyhocn=YHZHGGI
http://homewoodsuites.hilton.com/en/hw/hotels/index.jhtml?ctyhocn=ONTBUHW
http://hiltongardeninn.hilton.com/en/gi/hotels/index.jhtml?ctyhocn=YYZBUGI
http://hiltongardeninn.hilton.com/en/gi/hotels/index.jhtml?ctyhocn=YYZCMGI
http://www.hilton.com/en/hi/hotels/index.jhtml?ctyhocn=YXULOHF

I realize we meant exactly the same thing, when referring to tutorial 7 - i.e. creating a page with all the links and scraping out of this, according to the procedure in tutorial 7. I am glad there is a solution, even though I would hope for a more elegant and simple one - which, as I mentioned before, would be to read URLs directly from a spreadsheet column, or to read URLs imported manually in the script data from a spreadsheet column. because to do that with ex. 200,000 URLs (ex. HomeDepot) is messy, I believe.

but maybe there *is* some better solution available as the script stands now?

thanks,
heisje

.

.

many thanks Todd for replying to this post, much appreciated.
I hate to give the impression that I have not tried to give the solution by myself - and please be assured that this is not the case - I decided to ask for assistance in this forum only after many hours of trying to find the solution myself, or in previous forum-posts.

I had gone through all tutorials many times, including tutorial number 7.
the only hint surfacing (indirectly) from tutorial 7 is to create a page with all the required links, and scrape accordingly.

if this is what you imply, so far so good.
but I am talking about 40,000 links, 100,000 links - and so on.
if creating a page with 100,000 links is the solution, I would rather call for a more elegant one, like reading and importing from a spreadsheet column.

kindly confirm that what you implied as a solution, from tutorial 7, is creating a page with many thousands of links - or if you had something more elegant in mind which I could not discover.

many thanks in advance,
heisje

.