Home » Forums » screen-scraper » screen-scraper support for licensed users

Help with scraping website

Hello,
I'm trying to scrape website, but main structure is a bit complicated for me. Please help me to understand how should I set scraper.
The structure is:
-Search page (with pagination)
|-Product page (i need to scrape details)
|-Review tab on Product page i.e http://prod_link/reviews (also with pagination)

Main structure is pretty much the same like on YP website.
Please give me a tip how can i scrape this. Should i create 1 scrapping session and work with scrapeable files or should i create 2 scrapping session (1 for Search page and 1 for Review tab)

The main problem that I can't imaging how can i work with 2 page iterations.

Can you please provide schematically what should i do.

Really appreciate your help.

‹ Writing to CSV Multi-threading and performance ›
tellieno on 02/18/2011 at 10:43 am

Tellieno, Here is one

Tellieno,

Here is one approach you could take...

  1. Results (scrapeable file)
    1. Details link (extractor pattern) & Call Details scrapeable file (script)
      1. Details (scrapeable file)
        1. Reviews link (extractor pattern) & Call Reviews scrapeable file (script)
          1. Reviews (scrapeable file)
        2. Next reviews page (extractor pattern) & Call Reviews scrapeable file (script)
  2. Next results page (extractor pattern) & Call Results scrapeable file (script)

Because screen-scraper works on the principle of recursion as each level runs out of data to scrape your scraping session will automatically roll back up one level above.

 swilsonmc on 02/18/2011 at 2:24 pm