Home » Forums » screen-scraper » screen-scraper support for licensed users

scraping next link pages (navigation problem)

I am working on a project to scrape the data automatically.

I am scraping a website called "www.uspto.gov".

These are the two links:

http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&u=%2Fnetahtml%2FPTO%2Fsearch-adv.htm&r=0&f=S&l=50&d=PTXT&OS=%22social+networking%22&RS=%22social+networking%22&Query=%22social+networking%22&TD=6908&Srch1=%22social+networking%22&NextList2=Next+50+Hits

http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&u=%2Fnetahtml%2FPTO%2Fsearch-adv.htm&r=0&f=S&l=50&d=PTXT&OS=%22social+networking%22&RS=%22social+networking%22&Query=%22social+networking%22&TD=6908&Srch1=%22social+networking%22&NextList3=Next+50+Hits

Now we can see from these urls that there is no direct parameter for next page like "page = 2". Instead of that I need to take the whole url into variable and then somehow change the parameters which are called as NextList2 and NextList3.

I tried a lot of different codes but nothing worked out.

Can you help me by providing the solution to this problem?

‹ Ignoring html tags like <b></b> while scraping Scaping confusing Ajax? site ›
patela7014 on 09/16/2014 at 1:13 pm

See the attached scraping

See the attached scraping session for an example.

 jason on 09/17/2014 at 9:35 am

Thank you so much. Need one more help.

Thank you so much sir for showing me a path towards success of my project. Actually I am new to screen-scraper so I keep asking some stupid questions too.

Now in this iteration of pages, I want to scrape all the details of the patent for all pages. Is it possible that I can solve it in any easy way?

 patela7014 on 09/21/2014 at 12:35 am

Patel, I've seen your

Patel,

I've seen your questions on StackOverflow also, and it seems that you might be missing some of the basics. Have you need through the tutorials? If you're interested in engaging my people to build it for you, email me and we can discuss the specs/costs.

 jason on 09/22/2014 at 7:42 am

Please help me by giving tips

Thank you so much for the help you have provided so far. Actually I am a student and I am learning screen-scraper. I just need tips from you so that I can achieve my goals of automatic scraping details from all patents with navigation.

I am done with the page navigation. Now there are thousands of patents for each keyword searched. And so there are multiple pages too. So I need to scrape the details of each patents with the working navigation. Can you give me simple tips to do so?

 patela7014 on 09/23/2014 at 9:27 am

Pagination like this is not

Pagination like this is not uncommon. What I would do is extract the total results from:

Hits 51 through 100 out of 6908

And make a script to:
total = Integer.parseInt(dataRecord.get("TOTAL"));
perPage = 100
pages = total/perPage;
if (tota%perPage>0)
    pages++;

for (i=2; i<=pages; i++)
{
    param = "NextList" + String.valueOf(i);
    session.log("Scraping " + param);
    session.setv("NEXT_PARAM", param);
    session.scrapeFile("Next search results");
}

 jason on 09/17/2014 at 7:02 am