how to iterate different URL pages

Hi Everybody,

Please I would ask help you about the following request. I'm really new in scraping so probably my request is really simple but I don't succeed to resolve this point.

I'm building a web site (www.mysite.com) and I want to verify the behaviour if somebody tryes to do scraping.
For example I want to test the behaviour when somebody try to scrape my database requiring a list of product (for example pharmacy in Frankfurt).

Requiring pharmacy in Frankfurt you receive a response regarding the first 20 pharmacy and also how many pharmacy are in Frankfurt (and the number of the pages).
So, let's suppose that you can find 200 pharmacy (10 pages, in every page 20 pharmacy).

The URL format looks like this one: "http://www.mysite.com/pharmacy/Frankfurt/p-X", where "X" is the number of the page you're visiting.

So if you visit for example the 4th page the URL format is "http://www.mysite.com/pharmacy/Frankfurt/p-4".

I would iterate for the 20 pages of this site (I know the number of the pages because I can recover this value from the first page), but I don't know how because the number of the page is not a variable and if I create a scrapefile specifying in the URL the following information "http://www.mysite.com/pharmacy/Frankfurt/p-~#Next#~" (Next is the value I recovered from the first page regarding the following page with ~@Next@~, information recovered form the HTTP response) the script doesn't work (the script doesn't recognize the "Next" value and so the URL I see is "http://www.mysite.com/pharmacy/Frankfurt/p-").

Please have you any suggestion about how can I resolve this point?

Hoping my explaination is clear, thanks in advance!
Best Regards

MarFilAur

If you haven't gone through

If you haven't gone through the tutorials yet, I suggest you do so.

Once that is done we have a collection of examples and scripts for page iteration that you can look at, and it should get you started.