Page incrementing by 20

You've almost got it right, except that setting NEXTR to 2 in an initialize script won't do anything for you, because the script that we've posted here (in its variants) only runs the scrapeableFile "Query result - temp for testing" *after* overwritting NEXTR with '(currentPage - 1) * 20 + 1. You could put the word 'moo' in there to prime it, but it would be overwritten by that for loop before the scrapeableFile ever runs.

POST parameters will actually *not* show up in as part of the URL. POST is not part of a URI request, but rather background information for use by the server receiving it. GET parameters *are* a part of the URI request, because their intended purpose is very different from that of POST. (Don't ask a .NET web designer that question, because he'll lie to your face about how POST should be used at all times :P) The purpose of GET and POST are very different, although they are effectively both used simply as variables to send to the target page.

So as for the issue at hand... I'm running a dummy scrape modeled after yours (with that exact script, copied and pasted, except that I simply log the value of "NEXTR" right after that 'scrapeFile' command. (had to do '.toString()' on it, since 'currentPage' is an integer). I just made it post out to a local page of mine with a NEXTR variable, and it resolves just fine... here's my entire log:

Starting scraper.
Running scraping session: ~~Dummy
Processing scripts before scraping session begins.
Processing script: "~~init"
Scraping file: "Query result - temp for testing"
Query result - temp for testing: Preliminary URL: http://localhost:8080/johan.php
Query result - temp for testing: Using strict mode.
Query result - temp for testing: POST data: NEXTR=1
Query result - temp for testing: Resolved URL: http://localhost:8080/johan.php
Query result - temp for testing: Sending request.
Query result - temp for testing: Extracting data for pattern "detect 'next page' link"
Query result - temp for testing: The following data elements were found:
detect 'next page' link--DataRecord 0:
NEXT_PAGE=

Storing this value in a session variable.
1
Scraping file: "Query result - temp for testing"
Query result - temp for testing: Preliminary URL: http://localhost:8080/johan.php
Query result - temp for testing: Using strict mode.
Query result - temp for testing: POST data: NEXTR=21
Query result - temp for testing: Resolved URL: http://localhost:8080/johan.php
Query result - temp for testing: Sending request.
Query result - temp for testing: Extracting data for pattern "detect 'next page' link"
Query result - temp for testing: The following data elements were found:
detect 'next page' link--DataRecord 0:
NEXT_PAGE=
Storing this value in a session variable.
21
Scraping file: "Query result - temp for testing"
Query result - temp for testing: Preliminary URL: http://localhost:8080/johan.php
Query result - temp for testing: Using strict mode.
Query result - temp for testing: POST data: NEXTR=41
Query result - temp for testing: Resolved URL: http://localhost:8080/johan.php
Query result - temp for testing: Sending request.
Query result - temp for testing: Extracting data for pattern "detect 'next page' link"
Query result - temp for testing: The following data elements were found:
detect 'next page' link--DataRecord 0:
NEXT_PAGE=
Storing this value in a session variable.
41
Scraping file: "Query result - temp for testing"
Query result - temp for testing: Preliminary URL: http://localhost:8080/johan.php
Query result - temp for testing: Using strict mode.
Query result - temp for testing: POST data: NEXTR=61
Query result - temp for testing: Resolved URL: http://localhost:8080/johan.php
Query result - temp for testing: Sending request.
Query result - temp for testing: Extracting data for pattern "detect 'next page' link"
Query result - temp for testing: The following data elements were found:
detect 'next page' link--DataRecord 0:
NEXT_PAGE=
Storing this value in a session variable.
61
Scraping file: "Query result - temp for testing"
Query result - temp for testing: Preliminary URL: http://localhost:8080/johan.php
Query result - temp for testing: Using strict mode.
Query result - temp for testing: POST data: NEXTR=81
Query result - temp for testing: Resolved URL: http://localhost:8080/johan.php
Query result - temp for testing: Sending request.
Query result - temp for testing: Extracting data for pattern "detect 'next page' link"
Query result - temp for testing: The following data elements were found:
detect 'next page' link--DataRecord 0:
NEXT_PAGE=
Storing this value in a session variable.
81
Scraping file: "Query result - temp for testing"
Query result - temp for testing: Preliminary URL: http://localhost:8080/johan.php
Query result - temp for testing: Using strict mode.
Query result - temp for testing: POST data: NEXTR=101
Query result - temp for testing: Resolved URL: http://localhost:8080/johan.php
Query result - temp for testing: Sending request.
Query result - temp for testing: Extracting data for pattern "detect 'next page' link"
Query result - temp for testing: The following data elements were found:
detect 'next page' link--DataRecord 0:
NEXT_PAGE=
Storing this value in a session variable.
101
Scraping session "~~Dummy" finished.

I made it break in the middle. I made it always detect a next page by just making it match a '<ul>' tag, so that it just kept on going and going until I made it quit. As you can see, it's posting the data correctly, incrementing the value by 20 each time, starting from 1.

I'm not sure what could be wrong with your setup, since that script performs completely as expected. Semantically speaking, though, you could theoretically be NOT matching your 'NEXT_PAGE' variable, and so it just moves on to the next category or something, which makes it reset to '1' every time that script is executed. Does that make sense? Are you iterating 'categories' of sorts, where each 'category' needs to deal with pages?

It just feels like either *something* is not getting saved as a session variable, or you're not matching a 'NEXT_PAGE' indicator, or nothing is actually wrong and you're just seeing it attempting the first page for every category, and there simply *isn't* a next page for the categories you're seeing scraped. (In that latter case, you should eventually see it hit some 'NEXT_PAGE' action, once it comes upon a category that does in fact have a next page.)

Is any of this helping? If you can, post a log (or the relevant segment thereof) from the scrape. You can copy it out of the window in screen-scraper. Please be sure to put "<code>" and "</code>" around the log that you post, so that any HTML in the log doesn't get interpreted as actual HTML.

Hope we can sort out what's going wrong!
Tim