storing session variable not working, problem with tokens?
I'm a NEWBIE tring the following scripts, following your tutorials but on my text file, i'm getting just nulls, plus when i create an extractor pattern "edit token" is gray, any help is much appreciate
here are scripts, extractor patterns and logs
SCRIPTS
TITOLO_SPIAGGE
session.log( "SCRIVO IL TESTO." );
// Open up the file to be appended to.
out = new FileWriter( "testo_spiagge.txt", true );
// Write out the data to the file.
out.write(session.getVariable("TITOLO_SPIAGGE")+ "," );
out.write("\n" );
// Close up the file.
out.close();
TESTO_SPIAGGE
session.log( "SCRIVO IL TESTO." );
// Open up the file to be appended to.
out = new FileWriter( "testo_spiagge.txt", true );
// Write out the data to the file.
out.write(session.getVariable("TESTO_SPIAGGE")+ "," );
out.write("\n" );
// Close up the file.
out.close();
NEXT
// The variable "next" is the result of attempting to match for the word "Next"
// Check to see if we found the word or phrase that flags a "Next" page
if (session.getVariable("NEXT") != null)
{
// Retrieve the page number of the page just scraped
currentPage = session.getVariable("PAGE");
if (currentPage == null)
currentPage = 1;
else
currentPage = (Integer.parseInt(currentPage).toString());
// write out the page number of the page just scraped
session.log("Last page was: " + currentPage);
// Increment the page number
currentPage++;
// write out the page number of the next page to be scraped
session.log("Next page is: " + currentPage);
// Set the "PAGE" variable with the incremented page number
session.setVariable("PAGE", currentPage);
// Clear the "NEXT" variable so that the next page is allowed to find it's own value for "NEXT"
session.setVariable("NEXT", null);
// Scrape the next page
session.scrapeFile("spiagge");
}
EXTRACTOR PATTERNS
~@TESTO_SPIAGGIA@~
~@NEXT@~ standing for: "class="blu bold interlinea28">Spiaggia Successiva
"
Starting scraper.
Running scraping session: spiagge
Processing scripts before scraping session begins.
Scraping file: "spiagge"
spiagge: Preliminary URL: http://www.unionesarda.it/collane/spiagge/scheda_spiaggia.aspx
spiagge: Using strict mode.
spiagge: Resolved URL: http://www.unionesarda.it/collane/spiagge/scheda_spiaggia.aspx?sp=13
spiagge: Sending request.
spiagge: Processing scripts before all pattern applications.
spiagge: Applying extractor pattern: TITOLO_SPIAGGIA
spiagge: Extracting data for pattern "TITOLO_SPIAGGIA"
spiagge: The following data elements were found:
TITOLO_SPIAGGIA--DataRecord 0:
TITOLO_SPIAGGIA=Spiaggia La Pelosa - Stintino - Isola Asinara 0 - Spiagge della Sardegna - L'Unione Sarda
Storing this value in a session variable.
spiagge: Processing scripts after a pattern application.
spiagge: Processing scripts once if pattern matches.
spiagge: Processing scripts after all pattern applications.
Processing script: "TITOLO SPIAGGE"
SCRIVO IL TESTO.
spiagge: Processing scripts before all pattern applications.
spiagge: Applying extractor pattern: TESTO_SPIAGGIA
spiagge: Extracting data for pattern "TESTO_SPIAGGIA"
spiagge: The following data elements were found:
TESTO_SPIAGGIA--DataRecord 0:
TESTO_SPIAGGIA=Stintino (SS) - Si trova a nord di Stintino, davanti al Villaggio Capo Falcone. Deviare a destra, sulla provinciale Porto Torres-Stintino-Capo Falcone, se si proviene dal centro abitato; proseguire diritti se si proviene da Porto Torres, cio da sud. In pochi minuti si raggiunge la stupenda spiaggia; parcheggio a pagamento nella stagione estiva e possibilitˆ di limitazione allաccesso delle auto. 頲itenuto il pi bellաrenile della zona ed uno dei pi incantevoli dellՉsola, per le trasparenze dellաcqua e per la caratteristica Torre Pelosa che sorge sul lato nord, mentre a nord-est il panorama comprende lթsola Piana e lթsola Asinara. Di sabbia finissima, quasi impalpabile, con le morbide formazioni dunali mirabilmente disegnate dal vento, contrasta con lo scuro colore delle rocce scistose circostanti della penisola di Capo Falcone, che si trova subito alle spalle, con rocce alte 110 metri a picco sul mare. Disponibilitˆ di bar, alberghi, ristoranti, negozi e servizi vari. Natura in parte intatta e selvaggia nei dintorni; dimensioni piccole; mare azzurro, poco profondo; affollatissima in alta stagione (assolutamente da evitare a luglio e agosto); pulita, con scarsa presenza di posidonie.
Storing this value in a session variable.
spiagge: Processing scripts after a pattern application.
spiagge: Processing scripts once if pattern matches.
spiagge: Processing scripts after all pattern applications.
Processing script: "TESTO SPIAGGE"
SCRIVO IL TESTO.
spiagge: Processing scripts before all pattern applications.
spiagge: Applying extractor pattern: NEXT
spiagge: Extracting data for pattern "NEXT"
spiagge: The following data elements were found:
NEXT--DataRecord 0:
NEXT=class="blu bold interlinea28">Spiaggia Successiva
Storing this value in a session variable.
spiagge: Processing scripts after a pattern application.
spiagge: Processing scripts once if pattern matches.
spiagge: Processing scripts after all pattern applications.
Processing script: "Next"
Last page was: 1
Next page is: 2
Scraping file: "spiagge"
spiagge: Preliminary URL: http://www.unionesarda.it/collane/spiagge/scheda_spiaggia.aspx
spiagge: Using strict mode.
spiagge: Resolved URL: http://www.unionesarda.it/collane/spiagge/scheda_spiaggia.aspx?sp=13
spiagge: Sending request.
spiagge: Processing scripts before all pattern applications.
spiagge: Applying extractor pattern: TITOLO_SPIAGGIA
spiagge: Extracting data for pattern "TITOLO_SPIAGGIA"
spiagge: The following data elements were found:
TITOLO_SPIAGGIA--DataRecord 0:
TITOLO_SPIAGGIA=Spiaggia La Pelosa - Stintino - Isola Asinara 0 - Spiagge della Sardegna - L'Unione Sarda
Storing this value in a session variable.
spiagge: Processing scripts after a pattern application.
spiagge: Processing scripts once if pattern matches.
spiagge: Processing scripts after all pattern applications.
Processing script: "TITOLO SPIAGGE"
SCRIVO IL TESTO.
spiagge: Processing scripts before all pattern applications.
spiagge: Applying extractor pattern: TESTO_SPIAGGIA
spiagge: Extracting data for pattern "TESTO_SPIAGGIA"
spiagge: The following data elements were found:
TESTO_SPIAGGIA--DataRecord 0:
TESTO_SPIAGGIA=Stintino (SS) - Si trova a nord di Stintino, davanti al Villaggio Capo Falcone. Deviare a destra, sulla provinciale Porto Torres-Stintino-Capo Falcone, se si proviene dal centro abitato; proseguire diritti se si proviene da Porto Torres, cio da sud. In pochi minuti si raggiunge la stupenda spiaggia; parcheggio a pagamento nella stagione estiva e possibilitˆ di limitazione allաccesso delle auto. 頲itenuto il pi bellաrenile della zona ed uno dei pi incantevoli dellՉsola, per le trasparenze dellաcqua e per la caratteristica Torre Pelosa che sorge sul lato nord, mentre a nord-est il panorama comprende lթsola Piana e lթsola Asinara. Di sabbia finissima, quasi impalpabile, con le morbide formazioni dunali mirabilmente disegnate dal vento, contrasta con lo scuro colore delle rocce scistose circostanti della penisola di Capo Falcone, che si trova subito alle spalle, con rocce alte 110 metri a picco sul mare. Disponibilitˆ di bar, alberghi, ristoranti, negozi e servizi vari. Natura in parte intatta e selvaggia nei dintorni; dimensioni piccole; mare azzurro, poco profondo; affollatissima in alta stagione (assolutamente da evitare a luglio e agosto); pulita, con scarsa presenza di posidonie.
Storing this value in a session variable.
spiagge: Processing scripts after a pattern application.
spiagge: Processing scripts once if pattern matches.
spiagge: Processing scripts after all pattern applications.
Processing script: "TESTO SPIAGGE"
SCRIVO IL TESTO.
spiagge: Processing scripts before all pattern applications.
spiagge: Applying extractor pattern: NEXT
spiagge: Extracting data for pattern "NEXT"
spiagge: The following data elements were found:
NEXT--DataRecord 0:
NEXT=class="blu bold interlinea28">Spiaggia Successiva
Storing this value in a session variable.
spiagge: Processing scripts after a pattern application.
spiagge: Processing scripts once if pattern matches.
spiagge: Processing scripts after all pattern applications.
Processing script: "Next"
An error occurred while processing the script: Next
The error message was: class bsh.EvalError (line 10): Integer .parseInt ( currentPage ) -- Error in method invocation: Static method parseInt( java.lang.Integer ) not found in class'java.lang.Integer'
Processing scripts after scraping session has ended.
Scraping session "spiagge" finished.
thank you in advance
There might be a couple
There might be a couple things going on here, but there's a lot of stuff in the post so it's hard to tell. What I see first is an issue in casting your page numbers.
// Check to see if we found the word or phrase that flags a "Next" page
if (session.getVariable("NEXT")!=null)
{
// Retrieve the page number of the page just scraped
currentPage = Integer.parseInt(session.getVariable("PAGE"));
// Clear the flag so it won't linger
session.setv("NEXT", null);
}
if (currentPage == null)
currentPage = 1;
// write out the page number of the page just scraped
session.log("Last page was: " + currentPage);
// Increment the page number (you can't increment a String)
currentPage++;
// write out the page number of the next page to be scraped
session.log("Next page is: " + currentPage);
// Set the "PAGE" variable with the incremented page number
session.setVariable("PAGE", currentPage);
// Scrape the next page
session.scrapeFile("spiagge");
extractor patterns
thank you jason for your help, have you any idea why extractor patterns are working but i get just nulls? why my edit token button is grey?
I've tried your script and now i'm getting this error
Processing script: "Next"
An error occurred while processing the script: Next
The error message was: NumberFormatException (line 35): null-- Method Invocation Integer.parseInt
Processing scripts after scraping session has ended.
Scraping session "spiagge" finished.
thank you again
The parseInt is looking for a
The parseInt is looking for a session variable "PAGE" with a numeric value, and you get this error because it's not set.
I just tried to copy the logic out of your script, so I don't really recall from whence that came.
extractor partners
ok, my main problem now is that I can't get any data, i get only nulls and i just copied your examples!! why is that?
Are you scraping values PAGE
Are you scraping values PAGE and NEXT? If so, did you go into each token properties and set as a session variable?
session variables
I'm scraping TITOLO_SPIAGGIA and TESTO_SPIAGGIA, I can't set a session variable as you explain in your tutorials cause right clicking i get a gray "edit token" button, so i did it writing manually the variable in this way "tag with unique id" ~@TESTO_SPIAGGIA@~ "closing tag", then double clicking and storing as variable. I've tested the extractor pattern and it seems to work but when i run my scrapign session i get only nulls!!!