Home » Forums » screen-scraper » screen-scraper public support

Using multiple search terms (extension of Tutorial 7)

Hi,

I want to extend the features Tutorial 7 and wanted to know if it was possible.

The tutorial shows one search term per line.

Is there a way to add a variable to it that will be used in the outputting of a file? Let me explain.

I want to scrape a website with multiple search terms, but I also want to record the context of the search terms. For example:

the search file would be set up something like this:

IMCL, Imclone
Hsp90, Imclone
MEK, Array
MEK, AstraZeneca
AZD, AstraZeneca

for each hit on IMCL, I want to insert Imclone into the row of data. For each hit on MEK, I would want Array inserted the first time the search is run, and AstraZeneca inserted the 2nd time the search is run.

So the output might be something like this:

IMCL, Imclone, IMCL-383, Sunday, June 1
IMCL, Imclone, IMCL-212, Monday, June 2
Hsp90, Imclone, Hsp90 inhibitor, Sunday, June 1
MEK, Array, MEK inhibitor, Tuesday, June 3
MEK, AstraZeneca, MEK inhibitor, Tuesday, June 3
AZD, AstraZeneca, AZD6244, Monday, June 2

So basically, I want to read 2 variables from my search-terms.txt file and output both of them with the search results I get from scraping.

thanks!

Bhavesh

‹ Ignore HTML tags removes text also Writing the Search Term to a file? ›
veshman on 04/25/2007 at 9:31 am

works beautifully

Noelle,

The code works beautifully. Thanks for your help!

Bhavesh

 veshman on 04/25/2007 at 1:35 pm

duh

Noelle,

duh....!

I didn't capitalize String in the first line.

 veshman on 04/25/2007 at 12:36 pm

didn't work

Noelle,

I changed the code as you suggested, but I got the following error

Processing script: "ASCO Read Search terms"
ASCO 2007: An error occurred while processing the script: ASCO Read Search terms
ASCO 2007: The error message was: Typed variable declaration : Class: string not found in namespace : at Line: 12.
Processing scripts after scraping session has ended.

string searchTerm;
while( ( searchTerm = buffRead.readLine() )!=null)
{

  String[] st = searchTerm.split(",");
  String searchPhrase = st[0];
  String comprelevance = st[1];

  // Set a session variable corresponding to the search term.
  session.setVariable( "SEARCH", searchPhrase );
  session.setVariable( "COMPANY", comprelevance );

        // Remember we need to initialize the PAGE session variable, just

 veshman on 04/25/2007 at 12:28 pm

thanks

Noelle,

Thanks. I am just learning how to do the split as we speak. I think I've got it coded right. Here is what I have now:

string searchTerm;
while( ( searchTerm = buffRead.readLine() )!=null)
{
  String st[] = searchTerm.split(",");
  String searchPhrase = st[0];
  String comprelevance = st[1];
  // Set a session variable corresponding to the search term.
  session.setVariable( "SEARCH", searchPhrase );
  session.setVariable( "COMPANY", comprelevance );

        // Remember we need to initialize the PAGE session variable, just
        // in case we need to iterate through multiple pages of search results.
        // We begin at page 1 for each search.
        session.setVariable( "PAGE", "1" );

So far, even before modifyng this code, I have been unable to save searchTerm to the output file, as discussed in my other post.

Bhavesh

 veshman on 04/25/2007 at 11:59 am