Read from Database, then set SessionVariable
At last I got my scrape running and dumping the results into my database!! Hurrah! But now as part of my two part scrape I need to read records from my database.
I have done the intital scrape which was to retreive a list of brands (Brand) and the URL to the Brands page (BrandURL).
My plan is to read out the next BrandURL to scrape, Set a Session with it, then use the session to remove that Brand record from the table. Pretty sure a Delete will be much the same as an insert, BUT how do I set a session from a select query? (mysqlstring="SELECT Brand FROM `Brands` LIMIT 0 , 1";)
Thanks in advance for your assistance
Stu :-/
I have a scrape here where I
I have a scrape here where I did just that. I used the SQLDataManager to connect to the database, and here's the script where I query and scrape:
batchSize = 50;
session.setv("BATCH_SIZE", batchSize);
// Get datamanager
dm = session.getv("_DM");
conn = dm.getConnection();
// Get batch of businesses to scrape
try
{
getBatch = "SELECT `url`"
+ " FROM `urls`"
+ " WHERE `scraped` = 0"
+ " ORDER BY `updated` ASC"
+ " LIMIT ?";
PreparedStatement batch = conn.prepareStatement(getBatch);
batch.setInt(1, batchSize);
ResultSet resultSet = batch.executeQuery();
while (resultSet.next())
{
url = resultSet.getString("url");
// Mark the businesses as started
setClaim = "UPDATE `urls`"
+ " SET `scraped` = 2"
+ " WHERE `url` = ?";
PreparedStatement claim = conn.prepareStatement(setClaim);
claim.setString(1, url);
claim.executeUpdate();
// To validate results
session.setv("GOT_RESULTS", false);
// Scrape
session.setv("START", 0);
session.setv("URL", url);
session.scrapeFile("Details");
if (session.getv("GOT_RESULTS"))
{
// Mark the businesses as done
setClaim = "UPDATE `urls`"
+ " SET `scraped` = 3"
+ " WHERE `url` = ?";
PreparedStatement claim = conn.prepareStatement(setClaim);
claim.setString(1, url);
claim.executeUpdate();
}
}
}
finally
{
conn.close();
}