Plugable Scraping
is it possible for an external java program to upload scrapefiles to the server?
I want to hold a set of scrapfiles in exported XML format in a database, then get a Java program to load the files onto the server , before getting the server to serve each request.
how feasible is that.
Plugable Scraping
Hi,
In order to completely remove a scraping session from a running server, you would need to make a SOAP request
http//www.screen-scraper.com/support/docs/soap_interface.php#removeScrapingSession(string)
If you overwrite the scraping session with another that has the same name, it will hot swap it in (i.e., the new version will get used the next time the scraping session is invoked).
Kind regards,
Todd
Plugable Scraping
Ok so Im acting like columbo and cos i always got one more question....
BUT
In terms of scalabilty how does one remove a scrapable session from the servers repository of sessions.
Can I overwrite it with the same name and does all of that thread goodness still apply?
Plugable Scraping
That's correct. The update won't occur until the next scraping session is initiated. Currently running scraping sessions will be unaffected.
Best wishes,
Todd
Plugable Scraping
Right! So can you confirm that any currently executing threads will not be affected by the update?
Plugable Scraping
Hi,
After you issue the "update" method, the next time that scraping session is invoked it will use the updated version.
Best wishes,
Todd
Plugable Scraping
update method looks like what im looking for, i want to scrape many different sites but the scrapefile names will be the same.
i guess its like defining a common interface and different implementations
one question whats happens if the scrapefile is currently running when you update?
Plugable Scraping
Hi,
Are you referring to full scraping sessions, or simply a scrapeable file within a scraping session? That is, do you want to be able to completely replace an existing scraping session on the server? Or do you simply want to have an existing scrapeable file go to a URL you would pull from your database?
In the former case you could either copy an exported scraping session to screen-scraper's "import" folder. You could also use the "update" method on our SOAP interface: here.
In the latter case, I would recommend setting a session variable with a URL from your database on a RemoteScrapingSession, then using something like this as the URL for your scrapeable file:
~#URL#~
Kind regards,
Todd Wilson