Broken pipe
Hi,
Screen-scraper just stops scraping without any clue. We call screen-scraper using a php script. Execution time is set to a high value and memory is also set to a high value in php.ini. I see a lot of broken pipe errors in screen-scraper's error.log. Screen-scraper is run thourgh tor + privoxy. Any ideas ?
---------------------------------------------------------- error.log -----------
An IOException occurred. The message was: Broken pipe
java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at sun.nio.cs.StreamEncoder$CharsetSE.writeBytes(StreamEncoder.java:336)
at sun.nio.cs.StreamEncoder$CharsetSE.implFlushBuffer(StreamEncoder.java:404)
at sun.nio.cs.StreamEncoder$CharsetSE.implFlush(StreamEncoder.java:408)
at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:152)
at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:213)
at java.io.BufferedWriter.flush(BufferedWriter.java:236)
at com.screenscraper.scraper.RemoteScrapingSessionHandler.sendToClient(RemoteScrapingSessio
nHandler.java:639)
at com.screenscraper.scraper.RemoteScrapingSessionHandler.run(RemoteScrapingSessionHandler.
java:227)
----------------------------------------------------------------------------------
Broken pipe
Hi,
Just to narrow down the problem a bit more, if you go through the steps described in our third tutorial (here) do you get the same problem? The second tutorial presents a very simple case of what you're trying to do, so hopefully it helps determine more precisely what the issue is.
Thanks,
Todd
Broken pipe
Hi,
I still get the same 'Broken Pipe' error. I have tried everything you said.
Am pasting the php scipt we use below.
----------------------------------------------------------------scrape.php---
require( 'remote_scraping_session.php' );
include("../config.php");
$session = new RemoteScrapingSession;
$session->initialize( "xyz.co.uk, qwerty ", SCRAPPER_HOST, SCRAPPER_HOST_PORT );
$fetcchsitelink = "select xyz,abc,dfg,wer from siteconfig where status=1";
$res = mysql_query($fetcchsitelink);
while($row = mysql_fetch_assoc($res))
$mylinksArray[] = $row;
foreach($mylinksArray as $linkArray){
echo $linkArray['subLocationId']."\n";
$session->setVariable( "XYZ", $linkArray['xyz']);
$session->setVariable( "ABC", '"'.$linkArray['abc'].'"');
$session->setVariable( "DFG", $linkArray['dfg']);
$session->setVariable( "WER", $linkArray['wer']);
$session->setVariable( "PAGE", "1" );
$session->scrape();
print_r($data_set = $session->getVariable( "DETAILS" ));
}
if( $session->isError() )
{
echo "An error occurred: " . $session->getErrorMessage() . "
";
exit();
}
$session->setBufferSize( 64000 );
$data_set = $session->getVariable( "DETAILS" );
echo "Finished Scrapping Session ";
?>
----------------------------------------------------------------scrape.php---
I have checked mysql config and it seems to be ok. Also we are not writing to DB, just fetching data from it. Any ideas ?
Broken pipe
Hi,
I can think of a couple of possibilities offhand. The error message indicates that screen-scraper is trying to send data back to the calling program, but the calling program has closed the socket. Is there a chance your PHP script could be timing out? I believe there is both a socket time out and a script execution time out setting. You'll want to be sure to increase those values, depending on how long your scrapes take to run. Another less likely possibility may be that you're trying to send a very large data set from screen-scraper to your PHP script, but the buffer size isn't large enough. You can set the buffer size via the 'setBufferSize' method on the RemoteScrapingSession PHP class.
Feel free to post a reply if neither of those options seems to help.
Kind regards,
Todd Wilson