CSV query results from site
A site I am scraping only produces results in a CSV document. Screen Scraper handles the results, but I can't seem to figure out a good way to parse the CSV results in an effective manner. My primary problem is that I'm not getting line breaks.
If I could at least use a session variable to grab everything between the body tags in the below sample scrape, I could then parse the content later. But the results I'm getting don't seem to be finding a character I can use to create a line break.
Any suggestions?
____________________________________________________
HTTP/1.1 200 OK
content-disposition: filename=SearchResults.csv
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Set-Cookie: ezSessionID=469n9nlpc7h9b;Path=/dbsight
Content-Type: application/vnd.ms-excel;charset=ISO-8859-1
Server: Jetty/5.1.4 (Linux/2.6.9-42.ELsmp amd64 java/1.5.0
Content-Length: 3517
Date: Fri, 04 Feb 2011 20:41:04 GMT
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
Contract ID,Award/IDV Type,Vendor Name,Contracting Agency,Date Signed,Action Obligation ($),Reference IDV,Contracting Office Name,NAICS,NAICS Description,PSC,PSC Description,Vendor City,Vendor DUNS,Vendor State,Vendor ZIP Code, "TOS10F0190001","Delivery Order","UNDISCLOSED INC","DEPARTMENTAL OFFICES","Sep 14, 2010","$651,210.44","TOS10F019","TEOAF BRANCH","561410","DOCUMENT PREPARATION SERVICES","R499","OTHER PROFESSIONAL SERVICES","LOWELL","1","AR","727459632", "TOS10F0190002","Delivery Order","CENTRAL RESEARCH INC","DEPARTMENTAL OFFICES","Sep 15, 2010","$202,639.14","TOS10F019","TEOAF BRANCH","561410","DOCUMENT PREPARATION SERVICES","R499","OTHER PROFESSIONAL SERVICES","LOWELL","1","AR","727459632", "GS23F0320P","FSS","CENTRAL RESEARCH INCORPORATED","FEDERAL ACQUISITION SERVICE","Feb 8, 2008","$0.00",,"GSA/FSS SERVICE CONTRACT DIVISION","541211","OFFICES OF CERTIFIED PUBLIC ACCOUNTANTS","R708","PUBLIC RELATIONS SERVICES","LOWELL","1","AR","727459632", "TOS10F0190005","Delivery Order","CENTRAL RESEARCH INC","DEPARTMENTAL OFFICES","Sep 29, 2010","$292,295.47","TOS10F019","TEOAF BRANCH","561410","DOCUMENT PREPARATION SERVICES","R499","OTHER PROFESSIONAL SERVICES","LOWELL","1","AR","727459632", "TOS10F0190006","Delivery Order","CENTRAL RESEARCH INC","DEPARTMENTAL OFFICES","Sep 30, 2010","$170,000.00","TOS10F019","TEOAF BRANCH","561410","DOCUMENT PREPARATION SERVICES","R499","OTHER PROFESSIONAL SERVICES","LOWELL","1","AR","727459632", "GS23F0320P","FSS","CENTRAL RESEARCH INCORPORATED","FEDERAL ACQUISITION SERVICE","Sep 9, 2009","$0.00",,"GSA/FSS SERVICE CONTRACT DIVISION","541211","OFFICES OF CERTIFIED PUBLIC ACCOUNTANTS","R708","PUBLIC RELATIONS SERVICES","LOWELL","1","AR","727459632", "TOS10F019","IDC","CENTRAL RESEARCH INC","DEPARTMENTAL OFFICES","Sep 3, 2010","$0.00",,"TEOAF BRANCH","561410","DOCUMENT PREPARATION SERVICES","R499","OTHER PROFESSIONAL SERVICES","LOWELL","1","AR","727459632", "TOS10F0190001","Delivery Order","CENTRAL RESEARCH INC","DEPARTMENTAL OFFICES","Sep 15, 2010","$0.00","TOS10F019","TEOAF BRANCH","561410","DOCUMENT PREPARATION SERVICES","R499","OTHER PROFESSIONAL SERVICES","LOWELL","1","AR","727459632", "TOS10F0190004","Delivery Order","CENTRAL RESEARCH INC","DEPARTMENTAL OFFICES","Sep 21, 2010","$162,802.61","TOS10F019","TEOAF BRANCH","561410","DOCUMENT PREPARATION SERVICES","R499","OTHER PROFESSIONAL SERVICES","LOWELL","1","AR","727459632", "DTDTMA2C10025","Definitive Contract","CENTRAL RESEARCH INC","MARITIME ADMINISTRATION","Sep 22, 2010","$332,132.00",,"DEPT OF TRANS/MARITIME ADMINISTRATION","561410","DOCUMENT PREPARATION SERVICES","R499","OTHER PROFESSIONAL SERVICES","LOWELL","1","AR","727459632", "TOS10F0190003","Delivery Order","CENTRAL RESEARCH INC","DEPARTMENTAL OFFICES","Sep 17, 2010","$0.00","TOS10F019","TEOAF BRANCH","561410","DOCUMENT PREPARATION SERVICES","R499","OTHER PROFESSIONAL SERVICES","LOWELL","1","AR","727459632", "TOS10F0190001","Delivery Order","CENTRAL RESEARCH INC","DEPARTMENTAL OFFICES","Jan 18, 2011","$0.00","TOS10F019","TEOAF BRANCH","561410","DOCUMENT PREPARATION SERVICES","R499","OTHER PROFESSIONAL SERVICES","LOWELL","1","AR","727459632", "GS23F0320P","FSS","CENTRAL RESEARCH INCORPORATED","FEDERAL ACQUISITION SERVICE","Nov 15, 2010","$0.00",,"GSA/FSS SERVICE CONTRACT DIVISION","541211","OFFICES OF CERTIFIED PUBLIC ACCOUNTANTS","R708","PUBLIC RELATIONS SERVICES","LOWELL","1","AR","727459632",
In your place, I would use
In your place, I would use session.downloadFile to save it to the disk, then use a script (like we have in on the samples page) to parse it.