Badly formed HTTP header with file upload
I'm trying to scrape and ASP page basically to login and upload a CSV file.
I've got around all the hairy ASP stuff like setting the multitude of meaningless variables. I'm having trouble with the actual upload request.
If I capture the HTTP header with the Live HTTP Headers firefox extension or look at in the proxy server this is the correctly formed request:
/
POST /content/transactions_import.aspx HTTP/1.1
Host: estore.omsonline.net
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.9) Gecko/2009040821 Firefox/3.0.9
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: https://estore.omsonline.net/content/transactions_import.aspx
Cookie: ASP.NET_SessionId=ta2hdnap5ztmytnk4wee0faa; ShoppingBag-OMSBACKEND=788597258.20480.0000; [email protected]; OMSADMINGUID=3fd52bc6-ba09-4947-9ae4-ca6ff99b1da7
Content-Type: multipart/form-data; boundary=---------------------------19326494824544
Content-Length: 4853
-----------------------------19326494824544
Content-Disposition: form-data; name="__EVENTTARGET"
-----------------------------19326494824544
Content-Disposition: form-data; name="__EVENTARGUMENT"
-----------------------------19326494824544
Content-Disposition: form-data; name="__LASTFOCUS"
-----------------------------19326494824544
Content-Disposition: form-data; name="__VIEWSTATE"
/wEPaA8FDzhjYjk2YWVhM2E1MTk4NhgBBR5fX0NvbnRyb2xzUmVxdWlyZVBvc3RCYWNrS2V5X18WAQUOY3RsMDAkUmFkTWVudTGiSuqpu0gRp7eM3bvB2/9Mflv1Mw==
-----------------------------19326494824544
Content-Disposition: form-data; name="__EVENTVALIDATION"
/wEWEwKAx6K0DALxyffGAgLnye+cBQKH6dbtCAK8nNTYBwLMq6PbBgL40JWiCgLliJiDAgLi8LWEDQLgwOf2DwKVvp+tCgKu6ua7DgK6ntj5BgL/rb+qCwK+ntj5BgLejqnYDQLM/bSACAKe+vH1CwLa2Ya4BMlfhsWijIqMpVqxyYiEJiefhYE6
-----------------------------19326494824544
Content-Disposition: form-data; name="ctl00$ddlSwitchClients"
534493
-----------------------------19326494824544
Content-Disposition: form-data; name="ctl00_RadMenu1_ClientState"
-----------------------------19326494824544
Content-Disposition: form-data; name="ctl00$ContentPlaceHolder1$FileUpload1"; filename="blank1.csv"
Content-Type: text/plain
CSV TEXT GOES HERE
-----------------------------19326494824544
Content-Disposition: form-data; name="ctl00$ContentPlaceHolder1$btnSubmit"
Import Orders
-----------------------------19326494824544
Content-Disposition: form-data; name="ctl00$ContentPlaceHolder1$hdnPageID"
780
-----------------------------19326494824544
Content-Disposition: form-data; name="ctl00$tbReportName"
-----------------------------19326494824544
Content-Disposition: form-data; name="ctl00$ddlExportType"
excel
-----------------------------19326494824544
Content-Disposition: form-data; name="ctl00$ddlExportFrequency"
daily
-----------------------------19326494824544--
now when I actually run the scrape (I've set the "ctl00$ContentPlaceHolder1$FileUpload1" parameter to type file and pointed it to the same CSV file that I used with the proxy request) I get nothing in the 'request' or 'response' tab of the scrapeable file for that page. All the other pages are scraping fine and showing the correct content (which they wouldn't if the login hadn't worked).
The log show's that it's set to a multi part request and says that it's including the file contents but I can see from looking manually at the website that the file hasn't uploaded...
Since I can't see what's going on through the response tab I pointed the scraping session back through the proxy server so I could see the request... This is what it's showing:
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)
Content-Length: 0
Accept-Language: en-us,en;q=0.5
Referer: https://estore.omsonline.net/default.aspx
Cookie: ASP.NET_SessionId=hpgitx3e104cpk31mm2uvp45
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Host: estore.omsonline.net
Accept-Encoding: gzip
Content-Type: multipart/form-data; boundary=6DcdF2vJSm0ZLIJUEtAbE8ocr-CcBwl
The session log all looks ok (the blank parameters are supposed to be blank):
Processing script: "a12 - OMS import"
Copy of submit upload1: Preliminary URL: https://estore.omsonline.net/content/transactions_import.aspx
Copy of submit upload1: Using strict mode.
Copy of submit upload1: This will be a multi-part request.
Copy of submit upload1: Will upload the file "C:\dl\diy-eco-life\code\blank1.csv" using the parameter name "ctl00$ContentPlaceHolder1$FileUpload1".
Copy of submit upload1: POST data: __EVENTTARGET=&__EVENTARGUMENT=&__LASTFOCUS=&__VIEWSTATE=%2FwEPaA8FDzhjYjk2Y2JmYjc4YjMyMhgBBR5fX0NvbnRyb2xzUmVxdWlyZVBvc3RCYWNrS2V5X18WAQUNY2hrUmVtZW1iZXJNZRsim7D0YLrtxm2Qi9KYhiNvernM&__EVENTVALIDATION=%2FwEWBwLi%2F%2BKNCALtk%2FaGBgL8mJPvCAKDxenQDQK3jsrkBALC8qqaCgKC3IeGDM6fU6h2ohmPqieuclnj4X8C6g5b&ctl00%24ddlSwitchClients=534493&ctl00_RadMenu1_ClientState=&ctl00%24ContentPlaceHolder1%24btnSubmit=Import%0AOrders&ctl00%24ContentPlaceHolder1%24hdnPageID=780&ctl00%24tbReportName=&ctl00%24ddlExportType=excel&ctl00%24ddlExportFrequency=daily
Copy of submit upload1: Resolved URL: https://estore.omsonline.net/content/transactions_import.aspx
Using proxy server: localhost:8777
Copy of submit upload1: Sending request.
I'm pretty stuck... any suggestions?