SSL
Hi Jason,
What browser are you recommending these days for for proxy? I have tried Opera 10.0 and I am unable to connect to https://www.aliexpress.com or any other secure site. I have unchecked all but SSL3.
I have been able to get a log-in script about a year ago using version 12.0 it would ask me to accept each certificate as it loaded. I no longer have that functionality.
Now I am needing to add some functionality to the script (payments). I have been trying to use Google Chrome developer tools which has proven to be tedious.
For example here is my request in scraper:
POST /openapi/param2/1/com.aliexpress.pg/alibaba.pg.ipay.token?requestContext=%257B%2522context%2522%253A%257B%2522cardNo%2*********sensitiveinfo*******252C%2522expireMonth*********sensitiveinfo*******2522expireYear%2522*********sensitiveinfo*******22%252C%2522cvc%2522%253A%2522340%2522%252C%2522cardHolderF%2522%253A%2522Ben%2522%252C%2522cardHolderL%2522%253A%*********sensitiveinfo*******%2522%252C%2522billingCountry%2522%253A%2522US%2522%252C%2522billingState%2522%253A%2522California%2522%252C%2522billingAddress1%********sensitive**********%253A%2522111%********sensitive**********%2BDR%2522%252C%2522billingCity%2522%253A%2522Goleta%2522%252C%2522billingZipCode%2522%253A%252284118%2522%252C%2522brand%2522%253A%2522visa%2522%252C%2522cardHolderName%2522%253A%2522Ben%*********sensitiveinfo*******%2522%252C%2522billingFirstName%2522%253A%********sensitive**********%2522%252C%2522billingLastName%2522%253A%********sensitive**********%2522%257D%252C%2522ipayUserId%2522%253A%25222188202134201745%2522%252C%2522encrypted%2522%253A%2522false%2522%257D HTTP/1.1
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Cookie: _hvn_login=13; acs_usuc_t=acs_rt=6a787e24f44a49309af1e3aec0e5ba32&x_csrf=abho8hm3675z; aep_common_f=g0zdn+LSYS0Noswz0i9O6cvg0l5VAOecMjruSBnsffVRt9+jAFE9dw==; aep_usuc_f=isfm=y&site=glo&c_tp=USD&x_alimid=125210106&isb=y®ion=US&b_locale=en_US; aep_usuc_t=ber_l=A4; ali_apache_id=69.8.201.82.1526360110922.406032.8; ali_apache_track=ms=|mt=1|mid=us1006042259; ali_apache_tracktmp=W_signed=Y; intl_common_forever=2TLwIEs7yYnTtXirqp4XQhaN/YauRFbQExHg0eS8MifdYB1UqofPNw==; intl_locale=en_US; xman_f=87NoT/ELaNdp3P1DEc9B5JkoZHlpwl/LcUK4kypxULLoyo4NRbfDvCxBZqTQbjT8oAqbOr7TooYSS2euHAXauIQZyNO2wJ7MgzlnhrrHQ8lojVN/gd5uvoSTP7TlehZvGI3enIvDCDAaq+9lpfRDY4uGans9TvdUsQlrsR2nbFYiXn24oCkkoIToIjVJxILlOhiXBTvS02ePC12dLfonh38hjD0r/4B8yQZ0bpYTnJFdIfIT75LQDTVSDNBZS5TblhGMdlZON3akr9UjN+5zIvSiAgihSltsbp5/Og5JC69+KBfOv9u/bKBlbiEdYuT+twIJ7FhGoKL6GHq37NRAcf1bVIiHlnNtcpSnV3G7nIRhuuRIsO9GqD475XLTyubaVMFivKsobFp5s8whuQw2d9kNVCytr45R; xman_t=uk+4rXwIGJQ1DEjqBWhVnx4QArJxEAkc0vG1mlRalmEaqUNNJ3O0wbSs0l0yWdx8aNuFs8uNTuxvD2UDEEJxk3yFrtoQ3bwDVA1HCQV9Jtc5S++hoN2xlGQ3a18+o9XqDWH3Nbi8EQ03XqEBPyxd+knPZQZPN4O3ruSaKQRDPPbxxcA2qJ5WJRUI568eI04ujyfQZW+ab+UjCdRjtHOF84J0gIRQihTJNB8SFFDVwB9jfy7w95eGS58JvqG5PfSz44l3yMUPGrtjzHVYmXTHktAiJFq8n+Rass/oNzhKkTWOcJO9Oq1RpGgG42i1RT14VE5byddPWRKsIvQgM1t35TSdSP/LBxlzbbfFwosdohNE8AtWPyCpFhnvL24J6CoIZ63KUUceTvrngYFHm8qKNfLnlbDHFC0pwjob3E3ct39JYLKMW4u43Uty/MwJCUI+DaL2gcriCySm05hEcoar8MfAfJhGwopMVZeBBxM+3qyH9rhwlxFJjKq9O4Gahy9G9TPtR+WEE22ub5s4zNZL2y+/WHOxkOeBZOtziZWVtW+RPnXQFcvMqsJUKS8CzsQVxMQPC99NfnVeRVyze4nYqBDAsKm5QeLuYbIJSkLbF1OEeqwOMjwZEzc06z/E1qHv; xman_us_f=zero_order=n&x_locale=en_US&x_l=0&last_popup_time=1526360111039&x_user=US|Ben|Johnson|ifm|125210106&no_popup_today=n; xman_us_t=x_lid=us1006042259&sign=y&x_user=qh1oDkvhEayMKnzbuuxidfNoGOloAq3I3cu5V8VCqAA=&ctoken=17dbsc_ixme6q&need_popup=y&l_source=aliexpress
Referer: <a href="https://ilogisticsaddress.aliexpress.com/ajaxGetGlobalAddress.htm?callback=jQuery183042332076572879473_1526343892415&country=US&_csrf_token_=ud3xeubwx5we&_=1526344009139<br />
User-Agent:" title="https://ilogisticsaddress.aliexpress.com/ajaxGetGlobalAddress.htm?callback=jQuery183042332076572879473_1526343892415&country=US&_csrf_token_=ud3xeubwx5we&_=1526344009139<br />
User-Agent:">https://ilogisticsaddress.aliexpress.com/ajaxGetGlobalAddress.htm?callba...</a> Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36
Connection: Keep-Alive
Content-Length: 0
Content-Type: application/x-www-form-urlencoded; charset=UTF-8; application/json, text/javascript, */*; q=0.01
Host: pg.aliexpress.com
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Cookie: _hvn_login=13; acs_usuc_t=acs_rt=6a787e24f44a49309af1e3aec0e5ba32&x_csrf=abho8hm3675z; aep_common_f=g0zdn+LSYS0Noswz0i9O6cvg0l5VAOecMjruSBnsffVRt9+jAFE9dw==; aep_usuc_f=isfm=y&site=glo&c_tp=USD&x_alimid=125210106&isb=y®ion=US&b_locale=en_US; aep_usuc_t=ber_l=A4; ali_apache_id=69.8.201.82.1526360110922.406032.8; ali_apache_track=ms=|mt=1|mid=us1006042259; ali_apache_tracktmp=W_signed=Y; intl_common_forever=2TLwIEs7yYnTtXirqp4XQhaN/YauRFbQExHg0eS8MifdYB1UqofPNw==; intl_locale=en_US; xman_f=87NoT/ELaNdp3P1DEc9B5JkoZHlpwl/LcUK4kypxULLoyo4NRbfDvCxBZqTQbjT8oAqbOr7TooYSS2euHAXauIQZyNO2wJ7MgzlnhrrHQ8lojVN/gd5uvoSTP7TlehZvGI3enIvDCDAaq+9lpfRDY4uGans9TvdUsQlrsR2nbFYiXn24oCkkoIToIjVJxILlOhiXBTvS02ePC12dLfonh38hjD0r/4B8yQZ0bpYTnJFdIfIT75LQDTVSDNBZS5TblhGMdlZON3akr9UjN+5zIvSiAgihSltsbp5/Og5JC69+KBfOv9u/bKBlbiEdYuT+twIJ7FhGoKL6GHq37NRAcf1bVIiHlnNtcpSnV3G7nIRhuuRIsO9GqD475XLTyubaVMFivKsobFp5s8whuQw2d9kNVCytr45R; xman_t=uk+4rXwIGJQ1DEjqBWhVnx4QArJxEAkc0vG1mlRalmEaqUNNJ3O0wbSs0l0yWdx8aNuFs8uNTuxvD2UDEEJxk3yFrtoQ3bwDVA1HCQV9Jtc5S++hoN2xlGQ3a18+o9XqDWH3Nbi8EQ03XqEBPyxd+knPZQZPN4O3ruSaKQRDPPbxxcA2qJ5WJRUI568eI04ujyfQZW+ab+UjCdRjtHOF84J0gIRQihTJNB8SFFDVwB9jfy7w95eGS58JvqG5PfSz44l3yMUPGrtjzHVYmXTHktAiJFq8n+Rass/oNzhKkTWOcJO9Oq1RpGgG42i1RT14VE5byddPWRKsIvQgM1t35TSdSP/LBxlzbbfFwosdohNE8AtWPyCpFhnvL24J6CoIZ63KUUceTvrngYFHm8qKNfLnlbDHFC0pwjob3E3ct39JYLKMW4u43Uty/MwJCUI+DaL2gcriCySm05hEcoar8MfAfJhGwopMVZeBBxM+3qyH9rhwlxFJjKq9O4Gahy9G9TPtR+WEE22ub5s4zNZL2y+/WHOxkOeBZOtziZWVtW+RPnXQFcvMqsJUKS8CzsQVxMQPC99NfnVeRVyze4nYqBDAsKm5QeLuYbIJSkLbF1OEeqwOMjwZEzc06z/E1qHv; xman_us_f=zero_order=n&x_locale=en_US&x_l=0&last_popup_time=1526360111039&x_user=US|Ben|Johnson|ifm|125210106&no_popup_today=n; xman_us_t=x_lid=us1006042259&sign=y&x_user=qh1oDkvhEayMKnzbuuxidfNoGOloAq3I3cu5V8VCqAA=&ctoken=17dbsc_ixme6q&need_popup=y&l_source=aliexpress
Referer: <a href="https://ilogisticsaddress.aliexpress.com/ajaxGetGlobalAddress.htm?callback=jQuery183042332076572879473_1526343892415&country=US&_csrf_token_=ud3xeubwx5we&_=1526344009139<br />
User-Agent:" title="https://ilogisticsaddress.aliexpress.com/ajaxGetGlobalAddress.htm?callback=jQuery183042332076572879473_1526343892415&country=US&_csrf_token_=ud3xeubwx5we&_=1526344009139<br />
User-Agent:">https://ilogisticsaddress.aliexpress.com/ajaxGetGlobalAddress.htm?callba...</a> Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36
Connection: Keep-Alive
Content-Length: 0
Content-Type: application/x-www-form-urlencoded; charset=UTF-8; application/json, text/javascript, */*; q=0.01
Host: pg.aliexpress.com
Response:
{"succeeded":false,"responseCode":"0003","responseMsg":"com.aliexpress.pg.bridge.exception.CommandException: []ipay.CacheCard(152478)Error happened when executing JsonParseCommand[0] - JSONException: syntax error, pos 1\n\tat com.aliexpress.pg.bridge.core.impl.PipelineImpl.runCommand(PipelineImpl.java:394)\n\tat com.aliexpress.pg.bridge.core.impl.PipelineImpl.__run__(PipelineImpl.java:355)\n\tat com.aliexpress.pg.bridge.core.impl.PipelineImpl.run(PipelineImpl.java:330)\n\tat com.aliexpress.pg.bridge.core.impl.PipelineImpl.lambda$null$11(PipelineImpl.java:318)\n\tat com.al"}
Here is my request in Google developer tools which of course returns "success"
Accept: application/json, text/javascript, */*; q=0.01
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Connection: keep-alive
Content-Length: 645
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Host: pg.aliexpress.com
Origin: <a href="https://trade.aliexpress.com<br />
Referer:" title="https://trade.aliexpress.com<br />
Referer:">https://trade.aliexpress.com<br />
Referer:</a> <a href="https://trade.aliexpress.com/orderList.htm<br />
User-Agent:" title="https://trade.aliexpress.com/orderList.htm<br />
User-Agent:">https://trade.aliexpress.com/orderList.htm<br />
User-Agent:</a> Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36
requestContext=%7B%22context%22%3A%7B%22cardNo%2********sensitive**********8%22%2C%22expireMonth%22%********sensitive**********%22%2C%22expireYear%22%********sensitive**********0%22%2C%22cardHolderF%22%3A%22Ben%22%2C%22cardHolderL%22%3A%********sensitive**********%22%2C%22billingCountry%22%3A%22US%22%2C%22billingState%22%3A%22California%22%2C%22billingAddress1%22%3A%22111+********sensitive**********+DR%22%2C%22billingCity%22%3A%22Goleta%22%2C%22billingZipCode%22%3A%2284118%22%2C%22brand%22%3A%22visa%22%2C%22cardHolderName%22%3A%22Ben+Weber%22%2C%22billingFirstName%22%3A%22Ben%22%2C%22billingLastName%22%3A%********sensitive**********%22%7D%2C%22ipayUserId%22%3A%********sensitive**********%22%2C%22encrypted%22%3A%22false%22%7D
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Connection: keep-alive
Content-Length: 645
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Host: pg.aliexpress.com
Origin: <a href="https://trade.aliexpress.com<br />
Referer:" title="https://trade.aliexpress.com<br />
Referer:">https://trade.aliexpress.com<br />
Referer:</a> <a href="https://trade.aliexpress.com/orderList.htm<br />
User-Agent:" title="https://trade.aliexpress.com/orderList.htm<br />
User-Agent:">https://trade.aliexpress.com/orderList.htm<br />
User-Agent:</a> Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36
requestContext=%7B%22context%22%3A%7B%22cardNo%2********sensitive**********8%22%2C%22expireMonth%22%********sensitive**********%22%2C%22expireYear%22%********sensitive**********0%22%2C%22cardHolderF%22%3A%22Ben%22%2C%22cardHolderL%22%3A%********sensitive**********%22%2C%22billingCountry%22%3A%22US%22%2C%22billingState%22%3A%22California%22%2C%22billingAddress1%22%3A%22111+********sensitive**********+DR%22%2C%22billingCity%22%3A%22Goleta%22%2C%22billingZipCode%22%3A%2284118%22%2C%22brand%22%3A%22visa%22%2C%22cardHolderName%22%3A%22Ben+Weber%22%2C%22billingFirstName%22%3A%22Ben%22%2C%22billingLastName%22%3A%********sensitive**********%22%7D%2C%22ipayUserId%22%3A%********sensitive**********%22%2C%22encrypted%22%3A%22false%22%7D
Response from Google dev tools
{succeeded: true, responseCode: "0000",…}
responseCode
:
"0000"
responseContext
:
"{↵ "isSuccess": "Y",↵ "respCode": "IPAY_SUCCESS",↵ "respCodeMsg": "执行成功",↵ "cardToken": "***********sensitives**********",↵ "echoCardNo": "****************"↵}"
succeeded
:
true
responseCode
:
"0000"
responseContext
:
"{↵ "isSuccess": "Y",↵ "respCode": "IPAY_SUCCESS",↵ "respCodeMsg": "执行成功",↵ "cardToken": "***********sensitives**********",↵ "echoCardNo": "****************"↵}"
succeeded
:
true
SSL certificates that dislike
SSL certificates that dislike the screen-scraper proxy are getting more common. The SSL is correctly identifying screen-scraper as a man-in-the-middle of the site and your browser. I have to use Charles Proxy a lot, and in some cases I use the browser tools. In Firefox if you go to "web developer" > "network" you can see the HTTP request, and most of the time you can create the scrapeableFile with that data.