Dealing with Flash
Hello guys. Tim here, ex-employee. I never thought I'd be coming to the forums to ask questions one day :)
So, I've been doing some fixes for a past customer who approached me about working with me directly, and things have been okay. But recently we found that the new login process takes us through the ridiculous roller coaster of redirects and Flash cookie checks.
Now, I haven't gotten deep into the exploration process in this case, but I was at least curious if screen-scraper has developed any new tools for such Flash woes, or if there are any tips to be had amongst the developers. I'm sure it's going to be a looong evening to figure out a perfect solution for this. It's a stock brokerage site, so they don't play kiddie games with login security.
Happy to see some of the excellent new features in the 4.5 alpha updates. Glorious program.
Tim Valenta
Hey Tim, Someone here
Hey Tim,
Someone here recently had to fight something similar, but I forget who. I'll figure it out and let you know if he cane up with any cool tools, but aside from that Flash is still just a pain.
Opera A very
Opera
A very standards-compliant browser that has a number of features for monitoring http goings-on. Proxy-friendly.
Firefox 2 (no longer officially supported)
Having Firefox 2 installed is helpful for a few reasons: You can fill it up with lots of add-ons, you can empty it's cache anytime, Firefox 3 won't proxy certain ssl sites where Firefox 2 will.
Note: Be sure to install in separate directory from other Firefox installations.
After installation, create new profile called "two". Execute the following from Start > Run.
firefox -CreateProfile two
Create a batch file to run.
@echo off
set MOZ_NO_REMOTE=1
start "" "C:\Program Files\Mozilla Firefox 2\Firefox.exe" -p "two"
set MOZ_NO_REMOTE=0
Firefox add-ons
These are helpful in knowing what the video URL looks like. Lately they have been less effective and will often only grab the ad preroll. Be selective and avoid most that say Youtube in the title.
Most of these I've tried.
https://addons.mozilla.org/en-US/firefox/addon/3006
https://addons.mozilla.org/en-US/firefox/addon/6584
https://addons.mozilla.org/en-US/firefox/addon/8174
https://addons.mozilla.org/en-US/firefox/addon/53003
https://addons.mozilla.org/en-US/firefox/addon/7447
https://addons.mozilla.org/en-US/firefox/addon/75489
https://addons.mozilla.org/en-US/firefox/addon/1993
https://addons.mozilla.org/en-US/firefox/addon/65949
https://addons.mozilla.org/en-US/firefox/addon/220
https://addons.mozilla.org/en-US/firefox/addon/5258
https://addons.mozilla.org/en-US/firefox/addon/3242
https://addons.mozilla.org/en-US/firefox/addon/13465
https://addons.mozilla.org/en-US/firefox/addon/13539
SwitchProxy
Configure for the following:
screen-scraper: 8777 (default)
Charles Proxy: 8888
Fiddler: 7777
Clearing your Flash cache
Because it doesn't get cleared with your browser's cache and all browsers share the same cache. Recommend making a Bookmarklet.
URL Finders
URL Snooper
This is used to observe the actual video url(s) as they are streaming from the server. The program is pretty buggy at times but it's much easier to use than other packet sniffers. Outside of sniffing through the torrent of data that comes through normal packet sniffers (MNM, Wireshare, etc.) this may be the only way to see actual evidence of non-http (rtmp, mms, rtsp) video urls.
URL Helper
Similar to URL Snooper.
StreamTransport
Reveals often hard-to-find rtmp urls.
Net Transport
Very nice software similar to URL Snooper. Free trial (too bad it's not all free).
Charles Proxy
An indispensable tool for piecing together forensic evidence of the video url. Often the video urls are found in pieces from different files being sent over the wire. It is often difficult to put together the pieces needed to request the right pages in order to get to the data you need.
Fiddler
Pretty easy to use but does not show non-http streams.
Microsoft Network Monitor
This is much like Wireshark but a bit easier to use. It doesn't observe rtmp urls directly which I believe Wireshark will. Use this tool as a second to last resort.
Wireshark
I'm listing this tool just in case you're interested. It is much like MNM but a bit harder to use. However, it very widely supported amongst the open source hacker community. Use as last resort.
Others
Playing/Testing the URLs
VLC
Plays most any url you need to test. Always turn on messaging (ctrl+m) and up the verbosity output to 2.
Applian FLV Player
When the video crashes VLC, this player will often play it. Downside is there is no logging, so if a video doesn't play you won't know why.
SPlayer
Another flv player
Akamai Flash Support Player
There are three or four big CDN's that handle the delivery of content for many of the sites we scrape. Akamai is the biggest. You'll know them by the word "edge" in their urls. The Support Player is where you can test out the pieces you've found to see if they actually play.
rtmpdump (Linux only)
One of the only tools to download certain rtmp videos.
install:
sudo apt-get install libssl-dev
sudo apt-get install openssl
sudo apt-get install zlibc
sudo wget http://rtmpdump.mplayerhq.hu/download/rtmpdump-2.1.tar.gz
sudo tar xvzf rtmpdump-2.1.tar.gz
cd rtmpdump-2.1
sudo make linux
sudo cp rtmpdump /usr/local/bin/
usage:
rtmpdump -r rtmp://cp81304.edgefcs.net/ondemand/mp4:brightcove/35121343001/35121343001_63902710001_sc-be-the-hammer1.mp4 -o - &> 35121343001_63902710001_sc-be-the-hammer1.mp4.txt
Writes err and stdout to txt file. err contains meta-data similar to mediainfo.
wgetpro ("wpro") (Linux only)
Handles more protocols than wget
install:
sudo wget http://prdownloads.sourceforge.net/wgetpro/wgetpro-0.1.3.tgz?download
sudo tar xvzf wgetpro-0.1.3.tgz
cd wgetpro-0.1.3
sudo ./configure
sudo make install
Automatically places wpro in class path
usage:
wpro mms://wm-od.cdn.turner.com/nascar/big/cup/2007/08/26/cup.bri.mdm.c.nasscar.ws.wmv