• About

N1nja Hacks

~ Random assortment of solutions, sneaky hacks and technical HOWTOs

N1nja Hacks

Category Archives: video

How to Save HDS Flash Streams from any web page

29 Thursday Jan 2015

Posted by valblant in video

≈ 28 Comments

Tags

flash, Flash video, HDS

I came across a Flash video that I was not able to save with any Video Downloader app, including the ones that actually sniff traffic on your network adapter, such as Replay Media Catcher and many others.

Turns out that this particular page was using the new Adobe HTTP Dynamic Streaming (HDS) technology. With HDS, the original MP4 or FLV file is split up into many F4F segments, which are then served to the media player on the page one after the other, so there is no single video file to download like with most other video streaming technologies.

You can easily check if HDS is being used by using Firefox to watch the video.

  1. Clear Firefox cache (Tools -> Options -> Network, Clear Cached Web Content, Clear User Data)
  2. Load the page with the video
  3. Open a new tab and browse to about:cache?storage=disk
  4. Search for a bunch of files that have the word ‘Frag’ in them. They’ll look something like this:
http://ams-vp11.9c9media.com/hds-vod/ae/2015-01-29/3FA6DB15557BA5F0/CTVNews-546418-29-WPG-WEBPARKOUR08-SOT-Adaptive_08.mp4Seg1-Frag39 
http://ams-vp11.9c9media.com/hds-vod/ae/2015-01-29/3FA6DB15557BA5F0/CTVNews-546418-29-WPG-WEBPARKOUR08-SOT-Adaptive_08.mp4Seg1-Frag38 
http://ams-vp11.9c9media.com/hds-vod/ae/2015-01-29/3FA6DB15557BA5F0/CTVNews-546418-29-WPG-WEBPARKOUR08-SOT-Adaptive_08.mp4Seg1-Frag37 
http://ams-vp11.9c9media.com/hds-vod/ae/2015-01-29/3FA6DB15557BA5F0/CTVNews-546418-29-WPG-WEBPARKOUR08-SOT-Adaptive_08.mp4Seg1-Frag36

These are all the F4F fragments of the video. You could download them all and combine them together, but that’s not the best way to do this.

There is a script called AdobeHDS.php, which can automate the download process for you if you provide it with the F4M Manifest for the stream. You can download the script from https://github.com/K-S-V/Scripts

This manifest file is easy to obtain, b/c it is delivered via a plain GET request that is issued before the video starts playing. To find the URL:

  1. Open Firefox Console (Ctrl+Shift+K) or Tools -> Web Developer -> Web Console
  2. Make sure that “Net” filter is selected
  3. Clear the Console
  4. Open the video page and let the video load
  5. In the Filter text box type “f4m” and you should now see a few F4M requests. You want the first one, which will probably be called “manifest.f4m“. Mine looked like this:
GET http://capi.9c9media.com/destinations/ctvnews_web/platforms/desktop/contents/540901/contentpackages/546418/stacks/1130329/manifest.f4m

Now just run the script with the manifest URL and you should get the re-combined flv file:

$ php AdobeHDS.php --delete --manifest "http://capi.9c9media.com/destinations/ctvnews_web/platforms/desktop/contents/540901/contentpackages/546418/stacks/1130329/manifest.f4m"
 KSV Adobe HDS Downloader

Processing manifest info.... 
Quality Selection: 
 Available: 2048 1856 1536 1280 896 640 480 299
 Selected : 2048 
Fragments Total: 55, First: 1, Start: 1, Parallel: 8 
Downloading 55/55 fragments 
Found 55 fragments 
Finished

You should now have an FLV file waiting for you in the script directory.

For Mac Users

Posting some info from a comment by Eric L. Pheterson below:

To add a few more baby steps (for Mac users) :

  • When you view the AdobeHDS.php file at Sourceforge, copy/paste it into a file, and name it AdobeHDS.php
  • PHP should be installed alreadyon your mac
  • A dependency of AdobeHDS is not installed, so in Terminal run :
brew install homebrew/php/php55-mcrypt
  • After installing mcrypt, you must open a new terminal window or tab to use it
  • If you don’t have brew installed, in Terminal run :
/usr/bin/ruby -e “$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)”
  • After installing brew, run
brew update
  • This Firefox extension will perfectly present you with the php command you need : https://addons.mozilla.org/en-US/firefox/addon/hds-link-detector/

Exporting H.264 from Cinelerra

11 Monday Mar 2013

Posted by valblant in video

≈ 6 Comments

Tags

video

This article will discuss creation of H.264 videos from Cinelerra on Ubuntu 12.04. A lot of the information was synthesized from
Olson Video and Websites.

Built-in H.264 Export

In the Renderer select “Quicktime for Linux”, and then use the H.264 compression option for video, and MPEG-4 Audio for sound:

01 H.264 - built-in

The problem with this approach is that Cinelerra provides very few settings to customize the encoding. I can achieve a much higher quality for the same size by using 2-pass encoding with the x264 utility. Luckily for us, Cinelerra has YUV4MPEG Stream option.

YUV4MPEG Stream

YUV4MPEG Stream allows us to pipe the video stream out of Cinelerra into an arbitrary tool chain. In this case the tool chain is my script that uses named pipes to encode the video with x264.

The script was adopted from here: http://www.renomath.org/ejolson/video/hddvd/264wf.html
I used http://www.renomath.org/ejolson/video/hddvd/pipe-x264 as a starting point.

Preparation

  1. Make sure that you have the x264 package installed.
  2. Save the code provided in the last section of this post as /usr/local/bin/pipe-x264.sh
  3. Make /usr/local/bin/pipe-x264.sh executable.
  4. Specify the desired resolution and FPS in the script. See code in the last section, which shows you which lines to modify.
  5. Cinelerra must be launched from the command line, so that you can see the console output.

Export Steps

1) Export The Sound

In the Renderer select “Microsoft WAV” for File Format, check Audio and uncheck Video. Export.

2) First Pass Video

In the Renderer select “YUV4MPEG Stream” for File Format, check Video and uncheck Audio. In the settings check “Use Pipe” and type the following:

$ /usr/local/bin/pipe-x264.sh -p 1 -b 1000 "%"

The first parameter (-p) is the pass number. The second (-b) is the bitrate you want. Make sure that the path points to where you’ve saved the script, that I’ll give you below.

If all works well, the video should start encoding. To check if it is actually working, you must launch Cinelerra from the command line and look at the console output after you start rendering. You should see x264 command call printed and x264 should be running and printing stuff to the console. If something went wrong, you’ll see an error instead. If you can’t figure out what’s wrong from the output, leave a comment and I might try to help.

The output in the console should look something like this:

Running pass 1:
     x264 /tmp/cine_pipe --input-res 1920x1080 --fps 59.9401 --bitrate 1000     --pass 1 --stats "/mnt/hde/temp/work/gym_2013-03-07/test40.stats"     --bframes 2 --b-adapt 2     --direct auto     --threads auto     --output "/mnt/hde/temp/work/gym_2013-03-07/test40.m2v"

[yuv4mpegpipe @ 0x15ae720] Estimating duration from bitrate, this may be inaccurate
lavf [info]: 1920x1080p 0:1 @ 60000/1001 fps (cfr)
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.2
x264 [info]: profile Main, level 4.2
Render::render_single: Session finished.
x264 [info]: frame I:1     Avg QP:43.42  size: 10362
x264 [info]: frame P:49    Avg QP:39.33  size:  3682
x264 [info]: frame B:95    Avg QP:41.30  size:   788
x264 [info]: consecutive B-frames:  0.7%  4.1% 95.2%
x264 [info]: mb I  I16..4: 97.9%  0.0%  2.1%
x264 [info]: mb P  I16..4: 11.4%  0.0%  0.0%  P16..4:  9.3%  0.0%  0.0%  0.0%  0.0%    skip:79.2%
x264 [info]: mb B  I16..4:  0.7%  0.0%  0.0%  B16..8:  2.8%  0.0%  0.0%  direct: 1.0%  skip:95.6%  L0:39.7% L1:59.7% BI: 0.6%
x264 [info]: final ratefactor: 34.33
x264 [info]: direct mvs  spatial:88.4% temporal:11.6%
x264 [info]: coded y,uvDC,uvAC intra: 1.7% 25.6% 1.4% inter: 0.2% 1.5% 0.0%
x264 [info]: i16 v,h,dc,p: 46% 31% 16%  7%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 16% 25% 36%  2%  3%  6% 10%  2%  1%
x264 [info]: i8c dc,h,v,p: 76% 12% 11%  2%
x264 [info]: Weighted P-Frames: Y:0.0% UV:0.0%
x264 [info]: kb/s:878.42

3) Second Pass Video

Once the first pass render is finished, go back to Video Settings in the Renderer and change the pipe command to run the second pass (-p 2):

$ /usr/local/bin/pipe-x264.sh -p 2 -b 1000 "%"

Again check the console to make sure that encoding has started.

4) Multiplex Video and Sound

The previous steps should have produced two files: test40.m2v and test40.wav. We need to multiplex them into a container:

$ ffmpeg -i test40.m2v -i test40.wav -acodec libmp3lame -ab 128k -ar 48000 -vcodec copy test40.avi

Here is a handy script to keep around for this step:

#!/bin/bash
# Multiplex video and sound
#

if [[ $# -lt 2 ]] ; then
    echo
    echo "  Usage: $0 <video file> <sound file>"
    echo
    exit 1
fi

command="ffmpeg -i $1 -i $2 -acodec libmp3lame -ab 128k -ar 48000 -vcodec copy ${1%%.*}.avi"

echo $command
eval $command

Mission Accomplished

You should now have a nice H.264 avi file.

Note that the .m2v file will probably not play correctly in any player, but that is ok, b/c after we multiplex it with sound the resulting file plays everywhere.

pipe-x264.sh

Please note that you have to set the desired resolution and framerate on the lines highlighted below.

#!/bin/bash
#
# Use this script to export H.264 videos from Cinelerra
#

# Change these to suite your needs
######## Canon G6 ##############
#frame_rate=10
#resolution=640x480
######## Sony A57 ##############
frame_rate=59.9401
resolution=1920x1080

pass=1
bit_rate=10000

help(){
    cat <<END
Usage: $0 [options] filename.anything
Options:
    -b n      bitrate n                             ($bit_rate)
    -p n      pass n                                ($pass)
    -h        Print this help message
END
    exit 0
}

while getopts b:p:h name "$@"
do
    case $name in
b)
	bit_rate=$OPTARG ;;
p)
	pass=$OPTARG ;;
*)
    help ;;
    esac
done
let shiftind=$OPTIND-1
shift $shiftind
if test "$#" != "1"
then
    help
fi

outfile=$1
base=`echo $outfile | sed "s/\.[^.]*$//"`

command="x264 /tmp/cine_pipe --input-res $resolution --fps $frame_rate --bitrate $bit_rate \
    --pass $pass --stats \"$base.stats\" \
    --bframes 2 --b-adapt 2 \
    --direct auto \
    --threads auto \
    --output \"$outfile\""

# Make a named pipe
rm /tmp/cine_pipe 2> /dev/null
mkfifo /tmp/cine_pipe

echo "Running pass $pass:"
echo "     $command"
echo

# Run the encoding command. It will block and wait for cat to start feeding data into the pipe
eval "$command &"

cat > /tmp/cine_pipe

Hope this helps.

Installing Cinelerra 4.4 on Ubuntu 12.04

09 Saturday Mar 2013

Posted by valblant in video

≈ 16 Comments

This article provides step-by-step instructions for installing Cinelerra 4.4.

I upgraded my main workstation to Ubuntu 12.04 recently, briefly looked at PiTiVi and decided that Cinelerra is still the only capable video editor for Linux, despite its many flaws.

This post provides step-by-step instructions for compiling and installing Cinelerra 4.4.

Install Dependencies

Here is the list of dependencies that I know about. At a minimum you’ll need these packages, although there may be more. If the compile fails on a dependency, simply look at the last few lines of the output and the error should tell you what it’s missing.

$ sudo apt-get install wget bzip2 patch build-essential w32codecs w64codecs libtool nasm libncurses5-dev libbz2-dev libncursesw5-dev libxv-dev libxxf86vm-dev libogg-dev libvorbis-dev libtheora-dev libopenexr-dev libdv-dev libpng-dev libjpeg62-dev libfreetype6-dev libfaad-dev libsndfile1-dev uuid-dev libavutil-dev libmpeg3-dev libavcodec-dev libx264-dev libfaac-dev libglu1-mesa-dev libmjpegtools-dev

Download Source

$ wget http://downloads.sourceforge.net/project/heroines/cinelerra-4.4-src.tar.xz
$ tar xvf cinelerra-4.4-src.tar.xz

You should now have a directory called cinelerra-4.4

Apply Patches

You will need to apply two patches from the Community Version of Cinelerra to add dnxhd and dv50 codec support and YUV Stream support. This was easy, b/c the patches applied cleanly.

$ wget http://renomath.org/video/linux/cinelerra/patch-cinelerra-4.4-dnxhd
$ wget http://renomath.org/video/linux/cinelerra/patch-cinelerra-4.4-yuvstream
$ patch -Np1 -d cinelerra-4.4 < patch-cinelerra-4.4-dnxhd
$ patch -Np1 -d cinelerra-4.4 < patch-cinelerra-4.4-yuvstream

I found out about these patches from Eric Olson’s site: http://renomath.org/video/linux/cinelerra/

Build

$ cd cinelerra-4.4
$ ./configure
$ make

The install target does not work, so don’t bother with it. After make succeeds, the Cinelerra binary will be in the bin/ folder. So, to run Cinelerra:

$ cd bin
# ./cinelerra

Importing AVCHD

I mostly use Cinelerra to work with AVCHD video from my Sony DSLR, which shoots at 1080p with 60 FPS. Cinelerra cannot read the resulting MTS files, so I have to convert them to dnxhd (AVdn) format first:

$ ffmpeg -i 00014-360_2.MTS -b 185M -vcodec dnxhd -acodec pcm_s16le -threads 4 output.mov

Also, I could not get Cinelerra to read the sound from the resulting file (complains about missing ‘lpcm’ codec), so I had to take another intermediate step to extract the audio from the file, so I can import it separately:

$ mplayer output.mov -vo null -vc dummy -ao:file=sound.wav

Once I import output.mov into Cinelerra, I delete the audio tracks and I replace them with sound.wav, which work well.

Improvements

Aside from some new features here and there, the most obvious advantage of Cinelerra 4.4 over the Community Version is that the video processing and playback appears to be faster. I get about 10 FPS in cinelerra-cv, whereas Cinelerra 4.4 gives me 20 FPS, which makes the video manipulation and playback much smoother. Nice.

Missing Faders in Video Tracks

Aside from some minor problems, like missing icons for transitions and other resources, the most severe problem I ran into was the missing Fader line in video tracks. The audio tracks still had the Faders, but in Video tracks the horizontal white line was just gone.

Here is what it looks like:

Missing Faders

If this happens to you, the fix is to press the 3rd button from the right in the tool bar. It is called “Fit autos to display (Alt + f)”. What it will do is change the fade range in the drop down at the bottom from “-10.00 – 10.00” to something like “-33.00 – 133.00” in my case. You can also select “0 – 100” from that drop down as an alternative.

Blog at WordPress.com.

  • Follow Following
    • N1nja Hacks
    • Already have a WordPress.com account? Log in now.
    • N1nja Hacks
    • Customize
    • Follow Following
    • Sign up
    • Log in
    • Report this content
    • View site in Reader
    • Manage subscriptions
    • Collapse this bar
 

Loading Comments...