Introduction
I was reading about datamoshing and fell in love with it. I could only find one single datamoshing tool that is able to run on Linux: https://github.com/happyhorseskull/you-can-datamosh-on-linux
It is basically a Python script that wraps around the famous ffmpeg, a vast software tool suite of libraries and programs for handling video, audio, and other multimedia files and streams. Ffmpeg is used for the video format conversion. The Python script also has the algorithm to edit i-frames and p-frames.
The Python tool already needs some maintenance (found some broken stuff), and I thought it would be a nice exercise (C++, Python) to refactor the “ you-can-datamosh-on-linux ” as an openFrameworks addon. There’s no decent datamoshing addon written yet, and I think this could be interesting for a lot of people in the oF community.
What’s datamoshing?
In plain English, datamoshing is an umbrella term for an array of cool effects you get by glitching, altering, or otherwise breaking the fundamental structure of video files’ data. We datamosh because we can, and because it looks really cool. What’s happening in the Python datamosh tool is that first the video file is converted to AVI format which is glitch friendly as it sort-of doesn’t care if you delete frames from the middle willy-nilly (mp4 gets real mad if you delete stuff in a video file).
There are 2 types of frames that we’re dealing with: i-frames and p-frames. I-frames (aka key frames) give a full frame’s worth of information while p-frames are used to calculate the difference from frame to frame and avoid storing lots of redundant frame information. A video can be entirely i-frames but the file size is much larger than setting an i-frame every 10 or 20 frames and making the rest p-frames. The first i-frame is the only one that’s required and after that we use p-frames to calculate from frame to frame. The encoding algorithm then makes inter-frame calculations and sometimes interesting effects happen.
Initially datamoshing was just deleting the extra i-frames maybe smooshing some p-frames in from another video and seeing what you got. However the glitchers eventually grew bored of this and discovered if they repeated p-frames that the calculations would cause a blooming effect and the results were real rowdy. So that’s what the repeat_p_frames variable does and that’s why “it sounds like a dying printer”-@ksheely. Because we’re repeating p-frames the video length may get much longer. At ((25fps - 1 i-frame)) * 15 or (24 * 15) a single second of 24 frames turns into 360 frames which is (360 frames / 25 fps) = 14.4 seconds.
After we’re done mucking around with i-frames and p-frames the results are fed to ffmpeg which locks in the glitches and makes a twitter-ready video.
What needs to be done
I need to write a ofxDataMosh oF/C++ class that opens an AVI video file and edits its i-frames and p-frames to generate the effect.
In order to achieve such goal, the following questions need to be answered:
- How to use FFmpeg/libavcodec to open AVI video file.
- How to use FFmpeg/libavcodec to manipulate video frames in such a way that we can distinguish and edit video frames.
- How to use FFmpeg/libavcodec to save AVI video files.
Looking at “ you-can-datamosh-on-linux ” and the ofxAvCodec gave me some good insights on how to do this.
When writing ofxDataMosh we need to keep in mind how the user will invoke it (ofSetup(), ofUpdate(), ofDraw()). The “ you-can-datamosh-on-linux ” tool only writes a new video file to disk, which is a decent start. But an ideal goal would be the possibility of playing the actual glitched video on the screen from an ofApp.
Pseudo Algorithm
- Parse input arguments
- Load input video
- Initialize AVI video files (converted_input.avi + datamoshed_output.avi)
- Initialize ffmpeg
- Convert input video to AVI
- Open converted_input.avi
- Load frames as hex data
- For each frame:
- if not i_frame_yet or current time isn’t effect
- write out frame as is
- if current frame is i-frame
- set i_frame_yet flag
- else:
- if current frame is not i-frame:
- write repeatedly the p-frame X times
- if current frame is not i-frame:
- if not i_frame_yet or current time isn’t effect
- Close files
Comments:
I-frames are identified when frame[5:8] == 0x0001B0. On the other hand, p-frames are identified when frame[5:8] == 0x0001B6. The end of a frame is identified by0x30306463, which is used to split the single hex stream from input file into a structured list of separated frames.
Where to begin
I created a Bitbucket Git repo with my personal account so we can push some code https://bitbucket.org/bernardoaraujor/ofxdatamosh
I started with the template from https://github.com/openframeworks/ofxAddonTemplate. The “ you-can-datamosh-on-linux ” is released unlicensed. I’m using MIT (from the template) to keep things as free as possible on ofxDataMosh.
Once we have something stable we could move it to GitHub to make it popular.
Glossary
-
datamoshing : the process of intentionally “corrupting” AVI video data frames to make cool glitch effects on the video
-
ffmpeg : command-line-based processing of video and audio files, and widely used for format transcoding, basic editing (trimming and concatenation), video scaling, video post-production effects, and standards compliance (SMPTE, ITU). FFmpeg includes libavcodec , an audio/video codec library used by many commercial and free software products,
-
ofxDataMosh : openFrameworks addon to be written as a goal. The oF community will be able to use ofxDataMosh as a modular piece of code in their own oF projects. Also refers to the oF/C++ class that encapsulates the calls to ffmppeg inside the oF abstraction framework.
-
do_the_mosh.py : Python script that wraps ffmpeg to perform datamoshing.
-
i-frame: key frames that are the least compressible but don’t require other video frames to decode.
-
p-frame : can use data from previous frames to decompress and are more compressible than I‑frames.
Other resources
Turns out the do_the_mosh.py script is derived from https://github.com/amgadani/Datamosh-python/blob/master/standard.py, which in turn is derived from https://github.com/grampajoe/Autodatamosh/blob/master/autodatamosh.pl
A good introduction to FFmpeg: https://github.com/leandromoreira/ffmpeg-libav-tutorial