ofxDataMosh, an addon for datamoshing

[bernardoaraujor]

Introduction

I was reading about datamoshing and fell in love with it. I could only find one single datamoshing tool that is able to run on Linux: https://github.com/happyhorseskull/you-can-datamosh-on-linux

It is basically a Python script that wraps around the famous ffmpeg, a vast software tool suite of libraries and programs for handling video, audio, and other multimedia files and streams. Ffmpeg is used for the video format conversion. The Python script also has the algorithm to edit i-frames and p-frames.

The Python tool already needs some maintenance (found some broken stuff), and I thought it would be a nice exercise (C++, Python) to refactor the “ you-can-datamosh-on-linux ” as an openFrameworks addon. There’s no decent datamoshing addon written yet, and I think this could be interesting for a lot of people in the oF community.

What’s datamoshing?

In plain English, datamoshing is an umbrella term for an array of cool effects you get by glitching, altering, or otherwise breaking the fundamental structure of video files’ data. We datamosh because we can, and because it looks really cool. What’s happening in the Python datamosh tool is that first the video file is converted to AVI format which is glitch friendly as it sort-of doesn’t care if you delete frames from the middle willy-nilly (mp4 gets real mad if you delete stuff in a video file).

There are 2 types of frames that we’re dealing with: i-frames and p-frames. I-frames (aka key frames) give a full frame’s worth of information while p-frames are used to calculate the difference from frame to frame and avoid storing lots of redundant frame information. A video can be entirely i-frames but the file size is much larger than setting an i-frame every 10 or 20 frames and making the rest p-frames. The first i-frame is the only one that’s required and after that we use p-frames to calculate from frame to frame. The encoding algorithm then makes inter-frame calculations and sometimes interesting effects happen.

Initially datamoshing was just deleting the extra i-frames maybe smooshing some p-frames in from another video and seeing what you got. However the glitchers eventually grew bored of this and discovered if they repeated p-frames that the calculations would cause a blooming effect and the results were real rowdy. So that’s what the repeat_p_frames variable does and that’s why “it sounds like a dying printer”-@ksheely. Because we’re repeating p-frames the video length may get much longer. At ((25fps - 1 i-frame)) * 15 or (24 * 15) a single second of 24 frames turns into 360 frames which is (360 frames / 25 fps) = 14.4 seconds.

After we’re done mucking around with i-frames and p-frames the results are fed to ffmpeg which locks in the glitches and makes a twitter-ready video.

Video compression picture types

What needs to be done

I need to write a ofxDataMosh oF/C++ class that opens an AVI video file and edits its i-frames and p-frames to generate the effect.

In order to achieve such goal, the following questions need to be answered:

• How to use FFmpeg/libavcodec to open AVI video file.
• How to use FFmpeg/libavcodec to manipulate video frames in such a way that we can distinguish and edit video frames.
• How to use FFmpeg/libavcodec to save AVI video files.

Looking at “ you-can-datamosh-on-linux ” and the ofxAvCodec gave me some good insights on how to do this.

When writing ofxDataMosh we need to keep in mind how the user will invoke it (ofSetup(), ofUpdate(), ofDraw()). The “ you-can-datamosh-on-linux ” tool only writes a new video file to disk, which is a decent start. But an ideal goal would be the possibility of playing the actual glitched video on the screen from an ofApp.

Pseudo Algorithm

• Parse input arguments
• Load input video
• Initialize AVI video files (converted_input.avi + datamoshed_output.avi)
• Initialize ffmpeg
• Convert input video to AVI
• Open converted_input.avi
• Load frames as hex data
• For each frame:
  • if not i_frame_yet or current time isn’t effect
    • write out frame as is
    • if current frame is i-frame
      • set i_frame_yet flag
  • else:
    • if current frame is not i-frame:
      • write repeatedly the p-frame X times
• Close files

Comments:

I-frames are identified when frame[5:8] == 0x0001B0. On the other hand, p-frames are identified when frame[5:8] == 0x0001B6. The end of a frame is identified by0x30306463, which is used to split the single hex stream from input file into a structured list of separated frames.

Where to begin

I created a Bitbucket Git repo with my personal account so we can push some code https://bitbucket.org/bernardoaraujor/ofxdatamosh

I started with the template from https://github.com/openframeworks/ofxAddonTemplate. The “ you-can-datamosh-on-linux ” is released unlicensed. I’m using MIT (from the template) to keep things as free as possible on ofxDataMosh.

Once we have something stable we could move it to GitHub to make it popular.

Glossary

• datamoshing : the process of intentionally “corrupting” AVI video data frames to make cool glitch effects on the video

• ffmpeg : command-line-based processing of video and audio files, and widely used for format transcoding, basic editing (trimming and concatenation), video scaling, video post-production effects, and standards compliance (SMPTE, ITU). FFmpeg includes libavcodec , an audio/video codec library used by many commercial and free software products,

• ofxDataMosh : openFrameworks addon to be written as a goal. The oF community will be able to use ofxDataMosh as a modular piece of code in their own oF projects. Also refers to the oF/C++ class that encapsulates the calls to ffmppeg inside the oF abstraction framework.

• do_the_mosh.py : Python script that wraps ffmpeg to perform datamoshing.

• i-frame: key frames that are the least compressible but don’t require other video frames to decode.

• p-frame : can use data from previous frames to decompress and are more compressible than I‑frames.

Other resources

Turns out the do_the_mosh.py script is derived from https://github.com/amgadani/Datamosh-python/blob/master/standard.py, which in turn is derived from https://github.com/grampajoe/Autodatamosh/blob/master/autodatamosh.pl

A good introduction to FFmpeg: https://github.com/leandromoreira/ffmpeg-libav-tutorial

[bernardoaraujor]

anyone with sharp make skills?

seems to be some sort of linking problem between the ofApp and ffmpeg libs

https://paste.debian.net/1055739/

[bernardoaraujor]

maybe it has to do with how I compiled ffmpeg… not sure if these autoconf flags are ideal to make the libs link correctly to openFrameworks

this is how I did it:

git clone git://source.ffmpeg.org/ffmpeg.git
cd ffmpeg
# check out specific revision if you want
./configure  --prefix=`pwd`/dist/x86_64/ --enable-pic --enable-shared
make && make install

then copied /dist/x86_64/ * to ~/openFrameworks/addons/ofxDataMosh/libs

if I rename libs to something else make will run just fine.

[roymacdonald]

Hey!
just some thoughts:
Datamoshing is a kind of outdated aesthetic. It used to be the thing back in 2010. It was used and abused and eventually people stopped using it as much. This is why you probably found just outdated code.

Have you been able to compile and run ofxAvCodec?
if not I would start from there.

[bernardoaraujor]

I know but it still looks nice hahah

yeah I mentioned in the post… I did give it a look, I used the ffmpeg build instructions from there.

kritzikratzi/ofxAvCodec/blob/master/ffmpeg_src/readme.md

I built it and put the artifacts inside libs

https://bitbucket.org/bernardoaraujor/ofxdatamosh/src/5d4c929c7c20ec25603ba7099b50134ed3a32762/libs/ffmpeg/?at=master

I suspect there’s some mismatch between the autoconf parameters and my machine. Maybe I need to add some other parameters to autoconf that will match the oF make build environment.

[roymacdonald]

what OS are you using?
did you try the already compiled ffmpeg that is provided by ofxAvCoded?

Also it is important that you correctly setup the addons_config.mk file in your addon.
Take a look at ofxAvCodec’s addon_config.mk

[bernardoaraujor]

fixing addons_config.mk did the trick!

I’m using ubuntu, so I added the following to addons_config.mk:

linux:
	ADDON_LDFLAGS = -lavcodec -lavutil -lavformat -lswresample -lswscale
	ADDON_SOURCES_EXCLUDE = libs/%	
	ADDON_INCLUDES_EXCLUDE = libs/%	
	
linux64:
	ADDON_LDFLAGS = -lavcodec -lavutil -lavformat -lswresample -lswscale	
	ADDON_SOURCES_EXCLUDE = libs/%
	ADDON_INCLUDES_EXCLUDE = libs/% 

thanks @roymacdonald