I have a few things to add, Avfoundation does not permit that much freedom in speed changes, it is a built in limitation at the moment. Quicktime was actually better at this but is depreciated by apple.
Secondly your choice of codec will be really important for this. H264 is not a good choice for speed changes, go for a codec that does not use interpolation or i-frames. Higher end proress will be better, 422 is fine. Same with audio, less compression will help, so straight PCM. I would check out using windows instead and this addon https://github.com/secondstory/ofxDSHapVideoPlayer
Image sequences and audio may be a good way to go as well, but I think sync will be hard in that case.