Patch for generating video sequences with subtitles (.srt) and word2vec

https://montageautomat.handmadeproductions.de/
It uses this addon: https://github.com/perevalovds/ofxWord2Vec
This is a good set of trained data: https://github.com/eyaler/word2vec-slim/blob/master/GoogleNews-vectors-negative300-SLIM.bin.gz
But others work too.
This is the code: https://github.com/Jonathhhan/ofEmscriptenExamples/tree/main/subtitleW2vExample_v2
Any ideas how to improve the code / result?

1 Like

I see this but nothing else. No error on console.
What should I be able to see? or does it take long to load?

@roymacdonald You need to load 3 files before it works. An embedding file (it takes like 10 seconds to load), for example this one: https://github.com/eyaler/word2vec-slim/blob/master/GoogleNews-vectors-negative300-SLIM.bin.gz,
an subtitle file (.srt) and the corresponding movie file.
Here are some other embedding files (it works with the .bin files): NLPL word embeddings repository
Disadvantage is, it does not work out of the box, advantage is you can load the files you want, also your own trained data…
Another reason I did it like that is that you need to download much less data if the files are not included.
One disadvantage compared to the desktop version is, that the max embedded file size is limited to something like 500 MB…

I see! cool.
I will take a look at it

This is a generated subtitle sequence (from Jean Luc Godards Alphaville):

Weight: 0.106881, Subtitle: 302, Dialogue: Because they lack electricity
Weight: 0.103191, Subtitle: 308, Dialogue: ...or the power of finance...
Weight: 0.0982258, Subtitle: 453, Dialogue: Our 17th electricity plan is inadequate
Weight: 0.0982258, Subtitle: 482, Dialogue: electricity distribution... crime suppression... war operations
Weight: 0.107424, Subtitle: 452, Dialogue: I'd like to know what Grand Omega Minus is
Weight: 0.164631, Subtitle: 486, Dialogue: A simple instruction...
Weight: 0.144092, Subtitle: 203, Dialogue: He obeys logical orders
Weight: 0.118289, Subtitle: 799, Dialogue: Fuck yourself with your logic
Weight: 0.100215, Subtitle: 444, Dialogue: An order is a logical conclusion
Weight: 0.111706, Subtitle: 433, Dialogue: We record, calculate, draw conclusions
Weight: 0.0638173, Subtitle: 192, Dialogue: I see. People have become slaves of probabilities
Weight: 0.129091, Subtitle: 394, Dialogue: ...and those of love?
Weight: 0.108561, Subtitle: 693, Dialogue: No, sensuality is a consequence
Weight: 0.105488, Subtitle: 512, Dialogue: ...that the human imagination couldn't grasp
Weight: 0.306174, Subtitle: 634, Dialogue: La conscience
1 Like

I made a version that works different (and I think better).
It compares the vector of the current subtitle with the vectors of all other subtitles and chooses the most similar one.
This is a generated sequence:

Weight: 0.695931, Subtitle: 254, Dialogue: Everything has been said...
Weight: 0.689745, Subtitle: 676, Dialogue: Like everything else, one never knows
Weight: 0.89333, Subtitle: 108, Dialogue: It's always like that. You never understand anything
Weight: 0.96907, Subtitle: 109, Dialogue: Y es, it's always like that. You never understand anything
Weight: 0.833821, Subtitle: 201, Dialogue: Nothing. Like here
Weight: 0.805801, Subtitle: 563, Dialogue: Nothing. Just a thought
Weight: 0.856398, Subtitle: 286, Dialogue: I thought I'd never
Weight: 0.864327, Subtitle: 94, Dialogue: You've never been?
Weight: 0.825386, Subtitle: 855, Dialogue: I'll never betray the Outlands
Weight: 0.749523, Subtitle: 729, Dialogue: Are you going to betray me?
Weight: 0.77787, Subtitle: 276, Dialogue: I'm going
Weight: 0.834692, Subtitle: 536, Dialogue: Going up, sir?
Weight: 0.834692, Subtitle: 19, Dialogue: Here it is, sir
Weight: 0.883882, Subtitle: 558, Dialogue: Yes, sir
Weight: 0.883882, Subtitle: 130, Dialogue: Yes
Weight: 1, Subtitle: 225, Dialogue: Yes, yes, that's it
Weight: 1, Subtitle: 326, Dialogue: Yes
Weight: 1, Subtitle: 637, Dialogue: Yes
Weight: 1, Subtitle: 854, Dialogue: ...with yes or no
Weight: 0.817187, Subtitle: 23, Dialogue: Yes, of course
Weight: 0.764355, Subtitle: 146, Dialogue: Of course I know him. Don't be stupid
Weight: 0.818982, Subtitle: 62, Dialogue: How do you know?
Weight: 1, Subtitle: 228, Dialogue: You know where it is?
Weight: 1, Subtitle: 586, Dialogue: I don't know what
Weight: 1, Subtitle: 657, Dialogue: Where? I don't know
Weight: 0.897041, Subtitle: 663, Dialogue: You see, you do know the Outlands
Weight: 0.897041, Subtitle: 804, Dialogue: We'll see
Weight: 1, Subtitle: 824, Dialogue: I see
Weight: 0.868936, Subtitle: 339, Dialogue: We see the truth you no longer see
Weight: 0.779538, Subtitle: 159, Dialogue: I wanted to see it again
Weight: 0.883478, Subtitle: 555, Dialogue: I was forbidden to come, but I wanted to see you again
Weight: 0.803273, Subtitle: 837, Dialogue: You never want to see the Outlands again, Professor?
Weight: 0.824447, Subtitle: 136, Dialogue: I don't know. I've never seen him. I'll ask
Weight: 0.840559, Subtitle: 552, Dialogue: If anyone asks, say you don't know

https://montageautomat3.handmadeproductions.de/

I optimized some things, and I think now it works quite well for what it is.

hey @Jona ,
cool thanks!
sorry to ask without looking better inside… but
this can be built easily on Windows?
I would like “the part” about getting auto subtitles from a live-played video.

hey @moebiussurfing yes, I guess so. Only thing (that comes to my mind) is, that you need to replace ofxEmscriptenVideoPlayer() with ofVideoPlayer().
Newest version is this one: ofEmscriptenExamples/montageautomat_3b at main · Jonathhhan/ofEmscriptenExamples · GitHub
And this is a minimal example for getting subtitles: ofEmscriptenExamples/subtitleExample at main · Jonathhhan/ofEmscriptenExamples · GitHub just replace ofGetElapsedTimeMillis() with ofVideoPlayer.getPosition()

1 Like

Again some optimizations: https://montageautomat3.handmadeproductions.de/
ofEmscriptenExamples/montageautomat_3g at main · Jonathhhan/ofEmscriptenExamples · GitHub
Now it would be great to load several subtitles and movies (from one director or genre, for example) at once (for having a larger subtitle pool to choose from).

1 Like

Here is a non Emscripten version: ofEmscriptenExamples/montageautomat_3gApp3 at main · Jonathhhan/ofEmscriptenExamples · GitHub

It loads several movies and subtitles, just make sure that the video and the subtitle have the same name (because they get sorted alphabetically). And you need to create videos/subtitles/embedding file folders in the data folder and put the files there, because I don’t know how to create gitkeep files (for having those empty folders)… Compared with the Emscripten version loading files is much faster (and jumping to a new video position is much more accurate).

1 Like

I made a fast test on Windows 10.
OF video-player example works,
I tried with 2 different video files but I am getting this kind of errors when hitting play:

...
'montageautomat_3gApp3.exe' (Win32): Loaded 'C:\Windows\System32\quartz.dll'. Cannot find or open the PDB file.
'montageautomat_3gApp3.exe' (Win32): Loaded 'C:\Windows\System32\qedit.dll'. Cannot find or open the PDB file.
'montageautomat_3gApp3.exe' (Win32): Loaded 'C:\Windows\System32\comdlg32.dll'. Cannot find or open the PDB file.
'montageautomat_3gApp3.exe' (Win32): Loaded 'C:\Windows\System32\msvfw32.dll'. Cannot find or open the PDB file.
'montageautomat_3gApp3.exe' (Win32): Loaded 'C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_5.82.19041.1110_none_792d1c772443f647\comctl32.dll'. Cannot find or open the PDB file.
'montageautomat_3gApp3.exe' (Win32): Loaded 'C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_5.82.19041.1110_none_792d1c772443f647\comctl32.dll'. Cannot find or open the PDB file.
'montageautomat_3gApp3.exe' (Win32): Loaded 'C:\Windows\WinSxS\amd64_microsoft.windows.gdiplus_6595b64144ccf1df_1.1.19041.1566_none_91a49058cc87e409\GdiPlus.dll'. Cannot find or open the PDB file.
'montageautomat_3gApp3.exe' (Win32): Unloaded 'C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_5.82.19041.1110_none_792d1c772443f647\comctl32.dll'
'montageautomat_3gApp3.exe' (Win32): Loaded 'C:\Program Files (x86)\K-Lite Codec Pack\Filters\LAV64\LAVSplitter.ax'. Module was built without symbols.
'montageautomat_3gApp3.exe' (Win32): Loaded 'C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.19041.1110_none_60b5254171f9507e\comctl32.dll'. Cannot find or open the PDB file.
'montageautomat_3gApp3.exe' (Win32): Loaded 'C:\Program Files (x86)\K-Lite Codec Pack\Filters\LAV64\avcodec-lav-59.dll'. Module was built without symbols.
'montageautomat_3gApp3.exe' (Win32): Loaded 'C:\Program Files (x86)\K-Lite Codec Pack\Filters\LAV64\avformat-lav-59.dll'. Module was built without symbols.
'montageautomat_3gApp3.exe' (Win32): Loaded 'C:\Program Files (x86)\K-Lite Codec Pack\Filters\LAV64\avutil-lav-57.dll'. Module was built without symbols.
'montageautomat_3gApp3.exe' (Win32): Loaded 'C:\Program Files (x86)\K-Lite Codec Pack\Filters\LAV64\libbluray.dll'. Module was built without symbols.
'montageautomat_3gApp3.exe' (Win32): Loaded 'C:\Program Files (x86)\K-Lite Codec Pack\Filters\LAV64\swresample-lav-4.dll'. Module was built without symbols.
'montageautomat_3gApp3.exe' (Win32): Loaded 'C:\Windows\System32\ncrypt.dll'. Cannot find or open the PDB file.
'montageautomat_3gApp3.exe' (Win32): Loaded 'C:\Windows\System32\ntasn1.dll'. Cannot find or open the PDB file.
'montageautomat_3gApp3.exe' (Win32): Loaded 'C:\Windows\System32\devenum.dll'. Cannot find or open the PDB file.
'montageautomat_3gApp3.exe' (Win32): Loaded 'C:\Program Files (x86)\K-Lite Codec Pack\Filters\LAV64\LAVVideo.ax'. Module was built without symbols.
'montageautomat_3gApp3.exe' (Win32): Loaded 'C:\Windows\System32\d3d9.dll'. Cannot find or open the PDB file.
'montageautomat_3gApp3.exe' (Win32): Loaded 'C:\Program Files (x86)\K-Lite Codec Pack\Filters\LAV64\avfilter-lav-8.dll'. Module was built without symbols.
'montageautomat_3gApp3.exe' (Win32): Loaded 'C:\Program Files (x86)\K-Lite Codec Pack\Filters\LAV64\swscale-lav-6.dll'. Module was built without symbols.
'montageautomat_3gApp3.exe' (Win32): Loaded 'C:\Windows\System32\dxva2.dll'. Cannot find or open the PDB file.
'montageautomat_3gApp3.exe' (Win32): Loaded 'C:\Windows\System32\DriverStore\FileRepository\nv_dispsig.inf_amd64_145fe9c72c40de0a\nvldumdx.dll'. Cannot find or open the PDB file.
'montageautomat_3gApp3.exe' (Win32): Loaded 'C:\Windows\System32\imagehlp.dll'. Cannot find or open the PDB file.
'montageautomat_3gApp3.exe' (Win32): Loaded 'C:\Windows\System32\cryptsp.dll'. Cannot find or open the PDB file.
'montageautomat_3gApp3.exe' (Win32): Loaded 'C:\Windows\System32\rsaenh.dll'. Cannot find or open the PDB file.
'montageautomat_3gApp3.exe' (Win32): Loaded 'C:\Windows\System32\DriverStore\FileRepository\nv_dispsig.inf_amd64_145fe9c72c40de0a\nvd3dumx.dll'. Cannot find or open the PDB file.
The thread 0x9344 has exited with code 0 (0x0).
NVD3DREL: GR-805 : DX9 Overlay is DISABLED
NVD3DREL: GR-805 : DX9 Overlay is DISABLED
'montageautomat_3gApp3.exe' (Win32): Loaded 'C:\Windows\System32\ResourcePolicyClient.dll'. Cannot find or open the PDB file.
'montageautomat_3gApp3.exe' (Win32): Unloaded 'C:\Windows\System32\ResourcePolicyClient.dll'
NVD3DREL: GR-805 : DX9 Overlay is DISABLED
'montageautomat_3gApp3.exe' (Win32): Loaded 'C:\Windows\System32\gpapi.dll'. Cannot find or open the PDB file.
'montageautomat_3gApp3.exe' (Win32): Loaded 'C:\Windows\System32\DriverStore\FileRepository\nv_dispsig.inf_amd64_145fe9c72c40de0a\NvCamera\NvCameraAllowlisting64.dll'. Cannot find or open the PDB file.
'montageautomat_3gApp3.exe' (Win32): Unloaded 'C:\Windows\System32\DriverStore\FileRepository\nv_dispsig.inf_amd64_145fe9c72c40de0a\NvCamera\NvCameraAllowlisting64.dll'
The thread 0x30d0 has exited with code 0 (0x0).
...

That’s the console log:

Loading embeddings file: embedding file/GoogleNews-vectors-negative300-SLIM.bin
ofxWord2VecEmbedding - loading embedding file/GoogleNews-vectors-negative300-SLIM.bin
Words in embedding file/GoogleNews-vectors-negative300-SLIM.bin: 299567
Dimensions in embedding file/GoogleNews-vectors-negative300-SLIM.bin: 300
Subtitles: data/subtitles\dark_vinyl_record_spinning.srt, Subtitle size: 0, Subtitle number: 0
Video: data/videos\dark_vinyl_record_spinning.mp4, Video number: 0

@moebiussurfing I made some small changes, for me it also works with Windows now.

thanks @Jona .
it´s working now.
i´ll check deeper…

but for the moment I have seen that

  1. we can add a custom word (in my case ‘bomb’) and
  2. then the app does a search for this word on the subtitles and it jumps to that frame where the word is located?

what other useful tasks have you seen that can be useful to?

PS
Some time ago I have seen a super interesting project called “synopsis video/cinemanet” but maybe changed the name, I’ll search for it…
I only found this, maybe has been “source-closed”:

PS
I initially thought that your app was auto-generating subtitles derived from the analyzed images or derived from the .srt file.

1 Like

@moebiussurfing I am glad that it is working now.

The “custom words” works exactly as you described (and you can also enter several custom words at once. For example: “mountain peace earth”). I also wanted to add, that it not only searches for the exact same word, but also for words that appear often in the same context. If you take the word “bomb”: It searches for bomb, but also for weapon, conflict and war. But also for peace and pacifism (which are related, but kind of the opposite).Thats all the characteristic of Word2Vec / Doc2Vec and depends on the trained data.

If “custom words” is deselected it searches for the most similar subtitle except the currently choosen.
Another idea (which isnt implemented yet) is to “recreate / remix” a movie with the most similar subtitles from other movies…

Maybe it could be useful for video editors to find some (subtitled) raw material(?)… For me its more the fun of creating algorithmic video sequences that are somehow interesting for me (my master thesis was about algorithmic video editing - and I was studying film editing). Its kind of strange poetry. But still thinking about other (artistic) use cases…

And thanks for hinting to “synopsis video/cinemanet”, that sounds very interesting…

1 Like
1 Like

@moebiussurfing wow. Thanks for the links. I didnt know that. Looks like they could be very useful for my projects…

1 Like

@moebiussurfing the windows version wasnt working properly with one video (but with several, thats what I tested). Now it works also with a single video (creating a subtitle sequence)…
Its this version: ofEmscriptenExamples/montageautomat_3gApp3 at main · Jonathhhan/ofEmscriptenExamples · GitHub