GPT-2 and Emscripten

emscripten
1

I made a GPT-2 inference patch: GitHub - Jonathhhan/ofEmscriptenOnnxGPT-2
Download GPT-2-LM-HEAD from models/text/machine_comprehension/gpt-2/README.md at main · onnx/models · GitHub and put it into the onnx folder. Replace template.html and compile the patch with Emscripten. Not practical to include the model into a webpage, because it is 650 MB (quantization should be possible).
The patch uses this addon for text formatting: GitHub - braitsch/ofxParagraph: Paragraph renderer for openFrameworks
And this as the tokenizer: GitHub - niieani/gpt-tokenizer: JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4. Port of OpenAI's tiktoken with additional features.

Here a short example video:

2

Here is a slightly edited file ofEmscriptenOnnxGPT-2/bin/data/js/main_custom.js at main · Jonathhhan/ofEmscriptenOnnxGPT-2 · GitHub, that accepts self converted transformer models from huggingface: Export to ONNX

3

Here an online version: https://gpt2.handmadeproductions.de/
GitHub - Jonathhhan/ofEmscriptenOnnxGPT-2_2
The model needs to be loaded the locally.
Here are 3 models that I use: Jona0123456789/gpt2-onnx-models at main

4

With this optimed model I get the fastest inferences so far: Jona0123456789/distilgpt2opt · Hugging Face

5

Here an image to text patch: GitHub - Jonathhhan/ofEmscriptenOnnxVitGpt
Put some jpg. images into the images folder and the encoder and decoder model into the bin folder (quantized and non-quantized work): Xenova/vit-gpt2-image-captioning at main
And replace template.html before compiling. Then it should work.