I trained a model with this code: tf2-transformer-chatbot/tf2_tpu_transformer_chatbot.ipynb at master · bryanlimy/tf2-transformer-chatbot · GitHub

And I can load it with ofxTensorFlow2, this are the operations:

```
[notice ] ofxTensorFlow2: ====== Model Operations with Shapes ======
[notice ] ofxTensorFlow2: count with shape: []
[notice ] ofxTensorFlow2: count/Read/ReadVariableOp with shape: []
[notice ] ofxTensorFlow2: total with shape: []
[notice ] ofxTensorFlow2: total/Read/ReadVariableOp with shape: []
[notice ] ofxTensorFlow2: count_1 with shape: []
[notice ] ofxTensorFlow2: count_1/Read/ReadVariableOp with shape: []
[notice ] ofxTensorFlow2: total_1 with shape: []
[notice ] ofxTensorFlow2: total_1/Read/ReadVariableOp with shape: []
[notice ] ofxTensorFlow2: layer_normalization_9/beta with shape: []
[notice ] ofxTensorFlow2: layer_normalization_9/beta/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: layer_normalization_9/gamma with shape: []
[notice ] ofxTensorFlow2: layer_normalization_9/gamma/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: dense_31/bias with shape: []
[notice ] ofxTensorFlow2: dense_31/bias/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: dense_31/kernel with shape: []
[notice ] ofxTensorFlow2: dense_31/kernel/Read/ReadVariableOp with shape: [512, 256]
[notice ] ofxTensorFlow2: dense_30/bias with shape: []
[notice ] ofxTensorFlow2: dense_30/bias/Read/ReadVariableOp with shape: [512]
[notice ] ofxTensorFlow2: dense_30/kernel with shape: []
[notice ] ofxTensorFlow2: dense_30/kernel/Read/ReadVariableOp with shape: [256, 512]
[notice ] ofxTensorFlow2: layer_normalization_8/beta with shape: []
[notice ] ofxTensorFlow2: layer_normalization_8/beta/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: layer_normalization_8/gamma with shape: []
[notice ] ofxTensorFlow2: layer_normalization_8/gamma/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: attention_2/dense_29/bias with shape: []
[notice ] ofxTensorFlow2: attention_2/dense_29/bias/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: attention_2/dense_29/kernel with shape: []
[notice ] ofxTensorFlow2: attention_2/dense_29/kernel/Read/ReadVariableOp with shape: [256, 256]
[notice ] ofxTensorFlow2: attention_2/dense_28/bias with shape: []
[notice ] ofxTensorFlow2: attention_2/dense_28/bias/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: attention_2/dense_28/kernel with shape: []
[notice ] ofxTensorFlow2: attention_2/dense_28/kernel/Read/ReadVariableOp with shape: [256, 256]
[notice ] ofxTensorFlow2: attention_2/dense_27/bias with shape: []
[notice ] ofxTensorFlow2: attention_2/dense_27/bias/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: attention_2/dense_27/kernel with shape: []
[notice ] ofxTensorFlow2: attention_2/dense_27/kernel/Read/ReadVariableOp with shape: [256, 256]
[notice ] ofxTensorFlow2: attention_2/dense_26/bias with shape: []
[notice ] ofxTensorFlow2: attention_2/dense_26/bias/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: attention_2/dense_26/kernel with shape: []
[notice ] ofxTensorFlow2: attention_2/dense_26/kernel/Read/ReadVariableOp with shape: [256, 256]
[notice ] ofxTensorFlow2: layer_normalization_7/beta with shape: []
[notice ] ofxTensorFlow2: layer_normalization_7/beta/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: layer_normalization_7/gamma with shape: []
[notice ] ofxTensorFlow2: layer_normalization_7/gamma/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: attention_1/dense_25/bias with shape: []
[notice ] ofxTensorFlow2: attention_1/dense_25/bias/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: attention_1/dense_25/kernel with shape: []
[notice ] ofxTensorFlow2: attention_1/dense_25/kernel/Read/ReadVariableOp with shape: [256, 256]
[notice ] ofxTensorFlow2: attention_1/dense_24/bias with shape: []
[notice ] ofxTensorFlow2: attention_1/dense_24/bias/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: attention_1/dense_24/kernel with shape: []
[notice ] ofxTensorFlow2: attention_1/dense_24/kernel/Read/ReadVariableOp with shape: [256, 256]
[notice ] ofxTensorFlow2: attention_1/dense_23/bias with shape: []
[notice ] ofxTensorFlow2: attention_1/dense_23/bias/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: attention_1/dense_23/kernel with shape: []
[notice ] ofxTensorFlow2: attention_1/dense_23/kernel/Read/ReadVariableOp with shape: [256, 256]
[notice ] ofxTensorFlow2: attention_1/dense_22/bias with shape: []
[notice ] ofxTensorFlow2: attention_1/dense_22/bias/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: attention_1/dense_22/kernel with shape: []
[notice ] ofxTensorFlow2: attention_1/dense_22/kernel/Read/ReadVariableOp with shape: [256, 256]
[notice ] ofxTensorFlow2: layer_normalization_6/beta with shape: []
[notice ] ofxTensorFlow2: layer_normalization_6/beta/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: layer_normalization_6/gamma with shape: []
[notice ] ofxTensorFlow2: layer_normalization_6/gamma/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: dense_21/bias with shape: []
[notice ] ofxTensorFlow2: dense_21/bias/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: dense_21/kernel with shape: []
[notice ] ofxTensorFlow2: dense_21/kernel/Read/ReadVariableOp with shape: [512, 256]
[notice ] ofxTensorFlow2: dense_20/bias with shape: []
[notice ] ofxTensorFlow2: dense_20/bias/Read/ReadVariableOp with shape: [512]
[notice ] ofxTensorFlow2: dense_20/kernel with shape: []
[notice ] ofxTensorFlow2: dense_20/kernel/Read/ReadVariableOp with shape: [256, 512]
[notice ] ofxTensorFlow2: layer_normalization_5/beta with shape: []
[notice ] ofxTensorFlow2: layer_normalization_5/beta/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: layer_normalization_5/gamma with shape: []
[notice ] ofxTensorFlow2: layer_normalization_5/gamma/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: attention_2/dense_19/bias with shape: []
[notice ] ofxTensorFlow2: attention_2/dense_19/bias/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: attention_2/dense_19/kernel with shape: []
[notice ] ofxTensorFlow2: attention_2/dense_19/kernel/Read/ReadVariableOp with shape: [256, 256]
[notice ] ofxTensorFlow2: attention_2/dense_18/bias with shape: []
[notice ] ofxTensorFlow2: attention_2/dense_18/bias/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: attention_2/dense_18/kernel with shape: []
[notice ] ofxTensorFlow2: attention_2/dense_18/kernel/Read/ReadVariableOp with shape: [256, 256]
[notice ] ofxTensorFlow2: attention_2/dense_17/bias with shape: []
[notice ] ofxTensorFlow2: attention_2/dense_17/bias/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: attention_2/dense_17/kernel with shape: []
[notice ] ofxTensorFlow2: attention_2/dense_17/kernel/Read/ReadVariableOp with shape: [256, 256]
[notice ] ofxTensorFlow2: attention_2/dense_16/bias with shape: []
[notice ] ofxTensorFlow2: attention_2/dense_16/bias/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: attention_2/dense_16/kernel with shape: []
[notice ] ofxTensorFlow2: attention_2/dense_16/kernel/Read/ReadVariableOp with shape: [256, 256]
[notice ] ofxTensorFlow2: layer_normalization_4/beta with shape: []
[notice ] ofxTensorFlow2: layer_normalization_4/beta/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: layer_normalization_4/gamma with shape: []
[notice ] ofxTensorFlow2: layer_normalization_4/gamma/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: attention_1/dense_15/bias with shape: []
[notice ] ofxTensorFlow2: attention_1/dense_15/bias/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: attention_1/dense_15/kernel with shape: []
[notice ] ofxTensorFlow2: attention_1/dense_15/kernel/Read/ReadVariableOp with shape: [256, 256]
[notice ] ofxTensorFlow2: attention_1/dense_14/bias with shape: []
[notice ] ofxTensorFlow2: attention_1/dense_14/bias/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: attention_1/dense_14/kernel with shape: []
[notice ] ofxTensorFlow2: attention_1/dense_14/kernel/Read/ReadVariableOp with shape: [256, 256]
[notice ] ofxTensorFlow2: attention_1/dense_13/bias with shape: []
[notice ] ofxTensorFlow2: attention_1/dense_13/bias/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: attention_1/dense_13/kernel with shape: []
[notice ] ofxTensorFlow2: attention_1/dense_13/kernel/Read/ReadVariableOp with shape: [256, 256]
[notice ] ofxTensorFlow2: attention_1/dense_12/bias with shape: []
[notice ] ofxTensorFlow2: attention_1/dense_12/bias/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: attention_1/dense_12/kernel with shape: []
[notice ] ofxTensorFlow2: attention_1/dense_12/kernel/Read/ReadVariableOp with shape: [256, 256]
[notice ] ofxTensorFlow2: embedding_1/embeddings with shape: []
[notice ] ofxTensorFlow2: embedding_1/embeddings/Read/ReadVariableOp with shape: [8278, 256]
[notice ] ofxTensorFlow2: layer_normalization_3/beta with shape: []
[notice ] ofxTensorFlow2: layer_normalization_3/beta/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: layer_normalization_3/gamma with shape: []
[notice ] ofxTensorFlow2: layer_normalization_3/gamma/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: dense_11/bias with shape: []
[notice ] ofxTensorFlow2: dense_11/bias/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: dense_11/kernel with shape: []
[notice ] ofxTensorFlow2: dense_11/kernel/Read/ReadVariableOp with shape: [512, 256]
[notice ] ofxTensorFlow2: dense_10/bias with shape: []
[notice ] ofxTensorFlow2: dense_10/bias/Read/ReadVariableOp with shape: [512]
[notice ] ofxTensorFlow2: dense_10/kernel with shape: []
[notice ] ofxTensorFlow2: dense_10/kernel/Read/ReadVariableOp with shape: [256, 512]
[notice ] ofxTensorFlow2: layer_normalization_2/beta with shape: []
[notice ] ofxTensorFlow2: layer_normalization_2/beta/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: layer_normalization_2/gamma with shape: []
[notice ] ofxTensorFlow2: layer_normalization_2/gamma/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: attention/dense_9/bias with shape: []
[notice ] ofxTensorFlow2: attention/dense_9/bias/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: attention/dense_9/kernel with shape: []
[notice ] ofxTensorFlow2: attention/dense_9/kernel/Read/ReadVariableOp with shape: [256, 256]
[notice ] ofxTensorFlow2: attention/dense_8/bias with shape: []
[notice ] ofxTensorFlow2: attention/dense_8/bias/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: attention/dense_8/kernel with shape: []
[notice ] ofxTensorFlow2: attention/dense_8/kernel/Read/ReadVariableOp with shape: [256, 256]
[notice ] ofxTensorFlow2: attention/dense_7/bias with shape: []
[notice ] ofxTensorFlow2: attention/dense_7/bias/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: attention/dense_7/kernel with shape: []
[notice ] ofxTensorFlow2: attention/dense_7/kernel/Read/ReadVariableOp with shape: [256, 256]
[notice ] ofxTensorFlow2: attention/dense_6/bias with shape: []
[notice ] ofxTensorFlow2: attention/dense_6/bias/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: attention/dense_6/kernel with shape: []
[notice ] ofxTensorFlow2: attention/dense_6/kernel/Read/ReadVariableOp with shape: [256, 256]
[notice ] ofxTensorFlow2: layer_normalization_1/beta with shape: []
[notice ] ofxTensorFlow2: layer_normalization_1/beta/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: layer_normalization_1/gamma with shape: []
[notice ] ofxTensorFlow2: layer_normalization_1/gamma/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: dense_5/bias with shape: []
[notice ] ofxTensorFlow2: dense_5/bias/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: dense_5/kernel with shape: []
[notice ] ofxTensorFlow2: dense_5/kernel/Read/ReadVariableOp with shape: [512, 256]
[notice ] ofxTensorFlow2: dense_4/bias with shape: []
[notice ] ofxTensorFlow2: dense_4/bias/Read/ReadVariableOp with shape: [512]
[notice ] ofxTensorFlow2: dense_4/kernel with shape: []
[notice ] ofxTensorFlow2: dense_4/kernel/Read/ReadVariableOp with shape: [256, 512]
[notice ] ofxTensorFlow2: layer_normalization/beta with shape: []
[notice ] ofxTensorFlow2: layer_normalization/beta/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: layer_normalization/gamma with shape: []
[notice ] ofxTensorFlow2: layer_normalization/gamma/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: attention/dense_3/bias with shape: []
[notice ] ofxTensorFlow2: attention/dense_3/bias/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: attention/dense_3/kernel with shape: []
[notice ] ofxTensorFlow2: attention/dense_3/kernel/Read/ReadVariableOp with shape: [256, 256]
[notice ] ofxTensorFlow2: attention/dense_2/bias with shape: []
[notice ] ofxTensorFlow2: attention/dense_2/bias/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: attention/dense_2/kernel with shape: []
[notice ] ofxTensorFlow2: attention/dense_2/kernel/Read/ReadVariableOp with shape: [256, 256]
[notice ] ofxTensorFlow2: attention/dense_1/bias with shape: []
[notice ] ofxTensorFlow2: attention/dense_1/bias/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: attention/dense_1/kernel with shape: []
[notice ] ofxTensorFlow2: attention/dense_1/kernel/Read/ReadVariableOp with shape: [256, 256]
[notice ] ofxTensorFlow2: attention/dense/bias with shape: []
[notice ] ofxTensorFlow2: attention/dense/bias/Read/ReadVariableOp with shape: [256]
[notice ] ofxTensorFlow2: attention/dense/kernel with shape: []
[notice ] ofxTensorFlow2: attention/dense/kernel/Read/ReadVariableOp with shape: [256, 256]
[notice ] ofxTensorFlow2: embedding/embeddings with shape: []
[notice ] ofxTensorFlow2: embedding/embeddings/Read/ReadVariableOp with shape: [8278, 256]
[notice ] ofxTensorFlow2: outputs/bias with shape: []
[notice ] ofxTensorFlow2: outputs/bias/Read/ReadVariableOp with shape: [8278]
[notice ] ofxTensorFlow2: outputs/kernel with shape: []
[notice ] ofxTensorFlow2: outputs/kernel/Read/ReadVariableOp with shape: [256, 8278]
[notice ] ofxTensorFlow2: Const with shape: []
[notice ] ofxTensorFlow2: Const_1 with shape: [1, 8278, 256]
[notice ] ofxTensorFlow2: Const_2 with shape: []
[notice ] ofxTensorFlow2: Const_3 with shape: [1, 8278, 256]
[notice ] ofxTensorFlow2: Const_4 with shape: []
[notice ] ofxTensorFlow2: serving_default_dec_inputs with shape: [-1, -1]
[notice ] ofxTensorFlow2: serving_default_inputs with shape: [-1, -1]
[notice ] ofxTensorFlow2: StatefulPartitionedCall with shape: [-1, -1, 8278]
[notice ] ofxTensorFlow2: saver_filename with shape: []
[notice ] ofxTensorFlow2: StatefulPartitionedCall_1 with shape: []
[notice ] ofxTensorFlow2: StatefulPartitionedCall_2 with shape: []
[notice ] ofxTensorFlow2: ============ End ==============
```

And this how to calculate the inference with the notebook (it works with the generated model):

```
def evaluate(sentence):
sentence = preprocess_sentence(sentence)
sentence = tf.expand_dims(
START_TOKEN + tokenizer.encode(sentence) + END_TOKEN, axis=0)
output = tf.expand_dims(START_TOKEN, 0)
for i in range(MAX_LENGTH):
predictions = model(inputs=[sentence, output], training=False)
# select the last word from the seq_len dimension
predictions = predictions[:, -1:, :]
predicted_id = tf.cast(tf.argmax(predictions, axis=-1), tf.int32)
# return the result if the predicted_id is equal to the end token
if tf.equal(predicted_id, END_TOKEN[0]):
break
# concatenated the predicted_id to the output which is given to the decoder
# as its input.
output = tf.concat([output, predicted_id], axis=-1)
return tf.squeeze(output, axis=0)
def predict(sentence):
prediction = evaluate(sentence)
predicted_sentence = tokenizer.decode(
[i for i in prediction if i < tokenizer.vocab_size])
print('Input: {}'.format(sentence))
print('Output: {}'.format(predicted_sentence))
return predicted_sentence
```

I would be happy, if anybody can give me a hint how to preprocess the input and output with OF / ofxTensorFlow2.

Maybe this could be helpful for tokenizing the input? GitHub - Gavin-Development/CPP-SubwordTextEncoder: Based on the SubwordTextEncoder From Tensorflow Datasets, but I implemented it in CPP. https://www.tensorflow.org/datasets/api_docs/python/tfds/deprecated/text/SubwordTextEncoder