Making a utf8 unicode aware textbox

igiso · February 23, 2014, 2:31pm

I’ve been working on some unicode-aware gui for the OpenFrameworks :

Here is what I got thus fur:
unicode textboxes for inputing multiline autoformated unicode text in mac+ pc
with
drop down semitrasparent menus
lists for inputing values and auto-refresh dropdown menus
copy paste dropdown menu (this is released at my github)
and a calendar that is able to do stuff like goTo(40,“12/2/2014”) that means go to 40 days from that day or to the past etc
also many other useful calendar stuff like get Howmany days a month has etc.

here is an example of the gui in action:

The
Problem:

so in ascci a char = a character

in utf8 a character can be two chars or bytes etc

say you have a unicode text: “κείμενο”

it’s not an array of 8 chars but 16+

so let’s say you place your cursor at the end and you pressbackspace

if you only remove one char you will mess up your string , so you need to understand how many bytes / numbers are used to represent that string you are seeing

I’ve tried some ideas I had, but I couldn’t find a way to be 100% accurate

so right now I hard code stuff,

Is there a smart way to do this?

I mean what if you put a Chinese char in the text how do I know when my actual character begins in a utf8 string and where it begins so that I can

place the cursor in the correct place in the array
remove the correct number of bytes ??

here is a video of the problem in action
the bug occurs at the end of the video where I place a “Π” that messed up the hard coded algorithm I used

something irrelevant:

finally got around to create two examples for ofOpenFile function:
if anybody wants to use this:

Jordi · February 23, 2014, 9:10pm

Hi,

I’m really really interested on that addon!! Maybe I could give you a hand on that issue.
I actually had to deal with something similar last week because my program was crashing when emojis appeared in a UTF8 string. I didn’t know anything about UTF8 and all so I had to read a lot to find my way. I can reccomend you too good readings:

A very good introduction to the topic:

And a precise answer to how long is each glyph:

I uploaded my sketch (a bit dirty) that I used to understand a bit on the topic. Maybe it will help you to understand some things… It is very simple… just printing values to understand the size of a string and all…

One more thing here. I’m aware that @bakercp is working on the “next level” text tools for OF. I don’t know what is the state of his work but I’m totally confident that he is The Guru about this topic (and everything else;)

Please keep us posted on that addon

igiso · February 23, 2014, 10:01pm

yes bakercp is the utf8 guru indeed hopefully one day OF is going to be uft8 aware !

in the meanwhile
I have modified the ofsystemUtils OF files so that OF ofSystemAlert and textinut dialogs are working with non english characters sutils.zip (9.0 KB)

thanks for the
Example-parse_UTF8_string link I am going to study this

it might help me finish what I’ve started , it seems like a good place to start!!

maybe you can take a look at my utf8 ofCopyPaste example

The CopyPaste menu will be part of the textbox it’s unicode aware and works great both on windows and mac

my testing with utf8 strings for the past few months has been bug-free - you can try it as well

I’ve made a couple of textboxes thus far…
some using fontStash auto resize draw multiline and others using a heavily modified ofxTextInputField class… but you need to modify OF, and none of them is 100% perfect thus far I will let you know if I have a breakthrough!

Jordi · February 23, 2014, 10:42pm

Thank you for that addon and SystemUtils!! I will have a look at them!

bakercp · February 25, 2014, 1:43am

Hey friends – getting really close to dropping the latest utf8 aware ofxFont … I’m rushing to get it done so hopefully we can document it in time for ofBook. I’ll keep you posted … regarding clipboard support, this PR has been sitting around for a long while:

It’s done except for android support.

igiso · February 25, 2014, 6:30am

hey Chris ,

**

How are you guys planning to deal with inputing unicode text from windows?

windows apps are currently considered non Unicode by the OS unless you compile them in VS (???)

bakercp · February 25, 2014, 6:49am

The clipboard code is mostly implemented by glfw on windows for the moment – so I’d dig into GLFW source to see how it deals with … I’m not sure on specifics.

http://www.glfw.org/docs/latest/group__clipboard.html

github.com

glfw/glfw/blob/611006af185ffd35b99173d4ec2fe22d55d5b230/src/win32_clipboard.c#L85-L126


const char* _glfwPlatformGetClipboardString(_GLFWwindow* window)
{
HANDLE stringHandle;


if (!IsClipboardFormatAvailable(CF_UNICODETEXT))
{
    _glfwInputError(GLFW_FORMAT_UNAVAILABLE, NULL);
    return NULL;
}


if (!OpenClipboard(window->win32.handle))
{
    _glfwInputError(GLFW_PLATFORM_ERROR, "Win32: Failed to open clipboard");
    return NULL;
}


stringHandle = GetClipboardData(CF_UNICODETEXT);
if (!stringHandle)
{
    CloseClipboard();

This file has been truncated. show original

igiso · February 25, 2014, 8:45am

awesome!!
the returning string seems to be UTF8

The problem that I see remaining though is with keyboard input.

On Macintosh when you press a key on another language OF (glut) -> ofKeyPressed will return a unicode string with two int numbers or more depending on the bytes of the char

pressing the letter ‘λ’ on a PC Greek widechar keyboard glutKeyboardFunc will return a single number: 235 in our OpenFrameworks::keyPressed (compiled with Codeblocks)

So my question is. is that number remotely correct in any encoding format?

or is it inherently wrong because our app in CB is not compiled like VS wants it to be compiled? (widecharWindows)

AS IS

Can we transform that number to a unicode char λ?
or any other char whatsoever?

it’s really easy to test what I am saying by adding a language on your keyboard in ControlPanel language settings and run the oscText example

It will not mess up your whole system

igiso · February 25, 2014, 9:18am

everything could be solved by implementing correctly GLFW instead of GLUT but as is fixing this issue with glut might be easier

igiso · February 26, 2014, 12:16pm

Major Breakthrough,

So I’ve abandoned my attempt to re-invent the wheel

and I’ve decided to try and draw a Textbox directly from system classes in windows/mac.

So I now have a text box that works like any other texbox field in windows:

copy-paste select -multiline unicode (from windechar to UTF8)

It seems to work great I just need to create a good wrapper for it

So
here is the new CAVIAT

in the mac part of the code

I can’t find a way to grab a pointer to the OF app window

…in the windows I am grabbing a handle

by using :

WindowFromDC(wglGetCurrentDC());

and in iOS

with ofxiPhoneGetGLView()

but How Can we do the same in OSX ( Glut)

Hope this makes sense…

hey here is a video of the Textbox in action:

see that when I right-click I get the actual windows menu sweet!

and code is coming soon as well lads!!

//edit I think mac version is impossible to hack

///EDIT … WRONG!

Jordi · February 27, 2014, 12:08am

so do you think this will only work for windows? i really need this kind of features… but i need them also in mac and with any font type…
did you try to write kanji on it?

igiso · February 27, 2014, 6:55am

I believe Yes:

though I don’t really know what I am typing

Loading specific fonts and drawing Chinese/russian/greek/ whatever text is not an issue, we can do this already, (using ofxUnicode Fontstash etc)

so you can just take the string from the texbox in the video and write a skin above it and you can theoretically do whatever you please with it.

Right now
The problem is inputing text in a PERFECT manner in other languages

OpenFrameworks was created to implement easily OPEN GL
and it does so by using : GLUT (& very recently GLFW )

Glut as is implemented it does not have a way to connect to NStextField

I come from an Obj-c Background , implementing Textboxes is not very difficult but you need to use apple’s NSWINDOW instead of Glut

so theoritically you can achive a crossplatform textbox by using that in the mac code

There is an addon that combines OF with a NSWINDOW : ofxNSWindower
as well. (needs modification to the Core)

So as I see it.

If we want cross a platform textbox Openframeworks needs to officially support NSWINDOW .

Regarding GLFW.

not sure if this is the answer either, seems to have similar if not more issues with GLUT. I think @bakercp and the lads are waiting how the new version goes to see how they make their plans… (?)

If you ask me ofxUnicode +fontStash+ofutf8TruetypeFont should be a part of the CORE

If that happens then it’s just a matter of time till we have a proper powerfull unicode textbox.

igiso · March 1, 2014, 12:06pm

Finnally!!! I figured this out lads!

UNICODE - UTF8 UNIVERSAL Textbox for mac and pc in OF is on its way!

The Pc part of the code is 99% done now I just need to write a wrapper for the macitosh version of the text box:

lol we can even get spelling check on it

Sample code of How it will work

ofTextbox textbox;

textbox.draw(x,y,w,h);

Jordi · March 3, 2014, 11:40am

Waaaw that is sweet !
when can we have a look at the addon?
What did you use at the end? and did you need to modify the core?
Great work!!

igiso · March 3, 2014, 5:45pm

I’ve extended a few objective-c classes inside system Utils and overloaded some of their functions so that they can be used in a different way. (attached above openGl)

As is - the core is modified since SystemUtils.cpp is the only place that we are able to call windows or obj-functions, but it’s possible to just move everything to a single addon, we will see. I might not have a choice anyway because I need some values from the main window for the mac vrsion., like knowing if it’s focused or not. and I can’t do it as OF is structured right now. but this is solvable if I move the addon on a higher level , or use pointers etc.
surely the more advanced OF folks can help with these issues if we decide to move this to the core.

anyway
I have everything solved inside my head J, I just need time to sit down and write it,

The positive is that it seems that the text box is working with both Glut and GLFW.

I am making the object by default multiline but a single line version of it can be called using .setMultiline(false)

Also Password mode is available. and ofcourse a scrolbar.

Dee · September 4, 2015, 4:11pm

Hi,

I was wondering what happened to this addon, it’s just what I am looking for and it seems like it was never finished?

igiso · April 24, 2018, 9:35am

it works just fine I use it all the time
I just never got around to wrap it into official cloths

basically it uses the same philosophy as the iosTextbox in the ios openframeworks,
but works on windows and mac
maybe it should all be merged into one textbox itm

charli_e · April 24, 2018, 10:01pm

nice bump ^^
Just wanted to add here that new release OF 0.10.0 (almost there) supports utf8 ( check this OF blog post)