In typical order of learning, I would tend to suggest: algebra, geometry, trigonometry, vectors, matrices, fractals.
You only need to get algebra and some geometry to understand lots of useful things for making artistic graphical programs.
And you don’t even need that to play with OpenFrameworks and do artistic things… which may tend to give you the motivation to learn more math so you understand better how to do more artistic things.
Physics (mechanics) can also be useful.
Though you can also use libraries written by other people who had more math knowledge.
OpenFrameworks is wonderful in that it provides easy functions that can be used to quickly do a lot of visual things with very little need to know about how it’s done.