Vulkan Renderer


#1

Hi folks,

over the last couple of months, I’ve been working on adding Vulkan support to openFrameworks.

Huh?

Vulkan is a next generation graphics API, and set to succeed OpenGL and GLES. It’s a much more explicit API. It is a fundamental shift in rendering, and it allows a level of control and performance previously only possible on games consoles. With its sister standard SPIR-V it can revolutionise drawing, shading and compute.

It was released on February 16 this year, and there is already fairly solid driver support from most manufacturers.

  • In terms of programming, it’s like switching from JavaScript to C++: potentially faster, more explicit — and there’s a chance you end up with undefined behaviour.
  • In terms of motoring, it’s like trading in a Toyota for an unlimited supply of Ferrari car parts. You could build something to put the A-Team to shame, but it might take a really long time to hit the road.
  • In terms of child’s play, its like moving from Duplo to Lego. You can build things with much more attention to detail, but you might miss how quickly you could have built a simple wall with cruder bricks.

Programming Vulkan is not as straightforward as programming OpenGL. Tutorials and manuals are rare or only just being written. There are rough edges.

For example, with Vulkan you have to manage GPU memory yourself. You have to upload your matrices to an address in memory (no more glPushMatrix), you have to tell the GPU where to find it, and you have to make sure not to overwrite the memory while the GPU might still use it in the background. Did I mention that Vulkan supports (and encourages) multi-threaded drawing?

Writing a renderer for Vulkan is an interesting puzzle.

The Cambridge (UK) Connection

Throughout the last couple of months I tried to soak up anything I could find in terms of specs and presentations. I researched different rendering engines. I looked at how bgfx (which has some similarity to console-based engines, I’m told) structures and sorts their render calls; I looked at Cinder’s current approach on wrapping the Vulkan API, and I looked at presentations and talks on how next generation engines such as Nitrous take advantage of Vulkan.

Recently, I took a train to a Khronos conference in Cambridge to meet some of the amazing people behind the API. I hoped to hear some insider gossip, and to learn about their motivations as hardware engineers and driver writers for creating this new API. There was a lot of information, good talk, and coffee. I recommend it: everyone got a T-shirt.

Richer in textiles and ideas, and having recovered from the fallout of the Cambridge Beer Festival, I came to believe that an openFrameworks Vulkan renderer would be impossible without adding a middle layer which makes Vulkan accessible in a fun and intuitive way.

Middle Out

Such a middle layer - in a first step - aims to simulate “immediate mode” OpenGL rendering and makes it possible to fulfil the API promise of an openFrameworks renderer.

For lack of a better name, I’m calling a key module of this layer the Context for now. Vulkan is a more or less stateless API after all, and the Context is where state can be tracked & banished, (and in the future, even re-used!).

At the moment, the Vulkan renderer can draw dynamic openFrameworks meshes - and it responds gracefully to long (double / triple-buffered) swapchains. By virtue of Context and its sister module Allocator, it can deal with multi-buffered dynamic memory allocation of matrix uniforms, and vertex attributes. Allocator is a simple linear GPU memory allocator. It is rather fast.

What Works

  • Setting up a vk context with debug layers
  • oFCamera (with mouse interaction)
  • ofPushMatrix() ofPopMatrix(), rotate, translate. Remaining renderer matrix operations trivial to implement
  • ofMesh.draw()
  • Depth testing
  • Loading and reflecting SPIR-V shaders
  • Generating Pipelines from Pipeline prototypes

The new vk::Shader module also ties in SPIR-V cross, which is a library originally developed by ARM and open-sourced under Khronos which allows us to reflect pre-compiled shader code. This means, we can find out about our shader uniforms and attributes and all their metadata in C++, before we load them - and even manipulate them before we hand them over to the GPU. There is a lot of potential in there.

What’s missing

But Vulkan support in the renderer is far from complete. It’s missing texture support, dynamic pipeline creation, renderpass helpers, multisampling, compute. And probably a whole lot of other things, so there’s still plenty to do.

That said, I feel the renderer and the helper modules are at a point where there is a meaningful basis to open the discussion and to invite everyone who might want to join in.

What next

The utopian grand plan I’m hoping for is to build up a set of pre-fabricated modules similar to Context and Allocator, based on common design principles which can be used independently, together or not at all.

With these pre-fabs, you should be able to slap together an openFrameworks Vulkan app prototype quickly, and then, if needed, write your own customised code for elements that matter aesthetically or functionally to you or to your project.


Following the example of the dev pushes to cpp11 and rPi, I’m setting up a fork of the main repository under the organisation openFrameworks-vk. It might be easier to collect issues and Vulkan-specific PRs this way, until the fork is ready to send a combined PR upstream to mainline openFrameworks.

Find the repo here:
https://github.com/openframeworks-vk/openFrameworks/

Note that there is some further information on early design thoughts and a write-up on how to set things up in the README.md file in the libs/openFrameworks/vk folder.

The test vs2015 project files I’ve been using to debug & develop vulkan support are in apps/devApps/testVk/

The Vulkan renderer branch is currently running and tested on Windows10/64 bit (using an NVIDIA card with latest drivers), and it may be straightforward to port this to Linux (it’s mainly a matter of wiring up GLFW 3.2 to the renderer, and to add a linux SPIR-V Cross lib apothecary recipe). A port to Android could be exciting, too!


I’d love to see interest building for Vulkan support in openFrameworks.

For one, it’s an interesting challenge in terms of software architecture, and engine programming.

And, strategically, I believe investing in such a renderer is a good way to make sure openFrameworks stays relevant in the future - as a platform for teaching and learning, for art, and for taylor-made professional software and prototypes.


Thoughts on porting open frameworks to vulkan api by khronos
#2

i want to collaborate with this. please assign me some tasks (simple stuff please). and nice work, by the way


#3

Thanks @Tiago_Morais_Morgado! It’s hard, unfortunately, to find simple tasks to do at the moment, as things are still pretty much in flux and fairly interconnected - but I’ll let you know once things have settled a bit…


Time for an update over the last three weeks or so…

Before extremely good weather in London (who would have thought) brought development to a temporary lull, this, in no particular order, happened:

Linux support

  • The Vulkan renderer now works on Linux and Windows. (And sightly better even, on Linux)
  • Makefile and QT Creator build paths work for linux (tested on latest ubuntu with nvidia drivers)

GLM support

  • oF-vk tracks oF master very closely, and is current as of today - therefore glm is now available natively.

Basics

  • Window Resize works - the swapchain will be automatically re-created when the window resizes.
  • streamlined Allocator - frame rates are back in the thousands for the simple test project
  • the testVk test project has some more demo modes - mostly to test vsync, and render speeds. Use key ‘m’ to cycle through modes, use key ‘l’ to lock or unlock frame rate to 120 Hz.
  • implemented ofDrawRectangle()

A major overhaul of Context

  • add generic setters and getters for shader uniforms
  • add const& getters for uniforms
  • add dynamic CPU memory backing to uniforms, derived from shader
    inspection
  • stacked uniform groups (allow for per-Ubo push/pop of Context state)
  • only write to GPU memory when recording draw calls (lazy allocation and write)

There’s some more (and sometimes more detailed) information in the commit messages to oF-vk.

Also, the weather has turned grayer again…


HTC Vive on Linux using ofxOpenVR: low FPS
#4

openFrameworks Vulkan Renderer: The Journey So Far

May 2017: Hello World*

How it began, in May 2016: Hello (vulkan) Triangle

It’s been a while. Since I last wrote, the openFrameworks Vulkan renderer has evolved quite a bit.

This latest version hopefully feels more modern to use, and the underlying engine should fit Vulkan better.

I wanted the renderer to work well with design principles I found good & useful in openFrameworks, but it’s also ambitious to follow a modern, stateless approach to rendering, similar to projects such as regl.

To me, and probably many others, openFrameworks is very useful as a prototyping environment to bring ideas to life quickly, and then to go deep if needed, to re-work and optimise code for good performance. It’s the “make it work”, “make it pretty”, “make it fast” design/development principle.

Vulkan does not make this easy: it is very explicit, and so impatient to optimise, that it wants to know everything upfront. It is very much a “no-surprises”, “no-sugar” API.

While the oF Vulkan renderer embraces the “no-surprises” policy, it very much tries to sweeten the experience of writing vulkan powered graphics.

Live shader programming, for example, should be quite enjoyable with this renderer.

If you want to check it out straight away, the current dev repository is here, and I’ve added Windows and Linux setup breadcrumbs in the Vulkan renderer specific readme that can help you setting up.

I’ve added a couple of vulkan-based examples into devApps, you will recognise them by “vk” in their name.

I’ve also added a vulkan app template to the project generator, which means you can create an oF Vulkan app scaffold by typing:

projectGenerator/commandline/bin/commandline -t"vk" myNewVkApp

Still here? Brilliant! I might take the opportunity to go into some more detail then, about what the vk renderer currently can do, how some of the parts fit together, and what’s planned next.


Why Vulkan?

Some time before the second rewrite of the renderer, I began to wonder if Vulkan was worth all the extra effort, compared to just stick with OpenGL. And while it’s difficult to predict what you are going to use it for, I found some Vulkan features specially promising for openFrameworks applications:

  • Unified shading language for draw and compute (great for particle systems, computed geometry, hybrid ray-tracing and/or ray-marching scenarios, engines)
  • Better and finer sync between draw and compute (great for particle systems, physics, engines)
  • Head-less GPU rendering (This is great for offline-scenarios, like rendering out an animation to disk as fast as the GPU/CPU will let you)
  • Multi-threading by design (great for performance, and battery on mobile)
  • Console-style programming on desktop and mobile hardware (removes many limitations of GLES on mobile)
  • Run draw and compute in parallel to draw on desktop hardware (great for particle systems, and compute-based culling, for example)
  • Run DMA transfers parallel to draw on desktop hardware (this is important for video streamers)
  • Better handling of window system integration, tighter render latencies (useful for VR headsets/applications)

Besides, for production apps, Vulkan gives you lots of new ways to get the best out of hardware, and for tailor-made software, the extra verbosity and complexity may be well worth it.

From these opportunities, I tried to figure out some goals for the renderer design.

Goals for the Renderer

First of all, I think, the Vulkan renderer should not be seen as a replacement for the Programmable GL Renderer, just as Vulkan is not an immediate replacement for OpenGL. The Vulkan renderer should focus on the things that Vulkan can do well, and the OpenGL renderers should stay focused on that OpenGL can do well.

One of openFrameworks’ main goals I feeel has always been to make it quick to get to draw something on screen. That’s something the oF Vulkan renderer aims for, too.

And as with the OpenGL renderer, you should always be able to open an escape hatch, and call raw API commands.

What the Renderer Takes Care Of

Here then is a selection of what the openFrameworks Vulkan renderer currently provides. First in a bullet-point summary, then fleshed out to a bit more.

Summary

The renderer gives you a “batteries included” environment with:

  • Vulkan.hpp (official Vulkan c++) bindings
  • a properly initialised Vulkan device
    • loaded with debug layers for Debug App targets
  • a swap chain, backed by default render pass
  • a default renderpass
    • depth-testing and alpha blending on by default
  • a fully resizable window
    • may go fullscreen
  • frame-based memory synchronisation
  • a simplified shader input interface (Descriptor handling)
  • a modern shader code interface:
    • load & compile shaders straight from GLSL
    • friendly shader compile error messages
    • load shaders optionally from SPIR-V
    • re-compilable shaders
    • hot-swappable shaders
  • dynamic pipeline generation
  • simplified memory (sub)allocation with poolAllocators for buffer-, and image memory
  • simplified CPU-GPU memory transfers

Vulkan.hpp

The renderer uses vulkan-hpp to talk Vulkan. This C++ API header file was initiated and open-sourced by NVIDIA, and is now officialy hosted by Khronos. It comes with the official Vulkan SDK, and wraps the raw c-header in modern, use-friendly C++.

Default Swapchain

From the outset, you will have a “canvas” to draw with: The Vulkan renderer gives you a default swapchain. A swapchain is a list of images where Vulkan can render into, and which your operating system can grab to display on screen. It’s similar to the system of front- and back-buffers in OpenGL, but gives you more control.

There is, for example, an Image Swapchain. This swapchain allows you to render without a window, straight to an image sequence on disk. The default swapchain, of course, goes to screen.

The renderer will create a swapchain with a flexible number of frames (backed by the right kind of memory), so you can tell the renderer in main.cpp how many frames you want the GPU to have in- flight at any time.

Default Renderpass

The renderer also creates a default renderpass for you, and this default renderpass has a depth buffer, so that you can easily enable depth-testing, if you want.

Memory Allocators

In Vulkan, you have to allocate GPU memory for nearly everything. “Want to set your modelViewMatrix? - You’ll first have to have memory for that. Want to set the modelViewMatrix for another mesh? - You have to have separate memory for that…”

But the spec only guarantees 4096 memory allocations over the full lifetime of your app.

To help with this, the renderer gives you two types of Allocator, one for buffer, and one for image memory. These allocators allow you to grab large chunks of GPU memory, and hand out as many small virtual sub-allocations as you may want.

Frame-based Memory Synchronisation

Once you have allocated memory, and use it to queue up a frame, you need to make sure that this memory is not touched again until the GPU has finished rendering that frame. Many frames can be in flight (meaning, all their memory is owned by the GPU) at the same time. Some engines track the CPU and GPU life-time of every transient memory object, and this can quickly get very complicated, and slow.

Instead of herding and tracking all these tiny objects across frame boundaries, the renderer fences them in, and frees them all at once when the GPU has finished rendering. Then it returns the frame to the CPU.

That’s mainly what of::vk::Context is good for. The context owns a big chunk of pre-allocated memory, and that memory is split into equal partitions for each frame which is in flight. While you build up instructions for a frame, the context keeps writing only to the partition owned by that particular frame.

When you send the frame off to the GPU for rendering, and start recording the next frame, the context starts writing again, but now at the beginning of the next partition, which is owned by the next frame.

Once your first frame has finished rendering, all the memory for that first frame can be re-used, and the context starts writing again at the start of the partition owned by that frame. It doesn’t care about deleting anything, it starts writing again at the beginning of the partition. This follows the pattern of a “pool allocator” or an "arena allocator”, if you want.

This means that instead of having many many CPU-GPU sync objects to track memory, the renderer only needs one such sync element per frame, a Fence. Once the GPU crosses the fence, we (or better the CPU) know that the GPU has finished rendering that frame and all its memory can be recycled.

Pipeline Helpers

Pipelines are the beating heart of Vulkan. But sadly, setting them up sometimes feels like shaving a proverbial yak in preparation for major heart surgery.

To keep this somewhat under control, the renderer has a helper to create pipelines, and pipelines created that way come with sensible defaults (alpha blending on, for example). Of course, the helper allows you to further customise your pipelines.

Many Vulkan tutorials suggest you define everything related to the pipeline upfront. But there are good reasons not to want do this in an art / design code context:

Imagine you do some live-coding and update your shader. The shader is the centre part of any pipeline. If your pipeline was set in stone, you’d have to restart your app everytime your shader code changed. Also, the number of possible combinations of shader, pipeline settings, and render passes is huge, and it’s sheer impossible to predict all possible combinations you might want to use and try out during runtime in your code.

This is something that’s also recognised by some folks at Khronos:

That’s why the openFrameworks vk renderer creates Vulkan pipeline objects for you only when a pipeline is first used. After that, the pipeline is cached, and re-used.

Shader Inputs

A sometimes tricky part of pipeline creation is to figure out the correct inputs for your shader. With inputs I mean what you broadly would call “Uniforms” in OpenGL: uniforms, storage buffers, textures, etc.

In OpenGL, the driver was keeping track of all your shader inputs, and there were methods like setUniform(), which don’t exist in Vulkan. Instead, you are supposed to use Descriptors to tell Vulkan where to read data for your uniforms.

True to their name, which reminds me of shady creatures straight out of the Harry Potter franchise, Descriptors are not very much fun at all to interact with.

Plus, much of the code you have to write in a Vulkan application dealing with Descriptors is essentially repeating definitions which you already gave in the shader.

The openFrameworks Vulkan renderer takes care of this extra double accounting: It uses SPIRV-cross (more about this later) to analyse and look into your shaders, and deals with Descriptors in a very pedantic way, so you don’t have to.

Spriv-Cross

Spirv-Cross is a library hosted by Khronos, which was initiated by @themaister et al. at ARM, and then open-sourced. And it allows us to know an awful lot about shaders at runtime. The vk renderer uses it mainly to analyse shaders. It will print out a list of inputs, how input memory is laid out, and the renderer also knows how much memory to set aside for each binding. All this (and more) analysis is done in the Shader class, by reflecting SPIR-V shader code.

Vulkan, you see, uses an intermediary shading language, SPIR-V. SPIR-V is not exactly readable for humans, but many shading languages compile to it. GLSL, the OpenGL shading language, can be compiled to SPIR-V with the Khronos reference shader compiler, glslang, which comes bundled with the Vulkan SDK. Pretty soon, I bet, you will be able to compile HLSL into SPIR-V, and use HLSL with the openFrameworks Vulkan renderer.

A Modern Shader Interface

Live-coding with shaders can be very rewarding. Instead of waiting for the compiler to re-create your full application and then start it, you save and reload a shader file while your app is running and, bang!, you see the changes you made.

Unfortunately, pure Vulkan would be a pretty hostile environment for this, mainly because changing the shader changes the pipeline object, and the pipeline object needs to be specified upfront, and once it’s set up, a Vulkan pipeline object is compiled into GPU machine code by the driver - and unchangeable.

Plus, you’d still have to translate your GLSL shader code to SPIR-V, since that’s the only shader language Vulkan accepts.

This where the openFrameworks Vulkan renderer steps in: It can compile shader code directly from GLSL. For this, it uses an open source library called shaderc, which is essentially the Khronos reference shader compiler with some Google extensions (which allow you to #include shader includes for example). So yes, the renderer now also includes a compiler.

It will tell you if there is a syntax error in the shader code, and will print out an error message to the console, together with a shader code snippet from around the line where it expects the error to be. This even works for shader files which you included from inside your shader code using the GLSL #include directive.

Once a shader compiles, it gets reflected by spirv-cross, as I describe above, and examined for any “linker” errors. If some were found you’ll see these printed out in the console as well.

If a shader doesn’t compile, the previous shader version is kept running, so that your app doesn’t need to crash just because of a shader code error.

This means, with the openFrameworks Vulkan renderer, you can now do Vulkan shader live-coding.

It looks as if shaderc will soon support compiling HLSL shader code, too.

What I’ve Learned So Far

I’ll close with a list of points which summarise my experience with Vulkan so far. As with everything in life, the best things are priceless and less nice things overpriced:

Vulkan: The Nice Bits

  • better access to hardware
  • documentation is amazing, and keeps improving
  • nicer unified interface to compute shaders
  • control and sync for render/compute/transfer
  • allows multi-threaded rendering
  • fine control over memory
  • meaningful debug messages when using validation layers
  • validation layers maintainers are friendly, and quick at fixing issues brought to their attention
  • faster than OpenGL when running without validation layers, because then there’s no error checking in the driver
  • no surprises: it’s all your fault.

Vulkan: The Less Nice Bits

  • complexity (mostly due to synchronisation)
  • verbosity
  • because of 1, and 2: hard to provide a dynamic environment for prototyping
  • “undefined behaviour” - uncaught errors lead to surprises
  • surprises: it’s all your fault.

What’s Next

In a future post I’ll try to go into a bit more detail about the way drawing, and the underlaying engine works, and what led to the current look of the draw interface.

I’m also working on adapting external addons for the Vulkan renderer, such as ofxImGui, which brings dear imgui to oF.

Further References

Over time, I’ve gathered a few resources around Vulkan. I’ve collected them on are.na, here. Of all these Resources, the meta-list awesome-vulkan, curated by Vinjin Zhang, deserves a special mention.

Thanks

I’d like to thank Arturo Castro for mentoring & insights into software architecture. I’d also like to say thanks to James Acres for sharing his enthusiasm for rendering techniques, for ideas and urls around Vulkan and rendering engines, and for brilliant discussions across probably three time zones.

I’d also like to thank the folks at Khronos for organising the Vulkan developer meetups in Cambridge, and for making these open to anyone.


*You can find the code for the “Hello World” Demo in the of-vk repository, under devApps.


HTC Vive on Linux using ofxOpenVR: low FPS
#5

Epic work @tgfrerer. Just to add, the README is very useful as well.


#6

Wow this is awesome work!


#7

@tgfrerer Not only amazing work, also amazing and detailed post on your insights, really thanks to help everyone of us to start this road!


Apple deprecates OpenGL support
#8

any progress on vulkan rendering?


#9

I guess it’s a great time to revive this with the deprecation of openGL on Mac OS :slight_smile: