Raspberry Pi / Libcamera / OS Buster / IMX477 HiQ Camera

Hello!

There was recently a RPi HiQ camera released based on a Sony IMX477 sensor, which allows C/CS-mount lenses to be attached. Using Buster and Pi4 I’ve been trying out different approaches with film and CCTV lenses for an upcoming project.

Here are some notes I made, in particular toward finding a solution for grabbing on RPi 4 and Buster, and a legacy MMAL addon ofxRaspicam.

Raspberry Pi / Libcamera / OS Buster / IMX477 HiQ Camera

MMAL

MMAL (Multi-Media-Abstraction-Layer) is an API layer on top of OpenMAX, which interfaces with Broadcom camera chips. On Raspberry Pi OS, the default CL tools are using MMAL, which is available in userland:

userland/host_applications/linux/apps

When using raspistill or raspivid, 2 channels are created, one for processing the frame, and another for preview.

With raspiraw, only a single channel is made, and no processing done to the raw Bayer pixels, which can be dumped to file or RAM. Afterwards this can be converted into a video, which is how 660FPS videos are achieved.

This is why it’s also helpful to think of the camera sensor less as a camera, and more as a sensor - where it might run straight into OpenCV for robotics, without touching the GPU.

V4L2 Drivers

The Linux kernel also includes drivers for various camera chips, which are loaded via boot/config.txt:

dtoverlay=ov5647 # v1 camera
dtoverlay=imx219 # v2 camera
dtoverlay=imx477 # HiQ camera
dtoverlay=adv7282m # for the heads ;)

Once loaded, MMAL (raspivid/cam/raw) ceases to work, but the camera is recognised as a connected webcam of sorts - such as via the default ofVideoGrabber and ofVideoPlayer which use GStreamer.

Camera settings can be directly controlled via the driver:

v4l2-ctl --list-ctrls-menus

NB: IMX477 driver is only supported on kernel >= 5.4 (see below)

OpenMAX + GPU-acceleration

@jvcleaves did amazing work with ofxOMXCamera and ofxOMXPlayer addons which go deeper into OpenMAX to do HW-accelerated playback and grabbing (textures).

jvcleave/ofxOMXPlayer
jvcleave/ofxOMXCamera

For a time things were good: but since Buster, this is gone with GLES (though still working on Pi 3B+ with Stretch):

Raspberry PI 4 support · Issue #11

Legacy MMAL

As a temporary fix, I made an addon ofxRaspicam

autr/ofxRaspicam
cedricve/raspicam

It wraps raspicam, itself a wrapper of MMAL, with added modes for IMX477. On Pi4 30FPS 1080p is possible, and individual control of framerate, shutter speed etc. Parameters are broken out into an ofParameterGroup for testing different settings (ie. via ofxGUI).

The layers of abstraction though are less than ideal: ie. via MMAL the preview channel is being used for pixel buffer, where preview is a second channel used to measure and produce auto-exposure modes or display on-screen.

On Pi 4 its a good way to plug the gap (documentation and examples will be forthcoming), but I will park it once done (due to libcamera, see below).

Kernel >= 5.4 + Libcamera

With the release of the HiQ camera RPi Foundation also announced libcamera:

libcamera - RPi Foundation
libcamera.git - Linux camera framework
libcamera Github Fork

A complex camera support library for Linux, Android, and ChromeOS
Cameras are complex devices that need heavy hardware image processing operations. Control of the processing is based on advanced algorithms that must run on a programmable processor. This has traditionally been implemented in a dedicated MCU in the camera, but in embedded devices algorithms have been moved to the main CPU to save cost. Blurring the boundary between camera devices and Linux often left the user with no other option than a vendor-specific closed-source solution.
To address this problem the Linux media community is collaborating with the industry to develop a camera stack that is open-source-friendly while still protecting vendor core IP. libcamera was born out of that collaboration and offers modern camera support to Linux-based systems, including traditional Linux distributions, ChromeOS and Android.

It’s exciting stuff! And with a quick look at the library; MMAL-less, communicating directly with the chip registers (+ V4L2 drivers and gstreamer implementations):

libcamera
raspberrypi
cam_helper_imx477.cpp

Raspberry Pi OS is not yet at 5.4 kernel, but you can test it out by updating Buster via the guide here (and experimenting with Qcam app which is a GUI equivalent to raspistill and raspivid):

libcamera installation for Raspberry Pi - Raspberry Pi Documentation
rpi-update - Raspberry Pi Documentation
libcamera/src/qcam at master · kbingham/libcamera · GitHub

Future Approach

Before seeing the libcamera library I’d planned to start a separate implementation based off raspiraw. This was for a couple of reasons:

a) I want to record CRT screens, which requires syncing to 50/60HZ, via GENLOCK (an old standard that sends 300mV blips on each scanline refresh)
b) varying contexts of needing pixels, cv Mats, GPU textures or encoding to a codec; and finding the fastest combination of that
c) OF seems to have some Bayer pixel handling already

With libcamera, this seems like the future, and it’s also reassuring to see config files like this:

libcamera/imx477.json

So I’m planning something along the lines of ofxLibcameraGrabber. If anyone would like to help out, especially with GPU-side: g@sinnott.cc :slight_smile:

Misc Links

Raspberry Pi Camera - Stamm-Wilbrandt - very thorough notes on everything pi-camera
Raspberry Pi/Camera Streaming – Wiki - streaming guides (nb: netcat approach is FAST)
UV4L for ARM (Raspberry Pi) - UV4L install guide (for WebRTC and localhost GUI)
A Guide to Recording 660FPS Video On A $6 Raspberry Pi Camera - 660FPS guide

2 Likes

I’m trying out all kinds of combinations to get access to the new Hi-Q camera on RPI-4 and very little seems to be likely to work - i cloned and ran up the ofxraspicam example and it compiles and runs but the app fails when clicking ‘enable’ in the first gui panel. it then reports the error

 [notice  ] A
Segmentation Fault
make: *** [/home/pi/Developer/of_v0.11.0/libs/openFrameworksC ompiled/peroject/makefilecommon/compile.project.mk:191 RunRelease] Error 139

I’ve installed libcamera etc and the camera works from from the new software

does any clues on where to look to get this working?

thanks

Hi Dan,

I’m getting the same error on my rpi4.
I saw on youtube you managed to get it working. Any tips?

Hi @floftus i cannot remember exactly the steps i went thru - In the end i think bypassed using ofxraspicam and set the camera up using libcamera https://www.raspberrypi.org/documentation/linux/software/libcamera/README.md and then having it recognised using vidgrabber as usual…

i’ll dig back thru what i did and try and find time to make a step by step video to go with this one https://youtu.be/s6Rr2pzf99g

thanks for your reply - I’d really love love to watch that video! excited for it whenever you get a chance to make it.

I’ve tried to install libcamera but I’m running into problems on this step

meson configure -Dpipelines=raspberrypi -Dtest=false

I seem to be lacking some libraries…

did you

pip install meson

to load meson?

does it work from the raspistill command?

hey, I switched onto a rpi3B+ w/ OMXCamera and I haven’t come back to the rpi4 yet because my one is stuck in labs I can’t access! I think I did get all the packages for libcamera installed but it still wasn’t working on oF. Hopefully, I’ll be able to get it working when I get the camera back…

It’s been really fun getting the camera to work on the rpi3b+ though. feels great to take some computational photographic processes outside! Thanks for your help Dan!

1 Like

happy to be of help ! if/when i get back to it i’ll start again from scratch and document the steps i went thru to make it work - if you get it working then please do share here !

Hi Dan, just thought I’d message here to let everyone know that I got libcamera working on the rpi4 fine,taking videos and still images. However I still can’t get it working through openframeworks and get ofGstVideoUtils: errors. Happy to detail them if anyone is interested :slight_smile:

1 Like

agh - this has been on my list ! i will get to it and report back - at some point !

2 Likes

Just to revive this thread: In my current setup I have Libcam working and when I try to use the video grabber in openframeworks the program compiles without errors but no video is shown. This is the verbose log from the script

pi@raspberrypi:~/openFrameworks/examples/video/videoGrabberExample $ make RunRelease
VER ID IS 10
using newer build and GLFW window
[notice ] ofGstVideoGrabber: Probing devices with udev...
[notice ] ofGstVideoGrabber: Found device :, getting capabilities...
[notice ] ofGstVideoGrabber: detected v4l2 device: bcm2835-codec-decode
[notice ] ofGstVideoGrabber: driver: bcm2835-codec, version: 330257
[notice ] ofGstVideoGrabber: Capabilities: 0x84204000
[notice ] device /dev/video10 seems to not have the capture capability, (radio tuner?)
[notice ] removing it from device list
[notice ] ofGstVideoGrabber: Found device :, getting capabilities...
[notice ] ofGstVideoGrabber: detected v4l2 device: bcm2835-codec-encode
[notice ] ofGstVideoGrabber: driver: bcm2835-codec, version: 330257
[notice ] ofGstVideoGrabber: Capabilities: 0x84204000
[notice ] device /dev/video11 seems to not have the capture capability, (radio tuner?)
[notice ] removing it from device list
[notice ] ofGstVideoGrabber: Found device :, getting capabilities...
[notice ] ofGstVideoGrabber: detected v4l2 device: bcm2835-codec-isp
[notice ] ofGstVideoGrabber: driver: bcm2835-codec, version: 330257
[notice ] ofGstVideoGrabber: Capabilities: 0x84204000
[notice ] device /dev/video12 seems to not have the capture capability, (radio tuner?)
[notice ] removing it from device list
[notice ] ofGstVideoGrabber: Found device :, getting capabilities...
[notice ] ofGstVideoGrabber: detected v4l2 device: bcm2835-isp
[notice ] ofGstVideoGrabber: driver: bcm2835-isp, version: 330257
[notice ] ofGstVideoGrabber: Capabilities: 0x84200002
[notice ] device /dev/video13 seems to not have the capture capability, (radio tuner?)
[notice ] removing it from device list
[notice ] ofGstVideoGrabber: Found device :, getting capabilities...
[notice ] ofGstVideoGrabber: detected v4l2 device: bcm2835-isp
[notice ] ofGstVideoGrabber: driver: bcm2835-isp, version: 330257
[notice ] ofGstVideoGrabber: Capabilities: 0x84200001
[notice ] ofGstVideoGrabber: Found device :, getting capabilities...
[notice ] ofGstVideoGrabber: detected v4l2 device: bcm2835-isp
[notice ] ofGstVideoGrabber: driver: bcm2835-isp, version: 330257
[notice ] ofGstVideoGrabber: Capabilities: 0x84200001
[notice ] ofGstVideoGrabber: Found device :, getting capabilities...
[notice ] ofGstVideoGrabber: detected v4l2 device: bcm2835-isp
[notice ] ofGstVideoGrabber: driver: bcm2835-isp, version: 330257
[notice ] ofGstVideoGrabber: Capabilities: 0x84A00000
[notice ] device /dev/video16 seems to not have the capture capability, (radio tuner?)
[notice ] removing it from device list
[notice ] ofGstVideoGrabber: Found device :, getting capabilities...
[warning] ofGstVideoGrabber: Error while probing v4l capabilities for /dev/v4l-subdev0 Inappropriate ioctl for device
[notice ] ofGstVideoGrabber: Found device :, getting capabilities...
[notice ] ofGstVideoGrabber: detected v4l2 device: unicam
[notice ] ofGstVideoGrabber: driver: unicam, version: 330257
[notice ] ofGstVideoGrabber: Capabilities: 0x85A00001
[notice ] ofGstVideoGrabber: Found device :, getting capabilities...
[notice ] ofGstVideoGrabber: detected v4l2 device: unicam
[notice ] ofGstVideoGrabber: driver: unicam, version: 330257
[notice ] ofGstVideoGrabber: Capabilities: 0x85A00001
[notice ] 0: bcm2835-isp
[notice ] 1: bcm2835-isp
[notice ] 2: unicam
[notice ] 3: unicam
[notice ] ofGstVideoGrabber: initGrabber(): selected device: bcm2835-isp
[notice ] ofGstVideoGrabber: initGrabber(): selected format: 256x256 video/x-raw RGB framerate: 60/1
[notice ] ofGstUtils: setPipelineWithSink(): gstreamer pipeline: v4l2src name=video_source device=/dev/video14 ! video/x-raw,format=RGB,width=256,height=256,framerate=60/1  ! videoscale method=2   ! appsink name=ofappsink enable-last-sample=0 caps="video/x-raw, format=RGB, width=320, height=240"

I’ve also tried the legacy driver but run into this error

pi@raspberrypi:~/openFrameworks/examples/video/videoGrabberExample $ make run
VER ID IS 10
using legacy build
[warning] ofAppEGLWindow: init(): X11 not availble on RPI yet, using a native window instead
[notice ] ofAppEGLWindow: setupRPiNativeWindow(): screenRect: 1920x1080
[notice ] ofAppEGLWindow: setupRPiNativeWindow(): windowRect: 1024x768
[notice ] ofAppEGLWindow: createSurface(): setting up EGL Display
* failed to add service - already in use?
make: *** [/home/pi/openFrameworks/libs/openFrameworksCompiled/project/makefileCommon/compile.project.mk:184: run] Error 1

If anyone has any advice that’d be greatly appreciated :slight_smile:

1 Like

Hey @floftus - that looks like a general issue with using RPi4 because you need to launch the app with an X11 desktop (and use DISPLAY=:0 when via ssh).

There’s quite a few threads on this and also an addon for creating apps that don’t need X: jvcleave/ofxRPI4Window but by far its easiest to just set raspi-config to boot into desktop.

Re. ofxRaspicam, apologies to @danb and others as I think I left it in-flux / undocumented as I moved over to trying other things.

ofVideoGrabber should work fine with the V4L2 driver however (enabled with dtoverlay=imx477 in boot config) - and you can also control a few more things via v4l2-ctl and ofSystem if needed.

In the last months RPi have added some example applications for libcamera so this would be the basis for a future addon:

Hi everyone!
I’m trying to make similar project, with RPi4 but with ov5647 camera.

Using libcamera I’m stuck with same error as @floftus. The libcam is working, but nothing shown in videograbber in openframeworks.

Using ofxRaspicam I find same error as first described by @danb , where the example compiles and runs but the app fails when clicking ‘enable’ in the first gui panel.
Later you said that you bypassed using ofxraspicam and using libcamera instead. But I noticed that earlier in your YT video, you test the HQ cam using raspistill, which means the MMAL is still working and you’re not using libcamera (yet ?)

just trying to help your memory about how you manage this to work :sweat_smile:

1 Like

Thanks so much for your prompt reply. I was working with @floftus on this and we were definitely booting into desktop in both instances (we just used the GLFW driver in the first and the Legacy in the second).

We did not try the jvcleave/ofxRPI4Window addon however. I wonder if this is what @danb used to get it to work?

been racking my brain as to exactly how I made it work - i didnt write down the exact steps and now I cant remember - sorry ! but pretty sure i didnt use the ofxRPI4Window addon -

1 Like

What video resolution are the HQ camera capable, 4k? Also if used via v4l2 is used as YUV not MJPG?

Basically i wan to know what is the max resolution, fps and if is jpg or yuv on videomode

Thanks!

EDIT: Im trying myself but found same problem as others