Math.h not found, QtCreator crash and corrupted SSD

Hi! A hard to debug one:

Symptoms

Sometimes (not every day), while compiling from QtCreator, the whole OS becomes unresponsive. I see the available RAM go down to 0. If I notice this on time I can stop the compilation by pressing the red stop button in QtCreator. If I don’t, then the computer may recover, or may become extremely slow taking it many seconds to switch to another window or open a terminal to try kill the clangbackend.

Current status

This kind of crash happened two days ago. When I rebooted the computer, I was instructed to repair the disk. Dozens of screens like this were shown:


Fortunately the system booted but I couldn’t compile my project. I bought a new SSD and cloned the drive.

A source file in an OF addon was full of binary gibberish. Strangely, git did not notice that this file had changed when doing git status. I had to delete the file, then do git checkout file to get the file back.

I still can not compile the project. I now get this error:

In file included from /home/funpro/src/openFrameworks/examples/3d/3DPrimitivesExample/src/ofApp.cpp:1:
In file included from /home/funpro/src/openFrameworks/examples/3d/3DPrimitivesExample/src/ofApp.h:3:
In file included from /home/funpro/src/openFrameworks/libs/openFrameworks/ofMain.h:5:
In file included from /home/funpro/src/openFrameworks/libs/openFrameworks/utils/ofConstants.h:53:
/bin/../lib64/gcc/x86_64-pc-linux-gnu/7.2.1/../../../../include/c++/7.2.1/cmath:45:15: fatal error: 'math.h' file not found
#include_next <math.h>
              ^~~~~~~~
1 error generated.
Process failed with exit code 1.

But math.h seems to be there:

ls -la /bin/../lib64/gcc/x86_64-pc-linux-gnu/7.2.1/../../../../include/c++/7.2.1/*math*
-rw-r--r-- 1 root root 48484 Nov 28 22:27 /bin/../lib64/gcc/x86_64-pc-linux-gnu/7.2.1/../../../../include/c++/7.2.1/cmath
-rw-r--r-- 1 root root  1360 Nov 28 22:27 /bin/../lib64/gcc/x86_64-pc-linux-gnu/7.2.1/../../../../include/c++/7.2.1/ctgmath
-rw-r--r-- 1 root root  4380 Nov 28 22:27 /bin/../lib64/gcc/x86_64-pc-linux-gnu/7.2.1/../../../../include/c++/7.2.1/math.h
-rw-r--r-- 1 root root  1360 Nov 28 22:27 /bin/../lib64/gcc/x86_64-pc-linux-gnu/7.2.1/../../../../include/c++/7.2.1/tgmath.h

Questions

Does QtCreator become unresponsive like this to anyone with medium sized projects? This used to happen when I had only 8Gb on my system, so I upgraded to 16Gb. It happens less often now, but still happens.

Any ideas about fixing compilation and the missing math.h? I tried reinstalling a few packages like clang, qtcreator, but I guess I didn’t find the right one.

Tips or suggestions?

Idea: scan all .c .cpp .h files looking if any of those contain binary content. This way maybe I can find corrupted ones.

An example of a corrupted file in ofxCv not being detected by git:

$ pwd
/home/funpro/src/openFrameworks/addons/ofxCv/example-calibration-lcp/src
$ cat ofApp.h 
�k�2��o���k�2��o��0����WW2�xp��e�~�o������WW2�p��~�k��Xp��au��i���p���p�� X2�A���`{2�⏎
$ git diff ofApp.h 
$ 

It should have shown the differences in that file but none are displayed. git status also doesn’t show it as changed.

This proved useful to find corrupted files:

$ find ./ -iname "*.h" | xargs file "$1"
...
./example-kalman-euler/src/ofApp.h:                  C++ source, ASCII text
./example-blur/src/ofApp.h:                          C++ source, ASCII text
./example-smile/src/ofApp.h:                         C++ source, ASCII text
./example-calibration-lcp/src/ofApp.h:               data                  <---- BROKEN
./example-contours-color/src/ofApp.h:                C++ source, ASCII text
./example-ar/src/ofApp.h:                            C++ source, ASCII text
...

i’m fixing memory usage along with other compilation problems atm here: https://github.com/openframeworks/openFrameworks/pull/5821 the problem was that we were including so much stuff from the different headers in the core that each instance of the compiler used up a lot of memory. at some point when it has not enough ram it begins to swap and that freezes the system. usually if you let it run it ends up recovering.

this seems to be solved in that branch along with other problems like compilation times, autocomplete…

in any case with the current master a way to avoid this problems is by reducing the amount of processes that run at once, with makefiles by using -j4 (or less if that’s still problematic) and in qtcreator in the settings of the project in parallel jobs.

If you need to boot the computer in a situation like this in linux using AltGr + PrintScreen + U (to sync the disks) waiting a couple of seconds and then AltGr + PrintScreen + B avoids any disk corruption by writing everything to disk before rebooting

Thank you, I didn’t know about those Magic SysRq keys.

You mentioned AltGr + PrintScreen + U and AltGr + PrintScreen + B, but according to


maybe you meant AltGr + PrintScreen + S (to sync disks) and AltGr + PrintScreen + B to reboot?

So good to read about your efforts to solve those compilation-performance issues! :slight_smile:

Good news: I got things running again! The key was to run commands like these to find corrupted files:

find ./ -iname "*.h" | xargs file "$1" | grep -v text
find ./ -iname "*.cpp" | xargs file "$1" | grep -v text
find ./ -iname "*.hpp" | xargs file "$1" | grep -v text
find ./ -iname "*.inl" | xargs file "$1" | grep -v text

inside the OpenFrameworks folder, but also inside /usr/include. If a file was shown as data then it was bad.

Many of the system header files were corrupted, I assume all the files that were open at the moment of the crash. So I reinstalled all the packages that provided those header files (sdl, sdl-ttf, taglib, js, libopenshot-audio, glibc and others) and things work again! :slight_smile: I hope this doesn’t happen to anyone!