Possible integer overflow in bytesFromPixelFormat when loading large images

bugs, errors
#1

Hi,

I was re-compiling some old code on a new machine and found an unexpected problem loading large textures. When loading a 10922x8498x3 channel jpg , I get a bad_alloc error:

libc++abi.dylib: terminating with uncaught exception of type std::bad_alloc: std::bad_alloc

Debugging the problem, bytesFromPixelFormat in the function void ofPixels_::allocate(int w, int h, ofPixelFormat format) is returning a negative number, -258425444.

Looking further, bytesFromPixelFormat is a one-liner:

static int bytesFromPixelFormat(int w, int h, ofPixelFormat format){
	return w*h*pixelBitsFromPixelFormat<PixelType>(format)/8;
}

I’m thinking that for w * h * pixelBitsFromPixelFormat(format) > 2,147,483,647, the calculation overflows the int return type. With the above image, the function returns -258425444 as opposed to 278445468 like it should.

I fixed it with by enclosing (pixelBitsFromPixelFormat / 8) in parentheses, so you divide by 8 before the multiplication with w*h:

static int bytesFromPixelFormat(int w, int h, ofPixelFormat format){
	return w*h*(pixelBitsFromPixelFormat<PixelType>(format)/8);
}

This could be an issue with my compiler settings, or with xcode. To test, just use the imageLoaderExample and a large jpg. (I resized the bikers.jpg image).

I’m curious if others experience this either on OS X, or other oF platforms: I am using oF 0.9.3, OS X 10.11.6, with XCode 7.3.1.

Thanks!

Robert