GetHistoBits: fix integer overflow

huff_image_size was a size_t (=32 bits with 32-bit builds) which could
rollover causing an incorrectly sized allocation and a crash in lossless
encoding.
fixes issue #128

Change-Id: I175c8c6132ba9792034807c5c1028dfddfeb4ea5
This commit is contained in:
James Zern 2012-10-03 12:14:44 -07:00
parent a792b913bd
commit 80237c4371

View File

@ -899,11 +899,11 @@ static int GetHistoBits(const WebPConfig* const config,
const WebPPicture* const pic) {
const int width = pic->width;
const int height = pic->height;
const size_t hist_size = sizeof(VP8LHistogram);
const uint64_t hist_size = sizeof(VP8LHistogram);
// Make tile size a function of encoding method (Range: 0 to 6).
int histo_bits = 7 - config->method;
while (1) {
const size_t huff_image_size = VP8LSubSampleSize(width, histo_bits) *
const uint64_t huff_image_size = VP8LSubSampleSize(width, histo_bits) *
VP8LSubSampleSize(height, histo_bits) *
hist_size;
if (huff_image_size <= MAX_HUFF_IMAGE_SIZE) break;